Cruxly

Cruxly › Cloud and DevOps › Cloud & DevOps: Networking

Service Mesh: What a Sidecar Proxy Actually Buys You, and What It Costs

mTLS, uniform telemetry, and the sidecar tax

Open the interactive lesson

Prompt: a platform team is deciding whether to adopt a service mesh for a cluster running 12 services, primarily to get mutual TLS between them without every service implementing TLS itself. Evaluate whether a mesh is the right tool at this scale, and what the actual cost — not just the benefit — would be if adopted anyway.

30 minutes first. The naive framing treats "should we use a service mesh" as binary — adopt it for the mTLS benefit, or don't and go without inter-service encryption. There's a real cost to weigh against a real benefit, and the answer depends on scale, not just on wanting the feature.

What the sidecar actually provides, concretely

Mutual TLS (mTLS) between every service, enforced by the infrastructure layer rather than reimplemented inside each service's own code. Without a mesh, giving every service-to-service call TLS with mutual authentication means every team, in every language their services happen to be written in, correctly implementing certificate management and rotation themselves — a mesh centralizes that into the sidecar, applied uniformly regardless of what language or framework any given service uses.

Consistent retry, timeout, and circuit-breaking behavior at the network layer, instead of each service's own application code implementing (or forgetting to implement, or implementing slightly differently) its own version. This is a real consistency win in an organization with many services in many languages, where a resilience pattern implemented per-service tends to drift into several subtly different, inconsistently-tested implementations over time.

Uniform L4/L7 telemetry across every service, regardless of what instrumentation any individual service's own code does or doesn't have — useful specifically when the alternative is "some services are well instrumented and some aren't," which a mesh's infrastructure-layer telemetry doesn't depend on.

The sidecar tax, in real numbers, and the scale heuristic

A sidecar proxy consumes real memory on every pod it runs alongside, and the commonly-cited figures are not trivial at scale: roughly 50-200MB for Envoy-based sidecars (the proxy Istio uses), roughly 20-100MB for Linkerd's deliberately lighter proxy. At a few hundred pods, that's multiple gigabytes of cluster memory spent on mesh infrastructure rather than on the application workloads the cluster actually exists to run — real, budgetable cost, not a rounding error.

Each hop also adds latency, and the proxies cause most of it, not mTLS. Every call passes through two proxies (the caller's sidecar and the callee's). The TLS handshake happens once per long-lived connection, and symmetric encryption per request costs microseconds. Istio's published benchmarks show well under a millisecond at p90 per hop for sidecars, and less for ambient mode. Real numbers grow with load, with L7 features turned on, and at the tail. The cost adds up across a path: a request touching five mesh-enabled services in series pays it ten times. Measure p99 with your own traffic before and after, rather than trusting a quoted figure.

The widely-cited operational heuristic — a mesh is probably overkill under roughly 20 services, or without a genuine, current need for its specific traffic-management features (canary routing at the mesh layer, fine-grained circuit-breaking policy) — exists because below that scale and need, the sidecar tax is a real, ongoing cost paid for a benefit the team doesn't yet have enough service-to-service complexity to actually realize. Ambient mode is now the main alternative. Istio's ambient mode (GA in Istio 1.24, November 2024) splits the sidecar's job in two:

Teams that only want mTLS pay for the ztunnel alone, and no app pod has to restart to join the mesh. The tooling evolved this lighter answer because the per-pod sidecar tax was a real problem.

Licensing is part of the choice now. In 2024, Buoyant stopped publishing open-source stable Linkerd releases: the open-source project ships edge releases, and stable builds come through Buoyant Enterprise for Linkerd (free for small companies). Check the current terms before standardizing on any mesh.

Kubernetes networking basics the mesh debate skips

Ingress is moving to the Gateway API. The Ingress resource only handled host and path routing; anything else needed controller-specific annotations. The Gateway API (GA since late 2023) splits it into roles:

It also supports weighted traffic splitting and header matching as part of the standard. The community ingress-nginx controller was retired in March 2026 (announced in November 2025) and no longer gets fixes, so a cluster still running it should plan a move to a Gateway API implementation. The mesh question is converging here too: Istio, Cilium and others use the same Gateway API resources for both north-south and east-west traffic.

ndots:5 quietly multiplies DNS traffic. A pod's default /etc/resolv.conf has several search domains and ndots:5. Any name with fewer than five dots goes through the search list first, so resolving api.stripe.com first tries:

Each of those fails before the real lookup runs, and the whole sequence happens for both the A and the AAAA record. Fixes:

Pod IPs can run out before CPU does. On EKS, the default VPC CNI gives every pod a real VPC IP, so a /24 subnet (251 usable addresses, since AWS reserves 5) caps that subnet's nodes at about 251 pods no matter how big they are. Plan the CIDRs up front, or use one of these:

Cilium is the eBPF-based CNI that many clusters now default to. It can replace kube-proxy, enforce L3-L7 network policy, and provide mTLS-like encryption (WireGuard or IPsec) without sidecars. For some teams, that covers what they wanted a mesh for in the first place.

How to say it in the interview

"A service mesh genuinely does buy you mTLS between services and consistent retry/timeout behavior without reimplementing it per service - that part is real. But a sidecar proxy on every pod has a real memory cost that adds up fast at scale, and below roughly 20 services, that cost is usually being paid for capability the team doesn't have enough surface area to actually benefit from yet."

Checklist

Practice this interactively
Share this lessonWhatsAppLinkedInX

Related lessons