Cruxly › Cloud and DevOps › Cloud & DevOps: Networking

DNS Failover, Load Balancer Layers, and the Retry Storm That Follows an Outage

TTL trade-offs, L4 vs L7, and thundering-herd retries

Prompt: design failover and load-balancing for a service where a downstream dependency has occasional transient outages. Failover must happen within roughly a minute of a real outage being detected, and when the dependency recovers, the flood of simultaneously-retrying clients must not immediately re-trigger the same outage.

30 minutes first. The naive version sets a short DNS TTL for fast failover and trusts every client to retry sensibly on its own — both assumptions are individually reasonable and, combined, still leave the exact failure mode they were meant to prevent.

DNS TTL is a trade-off, and a failover plan shouldn't fully depend on it

A short TTL (30-60 seconds) enables fast redirection of traffic during a real failover, at the cost of higher DNS query volume — a reasonable trade for anything where failover speed matters. But the trade-off has a hard floor: some resolvers and client libraries cache more aggressively than the record's TTL technically permits, meaning even a well-chosen short TTL is a best-effort signal, not a guarantee every client re-resolves on schedule.

The more robust pattern avoids needing a DNS change to propagate at all during failover — routing behind a stable address fronted by a load balancer, or an anycast prefix, moves the actual "which backend serves this request" decision to a layer that isn't subject to a client's DNS cache lifetime. Two caveats keep this honest:

• The load balancer's failover isn't instant either. It marks a backend unhealthy only after the health-check interval × the unhealthy threshold (AWS ALB defaults: 30s × 2 = 60s; tune to e.g. 10s × 2), and open keep-alive or gRPC connections stay on the old backend until they break.
• Not every load balancer has a static IP. An AWS ALB's IPs change and are reached through DNS; an NLB gives one static IP per zone. For cross-region failover without depending on DNS, an anycast front door (AWS Global Accelerator, Cloudflare, or a GCP global load balancer) gives fixed IPs and moves traffic between regions behind them. DNS-based failover is

still useful as one layer of a broader strategy; treating it as the only failover mechanism is where teams get bitten by a client that cached longer than expected.

L4 vs L7, and the HTTP/2 gotcha that breaks naive round-robin

An L4 load balancer routes by IP and port with no visibility into application content — fast, simple, but blind to what's inside the connection. An L7 load balancer parses HTTP and can route by URL path, header, or cookie, enabling content-aware routing and centralizing TLS termination — at real additional per-request processing cost. Neither is universally correct; L4 suits high-volume, latency-sensitive, protocol- agnostic traffic, L7 suits anything needing content-based decisions or policy enforcement.

The specific trap: L4 balances per TCP connection, and HTTP/2 multiplexes many logical requests over one long-lived connection. A client that opens one HTTP/2 connection and sends its next thousand requests over it has all thousand pinned to whichever single backend an L4 load balancer assigned that connection to at setup time — an intended round-robin distribution silently becomes a hot spot on one backend the moment a client's traffic pattern doesn't match the "one request per connection" assumption the naive mental model of round-robin relies on. This isn't a corner case with HTTP/2 as the default for many modern clients and gRPC — it's the common case, and Kubernetes ClusterIP Services have exactly this problem (covered two sections down).

At real scale, using both layers together is the standard answer, not a choice between them: an L4 tier absorbing raw connection volume and surviving partial backend failure, in front of an L7 tier making the actual content-aware routing decisions — each layer doing the job it's actually suited for, rather than one layer stretched to cover both.

Backoff reduces load, jitter breaks synchronization — they fix different halves

Fixed-interval retries synchronize independent clients by construction — every client that failed at the same moment (because they were all mid-request when a shared dependency had a transient outage) waits the identical fixed delay and therefore retries at the identical moment again, recreating the load spike that may have caused the outage in the first place, now aimed at a service still in recovery.

Exponential backoff (doubling the wait after each failure) reduces each individual client's total request volume during a sustained outage — fewer total retries per client over the same outage window — but it does not, by itself, fix the synchronization problem: every client is still doubling its delay on the same clock, so a population of clients that started failing together stays synchronized together, just at longer and longer intervals.

Jitter — adding randomness to each client's specific wait time — is the piece that actually breaks synchronization, spreading a population of retries across a window of time instead of concentrating them at discrete, predictable moments. Backoff and jitter solve two different problems (total volume, and synchronization) and a resilient retry strategy needs both; backoff alone is a common, incomplete fix that still produces a retry storm, just a slightly less frequent one.

gRPC behind a ClusterIP, retry amplification, and deadlines

The HTTP/2 trap hits hardest inside Kubernetes. A ClusterIP Service is implemented by kube-proxy (iptables/IPVS) or an eBPF replacement like Cilium — all of them balance per connection. A gRPC client opens one connection to the Service, lands on one pod, and sends everything there. Scale the server from 3 to 10 pods and the new 7 get almost nothing until clients reconnect. The fixes:

• Client-side balancing: a headless Service (clusterIP: None) returns every pod IP, and the gRPC client's round_robin policy spreads requests across them.
• An L7 proxy that balances per request — a mesh sidecar, or a proxy like Envoy in front of the service.
• MaxConnectionAge on the server, which makes clients reconnect periodically so new pods get picked up. It's a cheap fix, but a partial one.

Retries multiply across layers. If five services in a chain each retry 3 times (4 attempts), one failing request at the bottom can become 4⁵ = 1,024 attempts against the struggling service. Three rules keep this under control:

• Retry at one layer, usually the one nearest the caller or the mesh, not at every hop.
• Use a retry budget: allow retries only up to a fixed share of normal traffic (around 10-20%), so retries stop growing when everything is failing. Envoy and Finagle both support this.
• Retry only idempotent operations, or ones that carry an idempotency key.

Propagate deadlines, not just timeouts. If the caller gave up after 2 seconds, work still running downstream is wasted load. gRPC passes the remaining deadline along with each call, so every hop knows how much time is left and can stop early. Pair this with circuit breakers: they fail fast instead of queueing requests onto a dependency that's already down.

How to say it in the interview

"A short DNS TTL gets you fast failover in theory, but some resolvers and clients cache more aggressively than the TTL allows, and the moment the dependency recovers, every client that failed during the outage retries - and if none of them add randomness to their retry timing, they all arrive back at the same moment, re-triggering the very outage that just ended."

Checklist

• Didn't rely on DNS TTL alone for failover speed, using a static IP behind a load balancer for the actual failover decision
• Chose L4 vs L7 based on whether content-aware routing was actually needed, not by default
• Accounted for HTTP/2 connection multiplexing when relying on an L4 load balancer's per-connection round-robin
• Used exponential backoff specifically to reduce each client's total retry volume during an outage
• Added jitter specifically to break synchronization across clients, on top of backoff
• Budgeted failover time as health-check interval × unhealthy threshold, plus draining of long-lived connections
• Fixed gRPC load imbalance behind a ClusterIP with client-side balancing, an L7 proxy, or MaxConnectionAge
• Retried at one layer with a retry budget and propagated deadlines down the call chain

Related lessons

• Service Mesh: What a Sidecar Proxy Actually Buys You, and What It Costs