Cruxly › System design › Scaling Building Blocks

Load Balancing, CDNs, and the API Gateway

everything the request meets before it reaches your code

Three components, and they get lumped together because they answer the same question at three different distances from the client: where is the nearest, cheapest, correct place to handle this request — as close to the edge as possible, and no closer to your database than it has to be.

Three layers, one job each

They compose in a fixed order, front to back. Each one absorbs what the next layer doesn't need to see.

Order matters: rejecting early is strictly cheaper than rejecting late. A request with bad auth or over its rate limit should never use app-instance compute. (In practice the gateway fleet also sits behind its own load balancer.)

Load balancer: algorithm, health, and stickiness

:::key Health checks make any of this safe Routing to a dead instance turns healthy capacity into guaranteed failures. Poll /health every few seconds, pull an instance after one or two missed checks, and probe to bring it back. Test real readiness: a process that's alive but stuck (deadlocked, connection pool exhausted) passes a naive TCP-connect check. :::

L4 vs L7: L4 routes by IP and port: fast, but it can't tell /api/search from /api/checkout. L7 reads the HTTP request and routes on path, header or cookie: slower, but it can send search traffic to a search-optimised fleet.

Hash-based, weighted and two-choice balancing

The previous section gave the defaults. Three refinements matter in a design review, and each answers a different failure of round robin: unequal machines, state or caches that live on one machine, and load information that is out of date by the time you use it.

Weighted: when machines are not equal. Round robin and least connections both treat backends as identical. Real fleets mix instance sizes, or are mid-rollout. A weight skews the rotation: NGINX's default is weighted round robin, and with weight=5 on one of three servers, 5 of every 7 requests go to it. Google's SRE chapter on datacenter load balancing shows why static weights are not the end of it. Plain round robin left up to a 2x spread in CPU between the least and most loaded task, mostly because queries cost different amounts. Least-loaded round robin did about as poorly in large services. Weighted round robin where backends report their own utilization (CPU, error rate) and clients adjust weights narrowed the spread significantly. Weights are a statement about capacity; the feedback is what keeps them honest.

Hash-based: when the destination must depend on the request. Hash a key (user ID, session cookie, URL, client IP) to pick the backend, and the same key reaches the same machine. Three reasons to want that:

The costs are the part to say aloud. Hash balancing gives up the one thing least-connections does: reacting to load. A heavy tenant hashes to one place and stays there. And plain hash mod N remaps nearly every key when N changes; NGINX warns of exactly this and offers a consistent option so that only a few keys move. How consistent placement works, and what it costs on rebalance, is covered in Cache Eviction and Topology and is not repeated here. Two practical notes: NGINX's ip_hash keys on the first three octets of an IPv4 address, so one office or mobile carrier behind a shared range lands on one server; and Envoy's ring hash only helps if the route supplies something to hash on. Envoy also notes that Maglev builds its table faster than ring hash but moves more keys when hosts change. A simpler design than either is to keep the state outside the app tier, so any backend can serve any request.

Power of two choices: when load information is stale. The obvious upgrade is "always pick the least loaded backend." It fails when load information is old or when many balancers decide at once: every balancer sees the same quiet backend and sends it all its traffic. Envoy's documentation names the benefit of two random samples as resistance to this herding. Picking two backends at random and sending the request to the less loaded of the two removes the herd, because no single backend is everyone's best choice. It also needs no global view: two lookups per request.

The intuition comes from the balls-into-bins result: throw n balls into n bins at random and the fullest bin holds about log n / log log n balls; place each ball in the less full of two random bins and the maximum drops to about log log n / log 2, an exponential improvement from one extra sample.

Worked arithmetic, with in-flight counts on four backends A=8, B=10, C=12, D=30. Two distinct backends are sampled, so there are six equally likely pairs, and the less loaded of each wins: AB, AC, AD go to A; BC and BD to B; CD to C. Shares of new requests: A 50%, B 33%, C 17%, D 0%. Random choice would give each 25%, and strict least-loaded would send 100% to A until its count changes. Two choices push traffic away from the overloaded backend without overcorrecting onto one.

:::key Verified November 2026

• Envoy: weighted least request uses power of two choices with two samples by default when weights are equal; ring hash and Maglev need a request value to hash on; Maglev uses a fixed table of 65,537 entries.
• NGINX: default is weighted round robin; random two least_conn samples two servers and defaults to least_conn; hash ... consistent uses ketama consistent hashing; least_time was a commercial-subscription feature before 1.31.0.
• Check the current docs of the proxy you run; defaults and licensing change.

:::

Further reading: Mitzenmacher, Richa and Sitaraman, The Power of Two Random Choices: A Survey of Techniques and Results; Eisenbud et al., Maglev: A Fast and Reliable Software Network Load Balancer (NSDI 2016); and the SRE book chapters 19 and 20.

Load balancing everywhere, not just at the front door

A real system usually has three balancers, one per tier boundary: each boundary is its own many-to-many fan-out needing the same "spread traffic, hide dead nodes" job.

Smart clients put the balancer in the caller's client library, which keeps the healthy-host list and picks one directly. It must drop unresponsive hosts, resume recovered ones, and pick up new hosts without a restart. One less hop, but the logic must be correct in every client. Redis and Cassandra clients often work this way.

CDN: what to cache, and the invalidation problem

The best trick here: make invalidation unnecessary by changing the cache key whenever the content changes.

:::warn Why a purge is the last resort A fixed URL (a CMS typo fix, a banned image) means purging across potentially hundreds of edge nodes you don't run: slow, eventually consistent, and occasionally failing silently on one node. Use TTL plus purge only for what can't be versioned. :::

API gateway: what actually belongs there

:::key Business logic does NOT belong there A rule for one product feature turns the shared front door into an undocumented, hard-to-test extra hop that every request pays for. If a rule matters to one service, put it in that service. :::

Every request passes the gateway, so it's a single point of failure by construction. Run it stateless and horizontally scaled behind its own load balancer.

How to say it in the interview

"I'd put a CDN in front of anything static, a load balancer distributing across stateless app instances behind it, and an API gateway doing auth, rate limiting and routing before any request reaches business logic — each one exists to stop work from reaching the layer behind it unnecessarily."

Checklist

• Named which of the three layers a given concern belongs to
• Chose a load-balancing algorithm and named when it breaks
• Required consistent hashing the moment any instance holds per-user state
• Specified what the health check actually tests
• Matched the balancing method to the failure: weights for unequal machines, hashing for cache or state affinity, two choices for stale load data
• Placed a load balancer at each tier boundary, not just the front door
• Named the hardware-vs-software LB tradeoff and where each fits
• Matched the balancing method to the failure: weights for unequal machines, hashing for affinity (and named its hot-key cost), two choices for stale load data
• Used content-hashed URLs for static assets rather than planning to invalidate
• Excluded per-user or auth-sensitive responses from edge caching by default
• Kept business logic out of the gateway
• Made the gateway itself stateless and load-balanced

Related lessons

• Scaling Up, Scaling Out and Splitting
• Caching
• Cache Eviction and Topology
• Distributed Cache Cluster
• Content Delivery Network
• Rate Limiting