Cruxly › System design › Microservices and Events

API Gateway

one front door for services that disagree about almost everything

Prompt: design an API gateway sitting in front of a fleet of internal microservices — some REST, some gRPC, each with different auth requirements and different rate limits. Clients should see one consistent API surface regardless of what's behind it.

40 minutes first. This exercise is mostly about composing patterns already covered elsewhere in this track, correctly, in one place.

Applying the four moves: Scope — one consistent API surface over REST and gRPC services with per-service auth and rate limits; the services themselves out. Estimate — the gateway sees every request, so its own added latency is multiplied by total traffic; a few milliseconds per hop is a real budget item. Sketch — routing, auth, rate limiting, and protocol translation as stages in one request path. Deep dive — this is composition, so the deep dive is about where it breaks as a whole: the gateway becomes a single point of failure and a shared blast radius, so it needs to be stateless, horizontally scaled, and must fail open or closed deliberately per concern.

Scope, and why this is a composition exercise

In scope: request routing to backend services, authentication/ authorization enforcement at the edge, per-client and per-route rate limiting, protocol translation, basic request/response transformation. Out of scope: the backend services' own internal logic, service mesh concerns (service-to-service traffic that never passes through this gateway — a related but distinct system), full API management/developer portal features.

This exercise deliberately does not introduce a new hard algorithmic problem. Its value is in composing several patterns already covered elsewhere in this track — rate limiting, caching, authentication — correctly, at a single chokepoint that every request in the system passes through, where a mistake or an added millisecond multiplies across all traffic. Recognising "I've solved each of these pieces already, the exercise is integration" is itself a strong, mature answer rather than treating each piece as a fresh problem.

The request pipeline, and why order matters

This is not an arbitrary pipeline order — each stage depends on information the previous one produced.

Authentication has to run before rate limiting, and this is not arbitrary. The rate limiter (covered in depth elsewhere in this track) needs a stable identity to key its buckets on — per-user or per-API-key limits, not per-IP, because many legitimate users can share one IP behind a NAT and one hostile client can rotate through many. Running rate limiting before authentication forces a weaker, IP-based key, which is both less accurate and easier to evade.

Routing decides which backend a validated, rate-limit-cleared request goes to — typically path-based (/users/* to the user service, /orders/* to the order service), sometimes header- or version-based for API versioning. This is a lookup table, not a novel data structure problem; the interesting design decisions are upstream of it.

Protocol translation and response caching

Clients speak REST/JSON; backends may speak gRPC, or an older SOAP service, or an internal binary protocol. The gateway is the place this translation happens once, rather than every client needing to speak every backend's native protocol — the same "one owner of a subsystem's real shape" motivation as the Facade pattern covered elsewhere in this track, here applied at the network-protocol level instead of a class's API surface.

client → gateway: POST /users/42  (REST/JSON)
gateway → backend: UserService.GetUser(id=42)  (gRPC)
backend → gateway: User{id, name, email}  (gRPC response)
gateway → client: {"id": 42, "name": ..., "email": ...}  (JSON)

Response caching at the gateway needs the same discipline as caching anywhere else in this track — a hit-rate justification, an explicit invalidation strategy, and honesty about which endpoints are safe to cache at all. A GET /products/42 product-catalogue lookup is a strong caching candidate; POST /orders obviously is not, and GET /users/42/balance should not be cached at the gateway layer regardless of how tempting the latency win looks, for the same strong-consistency reasons that ruled out caching in the seat-booking and payment-ledger exercises elsewhere in this track.

Why the gateway itself must not become the bottleneck

Every request in the entire system passes through this one component, which means any latency added here is a tax on all traffic, and any outage here is a total outage — this is the central operational fact the whole design has to respect.

Keep the gateway stateless and horizontally scalable. Auth token validation should not require a database round-trip on every request if avoidable — cache validated tokens briefly, or use a self-contained, verifiable token format (a signed JWT) that can be validated with no external call at all, trading a small revocation-latency window (a revoked token remains technically valid until its signature-embedded expiry) for eliminating a database hop from the hottest path in the entire system.

Circuit breakers per backend, not global. If the order service is degraded or down, the gateway should fail fast for order-service requests specifically — after a threshold of failures, stop forwarding to it for a cooldown period and return an immediate error rather than waiting out full timeouts — while requests to every other, healthy backend continue completely unaffected. A single global circuit breaker would take down routing to every service the moment any one of them has a bad minute, turning one backend's localized problem into a total outage.

Failure modes to raise unprompted:

The gateway's own deploy is a single point of failure for everything. Run multiple gateway instances behind a load balancer, with rolling deploys — never a full-fleet restart that drops all traffic simultaneously.

A misconfigured route silently black-holes traffic. Validate routing configuration before it takes effect (a staging/canary check), because a gateway with no traffic to a backend produces no errors on the gateway side at all — from the gateway's perspective, requests to a black-holed route simply vanish, which makes this a nasty, silent failure to detect without deliberately monitoring for it.

Aggregating multiple backend calls into one client-facing response (a common gateway feature — "get user profile plus their recent orders in one call") means the gateway now waits on the slowest of several backend calls, and a single slow backend call degrades every aggregated endpoint that includes it. Set aggressive per-backend timeouts specifically for aggregation paths, and decide explicitly whether a partial result (some backends responded, one timed out) is an acceptable degraded response or a hard failure — that decision should be made once, deliberately, not discovered during an incident.

How to say it in the interview

"An API gateway isn't one new problem — it's the place where routing, auth, rate limiting and protocol translation all have to compose correctly at the front door, and the interesting part is making sure adding one more concern here doesn't become the next latency bottleneck for every request in the system."

Checklist

Recognised the exercise as composing already-covered patterns rather than a new algorithmic problem
Ordered authentication before rate limiting and explained why the order is load-bearing
Treated protocol translation as the gateway's job, done once, rather than pushing it to every client
Applied the same caching discipline (hit-rate justification, explicit invalidation) at the gateway as anywhere else
Named the gateway as a total-outage single point of failure if handled carelessly
Used self-contained signed tokens to avoid a database round-trip on every authenticated request
Scoped circuit breakers per backend rather than one global breaker
Named a misconfigured route that silently black-holes traffic as a distinct, hard-to-detect failure mode
Set per-backend timeouts specifically for response-aggregation endpoints
Decided explicitly whether a partial aggregation result is acceptable, rather than discovering it during an incident

Practice this interactively