Cruxly › System design › Principles of Distributed Design

Failure Is a Mode, Not an Exception

for every dependency, the question is what the system does when it's down — not whether it can be

The principle: in a system of enough components, something is always failing. Treating failure as an exceptional branch produces designs that are correct on the happy path and undefined everywhere else. For every dependency, specify the behaviour when it is unavailable — that behaviour is a designed feature, not an error case.

The reframing is the whole principle: "the cache is down" is not an incident to be handled later, it is a mode the system operates in, and someone decides what that mode looks like. If you do not decide, it gets decided by whatever the code happens to do.

The violation: a design that only describes the happy path

The tell is a design where every arrow points forward and nothing describes what happens if the arrow fails.

Request → Auth service → Cache → Database → Response

Ask of each hop: and if this one is down? A design that has not answered acquires its behaviour by accident:

Auth down → does every request fail closed (site is down) or fail open (unauthenticated access)? These are opposite decisions with opposite risks, and nobody chose.
Cache down → every request now goes to the database. If the cache was absorbing 95% of reads, the database just received 20× its normal load and will likely fall over too. The cache failure became a database failure.
Database down → can anything still be served, or is the whole product offline?

The cache case deserves particular attention, because it is the most common way a small failure becomes a large one. A cache is usually justified as a latency optimisation, which makes it feel optional — but once the database is provisioned assuming the cache absorbs most reads, the cache is load-bearing. Losing it does not make the system slower; it makes the system dead.

The fix: name the mode, and choose it deliberately

For each dependency, write down the degraded behaviour as a decision:

Auth down      → fail CLOSED. Reject requests. (Serving data to
                 unauthenticated users is worse than serving nothing.)

Cache down     → serve from DB, but with a concurrency limit + load shedding,
                 so the DB receives survivable load and sheds the rest
                 rather than collapsing under all of it.

Database down  → serve stale cached reads where correctness permits;
                 reject writes with a clear error. Reads degrade, writes stop.

Payment provider
     unreachable → do NOT guess. Leave the transaction PENDING and
                 reconcile from the provider's own record afterwards.

Each line is a product decision as much as a technical one, which is exactly why it must be made explicitly. "Fail closed" for auth and "serve stale" for reads are choices about what the business prefers under duress, and an engineer who makes them silently has made a policy decision on the company's behalf without telling anyone.
Load shedding deserves its own emphasis, because it is counter-intuitive: when the database is receiving more load than it can serve, the correct behaviour is to reject some requests immediately rather than accept all of them slowly. A system that queues everything under overload serves nothing and dies; one that serves 30% and fast-fails the rest stays alive and recovers when load drops. Deliberately failing some requests is what keeps the system from failing all of them.

:::key The sentence that distinguishes a senior answer Not "we'd add a circuit breaker" — a mechanism — but "when the recommendation service is down we render the page without recommendations, because the page is still useful and the alternative is losing the whole page for a non-essential module." The mechanism follows from the decision; naming only the mechanism skips the part that was actually hard. :::

Timeouts, retries, and the failure the fix creates

Two mechanisms follow directly from this principle, and both cause outages when applied naively.

Every network call needs a timeout, and the default is almost always wrong. A call with no timeout does not fail — it hangs, holding a thread or connection. Enough hung calls and the caller runs out of capacity and fails too, which is how one slow dependency takes down a service that was merely talking to it. The timeout should be derived from the latency budget: if the end-to-end budget is 200ms and this call is one of three, it does not get to spend 5 seconds.
Retries help exactly until they become the attack. When a dependency is struggling, every caller retrying multiplies load against the thing that is already failing — a retry storm converts a brief degradation into a sustained outage. The three corrections, all necessary:

Exponential backoff with jitter. Without jitter, retries synchronise into waves; the jitter is not a refinement, it is what stops the waves.
A retry budget or circuit breaker. After enough consecutive failures, stop calling entirely for a while — give the dependency room to recover instead of holding it down.
Retry only idempotent operations. Retrying a non-idempotent write can double-charge a customer. If the operation is not naturally idempotent, make it so with an idempotency key before adding the retry.
Retry at one layer, not every layer. Retries multiply across a call chain: if the client, the API, and the service each retry 3 times, one user request can become 3 × 3 × 3 = 27 calls against the struggling dependency. Pick one layer to retry (usually the one closest to the failure), and have the others fail fast.
Propagate a deadline. Pass the caller's remaining time budget down the chain, so a downstream service stops working — and stops retrying — on a request the user has already given up on.

The general lesson underneath both: the fix for a failure mode introduces its own failure mode.

Timeouts create spurious failures for slow-but-succeeding calls.
Retries create load amplification.
Circuit breakers create a window where you refuse requests the dependency could now serve.
None of these are reasons to skip the mechanism — they are the reason to name the second-order cost rather than presenting the fix as free.

How to say it in the interview

"For each dependency here I want to say what happens when it is unavailable, because at this number of components something is always down — so 'the payment provider is unreachable' is a state this system spends real time in, and I would rather design that state than discover it."

Checklist

Stated the degraded behaviour for each dependency, rather than only the happy path
Recognised that a cache sized into the capacity plan is load-bearing, not optional
Chose fail-open versus fail-closed explicitly, as a product decision
Included load shedding, and explained why rejecting some requests prevents rejecting all
Set timeouts derived from the latency budget instead of accepting library defaults
Added jitter to backoff and named it as what prevents synchronised retry waves
Restricted retries to idempotent operations, or added an idempotency key first
Named the second-order cost each remedy introduces rather than presenting it as free

Practice this interactively