Cruxly › System design › Principles of Distributed Design

Make the Expensive Path Rare

if you cannot make an operation fast, make it uncommon

The principle: some operations are irreducibly expensive — a cross-shard join, a consensus round, a cold read from disk, a call to a slow third party. You usually cannot make them cheap. You can almost always make them rare, and rare-and-expensive is a system that performs well.

This is the principle underneath caching, denormalisation, batching and precomputation, all of which are usually taught as separate techniques. They are one idea wearing four costumes: move work off the common path.

The violation: optimising the wrong end

A candidate designing a feed notices that assembling it requires querying every followed account and merging results — expensive. They respond by optimising the query: better indexes, a faster database, more replicas.
This is effort spent making an expensive operation slightly less expensive, while leaving it on every single request. A 40% improvement on an operation that happens 40,000 times a second is a worse outcome than eliminating it from 95% of requests, and the second is usually the easier engineering.

The same error, in other shapes:

Recompute a user's recommendation set on every page load
  → it was expensive; now it is expensive 40,000 times/second

Count rows with SELECT COUNT(*) on every dashboard render
  → the count is expensive and the answer barely changes between renders

Call the pricing service for every line item in a cart
  → N calls on the hot path where one batched call would do

The diagnostic question is not "how do I make this faster" but "how often does this actually need to happen?" Most expensive operations are being performed far more often than their inputs change, which is precisely the condition that makes them removable.

The four ways to make a path rare

Precompute. Do the work when the inputs change instead of when the answer is requested. A trie node caching its own top-10 completions turns a subtree walk into a pointer dereference. The cost is freshness — the precomputed answer is as old as its last computation — and storage for the materialised result.
Cache. Do the work once and reuse the answer for a while. The distinguishing feature versus precomputation is who triggers it: a cache fills on the first request for a value (lazily, on demand), while precomputation fills on write. Caches suit unpredictable access; precomputation suits known-in-advance queries.
Batch. Do the work for many items in one expensive operation instead of many. One call fetching 200 prices costs far less than 200 calls, because most of the cost is per-call overhead — connection, round trip, query planning — not per-item work. The cost is latency for the first item, which now waits for the batch to fill.
Approximate. Accept an answer that is nearly right when nearly right is sufficient. An exact COUNT(*) over a billion rows is expensive; a count maintained incrementally, or a probabilistic sketch, is nearly free. The cost is exactness, which is often not needed — nobody makes a decision differently because a dashboard says 1,048,203 instead of "about 1.05 million."

:::key How to choose between them Ask what the expensive operation is expensive in. Expensive in computation → precompute or cache. Expensive in round trips → batch. Expensive in the amount of data that must be examined → approximate. Choosing the wrong one produces a solution that adds complexity without removing cost. :::

The discipline: the rare path must still work

The failure mode of this principle is a design that makes the expensive path rare and then stops thinking about it.
The rare path is the one that runs during your worst incident. A cache with a 99% hit rate means the database sees 1% of reads — until a deploy flushes the cache, or a cold region comes online, and suddenly it sees 100%. If the database was provisioned for 1%, the rare path has just become the only path and nothing survives it.

Three obligations follow:

Provision the rare path for more than its steady-state share. Not for 100% necessarily, but enough that a cache flush degrades service rather than ending it — combined with the load shedding from the failure-modes principle, which is what makes "not enough capacity" survivable.
Exercise it. A path that runs on 1% of requests in production runs on 0% in testing, and code that never runs is code that does not work. Cold-start behaviour after a cache flush should be something you have observed deliberately, not discovered.
Prevent the stampede. When a hot precomputed value expires, every concurrent request misses simultaneously and all of them recompute it at once — a thundering herd where one recomputation would have sufficed. The fix is a single-flight lock, or refreshing slightly before expiry so the old value serves while one request recomputes.

Stated as a rule: making a path rare changes its frequency, not its correctness obligations. A rare path that is broken is an outage waiting for the day its frequency changes — and its frequency will change, because that is what incidents are.

How to say it in the interview

"I can't make this operation cheap — a cross-shard query is going to cost what it costs. So the design question becomes how to make it rare rather than how to make it fast, which usually means precomputing the common case and accepting the expensive path only for the long tail."

Checklist

Asked how often the operation needs to happen before asking how to make it faster
Recognised caching, precomputation, batching and approximation as one principle in four forms
Chose the technique by what the operation is expensive in, not by familiarity
Distinguished precomputation (fills on write) from caching (fills on first read)
Named the cost each technique pays — freshness, storage, first-item latency, or exactness
Provisioned and exercised the rare path rather than assuming it stays rare
Prevented the thundering herd on expiry with single-flight or early refresh

Practice this interactively