Cruxly › System design › Scaling Building Blocks
Caching
buys latency, costs freshness
Open the interactive lessonA cache trades correctness-over-time for speed. Everything interesting about caching is in the trade: what goes stale, for how long, who notices, and what happens the moment the cache is empty.
What problem it actually solves
The pain: 60,000 feed reads/s × ~40 ms of database time each = 2,400 s of database work every second. That's 60+ replicas just to serve reads.
A cache only helps if reads repeat. If traffic is uniform (every key about equally popular), there's no hit rate to be had: the cache is just a second copy of the data plus a staleness problem. Saying "don't cache this" is a correct answer.
Real traffic is skewed. Here the top 5,000 conversations get 60% of reads, so caching them cuts database load by 60%. Be precise about what that buys:
- Load drops with the hit rate. 60% hits = 60% fewer database reads.
- p99 latency doesn't move at 60%. The 40% that miss still pay database latency (plus a cache hop).
- p99 only drops once the hit rate passes ~99%, because p99 is set by the slowest 1% of requests.
The number to carry into the interview is the hit rate, and skew is what creates it.
Cache-aside, write-through, write-behind
On a read or a write, who touches what, and in what order?
| Strategy | How it works | Use it when | The cost |
|---|---|---|---|
| Cache-aside (lazy loading), the default | The app reads the cache; on a miss, the app reads the DB and fills the cache (in read-through, the cache layer does that load itself) | Almost always | A miss is two hops, slower than no cache. A hot key expiring causes a stampede (see below) |
| Write-through | Every write updates the cache and the DB together | Reads must be fresher than a TTL allows, and writes are rare | Every write pays two round trips. Not atomic: a crash between the two, or two racing writers, leaves them disagreeing. Keep a backstop TTL |
| Write-behind | Write the cache, ack at once, flush to the DB later | Losing a few recent writes is fine: view counters, telemetry | If the cache dies before the flush, writes you already acked are lost. Never for anything a user would call a transaction |
Invalidation, which is the actual hard part
"There are only two hard problems in computer science: cache invalidation, naming things, and off-by-one errors."
Every strategy fails somehow. The skill is naming which failure you accept for this data.
| Strategy | Use it when | How it fails |
|---|---|---|
| TTL only | A few seconds or minutes stale is invisible: follower counts, trending, suggestions | Stale for up to the TTL, but there's no invalidation logic to get wrong |
| Invalidate on write | A write must show up immediately | Has a race (below). Always pair it with a short backstop TTL |
Versioned keys (user:42:v7) | You'd rather bump a version than delete | Readers must look up the current version, and that lookup can be stale. The race moves; it doesn't disappear |
The invalidate-on-write race: a reader misses just before the delete, reads the old value from the database, and fills the cache just after the delete. The stale value now lives for a full TTL. "Delete after commit" doesn't fix this. What does:
- Leases / compare-and-set fills (Facebook's memcache): a delete invalidates outstanding leases, so the stale fill is rejected.
- Delayed double-delete: delete again a moment later, catching a fill that raced in.
- A short backstop TTL, so whatever slips through heals itself.
Defend the choice for the data: "TTL of 30 s. A follower count half a minute stale is invisible, and I skip invalidation entirely."
Thundering herd and cache stampede
The most common caching follow-up, and a real outage pattern:
Two shapes worth telling apart:
- Hug of death: an unpredictable hot key. A viral link lands on a page whose cache entry is cold or just expired. You can't pre-warm what you didn't know would be hot.
- Fleet-wide config-key stampede: a shared key (feature flag, pricing table) read by every instance. Fine for a year, then the fleet grows 10x and hundreds of instances miss it at the same moment. The key didn't change; the fleet around it did.
| Failure | Fix | The interviewer's follow-up |
|---|---|---|
| Many keys filled together with the same TTL expire together (a cache avalanche) | TTL jitter: 300 ± random(30), never a flat 300 | "Does jitter help one hot key?" No. One key still expires at one moment |
| One hot key expires and N requests all refetch it | Single-flight: one request locks and refills, the rest wait. Or probabilistic early refresh (XFetch), which refreshes slightly before expiry | "What if the lock holder crashes?" The lock needs a timeout or lease, or every waiter blocks forever |
| Every waiter blocks on a slow refill | Serve stale while revalidating: return the old value, refresh in the background | "How stale can it get?" Name the bound, and data where it's unacceptable, like an account balance |
Common miss: offering one fix as the answer. Jitter and single-flight solve different problems.
:::key Mention cold start unprompted A cold cache (restart, failover) drops the hit rate to zero and sends all traffic to the database at once. Warm the cache before it takes traffic, or ramp traffic in gradually. :::
Where to put it
Caching is a decision at every layer a request passes through.
| Layer | Use it for | The catch |
|---|---|---|
| Client | Content that can't change under its name: hashed asset files, signed URLs | Can never be invalidated once cached |
| CDN / edge | Public static content: images, video, assets | Cuts latency and bandwidth, but a purge across many edge locations is slow and coarse |
| App memory (per instance) | Small, slow-changing config only | 10 instances = 10 independently stale copies, with no coordination |
| Shared tier (Redis / Memcached), the default | Most application data | One network hop, and one more system to operate |
| Database buffer pool | A working set that already fits in DB memory | Nothing. An external cache would add a hop and buy nothing. A legitimate answer, not a cop-out |
When caching is the wrong answer
The strongest signal in a caching discussion is naming when you wouldn't cache. Say at least one unprompted:
- Write-heavy (~1:1 reads to writes): you invalidate as fast as you fill, so the hit rate collapses.
- Uniform access: no skew, no hit rate.
- Strong consistency required (balances, inventory, seats): a stale read is a wrong answer served with confidence. Cache around it (the product catalogue), never the count itself.
- Already cheap: an indexed lookup under 1 ms. You save a millisecond and own an invalidation strategy forever.
- Tiny dataset: it already fits in the database's buffer pool.
How to say it in the interview
"Reads are 40x writes here and the same few thousand keys dominate, so a cache in front of the database is the highest-leverage thing I can add. It buys me database load — and p99 latency once the hit rate is high enough — and costs me a staleness window — let me say how long that window is and who sees it."
Checklist
- Justified the cache with a hit-rate argument from access skew
- Chose cache-aside / write-through / write-behind and said why
- Named the invalidation strategy and the failure you chose to keep
- Matched the fix to the stampede: single-flight, serve-stale or early refresh for one hot key; jitter for many keys expiring together
- Named which stampede shape you're defending against — unpredictable hot key or fleet-wide blast radius
- Mentioned cold-start behaviour after restart or failover
- Placed the cache at a specific layer and justified that layer
- Named a case in this system where caching would be wrong