Cruxly › System design › Principles of Distributed Design

Asymmetry First

find the axis where the distribution is wildly uneven, and design to that

The principle: real workloads are almost never uniform. Some users have a thousand times more followers, some keys are read a million times more often, some records are a hundred times larger. Find the axis with the most extreme skew and design for that axis first, because the average case was never the thing that was going to break.

A design built around the mean is a design that works in a spreadsheet and fails in production, where the tail is not an edge case — it is the traffic.

The violation: designing for the average

• A candidate is asked to design a social feed. They estimate 200 million users, average 500 followers, and confidently build fan-out-on-write: every post is copied into each follower's feed list.
• The arithmetic is correct. 4,000 posts/second × 500 followers = 2 million inserts/second. Large, but tractable.

Then the interviewer says: "what about a celebrity with 100 million followers?"

• One post now generates 100 million inserts. At any sustainable write rate this takes minutes to hours to propagate, during which the feed is wrong for most of those followers.
• The design did not degrade gracefully — it fell off a cliff, because the number it was built around (500) is not a number that describes the system. It describes the middle of a distribution whose maximum is 200,000 times larger.
• The general failure: an average is a summary statistic that deletes exactly the information the design depends on. Mean follower count, mean object size, mean request rate per key — each one hides the tail that determines whether the architecture survives.

"Average 500 followers, so fan-out costs 500 writes per post."
→ true of almost every user, and irrelevant to the one that breaks the system

The fix: split the design along the axis of skew

• Once the skew is named, the fix usually presents itself: handle the head and the tail with different mechanisms, split at a threshold you can defend.

followers < ~100,000  → fan-out on write  (cheap: bounded copies per post)
followers ≥ ~100,000  → fan-out on read   (cheap: post stored once, merged in)

Feed read = precomputed list  ⊕  merge of the few celebrities you follow

• Why this works is worth stating precisely, because it is not merely "handle two cases." The two mechanisms fail in opposite directions along the same axis: fan-out-on-write gets more expensive as follower count rises, fan-out-on-read gets more expensive as followed-count rises. Splitting on follower count puts each account on the mechanism that is cheap for it. Neither mechanism has an unbounded worst case any more.
• The threshold is derived, not decreed. It sits where the write cost of copying exceeds the read cost of merging — and the honest phrasing in an interview is "call it 100,000 and I would tune it from the observed distribution," which demonstrates that you know it is an empirical parameter rather than a magic constant.
• The same move recurs everywhere once you look for it: hot keys get their own cache tier while cold keys share one; large objects go to object storage while small ones live inline in a row; high-volume tenants get a dedicated shard while small ones pool. In each case the design has a head path and a tail path because the distribution had a head and a tail.

Finding the axis when nobody hands it to you

The skew is rarely stated in the prompt. Three questions surface it reliably:

1. "What is the ratio between the median and the maximum?" Ask it about every quantity the design touches — followers, object size, requests per key, rows per tenant. A ratio under about 10 means the average is a usable design input. A ratio in the thousands means the tail is the design.
2. "Which single entity, if it became extremely popular tomorrow, would hurt most?" This finds the hot-key problem before it finds you. In a URL shortener it is one viral link; in a chat system one enormous group; in a marketplace one product during a sale.
3. "Is the skew fixed, or does it move?" A celebrity account is a stable hot spot you can special-case in advance. A flash-sale product is a moving hot spot — the identity of the hot key changes hourly, so a static allowlist is useless and the mechanism must detect and adapt at runtime. These need different answers, and conflating them is a common error.

:::key Why this principle is listed second, not last

• Asymmetry determines the architecture, so it has to be found during estimation — before any boxes are drawn.
• A design sketched around the mean and then patched for the tail is almost always worse than one whose shape came from the skew in the first place, because the split usually wants to be structural rather than bolted on.

:::

How to say it in the interview

"Before I pick an architecture I want to know the shape of the distribution, not just its average — specifically the ratio between the median and the maximum, because if that ratio is four orders of magnitude then the whole design is really about handling the tail and everything else is scaffolding."

Checklist

• Asked for the ratio between median and maximum, not just the average
• Identified the specific axis with the most extreme skew before sketching any components
• Split the design so head and tail use mechanisms that get expensive in opposite directions
• Derived the split threshold from where the two costs cross, and called it tunable
• Asked which single entity going viral would hurt most, to find hot keys deliberately
• Distinguished a fixed hot spot from a moving one and chose the matching mechanism

Related lessons

• State Gravity
• Failure Is a Mode, Not an Exception
• Truth Has One Home
• Make the Expensive Path Rare
• Boundaries Are Where Guarantees Die