Cruxly › System design › Principles of Distributed Design

Truth Has One Home

for every fact, name the single component that owns it — everything else holds a copy

The principle: every fact in a distributed system should have exactly one authoritative owner. Every other place that fact appears is a derived copy, and must be treated as possibly stale. Two components that both believe they own the same fact will eventually disagree, and there will be no principled way to resolve it.

Copies are fine — they are how systems get fast. What is not fine is ambiguity about which copy is the truth.

The violation: two owners for one fact

This mistake hides inside designs that look well-factored.

Inventory service   → stores stock_count per product  (authoritative?)
Search index        → stores stock_count for filtering (authoritative?)
Cart service        → decrements its own stock_count on add-to-cart (!)

• The cart service writing stock_count is the fatal line. Now two components mutate the same fact independently, and the system has no way to say which is correct when they differ — because both are, locally, doing exactly what they were built to do.
• The consequence is not an inconsistency you can repair, it is an inconsistency you cannot even detect. The inventory service says 3 in stock; the cart service says 1. Neither is wrong by its own reckoning. Any "fix" is a policy decision made after the fact, and it will be made under pressure by someone guessing.
• The same violation in a subtler form: a denormalised copy that someone starts writing to. A search index holding stock_count for filtering is correct and useful — as a read-only derived copy. The moment any code path updates the index without going through inventory, the index has silently become a second owner.

The fix: one writer, many readers, and an explicit propagation path

Inventory service  → SOLE owner of stock_count. All mutations go here.
Search index       → derived copy, read-only, updated via change events
Cart service       → holds a RESERVATION (its own fact, which it does own),
                     and asks inventory to decrement

Propagation: inventory emits StockChanged → index and caches consume it

• Notice the cart service did not lose its job — it gained a clearer one. A reservation is a genuinely different fact from stock count, and the cart legitimately owns it. Most apparent ownership conflicts dissolve this way, by discovering that the two components were actually talking about two different facts that had been conflated under one name.
• Every copy must be explicitly labelled derived, and derived copies may be stale. This is not a weakness to apologise for; it is the contract. The search index showing an item as in-stock a second after it sold out is acceptable because the authority is consulted before the order is confirmed. Staleness is safe precisely when the design routes the decision that matters back through the owner.
• The rule that follows: never make an irreversible decision on a derived copy. Filter search results by the index — fine. Confirm an order against the index — not fine. The question to ask of any read is "if this value is five seconds stale, what breaks?" If the answer is "we oversell and refund a customer," that read belongs at the authority.

Where this principle is hardest: caches, and the reason invalidation is difficult

• A cache is a derived copy by definition, which makes cache invalidation an instance of this principle rather than a separate topic.
• The owner changed a fact; every derived copy must learn about it. The difficulty is that the owner often does not know who holds copies.

The three strategies, each conceding something different:

• TTL (expire after N seconds). Simple and needs no coordination; the owner does not have to know about the cache at all. Concedes a bounded window of guaranteed staleness — you have chosen to serve wrong data for up to N seconds.
• Write-through / explicit invalidation. The writer updates or evicts the cache as part of the write. Tighter, but now the writer must know every cache that exists, and a missed one is a permanently stale copy. Couples the owner to its readers.
• Event-driven invalidation. The owner emits a change event; caches subscribe. Decoupled and scales to many consumers — at the cost of running a delivery system whose own failures now produce stale copies, and whose ordering guarantees you must reason about.

• The honest framing in an interview is that TTL is usually correct and usually underrated. Candidates reach for event-driven invalidation because it sounds sophisticated, and a three-second TTL would have satisfied the requirement with a fraction of the machinery.
• The question is only ever "how stale can this specific fact be before something bad happens?" — asked per fact, not once for the whole system. A product's name can be a day stale; the price shown on a listing page can be seconds stale.
• And for the moment that commits money, don't rely on any cache. Every invalidation strategy here is asynchronous or can miss, so none of them makes a cached price exact. At checkout — an irreversible decision — read the price from its owner (the pricing service or database) and charge that. Caches are for browsing; the authority is for decisions.

:::key The connection back to the principle Every one of these strategies is an answer to the same question: how does a derived copy learn that the truth moved? Framing invalidation this way is what makes it tractable, rather than the folk wisdom that it is simply one of the two hard problems. :::

How to say it in the interview

"For each piece of data here I want to name which component is the authority and which are holding derived copies, because the failure mode I am trying to avoid is two places that both think they are right — at that point reconciliation is guesswork, and guessing about money or inventory is how you end up with an incident that has no clean fix."

Checklist

• Named the single authoritative owner for each significant fact in the design
• Labelled every other location holding that fact as a derived copy that may be stale
• Resolved an apparent ownership conflict by discovering two distinct facts under one name
• Kept irreversible decisions on the authority, never on a derived copy
• Framed cache invalidation as 'how does a derived copy learn the truth moved'
• Chose a staleness strategy per fact based on its tolerance, not once for the whole system
• Considered a plain TTL before reaching for event-driven invalidation

Related lessons

• State Gravity
• Asymmetry First
• Failure Is a Mode, Not an Exception
• Make the Expensive Path Rare
• Boundaries Are Where Guarantees Die