Cruxly › System design › Foundations of Data Systems

Replication and Consistency

copies disagree; decide who notices

Replication is how you survive machine failure and serve reads at scale. The moment there is more than one copy, the copies can disagree, and your job is to decide precisely who is allowed to see a disagreement and for how long.

Two reasons to replicate, and they are different

Reason	What it buys	When it applies
Durability and availability	N copies survive N−1 machine failures	Always, for anything you can't regenerate, whatever the read load
Read scaling	Replicas multiply read capacity	Only when reads dominate. At 1:1, every replica still applies every write, so you gain almost nothing

Keep them apart: "Three replicas for durability; reads go to replicas only where lag is acceptable." "I'll add replicas for scale" leaves the interviewer unsure you know the difference.

Single-leader

All writes go to one primary; replicas follow its log. Almost every relational deployment works this way, and for most designs it's the right answer.

You get no write conflicts (one writer, one order) and mature tooling. You pay:

Cost	Why	Fix
Write bottleneck	Replicas apply every write too, adding no write capacity	Partition
Replication lag	Async ack comes before followers have the data: ms normally, seconds under load. Users see "I posted it and it disappeared"; the write isn't lost, just not there yet	Read-your-own-writes
Risky failover	Detection needs a timeout: slow detection or false positives	Fencing

:::key Read-your-own-writes After a user writes, route that user's reads to the primary for a few seconds. You don't fix lag; you hide it from the one person who would notice. :::

:::race Split brain and fencing Promoting a follower that was behind loses acknowledged writes, and two nodes may both think they're primary. Fix: a monotonically increasing epoch the storage layer checks on every write, so the demoted primary's stale-epoch write is rejected (the fencing token from the distributed lock exercise). :::

Multi-leader and leaderless

Multi-leader: several nodes accept writes, usually one per region: local write latency, survives a region outage, and real write conflicts when two regions edit one record. Name the rule:

Rule	Trade-off
Last-write-wins	Simple, silently discards data; clock skew makes "last" a lie
Application merge	For structured data, e.g. union two shopping carts
CRDTs	Merge deterministically by construction; only counters, sets, text and similar

Leaderless (quorum): write to W of N nodes, read from R. If W + R > N, read and write sets overlap on at least one node, so a read sees the latest write. W=N, R=1 gives fast reads and fragile writes; W=1, R=N the reverse; W=R=2, N=3 is the balanced default.

:::warn W + R > N is not strict consistency (the differentiator)

Overlap means you saw the write, not which concurrent write wins.
A failed write can leave some replicas updated, with no rollback.
Sloppy-quorum writes, taken by stand-in nodes during a partition, can be invisible to a later quorum read.

:::

Synchronous versus asynchronous

Mode	Ack when	Lost if the primary dies	Write latency
Synchronous	A replica has it	Nothing	Replica's: 100 ms+ cross-region; writes block if it's slow or down
Asynchronous	Immediately	Acknowledged writes	Fast
Semi-synchronous	One replica has it, rest async	Nothing if one machine dies	Only the fastest replica's

:::key The answer being fished for Synchronous within a region (sub-millisecond), asynchronous across regions, and the exposure stated: a few seconds of writes if a whole region is lost. :::

Saying consistency precisely

Name the guarantee, the window, and who notices. Strongest first:

Guarantee	Meaning	Cost
Linearizable	Acts like one copy; every read sees the latest write	Coordination per operation; cross-region latency
Sequential	Everyone sees the same order, not necessarily current	Single-leader ordering
Read-your-own-writes	You see your own changes; others may lag	Pin the writer's reads to the primary
Monotonic reads	Time never goes backwards for you	Pin a session to one replica
Eventual	Converges once writes stop	Nothing, but explain the window

Monotonic reads is the underrated one: without it, reads from two replicas can show a comment appearing, then vanishing. Sticky sessions fix it.

Attach guarantees to data, not the system: "Seat inventory is linearizable, since double-selling is unacceptable; the 'N people viewing' badge is eventual, maybe ten seconds stale, which nobody can detect." That's what real designs do.

The vocabulary: failover, data replication, and shared-nothing

Interviewers use these as shorthand and expect them back precisely.

Term	Means	The precise part
Failover	A secondary takes over when the primary is detected dead	A mechanism, not a replication mode. Say what detects the failure and what stops the old primary acting as one
Data replication	An active node's writes are mirrored to standby copies	Makes failover safe: a secondary helps only if it already has (nearly) all the primary's data
Shared-nothing	Nodes share no memory and no disk, and coordinate only over the network	Contrast: shared-disk, many nodes on one storage array

Fencing a primary that owns its disk: an epoch only fences where something shared checks it (storage, a lock service). Otherwise use leader leases (the old primary stops accepting writes when its lease expires, before a new one is granted) or STONITH ("shoot the other node in the head": power off or isolate it).

Shared-nothing, precisely:

Scales by adding nodes, plus rebalancing: shards reassigned, data moved.
A failure is contained to that node's slice; without a replica, that shard is down.
Not "no coordinator": Kafka (KRaft), Vitess, HDFS (NameNode) and Spanner all run a metadata service. What's avoided is shared data storage.

How to say it in the interview

"I need replicas for durability regardless. The design question is whether reads may hit a replica — for the feed, yes, a two-second lag is invisible; for the author's own post, no, so I'll pin their reads to the primary briefly after a write."

Checklist

Separated replication-for-durability from replication-for-read-scale
Chose single-leader, multi-leader or leaderless and said why
Quantified replication lag and named who sees it
Solved read-your-own-writes explicitly
Described failover including fencing against split brain
Chose sync / async / semi-sync and said what is lost on failure
Named a conflict-resolution rule if more than one node accepts writes
Assigned a different consistency level per data class
Used failover, data replication, and shared-nothing as distinct, precise terms
Stated shared-nothing precisely, including what it doesn't give you

Practice this interactively