Cruxly › System design › Databases by Name: Internals and Terms

Cassandra and ScyllaDB: the ring, repair, compaction and Discord's migration

a masterless ring that scales writes until compaction, repair and one hot partition remind you what it costs

Cassandra has no leader, so there is nothing to fail over, and nothing that keeps the replicas identical either: hints, read repair and scheduled repair do that. This lesson covers the ring and its tokens, what a write actually does, the repair deadline set by gc_grace_seconds, compaction as a schema decision, the timeouts that cause retry storms, and what Discord's migration from Cassandra to ScyllaDB shows about hot partitions and application-layer protection.

The ring: tokens, vnodes and replica placement

Cassandra is a hash ring where a key's token decides its replicas, and no node is special. The Dynamo architecture page of the Apache Cassandra docs (the 5.0 documentation, labelled prerelease on the page) describes the model: nodes own one or more tokens on a continuous ring, a key hashes to a token, and replicas are found by walking the ring clockwise. When nodes join or leave, only a small fraction of keys moves. Conflicts between replicas are resolved last-write-wins using a timestamp on every mutation, deletes included. This is the model of the Dynamo paper that NoSQL modelling in practice: DynamoDB, Cassandra, MongoDB already used for partition-key design.

Virtual nodes. Instead of one big token range per machine, each node takes many tokens. The docs list the benefits: data is balanced when nodes are added or removed, and the load of a down node spreads across many peers. They list the costs too: each token has more neighbours, which raises the probability that an outage involves a pair of nodes that share data, and cluster-wide repair has more ranges to cover.

The default depends on the version, so check it. The Dynamo page says Cassandra 2.x used random token assignment with 256 tokens per node as the default, while 3.x and later added a deterministic allocator that needs far fewer. The current cassandra.yaml documentation (5.0) lists num_tokens with a default of 16. A fleet that started on 2.x with 256 tokens and one that started recently with 16 behave differently in repair duration and streaming, so "how many tokens do we have" is a real upgrade-planning question. Fewer tokens risk imbalance if the allocator is not used; more tokens make operations slower.

Replica placement. The replication factor (RF) is the number of copies, each on a distinct physical node. NetworkTopologyStrategy takes a factor per datacentre and prefers different racks; if there are fewer racks than the factor, some racks hold several replicas. The docs recommend it for all production use, including a single datacentre. SimpleStrategy has no datacentre or rack awareness, and the page calls it useful only for testing clusters of unknown layout. A cluster built on SimpleStrategy can put all three replicas in one rack, and the fault domain you thought you had does not exist.

What experts say in the room. "A leaderless ring trades coordination for repair work: there is no failover, and instead there is a standing obligation to reconcile replicas." The numbers: RF 3 with QUORUM needing 2, and 16 tokens in current defaults versus 256 on old clusters. The failure: an unbalanced ring after an expansion, where one node owns far more than its share and runs hot.

What a senior checks. Ownership percentage per node from nodetool status (effective ownership with the real keyspace), tokens per node against the version's recommendation, that every production keyspace uses NetworkTopologyStrategy with the right per-datacentre factors, rack layout against RF, and pending ranges during any topology change.

:::key Verified October 2026 Token defaults (256 in 2.x, deterministic allocation from 3.x, num_tokens 16 in the 5.0 docs) are from the Cassandra 5.0 pages read on 2026-10-29, which the site labels prerelease. Check your own cassandra.yaml. :::

What a write does: consistency levels, hints and speculative retry

Every write goes to every replica. The consistency level only decides how many acknowledgements the coordinator waits for. This is the part of the Cassandra docs people misremember. With RF 3 and QUORUM, the coordinator sends the mutation to all three replicas and replies to the client when two have acknowledged; the third is still being written, or failed and gets a hint.

The levels, from the Dynamo page:

Level	Acknowledgements needed
`ONE`, `TWO`, `THREE`	1, 2 or 3 replicas
`QUORUM`	A majority, n/2 + 1 (2 of 3)
`ALL`	Every replica
`LOCAL_QUORUM`	A majority in the coordinator's datacentre
`EACH_QUORUM`	A majority in each datacentre
`LOCAL_ONE`	One replica, in the local datacentre
`ANY`	Write only: one replica, or a stored hint

The overlap rule is W + R > RF: with RF 3, QUORUM writes (2) and QUORUM reads (2) overlap on at least one replica, so a read sees the latest acknowledged write. This is the quorum arithmetic from NoSQL modelling in practice: DynamoDB, Cassandra, MongoDB, and here is why it is a latency decision as well: the coordinator waits for the second-fastest of three replicas, so one slow replica does not slow the request, but two slow ones do. LOCAL_QUORUM avoids waiting for a remote datacentre, at the price that a remote datacentre may be briefly behind.

Reads contact only enough replicas to satisfy the level, with one exception the docs name: speculative retry, where the coordinator sends a redundant read to another replica if the first ones miss a time window. It cuts the tail latency caused by a single slow node, at the price of extra load. The window and the default vary by version, so check the table option for your version rather than quote a number.

Hinted handoff. When a replica is down, the coordinator stores a hint and replays it when the replica returns. It is a best-effort write-path mechanism. Two limits matter: hints are only created for a dead node for max_hint_window (3 hours by default in the 5.0 cassandra.yaml), and hinted_handoff_enabled defaults to true. The cassandra.yaml docs describe it as how long a dead host keeps receiving newly generated hints before hint creation stops. After that, only repair can bring it back. Note that ANY is the one level where a stored hint alone counts as success, which means a write can be acknowledged to the client when no replica has it yet.

Lightweight transactions are the exception to "no coordination". The Dynamo page describes LWT as a single-partition compare-and-set (IF NOT EXISTS, IF col = x) with its own serial consistency level. Use it for the rare uniqueness or claim operation, not as a general transaction: it covers one partition only and costs more coordination than a plain write.

Commit log durability. commitlog_sync defaults to periodic with a commitlog_sync_period of 10,000 ms (5.0 docs): writes are acknowledged immediately and the log is synced every 10 seconds, so one node losing power can lose that window, which replicas in other fault domains cover.

What experts say in the room. "We choose LOCAL_QUORUM for reads and writes, which gives read-your-writes inside a datacentre, and we accept that cross-datacentre reads may be stale for the length of replication lag." The numbers: 2 of 3, 3 hours, 10,000 ms. The failure: a team treats the consistency level as how many copies are written and is surprised to find all replicas receive every write.

What a senior checks. The consistency level per query path, not per cluster; whether ANY is used anywhere; the hint backlog and max_hint_window against the longest realistic node downtime; and the commit log sync mode relative to the failure domains of the replicas.

Hinted handoff, read repair and anti-entropy repair

Repair is the price of availability: with no leader to decide what is true, something has to compare the replicas. Cassandra has three mechanisms with very different reach.

Hinted handoff covers short outages of a replica during writes, up to max_hint_window (3 hours by default).
Read repair is a best-effort reconciliation during reads. It reconciles replicas that disagree on the data a read touches. It cannot fix data that nobody reads, so cold data stays divergent.
Anti-entropy repair is the real convergence tool. Replicas build Merkle trees (hash trees) over their data and compare them to find ranges that differ, then stream just those. The docs list three variants: full repair, which hashes the whole dataset as in the original Dynamo paper; sub-range repair, which uses more trees over smaller ranges, potentially down to single partitions; and incremental repair, which repairs only partitions changed since the last repair.

The time bomb: repair has a deadline. A delete in Cassandra writes a tombstone, a timestamped marker that is itself replicated. If a replica misses the tombstone (it was down, it was partitioned, hints were over) and the tombstone is later purged everywhere else, the stale value on the returning replica looks like new data and gets repaired back into the cluster. The compaction docs call this a zombie. The way to prevent it is to make sure every replica has seen every tombstone before the tombstone may be purged. That is what gc_grace_seconds sets: 864,000 seconds, ten days, by default and settable per table (compaction docs). Our arithmetic: 10 x 24 x 3,600 = 864,000.

So the real rule has two numbers. A full repair cycle of every table must finish inside gc_grace_seconds, and a node down longer than gc_grace_seconds must be rebuilt (or repaired before it rejoins), not simply restarted. The compaction docs say it directly: if a node is down or disconnected longer than gc_grace_seconds, its deleted data can be repaired back to the other nodes. If repair takes eight days on a large table with a ten-day grace, the schedule has two days of slack; a slow repair, a long stuck node or a changed topology can remove it.

Tombstones also have a purge condition that people forget. After gc_grace_seconds, a tombstone is removed only during compaction, only if it is older than the grace period, and only if all SSTables holding older data for that partition take part in the same compaction (compaction docs). So tombstones can outlive the grace period for a long time on tables that compact rarely. Reading those partitions steps over every tombstone, which is why NoSQL modelling in practice: DynamoDB, Cassandra, MongoDB gives the warn and failure thresholds of 1,000 and 100,000 tombstones per query; we do not repeat them here.

What experts say in the room. "Repair frequency is a function of gc_grace_seconds and how long repair takes; we alert when the last good repair is older than half the grace period." The half-grace rule is our suggestion, not a doc number. The failure: zombie rows, such as a deleted user or a revoked permission that reappears.

What a senior checks. Last successful repair time per table, repair duration against gc_grace_seconds with slack, the hint backlog, and the rule for a node that was down past the grace period (rebuild rather than restart). Lowering gc_grace_seconds to shed tombstones is only safe if the repair interval is shorter than the new value.

:::warn Resurrection, not loss Missing a repair deadline does not lose data. It brings deleted data back, which is often worse, because the system looks healthy while returning rows that should be gone. :::

Compaction strategies and what they do to p99

Compaction is where an LSM store pays its debts, and the strategy you set per table decides which debt. The mechanism is in Storage engines: B-trees, LSM-trees, row vs column: memtables flush to immutable SSTables and compaction merges them. This section is about the Cassandra choices, from the Cassandra 5.0 compaction overview.

Strategy	Purpose per the docs	Cost
STCS (size-tiered, the default)	A fallback for workloads that fit nothing else; suits mostly non-time-series data on spinning disks or where leveled I/O is too high. Defaults `min_threshold` 4, `max_threshold` 32	Low write amplification, higher space and read amplification
LCS (leveled)	Read-heavy tables and tables with many updates and deletes; the docs call it a poor fit for immutable time-series data	More compaction I/O, bounded read amplification
TWCS (time-window)	TTL'd, mostly immutable time series	Whole expired SSTables can be dropped at once; updates into old windows ruin that
UCS (unified)	The docs recommend it for new workloads, handling time-series and update-heavy data on HDD and SSD	Newer; read its docs for your version

Disk space. Old SSTables are deleted only after compaction finishes and pending reads end, so a compaction needs room for its output beside its inputs. The compaction docs do not state a headroom figure, so we cannot quote one; the practical consequence for size-tiered strategies, which can merge very large SSTables, is that a nearly full disk can make the biggest compaction impossible. Each data directory runs its own strategy instance, and repaired and unrepaired data are handled separately, so four data directories means eight instances (docs).

Throttles. From the 5.0 cassandra.yaml documentation: compaction_throughput defaults to 64 MiB/s per node (0 disables the throttle) and concurrent_compactors is listed in the generated reference with a default of 1, while the option's own description gives the rule: the smaller of the disk count and the core count, with a minimum of 2 and a maximum of 8. Treat the rule as the intent and the listed 1 as a documentation slip, and read the value your node actually runs with. Our arithmetic: 64 MiB/s x 86,400 s = about 5.27 TiB per day as a ceiling on compaction output per node. If the node ingests more than that per day, compaction can never catch up whatever you tune. If you raise the throttle, compaction competes with reads for disk I/O, and that shows up as p99.

How compaction debt becomes latency. Falling behind means more SSTables per partition; each read must consult them, so read amplification rises, then latency, then timeouts. It is invisible until it is not, because write latency stays good the whole time. That is why the fix is monitoring the cause rather than the symptom.

Tombstone-heavy tables. Single-SSTable tombstone compaction runs when the fraction of tombstones passes tombstone_threshold (0.2 by default), no more often than tombstone_compaction_interval (86,400 s by default); unchecked_tombstone_compaction (false) skips safety checks, and only_purge_repaired_tombstones (false) ties purging to repair state.

What experts say in the room. "Compaction strategy is a schema decision, like the partition key: choose it with the data's lifecycle." The numbers: STCS thresholds 4 and 32, throttle 64 MiB/s. The failure: compaction debt that surfaces as read latency a week after the write load changed.

What a senior checks. Pending compactions, SSTables touched per read, free disk on every data directory, compaction throughput against ingest rate, and whether a TWCS table is receiving updates or deletes outside its window.

:::key Verified October 2026 Strategy defaults and throttles come from the Cassandra 5.0 docs (prerelease label), read 2026-10-29. The docs gave no sstable size or fanout defaults for LCS and no sizing for TWCS, so this lesson does not quote any. :::

Timeouts, coordinators and the retry storm

Any node can coordinate a request, and coordinating is real work. A client connects to one node, which parses the request, finds the replicas, sends mutations or reads, waits for acknowledgements, merges results, and replies. A request that touches many partitions turns one node into a fan-out point.

The server-side timeouts (Cassandra 5.0 cassandra.yaml documentation):

Setting	Default	Meaning
`read_request_timeout`	5000 ms	How long the coordinator waits for reads
`write_request_timeout`	2000 ms	How long the coordinator waits for writes
`range_request_timeout`	10000 ms	Sequential and index scans
`native_transport_max_threads`	128 (commented out)	Threads handling client requests per node

Why the numbers form a trap. If a client sets a timeout shorter than the server's, the client gives up while the coordinator is still waiting for replicas. The client retries, and now the cluster does the same work twice, on a system that was already slow. The retry also lands, if the driver chooses another coordinator, on a node that has to fetch the same partitions again. With a write timeout of 2 seconds and a client timeout of 1 second, every slow write generates a second attempt before the first has finished. Whether a retry is safe depends on the statement being idempotent, so decide that before you enable automatic retries.

A bounded request thread pool is a backpressure mechanism. With native_transport_max_threads at 128, the node can have at most that many threads working on client requests; beyond that, requests queue. If the replicas are slow, threads sit waiting on them for up to the 5 or 2 second timeouts, and the pool saturates. Our arithmetic: if every request waits the full 5,000 ms, 128 threads complete at most 128 / 5 = 25.6 requests per second. A node's real capacity under a slow replica collapses from thousands per second to tens, and the cause lives on another node.

Multi-partition batches. A logged batch spanning many partitions makes its coordinator hold and forward the entire thing, and a failure leaves the client with a timeout and an unknown outcome. This is the design reason batches are discouraged for bulk loading: the load lands on one coordinator rather than being spread by the client, which can send each statement separately. The thresholds that warn or fail a large batch vary by version; look at cassandra.yaml for your version.

What experts say in the room. "The coordinator is real work, so the client spreads statements itself rather than batching them, and we keep client timeouts above server timeouts or accept the duplicate work." The numbers: 5 s read, 2 s write. The failure: a retry storm during a partial outage, which turns a slow node into a cluster-wide incident.

What a senior checks. Dropped message counts by type, pending and blocked native transport requests, client timeout and retry policy against the server values, whether retried statements are idempotent, and the use of multi-partition batches in application code.

Discord: hot partitions, GC and compaction debt on Cassandra

Discord's 2023 post describes a Cassandra cluster that was operated, not merely run. Bo Ingram's "How Discord Stores Trillions of Messages" (6 March 2023) gives the timeline: in 2017 the messages cluster had 12 Cassandra nodes; at the start of 2022 it had 177 nodes and trillions of messages. Messages are keyed by channel plus a time bucket, with Snowflake IDs that sort by time. Each partition is replicated across three nodes by default. The post does not state the bucket duration, so we do not either.

Failure mode one: the hot partition. A server with hundreds of thousands of members can send orders of magnitude more messages than a group of friends. Heavy reads on one channel-and-bucket pair created unbounded concurrency on the nodes holding it. Because reads and writes used quorum, the extra latency on the replicas of that partition affected other queries on those nodes too. This is the quorum cost from earlier in this lesson in practice: waiting for two of three means the slow replicas set your latency. NoSQL modelling in practice: DynamoDB, Cassandra, MongoDB covers the modelling side of bounded partitions; the new point is that bounding a partition by time does not bound it by traffic.

Failure mode two: compaction debt and the gossip dance. The post says the cluster often fell behind on compactions, which raised read costs and caused cascading latency. Engineers had a manual procedure they named the gossip dance: take a node out of rotation so it could compact, put it back to receive hinted handoff, and repeat until the backlog cleared. Hinted handoff here is the 3-hour mechanism from earlier doing double duty as a maintenance tool. A procedure like this works but does not scale with the number of nodes: with 177 nodes, some node was always due.

The JVM. The team spent significant time tuning garbage collection and heap settings, and long pauses sometimes required a manual node reboot.

The cost, in the post's words of effect: frequent on-call pages and unpredictable latency. That is the real number behind "177 nodes": every node adds a chance of a bad compaction, a GC pause, a hot partition.

Per-node arithmetic. The post reports an average of 4 TB per Cassandra node. Our arithmetic: 177 x 4 TB = 708 TB, against 72 ScyllaDB nodes at 9 TB each = 648 TB. We do not know that the data volumes were equal, so treat this as an order of magnitude rather than a claim about dataset size.

What experts say in the room. "Quorum plus one hot partition equals node-wide latency, and a JVM pause on one replica is everyone's problem." The numbers: 12 nodes in 2017, 177 at the start of 2022. The failure: compaction behind, read amplification rising, and an on-call rotation hand-running a node rotation.

What a senior checks. The distribution of partition size and cell count, not just the maximum; which partitions are hot per unit time; p99 per node rather than per cluster (a cluster average hides the one node with the pause); compaction backlog per node; and whether the runbook for a behind-on-compaction node needs a human.

:::key Verified October 2026 All Discord figures are from the company's March 2023 post as read on 2026-10-29. The 2017 and 2022 node counts are theirs; the totals computed above are ours. :::

ScyllaDB, request coalescing and the migration

Discord's fix was two things: an application-layer protection in front of the database, and a different engine behind it. Treat them separately.

The data services layer (Rust). The post describes intermediary services between the API monolith and the database clusters, with roughly one gRPC endpoint per database query and no business logic. Two techniques did the work:

Request coalescing. The first request for a row starts a worker task; later requests for the same row subscribe to that task. The database is queried once and the result is delivered to everyone who asked. For a hot channel, thousands of identical reads become one query.
Consistent hash routing. Requests carry a routing key (the channel ID for messages), so requests for one channel reach the same service instance. That is what makes coalescing effective, because the duplicates are in the same process.

Discord reports these reduced traffic spikes but did not eliminate hot partitions on Cassandra. That sentence is the architect's lesson: a smarter database does not remove the need to protect it from your own hot keys.

The engine. ScyllaDB is Cassandra-compatible and written in C++, so it has no garbage collector (post). The post credits its shard-per-core architecture for workload isolation but gives no implementation details, so this lesson does not explain shard-per-core. The ScyllaDB manual lists newer pages on tablets (data distribution), ring architecture and Raft consensus; check them for your version. A blocker, per the post, was reverse queries (reading a table backwards from its sort order); ScyllaDB implemented that and the last obstacle fell. By 2020 every database except the messages cluster had already moved.

The migration. Dual writes sent new data to both systems. The first plan, a Spark migrator, was estimated at three months. Discord instead wrote a Rust migrator in an afternoon by extending the data service library: it reads token ranges, checkpoints progress in SQLite and writes to ScyllaDB. The estimate fell to nine days, with throughput up to 3.2 million messages per second. A small percentage of reads went to both databases and were compared. Cutover was in May 2022.

The tombstone stall. The migration got stuck at 99.9999 percent. The last token ranges timed out, because they held large numbers of uncompacted tombstones in Cassandra, and reading them meant stepping over all of them. Compacting those ranges let the migration finish. It is the same mechanism as the tombstone thresholds in the nosql lesson, appearing at the worst time: a bulk read of old, deleted data.

The result (their figures): 177 Cassandra nodes became 72 ScyllaDB nodes, about 59 percent fewer (our arithmetic: 1 - 72/177 = 59.3 percent), with 9 TB per node. Historical message fetch p99 went from 40 to 125 ms down to 15 ms, and message insert p99 went from 5 to 70 ms down to a steady 5 ms. The post also reports the data services and ScyllaDB handling the message spikes of the 2022 World Cup final. Note these are Discord's own claims about its workload, and part of the gain comes from the coalescing layer, not only the engine.

What experts say in the room. The argument: the JVM and compaction debt on one side, a C++ engine on the other, and a coalescing layer that matters as much as either. Number: 177 to 72 nodes. Failure: a hot partition combined with quorum and compaction backlog. Be careful to say the post is one company's account, with the engine and the application change shipped together, so the effect of each alone is not separated.

What a senior checks before a similar move. Partition size and hotness distribution; a plan for tombstones before any bulk migration or mass delete (compact the affected ranges first, or the last 0.0001 percent will take days); dual-write and dual-read comparison to validate; a rollback path; and whether hot keys are protected at the application layer.

:::warn Do not generalise one post Discord's gains came with a rewrite of the access layer in Rust. "Move to ScyllaDB and get 15 ms p99" is not what the post says. :::

Before you run Cassandra at scale: the senior's runbook

The defaults above are not independent. They interact, and the interaction is where clusters fail. This section pulls the numbers into one table, then into the order a staff engineer would audit a cluster in.

Default (Cassandra 5.0 docs, read Oct 2026)	Value	The coupling
`max_hint_window`	3 hours (10,800 s)	Longer outages need repair
`gc_grace_seconds`	864,000 s (10 days)	Full repair must fit inside it
`commitlog_sync_period`	10,000 ms, periodic	Replicas cover a single-node crash
`read_request_timeout`	5,000 ms	Client timeout and retries must respect it
`write_request_timeout`	2,000 ms	Same
`native_transport_max_threads`	128	Slow replicas saturate it
`compaction_throughput`	64 MiB/s (about 5.27 TiB/day, our arithmetic)	Ingest above this means debt
STCS `min_threshold` / `max_threshold`	4 / 32	Compaction trigger shape
`num_tokens`	16 (256 on old 2.x clusters)	Repair and balance

The audit, in order.

Topology. NetworkTopologyStrategy everywhere, RF per datacentre, rack count against RF, token counts against the version, ownership balance.
Convergence. Last successful repair per table against gc_grace_seconds; repair duration trend; the policy for a node down longer than the grace.
Compaction. Strategy per table matches data lifecycle; pending compactions, SSTables per read, and free disk on each data directory; ingest rate against the compaction throttle.
Partitions. Largest and hottest partitions (by size, cell count and request rate), not averages.
Client behaviour. Timeouts and retry policy aligned with server values, idempotent writes, no wide multi-partition batches.
Deletes. A plan for tombstones before any mass delete, TTL change or migration.
Latency by node. p99 per node, so one paused or compaction-bound node is visible.

What experts say in the room. "Cassandra is cheap to write to and expensive to keep consistent; show me the repair schedule." The failure at the end of the chain: a node down for a long weekend, a repair that never finished, and a table whose deletes quietly return.

What a senior checks. Everything in the audit above, monitored as alerts rather than inspected by hand, with each alert tied to a number in the table.

How to say it in the interview

"Cassandra is a leaderless ring: a key hashes to a token, a replication factor of 3 with QUORUM needs 2 acknowledgements, and every write goes to all replicas whatever the consistency level. Convergence is not free, so I would schedule repair to finish inside gc_grace_seconds, which defaults to 864,000 seconds, ten days, or deleted data can return. I would pick the compaction strategy per table and watch pending compactions, because compaction debt shows up as read latency. Discord went from 177 Cassandra nodes to 72 ScyllaDB nodes, but a big part of the fix was a coalescing layer in front of the database."

Checklist

Confirmed num_tokens and token allocation against the Cassandra version in use (256 on old 2.x, 16 in current defaults)
Checked every production keyspace uses NetworkTopologyStrategy with per-datacentre factors matched to racks
Set consistency per query path and knew every write goes to all replicas whatever the level
Scheduled repair so every table completes a cycle well inside gc_grace_seconds (864,000 s by default)
Defined the rule for a node down longer than max_hint_window (3 hours) and longer than gc_grace_seconds
Chose a compaction strategy per table from the data's lifecycle, and monitored pending compactions and SSTables per read
Compared ingest rate with compaction_throughput (64 MiB/s default) and kept free disk on every data directory
Aligned client timeouts and retries with server timeouts (5 s read, 2 s write) and made retried writes idempotent
Measured partition size and request rate per partition, with p99 per node rather than per cluster
Planned for tombstones before any bulk delete, TTL change or migration, and protected hot keys at the application layer

Practice this interactively