Cruxly › System design › Databases by Name: Internals and Terms

Kafka as a log store: ISR, acks, compaction, transactions and LinkedIn scale

a replicated commit log whose guarantees are three config lines: acks, min.insync.replicas and unclean leader election

Kafka is a partitioned, replicated, append-only log, and almost everything people argue about in it (loss, duplicates, ordering, lag) comes from a handful of settings with weak defaults. This lesson teaches the replication protocol, what the producer can and cannot promise, what compaction does to a topic, and the limits the biggest operators have published.

Why a log is fast: page cache, segments and retention

A Kafka partition is a directory of segment files that is only ever appended to, and its speed comes from letting the operating system do the caching. This is the log-structured idea from Storage engines: B-trees, LSM-trees, row vs column taken to its limit: no in-place updates, no per-record index maintenance, no cache inside the process.

The Kafka 4.3 design page explains the read path. Brokers do not keep their own record cache; they lean on the OS page cache, which also stays warm across a broker process restart. To serve a consumer, the broker uses sendfile, which moves bytes from the page cache to the network card without copying through user space. The page is explicit about the catch: TLS runs in user space, Kafka does not use in-kernel SSL offload, so with SSL enabled sendfile is not used. Turning on encryption is therefore a throughput decision, not only a security one.

Offsets make replay a feature. A consumer's position in a partition is one integer, the offset of the next message to read. State that small is trivial to checkpoint, and a consumer can rewind and read again. That one property is why Kafka is used for rebuilding downstream stores, and it is what a queue that deletes on read cannot do.

Segments and retention. A partition is cut into segment files. Per the Kafka 4.3 broker configuration page, log.segment.bytes defaults to 1 GiB (1,073,741,824) and log.retention.hours defaults to 168, seven days. Retention works on whole segments, not individual records, so a record can outlive the retention window until its whole segment is eligible for deletion. A segment also closes by time: log.roll.hours defaults to 168 (7 days), so a quiet partition still rolls once a week, and the broker checks for deletable segments every log.retention.check.interval.ms, 5 minutes by default. Arithmetic: with both defaults, the oldest record in a segment can sit for up to roughly 7 days (until the segment rolls) plus 7 days (until its newest record ages out), so about 14 days.

Arithmetic, not the docs' number: a partition receiving 10 MB/s fills a 1 GiB segment in about 107 seconds, and holds about 6 TB (10 MB/s x 7 days x 86,400 s) at the 7-day default, per replica. Multiply by replication factor to get the cluster's disk bill.

Partitions are the unit of order and parallelism. The design page says a record's key is hashed to choose a partition, so records for one user land on one partition and keep their order. Within a consumer group each partition is read by exactly one member at a time, so the partition count is the ceiling on consumer parallelism. Because the key-to-partition mapping depends on the partition count, increasing the count later sends the same key to a different partition (this follows from hashing; the page gives no rule for choosing counts). Per-key order across the change is lost.

What experts say in the room. The argument is that sequential disk plus the page cache beats a clever in-process cache, and the number is 1 GiB segments and 7 days. The failure usually described is a co-located job that evicts the page cache, after which consumers reading recent data suddenly hit disk. What a senior checks: disk throughput and page-cache hit rate on brokers, balance of log directories, and the real disk need (write rate x retention x replication factor), not the retention number alone.

Replication: ISR, high-water mark, acks and min.insync.replicas

The ISR, not the replication factor, defines how many copies an acknowledged write really has. Each partition has a leader and followers. The in-sync replica set (ISR) is the followers with an active session that are not too far behind. Per the Kafka 4.3 broker docs, a follower that has not fetched or caught up to the leader's log end within replica.lag.time.max.ms (default 30,000 ms) is removed from the ISR.

Committed means all of the ISR has it. The 4.3 design page defines a message as committed once every ISR member has applied it, and only committed messages are given to consumers. A consumer therefore never reads something that can vanish at a leader change.

Why f+1 and not 2f+1. The design page contrasts majority-vote protocols, which need 2f+1 replicas to tolerate f failures (three copies for one failure, five for two), with Kafka's ISR, where f+1 replicas tolerate f failures without losing committed messages. The page names the price: a majority quorum's latency depends on the fastest servers, while an ISR write waits for all current members. This is the quorum arithmetic from Consistency models, for the person choosing applied to a log.

The three lines that decide durability.

The dangerous combination is acks=all with min.insync.replicas=1: when the ISR has shrunk to the leader alone, "all" means one copy, and the write is acknowledged anyway. Set min.insync.replicas=2 and the producer instead receives an error (NotEnoughReplicas) when the ISR drops below two. That error is the system refusing to lie.

Arithmetic, not a published number: with replication factor 3 and min.insync.replicas=2, writes with acks=all continue while at least two replicas are in the ISR, so one broker can be down. Reads of committed data continue while one ISR member survives and can be elected leader, so two can be down as long as the survivor was in the ISR. Raise replication factor to 4 with min 2 and you tolerate two down for writes, at a larger disk and network bill.

Eligible Leader Replicas, the 4.x change. Under the strict rule, once the ISR falls below min.insync.replicas the high watermark cannot advance, and a replica that is only just out of sync is not known to be safe to elect. ELR (KIP-966) lets the KRaft controller record replicas outside the ISR that are still safe to lead. Election then prefers an ISR member, then an unfenced ELR member, and only then the last known leader if it is unfenced. The cost is operational: with ELR on, min.insync.replicas becomes a cluster-level setting (broker-level overrides are not allowed), and updating the cluster or a topic value clears the ELR state, so change it deliberately.

Unclean leader election is a CAP choice made with one boolean. The design page frames it this way: if every ISR member is gone, you either wait (the partition may stay unavailable for a long time, or permanently if the data is lost) or let the first replica that returns lead, which can lose committed messages. Since 0.11.0.0 the default favours waiting for a consistent replica. In KRaft, the 4.3 docs add that a dynamic change takes effect when the unclean election thread next runs, by default every 5 minutes.

:::key Verified October 2026: version-sensitive Defaults above are from the Kafka 4.3 broker and producer configuration pages and the 4.3 design page. The 4.3 min.insync.replicas entry says its semantics change when the Eligible Leader Replicas (ELR) feature is enabled. The Kafka operations page on ELR says it is available from 4.0, off by default there, and on by default for new clusters from 4.1 (eligible.leader.replicas.version=1); an older cluster you upgrade does not necessarily have it on. :::

What experts say in the room. "Durability is configuration, not product." The numbers are 30 seconds of lag allowance and replication factor 3 with min 2. The failure they warn about is the single-copy acknowledged write: acks=all with min.insync.replicas left at 1 and an ISR shrunk to the leader. That is a configuration pattern worked out from the docs, not a named company incident. What a senior checks: under-replicated partitions, under-min-ISR partitions, ISR shrink and expand rate, and the unclean election count, plus a topic-creation policy that rejects replication factor below 3.

Producer guarantees: idempotence and transactions

Exactly-once in Kafka is a protocol between producer, broker and consumer, not a flag. The retry problem comes first: a producer sends a batch, the network drops the acknowledgement, the producer retries, and the log now has the batch twice.

Idempotent producer. Per the 4.3 design page (since 0.11.0.0), the broker assigns the producer an id and the producer attaches a sequence number per partition, so the broker can discard a retried duplicate. That turns at-least-once into no-duplicates per partition, for the life of that producer session. The Kafka 4.3 producer configuration page lists enable.idempotence default true, acks default all, retries default 2,147,483,647, max.in.flight.requests.per.connection default 5 (idempotence needs it at 5 or below), and delivery.timeout.ms default 120,000. Since Kafka 4.0 linger.ms defaults to 5 rather than 0, so a producer now waits a few milliseconds to fill batches (batch.size is 16,384 bytes). If you explicitly enable idempotence alongside a conflicting setting such as acks=1, the client throws a configuration error; if you did not enable it explicitly, it silently turns itself off. Both outcomes are worth knowing before an audit.

Transactions. For atomic writes across several partitions the producer sets transactional.id. The design page says restarting with the same id aborts any in-flight transaction of the earlier instance, which fences zombie producers. For a read-process-write pipeline the consumer needs isolation.level=read_committed and enable.auto.commit=false, with offsets committed inside the transaction. The default is read_uncommitted, and that level exposes records from aborted transactions. Exactly-once is only true for consumers that opt in.

The transaction coordinator keeps its state in a replicated log too. From the Kafka 4.3 broker docs:

So a one-broker or two-broker dev cluster cannot run transactions on defaults. Teams then lower these settings in dev, which hides the production behaviour.

Transaction timeouts. A producer's transaction.timeout.ms defaults to 60,000 and the broker caps requests at transaction.max.timeout.ms, 900,000 (15 minutes); a transaction open longer than its timeout is aborted by the coordinator. A consumer on read_committed cannot read past the first open transaction in a partition, so one hung or slow transaction delays every record behind it for up to that timeout.

What exactly-once covers. Inside Kafka, from consume to produce, with committed offsets in the transaction. It does not cover a side effect outside Kafka, such as sending an email or calling a payments API; those need their own idempotency key, the same retry problem Isolation levels, in depth warns about, solved with an idempotency key.

:::warn A consumer that forgot read_committed A service uses transactions on the producer side and ships; a downstream consumer left on the default read_uncommitted and processes records that were later aborted. Nothing errors. The fix is on the consumer, and it is why the check is "every consumer of a transactional topic", not "the producer". :::

What experts say in the room. "Exactly-once is a protocol, not a flag." The numbers are 3, 2 and 50 for the transaction log. The failure is a read_uncommitted consumer seeing aborted data. What a senior checks: the transaction timeout against real processing time, the aborted-transaction rate, producer-id and sequence errors in client logs, and that every consumer of a transactional topic sets the isolation level on purpose.

Log compaction: Kafka as a table

A compacted topic keeps at least the latest value for every key, so the log becomes a changelog you can read from the start to rebuild a table. The 4.3 design page says compaction preserves message order and never changes offsets; it just drops older values for keys that have a newer record. That is how a service restores a key-value cache, or a stream processor rebuilds state after a crash.

Tombstones delete. A record with a key and a null value says "this key is gone". Per the design page the tombstone itself is removed after delete.retention.ms, default 24 hours (broker-level log.cleaner.delete.retention.ms default 86,400,000 ms in the 4.3 docs), and the docs note that a consumer lagging longer than that can miss the delete marker. A consumer rebuilding from offset 0 has about that long to finish its scan, per the broker description.

Arithmetic, not the docs': with a ratio of 0.5, a log can be about half stale records before the cleaner even considers it, so disk use can approach twice the live data for a compacted topic, plus the active segment. Not compacted means not yet cleaned, not wrong.

The semantic limit. Compaction keeps the latest value per key, not a history. If a consumer needs every intermediate state, a compacted topic has discarded it for old keys. If it needs "the current state", compaction is exactly right. Pick the topic type by the question the consumer asks.

What experts say in the room. The argument is whether the log is the source of truth or a derived feed; section 7 takes it up. The number is 24 hours of tombstone life, the failure is a missed delete that leaves a ghost record in a rebuilt cache. What a senior checks: consumer lag against the tombstone window, cleaner thread health and dirty ratio per topic, and compaction lag settings for topics where consumers need recent history.

Groups, rebalances and coordinator state

Consumer lag and rebalances are the two operational signals of a Kafka estate. A consumer group splits partitions among members, so a membership change forces a reassignment (a rebalance), during which consumption stalls or slows. How fast the group notices a dead member is a timeout choice with a cost at both ends.

From the Kafka 4.3 broker docs (read October 2026):

Check which protocol a team's clients use before quoting a number to them. The trade-off the ranges imply: a very short session turns a long garbage-collection pause or a slow deploy into a rebalance storm, and a very long one means a dead consumer's partitions sit idle for that long.

__consumer_offsets is a compacted log. Committed positions are stored in an internal topic with replication factor 3 by default; the group coordinator is a broker leading one of its partitions. So the durability of "where was I" has the same ISR rules as your data.

Worked arithmetic: a group with a 45-second session whose members all restart in a rolling deploy can trigger a rebalance for each restart. With 20 members restarted one at a time, that is up to 20 reassignments, each pausing some consumption. This is the argument for static membership and cooperative rebalancing, which are features whose exact behaviour you should check for your client version.

What experts say in the room. "Lag per group is the ops signal." The number is 45 seconds. The failure is a rebalance storm during deploys, where lag rises on every restart and the team blames the broker. What a senior checks: consumer lag per group and partition, rebalance rate, and a deployment procedure that does not restart every member within one session window.

Running it at scale: LinkedIn, Slack and Cloudflare

The published ceilings and usage patterns come from three companies, and each says something different about operating Kafka.

LinkedIn, 2019 (its own engineering post on running Kafka at a trillion messages). The post gives a scale snapshot: over 100 Kafka clusters, more than 4,000 brokers, over 100,000 topics, 7 million partitions, and over 7 trillion messages per day. Arithmetic, not LinkedIn's: 7 x 10^12 / 86,400 is about 81 million messages per second on average.

The operational finding is where the pain began: some of LinkedIn's larger clusters have more than 140 brokers and host one million replicas in a single cluster, and there the post reports slow controllers and controller failure caused by memory pressure, which it says can cascade from one controller to the next. Their response was hotfix patches on their own branches, for example reusing UpdateMetadataRequest objects to cut controller memory, not a published rule that clusters must stay below 140 brokers. Arithmetic: 1,000,000 replicas over 140 brokers is about 7,100 replicas per broker. The post also reports that slow broker startup and shutdown made rolling deployments slow, since only one broker can be taken down at a time to keep partitions available, and describes more tooling: Cruise Control for cluster maintenance and self-healing, -li branches with patches for faster shutdown (less lock contention), a maintenance mode in which a broker is not assigned new partitions, and a minimum replication factor enforced at topic creation. The lesson is that at this scale a team builds tooling and carries patches.

Slack, December 2017 (Slack Engineering, "Scaling Slack's job queue"). The trigger, per the post, was database contention that slowed job execution about a year earlier: Redis hit its configured memory limit, new jobs could not be enqueued, and dequeuing failed too because it needed free memory, so the queue stayed locked after the database recovered and needed extensive manual recovery. Slack's change was to put Kafka in front of Redis as a durable buffer rather than replace Redis: a stateless Go service named Kafkagate accepts jobs over HTTP and writes them to Kafka, waiting only for the leader's acknowledgement (availability over consistency, the acks=1 end of section 2), and a second named JQRelay moves them from a topic into Redis, one relay per topic guarded by a Consul lock. The post reports 32 partitions per topic with replication factor 3 and two-day retention on 16 brokers, 50 topics (50 x 32 = 1,600 partitions) and a load of over 1.4 billion jobs at a peak of 33,000 per second. The pattern is Kafka as a durable intake in front of a faster store. This describes the post's 2017 design, not Slack's current setup.

Cloudflare, March 2018 (Cloudflare blog on HTTP analytics). The post says 106 Go consumers, one per partition of the HTTP logs topic, read the raw logs (average 6 million requests per second, peaks up to 8 million) and prepare them for ClickHouse. Arithmetic: 6,000,000 / 106 is about 56,600 requests per second per consumer on average. The shape is the standard one: partition count equals consumer count, so scaling the reader means scaling partitions first, which is the mapping caveat from section 1.

:::key Verified October 2026 These are the companies' own figures at the dates shown: LinkedIn 2019, Slack December 2017, Cloudflare March 2018. They describe those years, not current deployments. The 140-broker and million-replica figures describe LinkedIn's largest 2019 clusters, not a current Kafka limit. :::

What experts say in the room. One big cluster versus several is the standing argument; LinkedIn's data point is a cluster of more than 140 brokers and a million replicas, and the failure is controller memory pressure and slow failover. What a senior checks: partitions and replicas per broker and per cluster, controller failover time, replication throttling during partition moves so recovery does not starve production traffic, and a written plan for when to split a cluster.

Kafka versus a database as the system of record: the design question

Whether a Kafka topic can be the source of truth is a design question with settings that answer it, not a company claim. No company-sourced statement is used here; what follows is what the Kafka docs imply, framed as the argument people have.

The case for the log as truth. A replicated log with replication factor 3, min.insync.replicas 2 and acks=all gives ordered, durable, replayable events; downstream stores are rebuilt by reading from the start, so a bad deployment of a derived store is a replay, not a restore. With compaction, the latest state per key is retained indefinitely.

The case against. Look at the defaults you must override.

• Retention defaults to 7 days. Truth that expires is not truth. You must set retention to infinite or use compaction on purpose.
• Compaction discards history and deletes tombstones after 24 hours. A rebuild slower than that can resurrect deleted data (section 4).
• There is no query by anything but offset and key. Ad hoc questions need a second store.
• Transactions cover Kafka, not your other systems. A write to Kafka and a write to a database remain two writes (the outbox idea in Choosing the database: seven worked case studies is the usual remedy).
• Defaults are single-copy. Replication factor and min.insync.replicas default to 1.

How to answer. "Kafka is a good system of record for an event history if I configure for it: replication factor 3, min.insync.replicas 2, infinite or compacted retention, unclean election off, and I accept that I need a database for queries and constraints. For money balances I keep the constraint-enforcing database as the record and use Kafka as the feed." That last sentence is a position, not a quoted rule, and the right answer depends on the query needs.

:::warn Do not attribute this to a company No company is claimed here to treat Kafka as its sole source of truth. In an interview, say "the design question is". :::

What a senior checks: that retention, compaction and replication settings for a topic called a source of truth are written down and enforced by topic-creation policy; that a replay has been tested end to end and timed; and that the team can say which data store answers which question.

How to say it in the interview

"A Kafka topic is a set of partitions, each a replicated log, and ordering exists only inside one partition. Durability is configuration, not product: I want replication factor 3, min.insync.replicas 2 and acks=all, because the broker defaults for replication factor and min.insync.replicas are both 1, which is a single copy. I keep unclean leader election off, which is the default, and I keep the idempotent producer on, which is the default in 4.x, so retries do not duplicate. LinkedIn's 2019 post reports over 100 clusters, some with more than 140 brokers and a million replicas, and says controllers at that size became slow and failed under memory pressure."

Checklist

• Every durable topic has replication factor 3, min.insync.replicas 2 and producers on acks=all, enforced by a topic-creation policy
• Alerts exist for under-replicated and under-min-ISR partitions and for ISR shrink and expand rate
• unclean.leader.election.enable is false everywhere except topics where loss is explicitly accepted
• Retention is sized against the slowest consumer and the replay need, with disk computed as write rate x retention x replication factor
• Compacted topics have consumer lag and rebuild time checked against delete.retention.ms
• Every consumer of a transactional topic sets isolation.level=read_committed on purpose
• Lag per group and rebalance rate are on a dashboard, and deploys do not restart all members inside one session window
• Partition count is chosen for the key-ordering and consumer-parallelism need before launch
• Partitions and replicas per broker and per cluster are tracked against a written split point, with replication throttling in the partition-move runbook
• The cluster's min.insync.replicas semantics are checked against whether Eligible Leader Replicas is enabled

Related lessons

• PostgreSQL internals: MVCC cost, VACUUM, wraparound and WAL
• MySQL and InnoDB: locks, buffer pool, replication and Vitess
• Redis in depth: cluster slots, eviction, persistence and the lock debate
• Cassandra and ScyllaDB: the ring, repair, compaction and Discord's migration
• DynamoDB under the hood: partitions, adaptive capacity, quotas and the 2015 outage
• MongoDB internals: oplog, WiredTiger, shard keys, the balancer and Stripe's DocDB