Cruxly › System design › Design Exercises

Distributed Message Queue

a durable log, a partition count you cannot easily change, and delivery guarantees that stop at the edge of the broker

Prompt: design a distributed message queue as a product. Producers publish to named topics, consumers read them, a message must survive the loss of a machine, a slow consumer must not hurt a fast one, and the service must keep working while brokers are added, replaced and fail.

Give yourself a session of 40 minutes, a blank page and only the prompt line before reading on. The interesting part is not "put messages in a list". It is what you promise when a broker, a producer and a consumer can each die between any two steps.

Applying the four moves: Scope — topics, partitions, producers, consumer groups, acknowledgements, retention, and a dead-letter path; the first decision is queue (delete on ack) or log (retain, replay), and ordering is promised per partition only. Estimate — messages per second and bytes per second, then retention times replication factor for disk, then partitions from the slowest consumer's per-partition speed rather than from the producer. Sketch — producer, broker cluster of partition leaders with followers, a metadata and election service, consumer groups with offsets. Deep dive — what "acknowledged" means at each hop, how a group re-divides partitions without stalling for minutes, and why "exactly-once" is true only inside a boundary you can name.

This exercise builds the system from the interface down. It does not repeat the broker internals taught in Kafka as a log store: ISR, acks, compaction, transactions and LinkedIn scale, the queue-versus-log decision in Queues and Async Work, the majority-vote machinery in Distributed Consensus, or the copy-and-lag mechanics in Replication and Consistency. Where a number belongs to one of those, it is named and pointed at.

Further reading, after you have done the exercise yourself: System Design Interview: An Insider's Guide (Volumes 1 and 2) by Alex Xu, Volume 2 with Sahn Lam, for a second person's version of the same problem to compare decisions against.

Scope: queue or log, ordering, delivery semantics

The first question is whether a message is a task or a fact, because that decides whether the broker deletes it on acknowledgement or keeps it. A task ("resize this image") is wanted by exactly one worker and is finished once done. A fact ("order 17 shipped") may be wanted by billing today, by analytics tomorrow, and by a new service next quarter. Queues and Async Work gives the rule: ask whether anyone else needs this event, now or later. For a general-purpose product the answer is almost always yes, so the design below is a log with per-consumer positions, and a work-queue mode is a thin layer on top of it rather than a second system.

The interface is the contract, so state it before drawing anything.

In scope: publish, subscribe, consumer groups, replication, retention, replay by offset, dead-lettering, and capacity planning. Out of scope, said out loud: a stream-processing engine, schema registry, cross-region mirroring, multi-tenant billing, and a rich routing language. Naming routing matters: RabbitMQ's model (exchanges and bindings, from its AMQP 0-9-1 concepts page) puts routing logic in the broker, whereas the log design pushes it to the consumer side by letting every group read everything and filter. That is a real fork, and you choose it now.

Three delivery guarantees, in the words the Kafka design page uses: at-most-once ("may be lost but are never redelivered"), at-least-once ("never lost but may be redelivered"), and exactly-once ("processed once and only once"). The page says Kafka's default is at-least-once. Say which one you promise and where the promise ends. Two rules of thumb to commit to now: ordering is promised per partition only, and delivery is at-least-once.

What experts say in the room. "Do you need replay?" settles the shape. The trap is promising global order, which forces one partition and one consumer and caps the whole product at one machine's speed. What a senior checks: that the scope names ordering as per-partition, delivery as at-least-once, and what is out.

Numbers: messages per second, retention, partitions

Size three things in this order: throughput, disk, then partitions. Partitions come last because they depend on the consumer, not the producer. Everything on this page is worked arithmetic with assumed inputs, not a published figure.

Assumptions. Average 200,000 messages/s, peak 500,000 messages/s, average message 1 KB, retention 3 days, replication factor 3, three consumer groups each reading every message, and a consumer that handles 5,000 messages/s per partition (200 microseconds per message including its own downstream call).

Throughput. 200,000 x 1 KB = 200 MB/s average; 500,000 x 1 KB = 500 MB/s at peak. Replication triples the write traffic inside the cluster: followers fetch from leaders, so the leaders send out 2 x 500 = 1,000 MB/s of replication at peak.

Disk. 200 MB/s x 86,400 s x 3 days = 51.84 TB per copy. Times three copies = 155.5 TB. Plan disk at about 60 percent full so that a failed broker's partitions have room to be re-created: with 12 TB of usable disk per broker, 12 x 0.6 = 7.2 TB usable, and 155.5 / 7.2 = 21.6, so about 22 brokers by disk. Retention multiplied by replication factor is the disk bill; the retention number alone understates it threefold.

Network per broker at peak, spread over 22 brokers. Inbound: 500 MB/s from producers plus 1,000 MB/s from replication = 1,500 MB/s / 22 = about 68 MB/s. Outbound: 1,000 MB/s of replication plus 3 groups x 500 MB/s = 2,500 MB/s / 22 = about 114 MB/s. That is comfortably inside a 10 Gbit/s link (1,250 MB/s), so the disk, not the network, sets the broker count here. Change the consumer-group count to 20 and outbound becomes (1,000 + 10,000) / 22 = 500 MB/s per broker: fan-out is a network multiplier, and it is the first number to question.

Partitions. The minimum is peak rate over the slowest consumer's per-partition speed: 500,000 / 5,000 = 100 partitions before any headroom. Round up for growth and for rebalance slack: take 256. Then 256 x 3 = 768 replicas over 22 brokers is about 35 replicas per broker. Each partition at the average rate carries 200 MB/s / 256 = 0.78 MB/s, so a 1 GiB segment (the Kafka default log.segment.bytes, 1,073,741,824) takes 1,074 MB / 0.78 MB/s = about 1,374 s, 23 minutes to fill.

Why the partition count is the number you cannot walk back. It caps consumer parallelism (a group of 300 consumers on 256 partitions leaves 44 idle), and for keyed messages the key-to-partition mapping depends on the count, so raising it later sends the same key to a different partition and breaks per-key order across the change. This follows from hashing; the Kafka page gives the per-partition ordering and the one-reader-per-partition rule, not a count formula. So over-provision modestly at creation, and treat a later increase as a migration.

Catch-up time is a number worth stating. If consumers stop for 10 minutes at peak, the backlog is 500,000 x 600 = 300 million messages, 300 GB. With 256 partitions x 5,000 = 1.28 million messages/s of consumer capacity against 500,000 still arriving at peak, the net drain is 780,000/s and the backlog clears in 300,000,000 / 780,000 = about 385 s, a little over 6 minutes (if arrivals had already fallen to the 200,000 average it would be 300,000,000 / 1,080,000 = about 278 s). That assumes the downstream database can take 1.28 million writes/s, which it usually cannot; see the backpressure section.

What experts say in the room. "Retention times replication factor, and the consumer sets the partition count." What a senior checks: the fan-out multiplier on outbound network, and the headroom to rebuild a lost broker.

Storage: append-only segments, indexes, sequential IO

A partition is a directory of segment files that is only ever appended to, which turns every write into a sequential write and every read into a short scan. The internals (page cache, sendfile, retention timing) are in Kafka as a log store: ISR, acks, compaction, transactions and LinkedIn scale; here is only the layout a designer has to be able to draw.

The layout, from the Kafka 4.3 implementation page. Each partition has its own directory, such as my-topic-0. A log file is named after the offset of the first message it holds, so the first is 00000000000000000000.log. Appends go to the last file; when it reaches the configured size a new one starts. A message is addressed by a 64-bit offset. A read supplies an offset and a maximum chunk size; the broker finds the segment that holds the offset, converts it to a position inside the file, and reads from there. Deletion is one whole segment at a time, by time or size. On startup the broker scans the newest segment, checks each entry's size, offset and CRC32, and truncates the log at the last valid entry, which is how a crash mid-append is repaired.

Why a sparse index and not a per-message one. An index entry for every message would be as large as the data's metadata and would need updating on every append. The broker config page shows the trade: log.index.interval.bytes defaults to 4096, "how often an entry is added to the offset index", so one entry per 4 KiB of log. Arithmetic: a 1 GiB segment then has 1,073,741,824 / 4,096 = 262,144 index entries, which at 8 bytes each (a 4-byte relative offset and a 4-byte file position) is 2 MiB, well under the 10 MiB per-file cap in the verified box below. A lookup binary-searches the index for the nearest entry at or below the wanted offset, jumps to that file position, and scans forward at most about 4 KiB. The index is small enough to stay in memory, so one random read costs one disk seek at worst and usually zero when the data is in the page cache.

Sequential IO is the performance argument. Writes are appends, reads of recent data hit the cache, and a consumer catching up reads contiguous bytes. Random-access structures (a B-tree keyed by message id) would pay a seek per message. The cost of the design is that you cannot delete or update one message in the middle; the unit of deletion is the segment, so the segment size is a trade between the number of files and how precisely retention is enforced.

The record envelope. Keep it small and versioned: key, value, timestamp, headers, and a per-producer sequence number (used later for deduplication). Cap the batch size: the Kafka broker default message.max.bytes is 1,048,588 bytes (about 1 MiB), measured after compression. A cap protects the broker from one producer filling a fetch response; large payloads belong in an object store with the message holding a reference.

Flush policy is where durability lives, and the right answer is "not on every write". The broker defaults leave flushing to the operating system: log.flush.interval.messages is 9,223,372,036,854,775,807 (effectively never) and log.flush.interval.ms is unset. Durability then comes from replication, not from fsync per message. That is a deliberate position: a machine that loses power loses its unflushed tail, and the cluster recovers that tail from the other replicas. Say it, because "we fsync every message" is the common wrong answer and "we never fsync" is the common alarming one.

:::key Verified November 2026 Segment layout and recovery: Kafka 4.3 implementation page. log.segment.bytes 1 GiB, log.index.interval.bytes 4096, log.index.size.max.bytes 10 MiB (the cap on one index file), message.max.bytes 1,048,588, log.flush.interval.messages and log.flush.interval.ms as above: Kafka 4.3 broker configuration page, read 2026-11-02. Defaults change between versions; check yours. :::

What experts say in the room. "Sequential write, sparse index, delete by file." What a senior checks: that log directories are balanced across disks, that the index and segment counts per broker stay within file-handle limits, and what the flush policy actually is.

Partitioning, producers and consumer groups

Choosing the partition is the producer's ordering decision, and choosing who reads which partition is the group's scaling decision. Keep them separate in your head.

Producer side. The producer picks the partition. With a key, it hashes the key, so every message for one user, device or order lands in one partition and stays in order. With no key it spreads messages for balance, and gives up per-entity order. The Kafka page states that a record's key chooses the partition and that order holds per partition. Two things to name here (producer retries that duplicate and reorder are covered in the ordering deep dive):

• A hot key. One celebrity account or one tenant sends 30 percent of traffic. Its partition runs hot while others idle, and adding partitions does not help because the key still maps to exactly one. Fixes: salt the key into a few sub-keys and accept order per sub-key, or give that tenant a dedicated topic.
• Batching. Sending one message per request wastes the network. Producers batch by size and by a small wait; the Kafka producer's linger.ms default is 5 ms in 4.x, a latency you pay to fill batches (from the Kafka lesson).

Consumer side. A consumer group is a set of consumers with a shared name. The Kafka design page says each partition is consumed by exactly one consumer in each subscribing group at a time, so the partition count is the ceiling on useful consumers in a group, and several groups each get their own full copy of the stream. Kafka consumers pull: they fetch from the partition leader at their own offset. The design page argues for pull over push because a push system can overwhelm a consumer that is slower than the producer, while a pulling consumer "simply falls behind and catches up when it can", and a pull can batch everything available. Long polling avoids busy-waiting on an empty partition. Pull is also the first half of backpressure, covered below: the consumer sets the pace.

The offset is the consumer's whole state. The page says the position is one integer, the offset of the next message to consume, so acknowledgement state is tiny and rewinding is trivial. The group's committed offsets live in an internal replicated log (in Kafka, __consumer_offsets), whose durability follows the same rules as data. The design question is when to commit:

The Kafka page says the same: at-most-once is "disabling producer retries and committing offsets before processing". Defaults worth knowing from the Kafka 4.3 consumer page: enable.auto.commit is true with auto.commit.interval.ms 5,000, so by default offsets are committed every 5 seconds in the background, which is neither of the first two rows exactly: after a crash you may repeat up to 5 seconds of work, and you may also have committed a position for a message still being handled in another thread. Turn auto-commit off and commit after processing when the work matters.

Contrast with the other model. In Amazon SQS and RabbitMQ the broker tracks each message, not each consumer. SQS hides a received message for the visibility timeout (default 30 seconds, per the SQS guide) and expects a delete; if no delete arrives the message becomes visible again for any consumer. That is per-message state, which is why those systems give per-message retry and DLQ counters easily and give up cheap replay. Apache Pulsar sits between: brokers are stateless and storage is separate, with per-subscription cursors and acknowledgement that can be individual or cumulative (its docs note cumulative cannot be used with Shared or Key_Shared subscriptions).

What experts say in the room. "Order is a producer decision, parallelism is a consumer decision, and the offset is the only consumer state." What a senior checks: key skew per partition, the commit policy per consumer, and idle consumers versus partition count.

Replication, acknowledgement levels and leader election

The design question is what "the write succeeded" means, and the honest answer is a rule about how many copies exist at the moment of the confirm. The mechanics of the in-sync set, high-water mark and the settings that control them are in Kafka as a log store: ISR, acks, compaction, transactions and LinkedIn scale, and the vote counting is in Distributed Consensus. Here is the product-level decision.

Each partition has one leader and followers. The Kafka design page: all writes go to the leader, and followers pull from it and apply the same log in the same order (consumers normally fetch from the leader too, though the design page notes reads can be served by followers). Placing leaders evenly matters, because a broker that leads many hot partitions carries the load while others idle. With 256 partitions over 22 brokers, aim for about 12 leaders per broker (256 / 22 = 11.6, arithmetic).

Three acknowledgement levels the producer can ask for, and what each costs:

The page defines a message as committed when all in-sync replicas have applied it, and only committed messages are served to consumers, so a consumer never reads something a leader change can erase. The in-sync set is the thing to design around: a follower drops out of it if it lacks a session or falls behind the leader by more than replica.lag.time.max.ms, default 30,000 ms. Arithmetic on the quorum alternative from Distributed Consensus: a majority protocol needs 2f + 1 copies to survive f failures (3 for one, 5 for two), while an in-sync-set protocol needs f + 1 to avoid losing committed data, at the cost that the confirm waits for every in-sync member and not the fastest majority. The Kafka page names exactly this trade.

RabbitMQ chose the majority model. Its quorum queues (RabbitMQ docs) are replicated queues built on Raft. One leader and followers per queue; the default group size is 3; publisher confirms are sent only after the message has been replicated to a quorum; an odd group size is advised because an even one tolerates no more failures than the next smaller odd one (3 nodes tolerate 1, 5 tolerate 2, 7 tolerate 3). The docs say performance declines beyond 5 nodes and advise against more than 7. Same destination, two protocols: pick the in-sync set when you want f + 1 copies and can tolerate a straggler, pick the majority when you want latency set by the fast replicas and are willing to pay 2f + 1.

Leader election. When a leader dies, one of the remaining in-sync replicas becomes leader. In Kafka the cluster controller does the choosing, from the in-sync set. The risk is the case where no in-sync replica is left. The design page lays out the choice: wait for an in-sync replica to return (consistent, possibly unavailable for a long time) or let any replica that returns lead (available, may lose committed messages). Kafka favours waiting by default, and unclean.leader.election.enable flips it. In a RabbitMQ quorum queue, if a quorum cannot be recovered the queue is unavailable and the docs say it must be force-deleted and recreated. Whatever your product is, ask the owner which of the two, because both answers are defensible and they are different products: a payments pipeline chooses unavailable; a metrics pipeline may choose available.

Metadata and the control plane. Someone must know which broker leads which partition and what the in-sync sets are. That is a small, consistent store, kept by a consensus group (an external coordination service, or an internal Raft-based controller quorum). It is the one place you pay for strong consensus, and it is deliberately off the data path: a produce request goes to a partition leader, which the client learned from cached metadata. Cache the map on clients and refresh it on a not-leader error.

:::key Verified November 2026 Kafka: committed means all in-sync replicas have applied it; default favours waiting for a consistent replica; replica.lag.time.max.ms 30,000 (Kafka 4.3 design and broker configuration pages). RabbitMQ quorum queues: Raft based, default group size 3, confirms after quorum replication, performance declines beyond 5 nodes and more than 7 is advised against (RabbitMQ quorum queues documentation). All read 2026-11-02. :::

What experts say in the room. "Acknowledged means N copies; elections choose consistency or availability." What a senior checks: under-replicated partition count, leader skew, and that nobody can create a one-copy topic.

Delivery semantics: at-least-once, idempotent consumers, dead-letter queues

Every hop has its own acknowledgement, so "exactly-once" must be defined as a boundary: the broker plus one atomic sink, never the whole path. Queues and Async Work gives the rule (at-least-once plus an idempotent consumer) and the client-generated idempotency key; this section places it in the queue's design and shows what each product offers.

The five hops and what can repeat or vanish.

Where each product draws its line. The Kafka design page says exactly-once is supported in Kafka Streams and through the transactional producer with a read_committed consumer; the consumer default is read_uncommitted, which shows records from aborted transactions (Kafka 4.3 consumer configuration). The boundary is consume, process and produce within Kafka. SQS standard queues are at-least-once with best-effort ordering, and the guide tells you to design for duplicates. SQS FIFO queues "deliver each message once" and keep order within a message group; the producer-side duplicate defence is the deduplication ID, with a 5-minute deduplication interval: a retry of the send within five minutes adds nothing. After five minutes the same body is a new message, which is exactly the dedupe-window trap in Queues and Async Work. Throughput is the price: FIFO supports up to 300 API calls per second per action without batching and 3,000 messages per second with batching (SQS queue types page), with a high-throughput mode up to 30,000 transactions per second and relaxed ordering within groups.

The redelivery clock. In SQS the visibility timeout (default 30 seconds, maximum 12 hours from first receive, extendable with ChangeMessageVisibility) is the lease on a message. Too short and slow work runs twice; too long and a crashed consumer's message waits. The SQS guide says even within the timeout, at-least-once means there is no absolute guarantee that a message is not delivered twice. In Kafka the equivalent clock is the consumer's max.poll.interval.ms (default 300,000, five minutes): exceed it and the group treats the consumer as failed and rebalances, which re-delivers its uncommitted work to someone else. Long tasks need a heartbeat or must be split.

Dead-letter queues are the answer to a message that can never succeed, and they are a design feature with a monitoring owner. Worked rule: count delivery attempts per message, and after N move it aside. Product details to quote:

• SQS uses a redrive policy with maxReceiveCount. The DLQ must be in the same account and region. For standard queues the message keeps its original enqueue timestamp, so a DLQ with a retention shorter than the source queue's can delete the message soon after it arrives; the guide says to always set the DLQ retention longer than the source's. The SQS guide also warns against a DLQ on a FIFO queue if you need to keep the exact order of operations.
• RabbitMQ quorum queues have a delivery limit, default 20 since 4.0, after which a message is dropped or dead-lettered if an exchange is configured. The default dead-letter strategy is at-most-once: dead-lettering can lose a message. At-least-once dead-lettering exists but needs several settings together (strategy at-least-once, overflow reject-publish, a dead-letter exchange).
• Pulsar's dead letter topic has no subscription by default, so messages sent there can be lost unless you set initialSubscriptionName. The Pulsar docs also note that the redelivery counter is held in memory and resets on broker restart or consumer disconnect, so maxRedeliverCount may never be reached when only negative acknowledgements are used.

The lesson from three products: a DLQ that nobody reads is silent data loss, and one that can itself drop messages is not a safety net. Alarm on arrivals in the DLQ, not on its depth.

Retries need a policy. Transient failures (timeouts) retry with backoff; permanent ones (a malformed payload) go straight to the DLQ. A common structure is a short chain of retry topics with growing delays, then the DLQ, so a stuck message does not block a partition. The cost is order: a message pulled out and retried later arrives after its successors. That is the head-of-line trade in Queues and Async Work: block the partition to keep order, or skip and lose it.

:::key Verified November 2026 SQS visibility timeout default 30 s and 12 h maximum; FIFO deduplication interval 5 minutes; standard queues at-least-once with best-effort ordering; DLQ retention advice (Amazon SQS Developer Guide). RabbitMQ delivery limit 20 and at-most-once dead-letter default (quorum queues docs). Pulsar DLQ and redelivery caveats (Pulsar messaging concepts). All read 2026-11-02. :::

What experts say in the room. "Exactly-once is a boundary, not a property of the network." What a senior checks: the dedupe window against the longest retry, a DLQ with an owner and an arrival alarm, and visibility or poll timeouts longer than the p99 processing time.

Backpressure, retention and operating it

Backpressure is deciding in advance what gives when producers outrun consumers, and the log makes the cheap answer "the backlog grows", which is only a good answer until retention or disk says otherwise.

Where the pressure builds. Three places, each with a different signal:

• Consumers slower than producers. The backlog grows on the broker. Pull-based consumption makes this safe: the Kafka page's point is that the consumer falls behind and catches up rather than being overwhelmed. The danger is the backlog outliving retention: if the oldest unread message is older than the retention window, it is deleted unread. Arithmetic: with 3-day retention and a 200 MB/s average, a consumer more than 72 hours behind loses data; at that point lag in time is the alarm, not lag in messages.
• Brokers slower than producers. Produce requests queue up and latency rises. The broker needs quotas and a bounded request queue so one noisy producer cannot starve the rest; reject early with a clear throttle response rather than accept and time out. SQS makes the same edge visible: roughly 120,000 in-flight messages is the ceiling for a standard queue, after which receive calls error or return nothing (SQS visibility-timeout page).
• Downstream sink slower than consumers. Adding consumers makes it worse (see Queues and Async Work). The consumer should limit its own concurrency to the sink's capacity and, if it cannot keep up, pause fetching from the partitions rather than buffering unbounded messages in memory. Kafka consumers do this naturally through max.poll.records (default 500) and the poll loop: take fewer, process, poll again.

Pick a producer-side policy before it happens. Three honest options: block the producer (bounded buffer, back-pressure to the caller), shed load at the edge with an error, or drop low-value traffic first. Metrics can drop; payments cannot. Writing the policy per topic is part of the topic's definition.

Retention and compaction. Two models, chosen per topic:

Per the Kafka design page compaction never changes an offset or reorders messages, and tombstones are removed after delete.retention.ms (default 24 hours). A slow consumer that is offline longer than that may miss a delete. Consumer offsets themselves have retention too: offsets.retention.minutes is 10,080 (seven days) after a group becomes empty, so a group idle for eight days restarts from auto.offset.reset, default latest, and silently skips what was published meanwhile. Set it to earliest for groups where skipping is worse than reprocessing. Pulsar draws the line differently: by default acknowledged messages are deleted immediately and unacknowledged ones stay as backlog; retention keeps acknowledged messages, and expiry (TTL) drops unacknowledged ones.

Operating it: the signals. Consumer lag per group in seconds, not messages (the oldest unprocessed message's age). Under-replicated partitions. Leader skew. Disk used against the 60 percent plan. Request queue time. DLQ arrivals. Rebalance rate. Each has an owner and a threshold that was agreed before the incident.

Failure modes to name unprompted.

• Disk fills on one broker. Skewed partitions or a hot key fill one disk while the cluster has space. Brokers stop accepting writes; recovery is reassigning partitions, which moves bytes and adds load at the worst time.
• Reassignment storm. Moving partitions during an incident saturates the network. Throttle replication bandwidth for reassignments.
• Metadata outage. If the consensus group behind the control plane loses its majority, existing data traffic can continue on cached leaders, but no leader can be elected, so a broker failure during the outage becomes a partition outage.

What experts say in the room. "Alarm on the age of the oldest unread message against retention, and decide the shed order before the incident." What a senior checks: the ratio of lag to retention per group, and the one-line policy per topic for what is dropped first.

Deep dives: rebalancing, ordering, exactly-once claims

Three questions interviewers push on after the sketch, each with a trade-off to state and a number.

1. Rebalancing: how a group re-divides partitions without stalling. When a member joins, leaves or is judged dead, partitions are reassigned. In the classic eager protocol every member pauses and gives up its partitions while the group agrees on a new assignment, so a rebalance is a stop-the-world event for the group (the cooperative-sticky assignor and the newer consumer protocol shrink that pause). What triggers it, with Kafka 4.3 consumer defaults: a heartbeat gap beyond session.timeout.ms of 45,000 (heartbeats every 3,000), or a gap between polls beyond max.poll.interval.ms of 300,000. The trade at the edges: a short session turns a garbage-collection pause or a slow deploy into a rebalance; a long one leaves a dead member's partitions idle for that long. The Kafka consumer page also offers static membership (group.instance.id): a restarting consumer with the same id keeps its partitions until the session timeout, so a rolling restart does not trigger a rebalance per member. Arithmetic: 20 members restarted one at a time under dynamic membership can mean up to 40 reassignments (one when each leaves, one when it rejoins); with static ids and a restart quicker than the session timeout, zero. The consumer page also lists a group.protocol setting, classic by default, with a consumer protocol where the broker computes assignments; check which your clients use before quoting a number. Design rule that is yours to choose: assign by stickiness, so a rebalance moves the fewest partitions, because a moved partition means a cold cache and an offset hand-off.

2. Ordering: what it costs and how to keep it. Per partition, order is free: one writer appends, one reader reads. Three ways it breaks, each with a fix:

• Producer retries reorder. Batch 1 fails, batch 2 succeeds, batch 1 is retried and lands after it. Fix: the idempotent producer with sequence numbers, which the Kafka lesson explains.
• Consumer concurrency. A consumer that fans a partition's messages to a thread pool processes them out of order. Fix: parallelize across keys within the partition, not within a key.
• Retries and dead-lettering. A message retried later arrives after its successors. Fix: block the partition on failure, or accept that this stream is not strictly ordered.

Products differ here in useful ways. SQS FIFO orders per message group ID and holds back later messages in a group while one is in flight. Pulsar's Key_Shared subscription keeps per-key order across many consumers, while its Shared subscription gives no order guarantee, and its Failover subscription warns that a switch of the active consumer can duplicate or reorder. State which row your product is.

3. Exactly-once claims: what you can safely say. Make the claim in layers:

1. Inside the broker, an idempotent producer removes duplicates caused by producer retries, per partition and per producer session.
2. For consume-transform-produce between topics, a transaction commits the output and the consumer offset together; consumers read with read_committed (the default read_uncommitted exposes aborted data). The Kafka lesson covers the machinery and the caveat that one slow transaction delays readers behind it.
3. For any effect outside the broker (a database write, an email, a payment), the guarantee is at-least-once plus an idempotent sink: a unique key such as (topic, partition, offset) stored with the result, or an idempotency key at the provider. If the result and the offset are written in one database transaction, the effect is exactly-once from that database's point of view.

The honest sentence is: "exactly-once as an effect, inside a boundary I can draw". A product that advertises exactly-once without drawing the boundary has told you nothing.

Compare the designs once, then choose. Kafka: partitioned log, consumer-side offsets, replay, ISR replication; best for high-throughput streams and fan-out. SQS: managed per-message queue with a visibility timeout, DLQ and a FIFO mode with group IDs; best when you want no brokers to run and do not need replay. RabbitMQ quorum queues: Raft-replicated queues with routing and per-message ack; the docs themselves say they are not suited to large backlogs, high churn or large fan-outs and point to streams for those. Pulsar: stateless brokers over a separate storage layer, with several subscription modes. The architecture question under all four is where per-consumer state lives (offset in the consumer group, or message state in the broker), and that choice drives replay, retries and cost.

What a senior checks. Rebalance rate during deploys, per-key skew, the commit and dedupe contract for every consumer, and whether the dead-letter path has an owner.

How to say it in the interview

"I would build a partitioned, replicated, append-only log, not a delete-on-read list. Order exists only inside one partition, so the partition key is the ordering decision. Durability is the acknowledgement rule: a write is confirmed only after a majority or the in-sync set has it. Delivery is at-least-once by default, so the consumer must be idempotent, and anything called exactly-once holds only inside the broker's own boundary. The number I size from is the slowest consumer's throughput per partition, because partitions cap parallelism and cannot be changed cheaply later."

Checklist

• Decided queue or log first, from whether anyone else needs the message later
• Promised ordering per partition only, and made the partition key the ordering decision
• Sized disk as rate x retention x replication factor, and brokers from it with free-space headroom
• Derived the partition count from the slowest consumer's per-partition throughput, with headroom
• Described the log layout: segments named by base offset, a sparse index, deletion by whole segment
• Defined what a confirm means: the acknowledgement level and the minimum in-sync copies
• Chose what an election does when no in-sync copy remains: wait or elect an out-of-sync replica
• Committed offsets after processing and made every consumer idempotent
• Gave the dedupe window and compared it with the longest retry or visibility timeout
• Specified a dead-letter path with a retry cap, an owner, an arrival alarm and longer retention than the source
• Wrote a backpressure policy per topic and alarmed on lag in time against retention
• Stated exactly-once only as an effect inside a named boundary

Related lessons

• Short Links at Scale
• Activity Feed
• Real-time Messaging
• Seat Booking
• Notification System (HLD)
• Search Autocomplete (HLD)