Cruxly › System design › Microservices and Events
Queues and Async Work
decouple the spike from the work
Open the interactive lessonA queue lets you accept work faster than you can do it. It buys you spike absorption, failure isolation and retries, and it costs you end-to-end latency, duplicate deliveries and an entirely new class of operational problem.
What it buys
A queue makes the system degrade in latency instead of availability, which is almost always the better failure.
| Benefit | What it means |
|---|---|
| Spike absorption | A 10x spike becomes a longer queue, not a pile of timeouts |
| Failure isolation | Email provider down: emails wait, and the checkout that sends a receipt still works |
| Fan-out | One event, many consumers (fulfilment, analytics, the customer) the producer doesn't know exist |
| Retries with backoff | Transient failures retry with no user watching a spinner |
| Expensive work | Anything longer than a request timeout (transcoding, reports, bulk imports) was always going to be async |
The ACK pattern, and pull versus push work distribution
The client doesn't wait for the work. It gets a receipt (an ACK) once the request is durably queued, and carries on. The receipt isn't the result; it's a future reference to the result, looked up once a worker finishes.
| Push: a load balancer | Pull: workers read a queue | |
|---|---|---|
| Who decides | The balancer, at arrival time | Each worker, when it has capacity |
| Uneven job cost | Round robin keeps feeding a loaded server. Least-outstanding, EWMA or power-of-two-choices (front-door lesson) fix most of it | A busy worker just doesn't pull. Balanced by construction, no estimate to get wrong |
So the queue's edge is pull-based distribution for jobs whose cost varies wildly, not that synchronous serving can't be balanced.
Name a product: Kafka, the default log (or Pulsar, Redpanda); SQS or Pub/Sub for managed work queues; RabbitMQ for flexible routing; NATS JetStream or Redis Streams if you already run them. Not ZeroMQ: a brokerless socket library with no durability can't give a "durably queued" receipt.
Queue versus log
| Work queue | Append-only log | |
|---|---|---|
| A message goes to | One consumer, removed on ack | Every consumer group; retained for a period, each group tracks its offset |
| Readers | Competing consumers share the load | Independent groups read at their own pace |
| Order | Typically none | Within a partition only |
| Replay | No | Rewind to an offset and reprocess |
| Right for | Task dispatch: send this email | Event streams, change data capture |
Ask: does anyone else need this event, now or later? If yes, it's a log. Replay is underrated: after a six-hour consumer bug, you rebuild the derived store from an offset.
Delivery semantics, honestly
| Semantic | Behaviour | Use for |
|---|---|---|
| At-most-once | No retries. Fast, loses messages on failure | Metrics samples, never anything a user did |
| At-least-once | Retry until acknowledged. Never loses, delivers duplicates | The default. Assume this |
| Exactly-once | Not a thing you get from a network | — |
:::key Say this in exactly these words "Exactly-once delivery isn't achievable over a network. What is achievable is at-least-once delivery plus an idempotent consumer."
The duplicate comes from an ack lost after the work was done; the sender can't tell that from the work never happening. So make the duplicate harmless. :::
Idempotent consumer, in order of preference:
| Mechanism | The rule |
|---|---|
1. Idempotency key (payment_id, client_message_id) | Generated by the client, so a retry carries the same key. A server key is new per attempt and defeats it |
| 2. Uniqueness constraint on the key | The database rejects the duplicate, where concurrent consumers can't race past |
| 3. Dedupe table with a TTL | TTL longer than the max retry window, or a late retry passes as new work |
Say the window: "Dedupe keys live 24 hours, well beyond my retries. A later duplicate would reprocess; I accept that because the queue's retention is shorter anyway."
Ordering, which you usually should not need
Global order means one consumer, capping throughput at its speed. Almost never worth it.
Partition by entity key instead: one user's events stay ordered while different users run in parallel. That's what "ordered" means in practice.
Or make order not matter:
| Operation | Out of order? (needs commutativity or a version) | Duplicated? (needs idempotency) |
|---|---|---|
| Increment | Safe: +3 then +5 equals +5 then +3 | Unsafe: needs a dedupe key |
| "Set to X" | Unsafe: A then B differs from B then A | Safe |
| Versioned set (drop updates older than the applied one) | Safe | Safe: the usual robust choice |
:::key Raise this trade unprompted Retrying a failed message while processing later ones breaks order. Strict order means blocking the partition on failure, so one poisoned message stalls that user's stream (head-of-line blocking). Pick one and say which; naming it unprompted shows you've run one. :::
Operating it: the parts that bite
| Concern | Mechanism | The mistake |
|---|---|---|
| Dead letter queue | After N failures, move the message aside | Not monitoring it: silent data loss |
| Poison messages | Retry cap + DLQ | Retrying forever, blocking the partition |
| Visibility timeout | How long before the queue assumes a consumer died | Too short: slow work runs twice. Too long: a crash stalls it. Long tasks heartbeat or split |
| Consumer lag | Your primary alarm | Alarming on queue depth, a level, not a duration |
:::key Lag in seconds, not messages "The oldest message is 45 seconds old" is what a user feels. A depth of 10,000 could be two seconds of work or two hours. :::
:::warn "Scale up consumers" is not a backpressure policy If the database behind the consumers is the bottleneck, more consumers make it worse. Pick in advance: shed load at the edge with a clear error, drop low-value work first, or block producers. :::
How to say it in the interview
"Sending ten million push notifications is not something a request can wait for, so the write path publishes an event and returns; a worker pool drains it. That means at-least-once delivery, so the consumer has to be idempotent — let me show where the dedupe key comes from."
Checklist
- Justified the queue with a spike, a slow dependency, or a fan-out
- Described the ACK/receipt pattern as a future reference to a result, not the result itself
- Contrasted push-based load balancing with pull-based workers for variable-cost jobs
- Named a real queue or log product that fits the need
- Chose work queue vs append-only log deliberately
- Assumed at-least-once and made the consumer idempotent
- Stated the dedupe window and why it exceeds the retry window
- Partitioned by entity key rather than demanding global order
- Specified a dead letter queue and who monitors it
- Gave a backpressure policy for consumers falling behind
- Named consumer lag in seconds as the alarm