Cruxly › System design › Microservices and Events

Event-Driven Architecture: Choreography, Orchestration and Sagas

events decouple who knows about whom; they do not decouple who must be correct, and that is where the saga, the outbox and the inbox come in

"Event-driven" names four different designs, and most arguments about it are people using the word for different ones. This lesson separates them, then teaches the parts that decide whether an event-driven system is safe: choreography versus orchestration, sagas and their compensations, the outbox that fixes the dual write, idempotent consumers, event schemas, and the operating cost you take on. It builds on Queues and Async Work, Distributed Message Queue and Kafka as a log store, and does not repeat them.

Four things people mean by event-driven

"Event-driven" is four separate patterns that share a message bus, and the first design question is which one you are buying. Martin Fowler's February 2017 article "What do you mean by Event-Driven?" makes exactly this split, and the Azure Architecture Center's event-driven style page adds the topologies. Naming them is the cheapest way to stop a design review going in circles.

• Event notification. A service says something happened and does not care what anyone does with it. Fowler's point is that the source does not much care about a response, which gives low coupling and a simple setup, and also makes the overall flow implicit: nobody can read one file and see the process.
• Event-carried state transfer. The event carries the changed data, so receivers keep their own copy and never call back. Next section.
• Event sourcing. The log of events is the source of truth and current state is derived from it. microservices.io lists the price: the store is hard to query directly, so you usually add CQRS and accept eventually consistent reads. Fowler stresses that it does not have to be asynchronous and that not everyone needs to read the log.
• CQRS. Separate models for writes and reads. The Azure CQRS page says it does not require messaging, but when the read store is separate the usual link is events from write side to read side, which brings the dual-write problem (section 5).

Events, commands and queries are different messages. A command is a request addressed to one handler that may fail ("ReserveStock"). An event is a past-tense fact with no addressee ("StockReserved"). A query asks and changes nothing. Fowler warns of the trap of disguising a command as an event: a message called OrderPlaced that exists only because the warehouse must now ship is a command with a polite name, and the coupling it hides is real. Test: if the publisher breaks when the one consumer is removed, it was a command.

Two topologies. The Azure style page distinguishes a broker topology, where components broadcast and others react, from a mediator topology, where a component holds the state of the flow and sends commands. Broker is highly decoupled but, in the page's words, no component owns the state of a multistep transaction, so restarts and replays are manual. Mediator gives control and better error handling and costs coupling and a possible bottleneck. Sections 3 and 4 are that choice.

:::key Verified November 2026 Fowler's article is dated 7 February 2017 (updated 8 February 2017). The Azure Architecture Center pages cited in this lesson were dated between February 2025 and June 2026 when read on 2 November 2026. :::

What experts say in the room. "Which of the four are we doing?" The failure they warn about is the disguised command, a hidden dependency that no diagram shows. What a senior checks: that each event has a past-tense name, no addressee in its payload, and at least one consumer the publisher has never heard of; if not, call it a command and treat it as one.

Boundaries Are Where Guarantees Die explains why the bus itself guarantees less than people assume; this lesson is about the design on top of it.

Event notification versus event-carried state

The payload decides the coupling: a thin notification keeps one source of truth and costs a call-back, a fat event removes the call-back and creates copies you must keep honest. The Azure event-driven page names the two payload strategies directly.

The call-back failure. A notification-only design looks decoupled, then at 3 a.m. the source is slow and every consumer calls it at once. The bus meant to protect you has turned an event into a synchronous dependency with a retry storm attached (Overload, load shedding and cascading failure covers what follows). Worked arithmetic: 20 consumers each receiving 1,000 events per second and calling back once is 20,000 reads per second landing on the source, which the 1,000 writes per second never predicted.

State transfer's failure is staleness and ordering. The consumer's copy is only as good as the last event it processed. If AddressChanged v7 arrives before v6, a naive overwrite leaves the older value. The fix is a version or sequence number in the event and a rule that a consumer ignores anything not newer than what it holds (section 7). The Azure choreography page says the same: carry sequence or version data so consumers can reject stale events and detect gaps.

A middle road that works. Carry the fields consumers always need (id, status, version, timestamp) and let rarely needed detail be fetched. This is a judgement call, not a rule; the Azure page leaves it to you per consumer when you control them.

Privacy is a payload decision too. The Azure page notes that events are visible to many components, including ones that were not meant to consume them, and recommends an assume-breach mindset about what goes in. A fat event containing a phone number is copied into every consumer's database and every log of the broker.

What experts say in the room. "Do I want my consumer to survive the source being down?" If yes, state transfer; if the data changes rarely and the source is the only authority, notification. The number is the call-back fan-out: consumers times event rate. What a senior checks: the consumers' dependency on the source at 3 a.m., a version field on every state-carrying event, and who owns erasure requests when copies exist in six services.

Choreography versus orchestration, with trade-offs

Choreography spreads the flow across services and orchestration writes it down in one place; pick by how many steps there are and how often the flow changes. The Azure Saga page defines both: in choreography, services exchange events with no central controller and each local transaction publishes events that trigger others; in orchestration, a controller tells each participant which operation to run, stores the state of each task and handles failure with compensating transactions.

The failure choreography grows into. The Azure choreography page lists "emergent behavior and event storms": when many services react to each other's events, a small event can cascade or loop. Its recommended guardrails are event filtering, consumer concurrency limits, throttling and explicit rules. A concrete form: service A reacts to PriceChanged by emitting QuoteUpdated, and service B reacts to QuoteUpdated by emitting PriceChanged. Nothing in either repository shows the cycle.

The failure orchestration grows into. The orchestrator gradually absorbs business rules ("if the customer is premium and the item is digital, skip shipping") until every change to any service needs a change there, and a team owns the thing everybody queues behind. It also needs to be highly available itself; durable-execution engines exist for this, covered in Workflow Orchestration Engine.

A decision rule you can say aloud.

• Up to about three steps, one team, no compensation: choreography.
• A flow that must be explained to the business ("where is order 4411?"), has compensations, or has branches and timeouts: orchestration, with the orchestrator owning only sequence and state, never domain rules.
• A flow crossing bounded contexts that owners want decoupled: choreography between contexts, orchestration inside each. The Azure choreography page makes the same split: loose coupling across domain boundaries, an orchestrator inside one context when complexity warrants it.

The thresholds are a judgement, not a published number; change them with the team's pain, not before.

What experts say in the room. "Can anyone tell me where order 4411 is right now?" In choreography the answer needs tracing across services; in orchestration it is a query. What a senior checks: whether there is a status view, a rule that the orchestrator holds no domain logic, and an event-storm guard such as hop counts or consumer concurrency limits.

Sagas: compensations and failures they cannot undo

A saga replaces one distributed transaction with a chain of local ones plus a defined undo for each, and the undo is business logic, not a rollback. The idea and name come from Garcia-Molina and Salem's "Sagas" (ACM SIGMOD 1987). microservices.io states why it applies to microservices: each service owns its database, so one local ACID transaction cannot cover the work and two-phase commit is not practical.

Three kinds of step. The Azure Saga page gives the vocabulary that matters in a design review:

• Compensable transactions can be undone by an opposite action.
• The pivot transaction is the point of no return. After it succeeds, the saga must complete; it cannot be backed out.
• Retryable transactions come after the pivot, are idempotent, and are retried until they succeed.

Order the steps so everything that might fail comes before the pivot and everything after it can be retried forever. This is the single most useful design move in sagas.

Worked example: order booking with a failure in the middle. Steps: create order (Order service), charge card (Payment), reserve stock (Inventory), book courier (Shipping). Suppose stock is reserved at 11:00:02 and the courier API rejects the pickup at 11:00:03 because the postcode is outside its area.

Compensations run in reverse order. The customer sees PENDING for about five seconds and then CANCELLED, which is an outcome the product must be designed for; nothing was ever "rolled back".

Why charge as an authorisation, not a capture. Voiding an authorisation is a clean compensation. Capturing and then refunding takes days, shows twice on a statement and costs fees. The step you put before the pivot should be the easy-to-undo version of the operation. The capture belongs after the pivot.

What a compensation cannot undo. The Azure Saga page says compensating transactions might not always succeed and that data cannot be rolled back, because each participant has already committed. An email sent, an SMS delivered, a physical parcel dispatched, a third-party API call without a cancel: these are not compensable. The honest pattern is to postpone them until after the pivot, or to accept a second corrective action ("sorry, ignore the earlier message"). A compensation can itself fail, so it too must be idempotent and retried, with a human-visible dead-letter if it stays broken.

No isolation. The Azure Saga page lists the anomalies that appear because other transactions see the half-finished state: lost updates, dirty reads and non-repeatable reads. It lists countermeasures: a semantic lock (an application-level flag such as PENDING marking a record in flux), commutative updates, a pessimistic view that reorders the saga so changes sit in retryable steps, re-reading values before updating, version files, and risk-based concurrency. In the example above, the reserved-stock number is visible to other buyers for those two seconds; the semantic lock RESERVED_PENDING tells them the unit might return. Isolation levels, in depth explains the anomalies; the saga gives you none of the database's tools, so you build them.

Event sourcing and sagas are separate. Workflow Orchestration Engine covers event-sourced workflow state and its replay rules; here the point is only that a saga needs durable state (who has done what) somewhere, whether an orchestrator's table or the event log.

What experts say in the room. "Which step is the pivot, and what comes after it?" The number is the window during which other readers can see the half-done state. What a senior checks: that each step has a named compensation (or is declared a pivot or retryable), compensations are idempotent, non-compensable side effects sit after the pivot, and stuck sagas raise an alert instead of retrying silently.

The dual write and the outbox

A service that writes to its database and then publishes an event has made two writes with no shared transaction, and one of them will eventually be lost. microservices.io frames the problem plainly: sending inside the transaction is unreliable because the transaction might roll back, and sending after the commit is unreliable because the service could crash before sending. Two-phase commit across a database and a broker is usually unavailable or undesirable.

Worked failure without an outbox. An order service handles POST /orders: insert the row, commit, then publish OrderCreated. At 14:02:11 the process is killed during a deploy between the commit and the publish. The row exists; the event does not. Payment never charges, inventory never reserves, and the order sits in PENDING forever with no error anywhere. Worked arithmetic: at 1,000,000 orders a day, if one request in 10,000 lands in that window (deploys, OOM kills, broker blips), that is 100 orphan orders every day. Reverse the order (publish, then commit) and a rollback leaves a payment for an order that does not exist. No ordering of the two calls is safe.

The fix: one local transaction, two rows. The transactional outbox, per microservices.io and the Azure Cosmos DB guide, stores the outgoing message in an outbox table in the same database transaction as the business change. Either both commit or neither does. A separate process, the message relay, reads the outbox and publishes. Benefits listed on the page: messages are sent if and only if the transaction commits, and in the order the application wrote them.

Two ways to relay. microservices.io names both:

• Polling publisher. A loop selects unpublished rows, publishes, marks them. Simple, works on any database. It adds latency up to the poll interval (worked arithmetic: a 1 s poll adds 0 to 1 s, about 0.5 s on average) and puts query load on the table.
• Transaction log tailing, i.e. change data capture (CDC). A connector reads the database's own log and emits a message per outbox insert. Lower latency and no polling load, at the price of running connector infrastructure and keeping it in step with the database.

Debezium's outbox event router. In the Debezium reference (stable docs read 2 November 2026) the expected outbox columns are id (uuid), aggregatetype, aggregateid, type and payload (jsonb). The event is routed by default to a topic named outbox.event.<aggregatetype>, so customers goes to outbox.event.customers. The aggregateid becomes the message key, which keeps events for one entity ordered inside a Kafka partition (see Kafka as a log store). The id is sent as a header. The table is meant to be insert-only, and the router filters out delete operations, so you can clean old rows without emitting events. The same page notes the id can be used to remove duplicates but describes no built-in deduplication; that is the consumer's job.

:::key Verified November 2026 Debezium defaults above are from the stable outbox event router documentation read on 2 November 2026. Column names, the outbox.event. topic prefix and the warn default of table.op.invalid.behavior are configurable and can differ in other versions. :::

What the outbox does not give you. The relay can publish the same message twice, for example by crashing after publishing but before recording that it did (microservices.io). So the outbox converts "might lose" into "might duplicate", and the consumer must be idempotent (next section). Also, the outbox is only as ordered as its key: events for different aggregates are not ordered relative to each other.

What experts say in the room. "Never dual-write." The number is the window between commit and publish, which is zero only with an outbox. What a senior checks: that every event-producing write goes through the outbox (an easy rule to forget, as microservices.io says), outbox table growth and cleanup, relay lag as a metric, and the alert for the oldest unpublished row.

Exactly-once as an effect: idempotent consumers and the inbox

Delivery is at least once, so exactly-once is something the consumer builds out of deduplication, not something the broker hands over. The outbox relay can duplicate (section 5), brokers redeliver after an unacknowledged message, and consumers crash after acting but before acknowledging. microservices.io notes consumers usually need idempotence anyway because brokers can deliver more than once, and suggests tracking the ids of processed messages. Boundaries Are Where Guarantees Die covers why retries produce duplicates; Delivery semantics, honestly in Queues and Async Work covers the broker side. Here is the architecture-level answer.

The inbox pattern. The consumer keeps a table of processed event ids in its own database, and handles an event in one local transaction:

1. Begin transaction.
2. Insert the event id into inbox (a unique key). If the insert hits a duplicate, commit nothing and acknowledge: already done.
3. Apply the business change and write any outgoing events to the outbox.
4. Commit; then acknowledge the broker.

If the process dies after the commit but before the acknowledgement, the broker redelivers; step 2 sees the id and skips. If it dies before the commit, nothing happened and redelivery retries cleanly. The effect happens once even though delivery happened twice. The inbox plus the outbox on both sides is the closed loop that makes a chain of services effectively exactly-once.

Natural idempotence is better when you can get it. SET status = 'SHIPPED' repeated is harmless; balance = balance + 100 is not. An upsert keyed by entity id, or a version check ("apply only if event.version > row.version"), needs no inbox table. The Azure choreography example uses upserts so retries create no duplicate resources. Use the inbox for effects that are not naturally repeatable: money, counters, anything that sends.

Side effects outside your database. If the effect is a call to a payment provider, the inbox cannot make that call atomic with your commit. Pass the event id as the provider's idempotency key, so a repeated call is a no-op at their end. This is the same rule Payment Ledger uses.

The cost of the inbox. The table grows by every event. Worked arithmetic: 50 million events a day kept for 7 days is 350 million ids; at 16 bytes per uuid that is 5.6 GB of raw keys before index overhead, which is why a retention horizon is part of the design. The horizon must exceed the longest redelivery you will ever see (broker retention, replay of a backlog), or an old duplicate slips past. For a replay of three days of history through a consumer whose inbox keeps one day, the dedupe is gone exactly when you need it.

What experts say in the room. "I do not claim exactly-once delivery; I claim exactly-once effect." The number is the inbox retention against the longest possible redelivery. What a senior checks: a stable event id assigned at the source (not by the relay), the dedupe key stored in the same transaction as the effect, the retention horizon, and tests that deliver every event twice.

Schemas, ordering and evolution

An event is a public contract that outlives the code which wrote it, so treat its schema like an API with consumers you cannot see. Producers and consumers deploy independently. The Azure event-driven page says to define a schema versioning strategy early and design consumers to handle versions they do not recognise; the choreography page recommends a schema registry and backward-compatible evolution.

The rules, in order of cost.

• Add optional fields only. New fields get defaults, old consumers ignore them. Boundaries Are Where Guarantees Die covers how each encoding (JSON, Protobuf, Avro) makes that safe or unsafe, including never reusing a Protobuf field number and the registry's compatibility modes. Use that section rather than repeating it here.
• Never change a field's meaning. amount in rupees becoming amount in paise is a change no schema tool catches; make a new field.
• Version the event type, not just the schema, when a change is breaking: publish OrderPlaced.v2 alongside v1, migrate consumers, then stop publishing v1. The cost is producing two events for the migration window.
• Include an event id, a timestamp, an entity id and a version, because the inbox, ordering checks and tracing all need them.

Ordering is per key, not global. In a partitioned broker the order exists inside one partition (Distributed Message Queue). With the outbox keyed by aggregate id (the Debezium default key), events for one order stay ordered; events for two different orders do not, and an InvoiceIssued for order A may overtake OrderCancelled for order B without harm. A consumer that depends on a cross-entity order needs either a single key or a process manager that waits (the Azure choreography page uses a session identifier so the delivery service can pair the package and drone messages). When a message must be retried, the Azure event-driven page notes an error-handler processor resubmits events out of sequence, so ordering assumptions break exactly on the retry path.

Detecting gaps. Give each entity a version that increases by 1. The consumer applies version n only if it holds n-1; a version greater than that is parked until the gap fills or an alarm fires, and a version at or below its current one is ignored as stale or duplicate. This single rule handles reordering, duplicates and out-of-order replays together.

Replay is a schema test. Replaying a year of events, which sections 8 and 9 rely on, means today's consumer must read events written under every schema version you ever shipped. Teams that never replay discover their old versions are unreadable on the day they need to. Keep a sample of every historical version in the consumer's test suite.

What experts say in the room. "Who are the consumers of this field?" The unknown consumer is the problem. What a senior checks: a registry or contract tests in CI, a version number on each event, and one test that replays the oldest retained event through current code.

RabbitMQ's exchange model versus Kafka's log

RabbitMQ routes messages to queues and removes them once handled; Kafka appends to a log that consumers read at their own position and that is kept for a retention period. That one difference decides most event-driven choices. Queues and Async Work has the decision between Kafka, RabbitMQ and SQS and Kafka as a log store has Kafka's internals; this section only compares the architecture.

The AMQP 0-9-1 model, per the RabbitMQ concepts page. Producers publish to an exchange, never straight to a queue. The exchange applies bindings (rules linking it to queues, optionally with a routing key) to decide which queues get a copy. Four exchange types exist: direct (exact routing key match), fanout (every bound queue), topic (pattern match on the routing key) and headers (match on header values). A message with no matching binding is dropped or returned to the publisher. A consumer acknowledges explicitly, and the broker removes the message only then; if the consumer dies first, it is redelivered. Rejecting with requeue on a single failing message can loop forever, a poison message in the making.

Quorum queues. The RabbitMQ documentation describes them as durable, replicated queues based on Raft: one leader, followers, a quorum of (N/2)+1 members, with publisher confirms issued only after replication to a quorum. Three members tolerate one failure and five tolerate two, and the docs say they favour consistency over availability. For poison messages they have a delivery limit, 20 by default since RabbitMQ 4.0, after which a message is dropped or dead-lettered if a dead-letter exchange is set; setting it to -1 turns the limit off, which the docs advise against.

:::key Verified November 2026 RabbitMQ quorum queue defaults above were read on 2 November 2026: delivery limit 20 since RabbitMQ 4.0; from 4.3 it counts delivery failures rather than acquisitions. The default dead-letter strategy is at-most-once, so dead-lettered messages can be lost in transit unless at-least-once is configured with its prerequisites. :::

What this means for event-driven design. If your design needs a new service to catch up on history, or a bug-fix reprocessing run, the log's retention is the feature. If your design is mostly "route this job to the right worker and retry it independently", a broker with queues models it more directly. Many systems use both: events on a log, commands on queues.

What experts say in the room. "Do we ever need to read the past?" The numbers are the retention window and the delivery limit. What a senior checks: that dead-lettering is configured with the at-least-once strategy where loss is not acceptable, who watches the dead-letter queue, and the consumer's behaviour on an unbounded requeue loop.

Operating it: tracing, debugging, replays, poison events, and when not to

Event-driven systems move complexity from the call stack into operations: there is no stack trace across a bus, so you design the trail before you need it. The Azure event-driven page says it directly: in synchronous systems you can trace a request through a call stack, in event-driven ones a single business transaction spans producers, channels and consumers, with no shared call context. Its advice: include a correlation id in every event and plan this from the start, because retrofitting is much harder.

Tracing a flow. Mint a correlation id at the entry point and copy it into every event and log line; add a causation id (the id of the event that caused this one) so you can rebuild the tree. A trace is then a query. For orchestration, the orchestrator's state is the status view; for choreography, build a read model that listens to everything and answers "where is order 4411?" That read model is a real piece of the system, not a nice-to-have.

Debugging. The question is always one of: was the event published (check the outbox and relay lag), was it delivered (broker metrics, consumer lag), was it processed (inbox row, consumer logs), or was it processed wrongly (schema or logic). Each answer maps to one table or one metric, so give each its dashboard. The slowest question is "why did nothing happen?", because absence leaves no log line: alert on the age of the oldest unpublished outbox row and on sagas stuck in a non-terminal state for longer than their budget.

Poison events. One event that always throws will, on a log, block its partition, and on a queue, loop. Handle it with a bounded retry (RabbitMQ's quorum queue default of 20 deliveries, or your own count), then move it aside to a dead-letter destination with the failure reason, and let the rest flow. The Azure choreography page recommends flagging unacceptable messages with reason codes so they go straight to a dead-letter queue instead of cascading retries, and pairing that with a saga compensation, retry or pivot transaction. The dead-letter queue needs an owner and an alarm; otherwise it is a graveyard that hides an incident. Remember a parked event breaks per-key order for later events on that key; decide whether later events wait.

Replays. Replaying history rebuilds a read model or fixes a bug, and it is dangerous. Replayed events trigger side effects (emails, charges) in any consumer that is not idempotent; replays outrun an inbox with short retention (section 6); replays of old events hit old schemas (section 7). Rule: replays go to a new consumer group or a rebuild process that writes only to a derived store and has its outbound effects switched off. Worked arithmetic: replaying 30 days at 50 million events a day is 1.5 billion events; at 20,000 per second that is 75,000 seconds, about 21 hours, and your live traffic must not share that consumer.

Eventual consistency is a product problem. After the user clicks "place order", the read model has not caught up. The UI must show a pending state or read your own write; otherwise users submit twice. The Azure event-driven page makes this its central challenge.

When not to go event-driven. The Azure page lists the cases directly: simple request-response flows where synchronous calls meet the latency and throughput needs, because broker operations, asynchronous error handling and eventual consistency are overhead there; business transactions needing strong consistency across services; and teams without experience operating distributed asynchronous systems. Add one more of your own: a single team with a single database. A local transaction beats a saga every time, and a modular monolith (see Decomposing a System into Services) gives the same boundaries without the bus.

What experts say in the room. "Show me where order 4411 is, and show me the dead-letter queue." The numbers are retry limit and replay duration. What a senior checks: correlation and causation ids in every event, an owner and an alarm on every dead-letter destination, replay runbooks that disable side effects, and an honest answer to "is a synchronous call enough here?"

Further reading: Sam Newman, Building Microservices (2nd ed.), for the communication and workflow chapters (4 Microservice Communication Styles, 5 Implementing Microservice Communication, 6 Workflow); Neal Ford, Mark Richards, Pramod Sadalage and Zhamak Dehghani, Software Architecture: The Hard Parts, chapter 12 Transactional Sagas. Primary sources: the Azure Architecture Center pattern pages, microservices.io, the Debezium outbox event router reference and the RabbitMQ documentation.

How to say it in the interview

"I split 'event-driven' into notification, state transfer, sourcing and CQRS before I answer, because they have different costs. For a cross-service business flow I use a saga: local transactions with compensations, orchestrated once there are more than about three or four steps, because choreography hides the flow. Every service writes its state change and its outgoing event in one local transaction, an outbox, and a relay publishes it, so there is no dual write. Delivery is at least once, so every consumer records the event id in the same transaction as its effect. The thing I will not claim is exactly-once delivery; I claim exactly-once effect, and I name the steps a compensation cannot undo."

Checklist

• Named which of the four patterns (notification, state transfer, sourcing, CQRS) is in use and tested each event for being a disguised command
• Chose thin or fat payloads per consumer, with a version field on every state-carrying event
• Chose choreography or orchestration by step count and need for a status view, and kept domain rules out of the orchestrator
• Marked the pivot transaction in every saga and placed irreversible effects after it
• Gave each saga step a named, idempotent compensation and an alert for sagas stuck past their budget
• Wrote every state change and its outgoing event in one local transaction through an outbox, with relay lag and oldest-unpublished-row alerts
• Made every consumer idempotent using an inbox, a natural idempotent write or a version check, with the dedupe key committed with the effect
• Sized inbox retention above the longest redelivery or replay horizon
• Kept events backward compatible, versioned breaking changes as new event types, and replayed the oldest retained event through current code in CI
• Put correlation and causation ids in every event, an owner and alarm on every dead-letter destination, and a replay runbook with effects disabled

Related lessons

• Decomposing a System into Services
• Service-to-Service Communication: Discovery, Mesh, Identity and Compatibility
• API Gateway
• Queues and Async Work