Cruxly › System design › Principles of Distributed Design

Boundaries Are Where Guarantees Die

at every hop, name the guarantee you just lost

The principle: inside a single process you get ordering, atomicity and immediate consistency almost for free. Every boundary you cross — process, network, service, datacentre — silently removes some of them. At each hop, name which guarantee you just lost, because the bug you are about to write lives exactly there.

Most distributed systems bugs are not exotic. They are a developer reasoning with single-process intuitions about code that crossed a boundary where those intuitions stopped holding.

What each boundary actually takes away

• The row that catches people is service → service, because the code looks identical. A method call and a service call are both doTheThing(x), but one is atomic with its caller and one is not.

// Looks like two statements. Is not two statements.
orderService.createOrder(order);      // succeeds
paymentService.charge(order.total);   // times out — did it charge?

• There is no transaction here. The order exists, the charge outcome is unknown, and a naive retry may charge twice.
• Single-process intuition says "if the second line throws, the first is rolled back" — that intuition is simply false across a boundary, and the design must state what happens instead.
• The network adds failures with no local equivalent. A timeout is not a failure — it is the absence of information. The call may have succeeded, failed, or still be running. Code that treats a timeout as a failure and retries is making an assumption it has no basis for, which is where double-charges come from.

Designing across the boundary instead of pretending it isn't there

Three mechanisms recover most of what the boundary took, each recovering something different.

• Idempotency keys recover safe retry. The caller generates a key once per logical operation and sends it on every attempt; the callee records it and returns the original result for repeats. This converts "unknown outcome" from a correctness hazard into an inconvenience — retry freely, because a duplicate cannot take effect twice.
• The outbox pattern recovers atomicity between a local write and a message. The genuine problem: you must update your database and publish an event, and there is no transaction spanning both. Write the event into an outbox table inside the same database transaction as the state change — now they commit or fail together — and let a separate process read the outbox and publish. The event may be published twice, never zero times, which is exactly why consumers must be idempotent.
• Sagas recover a form of multi-step correctness. For a workflow spanning several services, replace the transaction you cannot have with an explicit sequence of steps, each with a compensating action that undoes it. Notably this gives eventual correctness, not atomicity — intermediate states are visible to other readers, and the compensation may itself fail. It is the honest tool for the job, not a distributed transaction in disguise.

:::key Why "just use a distributed transaction" is the wrong instinct Two-phase commit does exist and does provide atomicity — by holding locks across services for the duration and blocking if the coordinator dies mid-protocol. Across independent services and third-party APIs (your order service and a payment provider), that availability cost is almost always unacceptable, which is why the industry standardised on idempotency plus reconciliation there. Saying this out loud, with the reason, is a much stronger answer than either using 2PC uncritically or dismissing it without knowing why.

Inside a distributed database, though, 2PC is standard. Spanner and CockroachDB run two-phase commit across shards, but each participant is itself a consensus-replicated group, so a coordinator crash doesn't block anything — another replica takes over the coordinator's role. The problem isn't 2PC itself; it's 2PC across components that can't make themselves highly available. :::

Ordering, versioning, and the boundaries people forget

• Messages arrive out of order, and this breaks more designs than message loss does. A queue that guarantees delivery rarely guarantees global ordering. If ProfileUpdated(name=A) and ProfileUpdated(name=B) are processed in the wrong order, the profile ends up with a stale name and nothing reports an error. The fixes: include a version or timestamp and reject anything older than what you hold, or partition by entity so all events for one entity are ordered relative to each other. The second is usually cheaper and is why partition keys exist.
• Versioning is a boundary in time. Two services deploy independently, so at some moment during every rollout, v1 and v2 are both running and talking to each other. A field removed in v2 must therefore be tolerated by v1, and a field added in v2 must be ignorable by v1 — which makes backward compatibility a runtime requirement rather than a courtesy. This is why additive-only schema changes are the norm: add a field, migrate readers, remove it in a much later release.
• The boundary inside one database. Two transactions are also separated by a boundary, and isolation level decides which anomalies leak across it. Read-committed permits non-repeatable reads; a SELECT then UPDATE based on it is a lost-update bug unless the read takes a lock or the update is conditional. This is the same principle at a smaller scale, and it catches people precisely because it does not look like a distributed systems problem.
• The boundary between your code and a third party. You do not control their latency, their error semantics, or when they change behaviour. A provider updating a model, tightening a rate limit, or altering an error code is a change you did not deploy and will not see in your logs — which is why anything crossing this boundary needs its own timeout, its own circuit breaker, and monitoring of their behaviour rather than only yours.

Encoding formats and schema evolution

The versioning diagram above says "tolerate unknown fields" as if every format made that easy. It does not. Whether v1 can read v2's data is decided by the encoding, and you pick the encoding long before you ever deploy twice.

• Text encodings (JSON) carry their own description. Every message repeats its field names, so a reader needs no schema and a human can debug it with curl. The price is bytes and parsing work, plus loose typing: JSON has one number type, and RFC 8259 section 6 says good interoperability is only guaranteed for numbers that fit what an IEEE 754 binary64 double offers. Worked arithmetic: integers above 2^53 (9,007,199,254,740,992) cannot all be represented as doubles, so a consumer that parses a 64-bit id as a double turns 9007199254740993 into 9007199254740992. Send large ids as strings.
• Binary encodings (Protobuf, Avro) leave the description out. Worked arithmetic: {"id":1234} is 11 bytes of JSON. In Protobuf the same value is a one-byte tag plus a two-byte varint, 3 bytes. Across a service that sends a billion such messages a day that is the difference between 11 GB and 3 GB of payload, before compression. The saving is real but modest next to the cost you take on: the bytes are meaningless without the schema, so the schema becomes a deployed artifact with its own lifecycle.
• Protobuf identifies fields by number, not name. The wire carries the field number and a value; names exist only in the .proto file. That gives the single hardest rule in the format: never reuse a field number. If 4 once meant price as an integer and is later reused for region as a string, a reader with the old schema decodes new data as garbage, or old stored data as the wrong field, and the wire format has no way to notice. When you delete a field, reserved its number so nobody can reuse it by accident (the compiler then rejects it); reserving the name matters mainly for JSON and text formats, where names appear.
• Protobuf compatibility comes from unknown-field tolerance. A reader that meets a number it does not know keeps it and carries on, and a field missing from the data takes its type's default. So adding a field is safe both ways. Two caveats a staff engineer volunteers: the default for "absent" is indistinguishable from "set to zero" unless the field is declared with presence, and converting through JSON or copying field by field can silently drop the unknown fields you were relying on to survive a hop.
• Avro puts no tags on the wire at all. The bytes are bare values in schema order, so a reader must resolve two schemas: the writer's schema (what produced the bytes) and the reader's schema (what the code wants). Fields are matched by name; a field the writer has and the reader lacks is skipped; a field the reader wants and the writer lacks takes the reader's default, and with no default decoding fails. This is why, in Avro, a new field without a default is a breaking change in one direction, and why the writer's schema must travel with the data.
• A schema registry is how the writer's schema travels cheaply. Instead of embedding the schema in every message, the producer registers it and sends a small id; the consumer fetches the schema by id. The registry also enforces a compatibility mode at registration time, so a breaking schema is rejected in CI instead of discovered in production.

:::key Verified November 2026 Confluent Schema Registry compatibility modes: BACKWARD (the default) means consumers on the new schema can read data written with the old one, so upgrade consumers before producers. FORWARD means consumers on the old schema can read data written with the new one, so upgrade producers first. FULL means both, which lets producers and consumers upgrade in any order, and only adding or removing optional fields qualifies. The TRANSITIVE variants check against every earlier version, not just the last. NONE disables the check. The default is non-transitive, which matters if old data or old consumers live for months. :::

The rolling deploy is where this stops being theory. While the rollout runs, old and new code are both writing, and anything they write (rows, queue messages, cache entries) may be read later by either. You need both directions at once: new code reading old data (backward) and old code reading new data (forward). That is the FULL case, and it is why the safe recipe is the same in every format: add only optional fields with defaults, never repurpose or renumber, remove in a later release after every writer stopped using the field, and keep the removed number reserved.

Further reading: Martin Kleppmann, Designing Data-Intensive Applications, on encoding and evolution. Primary sources: the Protocol Buffers language guide (protobuf.dev), the Avro specification (avro.apache.org) and the Confluent Schema Registry compatibility documentation.

How to say it in the interview

"The moment this crosses a service boundary I lose atomicity between the two writes, so I want to be explicit about what happens if the first succeeds and the second fails — because that is not a rare case, it is a case that will happen thousands of times a day at this volume."

Checklist

• Named which specific guarantee was lost at each boundary the design crosses
• Treated a timeout as an unknown outcome rather than as a failure
• Used an idempotency key to make retries safe instead of trying to avoid retries
• Used an outbox to make a local write and an event publish atomic
• Said that outbox delivery is at-least-once and therefore consumers must be idempotent
• Chose a saga over a distributed transaction and explained the availability reason
• Handled out-of-order delivery with versioning or entity partitioning
• Accounted for v1 and v2 running simultaneously during a rollout
• Chose an encoding and a schema-evolution rule, never reusing a field number and adding only optional fields with defaults

Related lessons

• State Gravity
• Asymmetry First
• Failure Is a Mode, Not an Exception
• Truth Has One Home
• Make the Expensive Path Rare