Cruxly › System design › Design Exercises

Payment Ledger

money cannot be lost, double-spent, or invented

Prompt: design the core of a payment system — move money from one account to another, record it permanently, and expose a balance. Handle the case where the external card network times out and you genuinely do not know whether the charge succeeded.

40 minutes first. The interesting part is what you do when you don't know what happened.

Applying the four moves: Scope — transfer between accounts, permanent record, balance, and the timeout where you don't know if the charge happened; fraud scoring and FX out. Estimate — volume is modest compared with the other exercises; what's sized here is the cost of being wrong, which is unbounded, so correctness dominates every trade-off. Sketch — an append-only double-entry ledger where a balance is derived from entries rather than stored and mutated, plus idempotency keys on every inbound request. Deep dive — the unknown outcome: never guess, record the attempt as pending, reconcile against the network's own record, and make every retry safe because the key makes it the same operation.

Scope, and the one property everything hangs on

In scope: an internal ledger, transfers between accounts, balance reads, idempotent payment initiation, reconciliation against an external processor. Out of scope: fraud scoring, KYC, currency conversion rates (treat rates as an input, not something to design), chargeback dispute workflow. Naming these keeps the conversation on the ledger rather than on the surrounding compliance estate, which is genuinely enormous.

Estimate. 10 million transactions/day, spiky (heavily weighted toward business hours and month-end).

Writes: 10^7 / 10^5 s = 100 transfers/s average, maybe 1,000 peak.
These numbers are small on purpose. Payments is almost never a throughput problem — it's a correctness problem at modest scale. Saying that out loud early prevents you from spending the interview sharding something that doesn't need sharding.
Reads: balance checks maybe 10x writes, and every read is a point lookup.

The property everything hangs on: double-entry. A transfer is never "subtract 50 from A." It is two entries — A: -50, B: +50 — written atomically, that sum to exactly zero. Every entry belongs to a transaction; every transaction's entries sum to zero; therefore the sum of all entries in the system is always zero, forever. That invariant is checkable with one query, which makes "did we lose money" a question with an actual answer rather than a hope.

Balances are derived, not stored. A balance is the sum of an account's entries. Storing a running balance creates a second source of truth that can silently disagree with the ledger, and when they disagree you cannot tell which one is wrong. If summing gets slow, store periodic checkpoints (balance as of midnight) and sum only entries since — a cache that is rebuildable from the ledger, not an independent authority.

Every transaction ever written sums to zero individually, which means the whole ledger sums to zero collectively — the invariant scales from one transfer to the entire system's history for free.

The ledger schema, and why entries are immutable

transaction(id, idempotency_key UNIQUE, status, created_at, external_ref)
entry(id, transaction_id, account_id, amount_minor, currency, created_at)

Entries are append-only and immutable. There is no UPDATE and no DELETE on this table, ever. A mistake is corrected by writing a reversing transaction (equal and opposite entries), not by editing history — which means the ledger is a complete, auditable record of everything that ever happened, including the mistakes and their corrections. Auditors require this; more practically, it's the only way to answer "why is this balance what it is" six months later.

Money is integer minor units, never floating point. 0.1 + 0.2 != 0.3 in binary floating point, and over millions of transactions that drift is both real and completely unexplainable to a regulator. Store amount_minor as an integer (cents) with an explicit currency, so adding USD to EUR is a type error rather than a silent corruption.

Atomicity comes from putting both entries in one transaction on one partition. Both entries of a transfer are written in a single database transaction. If accounts live on different shards, this becomes a distributed transaction — which is why, at this scale (100/s), the right answer is usually don't shard the ledger: a single well-provisioned relational primary handles this volume comfortably, and you have preserved single-node ACID, which is worth vastly more here than horizontal headroom you don't need. Say that explicitly — choosing not to distribute is a design decision, and defending it is stronger than reflexively sharding.

Immutability isn't a restriction to work around — the ledger's entire value as a record depends on never being able to make a mistake disappear.

Idempotency and the unknown-outcome problem

A timeout is a third outcome, not a failure. Treating it as failure double-charges the customer on retry; treating it as success invents money. The only correct move is to stay pending and find out.

Every payment initiation carries a client-supplied idempotency key. Store it with a UNIQUE constraint on the transaction table. A retried request with the same key finds the existing transaction and returns its current state instead of starting a second charge — the constraint does the work, so two concurrent retries cannot both pass a check-then-insert race.

The three outcomes, and why there are three. A call to the card network returns success, explicit failure, or nothing (timeout, connection reset). That third case is the one that matters: the charge may well have succeeded on their side. Retrying blindly risks charging twice; marking it failed risks telling a customer their payment failed after their card was debited.

The rule: never guess, stay pending, reconcile. Leave the transaction PENDING, return "processing" to the caller, and let a reconciliation job query the processor by the idempotency key to learn the real outcome. Pass your idempotency key to the processor too, so their side deduplicates identically.

Reconciliation is not an afterthought — it's a first-class component. A scheduled job pulls the processor's settlement report and compares it against your ledger, in both directions: transactions you have that they don't (you think money moved and it didn't) and transactions they have that you don't (money moved and you never recorded it). The second is the scarier one, and only a two-way comparison finds it.

Consistency, failure modes, and what not to build

Reads. Balance reads can come from a replica, but a read immediately after a transfer by the same user must not show the old balance — the classic read-your-own-writes problem. Route a user's reads to the primary for a short window after they write, or have the client pass the transaction's position and the replica wait for it. Either is fine; not noticing the problem is not.

Never use eventual consistency for the ledger itself. This is the same argument as seat booking elsewhere in this track, but sharper: a stale seat map annoys someone, a stale balance lets them spend money twice. The ledger writes are the small, strongly-consistent core; everything derived from it (search, analytics, notifications, monthly statements) can be eventual and rebuildable.

Failure modes worth raising unprompted:

Processor is down entirely. Queue the intent durably and fail gracefully to the user ("payment processing, we'll confirm shortly") rather than dropping it. This is only safe because reconciliation exists — without it, a queued intent is a payment you may never resolve.

Partial batch settlement. The processor settles 900 of your 1,000 transactions and is silent about the rest. Your reconciliation must handle "missing from their report" as its own state, distinct from "they say it failed."

Duplicate webhook. Processors retry webhooks, and will deliver the same settlement notification more than once. Webhook handlers must be idempotent on the processor's event ID — the same discipline as the notification system's dedupe, applied to inbound events.

Clock skew on expiry. If a pending transaction expires after N minutes, use the database's clock for that comparison, not an application server's — skewed app clocks disagree about whether something expired, and here that disagreement is money.

What not to build: don't reach for a distributed transaction or a two-phase commit across your ledger and the external processor. You cannot make an external card network participate in your transaction. The idempotency-key-plus-reconciliation pattern above is what the industry actually does because 2PC across an organisational boundary isn't available — saying that directly is a strong, specific answer to "how do you keep them in sync."

2PC fails here for an organizational reason, not a technical one — no external processor will ever agree to hold a lock for your transaction, so the design has to route around that boundary entirely.

How to say it in the interview

"Two properties drive everything: money is conserved, so every movement is two entries that sum to zero; and I will at some point not know whether an external call succeeded, so the design has to make 'find out later' safe rather than guessing."

Checklist

Led with correctness, and said explicitly that payments is not a throughput problem at this scale
Used double-entry so every transaction's entries sum to zero
Derived balances from entries rather than storing a mutable running balance
Used checkpoints plus recent entries if summing gets slow, framed as a rebuildable cache
Made entries append-only, correcting mistakes with reversing transactions
Stored money as integer minor units with an explicit currency
Named the timeout as a distinct third outcome, not a failure
Stayed PENDING on unknown outcomes and resolved via reconciliation
Enforced idempotency with a UNIQUE constraint, not a check-then-insert
Made reconciliation two-way, including transactions they have that you don't
Handled read-your-own-writes for a balance read right after a transfer
Rejected distributed transactions across the external processor boundary

Practice this interactively