Cruxly

Cruxly › System design › Design Exercises

Prepaid Recharge & Wallet System

the money moves exactly once, even when your own server retries the request that moves it

Open the interactive lesson

Prompt: design the backend for a telecom prepaid recharge and wallet system — a user tops up their wallet or directly recharges a mobile number with a plan, the plan has a validity window and a set of benefits (data, calls, SMS), and the system must survive payment-gateway retries, network timeouts, and a promotional-offer day where recharge volume spikes 20x.

40 minutes first. This looks like a CRUD app until you say the sentence out loud: the same recharge request can arrive twice, and the wallet must end up debited exactly once either way.

Applying the four moves: Scope — wallet top-up, direct recharge, plan validity and benefits, gateway retries and timeouts, a 20× promotional spike; the plan catalogue's commercial logic out. Estimate — peak recharge rate on the promotion day, sized against 20× the normal rate, plus how often the payment gateway's response is ambiguous. Sketch — an idempotent recharge API, a ledger-style wallet with the balance derived from entries, and a reconciliation job against the gateway. Deep dive — the sentence the prompt asks you to say out loud: the same request can arrive twice, so an idempotency key per request makes the debit happen exactly once, and an ambiguous gateway result is held as pending and reconciled rather than guessed.

Scope, and the failure that defines the whole design

In scope: wallet top-up, direct mobile recharge with a plan (data/calls/SMS bundle plus a validity window), idempotent payment processing, plan activation and expiry, handling a payment gateway that can time out with an unknown outcome. Out of scope (say it): the payment gateway's own internals (card networks, UPI rails — treat it as a third-party API with its own failure modes), customer support/refund workflows beyond the core reconciliation logic, tax and regulatory reporting.

Estimate. Say 300 million prepaid subscribers, 2 recharges/month on average, concentrated heavily around salary dates and promotional days.

The failure that defines the design: the client calls "recharge," the request reaches the server, the server calls the payment gateway, and then the network times out before the gateway's response comes back. The money may or may not have moved. The client, seeing a timeout, will retry — and now the server must not double-debit the wallet or double-apply the plan, without actually knowing yet whether the first attempt succeeded.

This is not a wallet-design problem, it is an idempotency-and-reconciliation problem that happens to have a wallet attached to it. Every other decision in this exercise — the ledger schema, the state machine, the async reconciliation job — exists to answer one question honestly: what do we do when we don't know if the money already moved?

The ledger: idempotency keys and an append-only balance

A wallet balance is never stored as a single mutable integer updated in place. Every recharge, top-up, and debit is an immutable, append-only ledger entry (account_id, amount, type, idempotency_key, status, created_at), and the balance shown to a user is the sum (or a periodically-materialized running total) over their entries — the exact same "immutable entries, derived balance" pattern as this track's payment-ledger exercise, applied here to a wallet instead of a bank transfer.

The client generates an idempotency key once per user-initiated recharge attempt (a UUID created client-side before the first request) and sends the same key on every retry of that same logical attempt. The server's write path is a single conditional insert: INSERT INTO ledger (..., idempotency_key) VALUES (...) ON CONFLICT (idempotency_key) DO NOTHING, then read back whatever row now exists for that key. A retry with the same key either sees its own original row (safe to return the same result) or, if the first request is still in flight, briefly conflicts and can poll — but it can never create a second debit.

public class LedgerEntry {
    String id, accountId, idempotencyKey;
    long amountPaisa;                 // integer minor units, never floating point
    EntryType type;                   // TOPUP, RECHARGE_DEBIT, REVERSAL
    EntryStatus status;               // PENDING, CONFIRMED, FAILED
    Instant createdAt;
}

public class RechargeService {
    LedgerEntry initiate(String accountId, long amountPaisa, String idempotencyKey) {
        LedgerEntry existing = ledger.findByIdempotencyKey(idempotencyKey);
        if (existing != null) return existing;              // retry — return prior outcome
        return ledger.insertPending(accountId, amountPaisa, idempotencyKey);
    }
}

Why append-only, not a mutable balance column: a mutable UPDATE balance = balance - amount cannot tell you, after the fact, whether a given retry already applied — the column has no memory of individual attempts. An append-only ledger keyed on idempotency key makes "did this exact attempt already happen" a single indexed lookup instead of an unanswerable question.

The unknown-outcome problem: when the gateway itself times out

The idempotency key protects against a client retry — it does not solve the harder case where the server itself doesn't know if its own call to the external payment gateway succeeded. The server sent a debit request to the gateway, the connection dropped before a response arrived, and now the ledger entry sits in PENDING with no way to know, synchronously, whether the money actually moved on the gateway's side.

The honest fix is a reconciliation job, not a smarter retry. The pending entry is left exactly as PENDING (never guessed into CONFIRMED or FAILED). A background job periodically queries the payment gateway's own status-check API (every real gateway offers one, precisely because this failure mode is universal) for any entry PENDING longer than a short grace period, and transitions it to CONFIRMED or FAILED based on the gateway's authoritative answer — never based on a guess.

The plan is not activated until the ledger entry reaches CONFIRMED. This is the one invariant that must never be relaxed for latency: activating data/call benefits on an optimistic PENDING entry, then discovering the payment failed, means giving away service that was never paid for — a direct revenue leak at the volume this system runs at. A user-facing "processing, we'll confirm shortly" state is a better product tradeoff than an optimistic activation that sometimes has to be clawed back.

If reconciliation finds the gateway actually succeeded but the local entry is stuck PENDING, it transitions to CONFIRMED and then triggers activation — late, but correct. If reconciliation finds the gateway failed or never received the request, it transitions to FAILED and nothing was ever charged. Either outcome is derived from the gateway's own record, never assumed.

Absorbing the promotional-day spike, and plan expiry

A 20x spike concentrated in a few hours is a queueing problem, not a provisioning problem — provisioning 20x baseline capacity that sits idle 99% of the month is wasteful, and the actual traffic shape (a burst, not a sustained plateau) is exactly what a queue absorbs well. The recharge-initiation API accepts the request, writes the PENDING ledger entry (fast, local, cheap), and returns immediately; the actual gateway call is dispatched through a work queue that a fixed-size worker pool drains at a steady rate, so the payment gateway itself — which has its own rate limits — is never hit with a 20x burst directly.

Plan validity and expiry is a scheduled-transition problem, the same primitive as this track's job-scheduler exercise. Each active plan has an expires_at; rather than a sweeper job scanning for expired plans (expensive at hundreds of millions of active plans, and racy against a renewal happening at the same instant), expiry is checked lazily on read — "is this plan currently valid" is expires_at > now(), evaluated at the moment a data/call/SMS session actually needs to check entitlement, with a low-priority batch job only for the housekeeping of marking obviously-stale rows for archival, not for correctness.

Renewal at expiry is the same idempotency-keyed recharge path, not a special case. An auto-renewal at the exact expiry instant and a user manually recharging early both funnel through the identical initiate() call from earlier — the idempotency key here can even be derived from (account_id, plan_id, billing_cycle_id) for auto-renewals, so a renewal worker that crashes and restarts cannot double-charge for the same cycle.

How to say it in the interview

"A recharge is a distributed transaction across three parties who don't share one database: the user's client, our own ledger, and an external payment gateway or telecom core. Any one of those three can time out without telling the others what actually happened. So the design isn't really about wallets or plans — it's about making every step idempotent and reconciling the one case that idempotency alone can't fix: we genuinely don't know if the far side succeeded."

Checklist

Practice this interactively
Share this lessonWhatsAppLinkedInX

Related lessons