Cruxly › System design › Design Exercises
Webhook Delivery System
you are the client now, and the server you're calling can be down for as long as it wants
Open the interactive lessonPrompt: design a webhook delivery system — when an event happens in your system (an order ships, a payment completes), you POST it to a URL each customer configured, reliably, even though their endpoint might be down, slow, or misconfigured, and you don't control it at all.
40 minutes first. Every other exercise in this track assumes you control both ends of the request. This one flips that — the receiving end is somebody else's code, of unknown quality, that you cannot fix.
Applying the four moves: Scope — POST events to customer-configured URLs reliably, even when endpoints are down, slow, or broken; the customers' endpoints are out of your control by definition. Estimate — event rate times the number of subscribed endpoints, plus how long a failed endpoint can stay down while retries queue up behind it. Sketch — an event log, per-endpoint delivery queues, workers with retry and exponential backoff, and a dead-letter store. Deep dive — you don't control the other end, so one slow endpoint must not block anyone else's deliveries (per-endpoint isolation), delivery is at-least-once so receivers need a signed payload and an event ID to dedupe on, and repeatedly failing endpoints get disabled with notice rather than retried forever.
Why this is the mirror image of every other exercise here
Scope. In scope: reliable outbound delivery, retry/backoff policy, delivery status visibility for the customer, payload signing for authenticity. Out of scope: the receiving endpoint's own implementation (explicitly not this system's problem, though its failure modes are).
Every other exercise in this track assumes reasonable control over both sides of a request — the payment exercise's external payment provider is at least a professionally-run, generally-reliable service; here, the receiving endpoint is one of potentially thousands of different customers' own servers, of wildly varying quality, uptime, and correctness, none of which this system can fix, patch, or influence. This reframes the entire exercise around one question: how much retry effort is owed to a receiving endpoint that might simply be broken, and how does the sender communicate delivery status back to a customer who needs to know whether their own system actually got the event.
Retry policy: exponential backoff with a real ceiling
Exponential backoff (retry after 1 minute, then 5, then 30, then 2 hours, then 12 hours — roughly doubling-and-then-some each time) gives a temporarily struggling endpoint increasing room to recover without hammering it at a fixed interval, which would be both ineffective against a genuinely down server and actively harmful if the endpoint is struggling specifically because of load. The retry window has a real ceiling — a day, typically — after which the event is marked permanently failed rather than retried indefinitely. An endpoint that's been unreachable for 24 hours is very unlikely to suddenly become reachable in a way that makes attempt number 40 succeed where attempts 1 through 39 didn't, and retrying forever wastes resources while giving the customer no clear signal that something is actually wrong on their end.
Distinguishing retryable from non-retryable failures matters as much as the backoff schedule itself. A 500 or connection timeout suggests a transient problem worth retrying; a 410 Gone or a 4xx indicating the endpoint itself rejects the payload's shape is not something retrying will fix — retrying a request the receiver has explicitly and correctly rejected wastes effort and delays the customer's visibility into a configuration problem they actually need to fix on their end.
Delivery visibility and payload authenticity
The customer needs a delivery log — every attempt, its outcome, and the current status of each event — because from their side, "did my webhook fire" is otherwise a black box. A dashboard or API surfacing delivery status (delivered, retrying, permanently failed) with response codes and timestamps per attempt is what lets a customer's own engineers debug their receiving endpoint, rather than emailing support asking "did you send this."
Manual redelivery (a customer fixes a bug in their endpoint and wants to replay events that failed during the outage) is a real, frequently- needed feature, not an edge case — events should be retained and replayable on demand for some retention window after permanent failure, not deleted the moment the retry budget is exhausted.
Payload signing (HMAC of the payload body using a per-customer shared secret, sent as a request header) lets the receiving endpoint verify a webhook actually came from this system and wasn't forged or replayed by a third party who discovered or guessed the customer's endpoint URL. Without signing, any anonymous internet request to that URL with a plausible-looking payload is indistinguishable from a genuine webhook — a real security gap for any customer whose webhook triggers a meaningful action (like fulfilling an order) on receipt.
Failure modes and the honest edges
Duplicate delivery is a near-certainty over enough volume, not a rare edge case — a delivery attempt can succeed on the receiving end while the response is lost in transit (network failure after the receiver processed it but before the sender saw the 200), causing this system to retry a webhook the receiver already handled. The receiving endpoint must be told to expect duplicates and build its own idempotency handling using a unique event ID sent with every delivery — this system can minimize duplicates but, per the same fundamental at-least-once-delivery limit covered in this track's ad-click exercise, cannot eliminate them, and pretending otherwise sets the customer up to build a fragile integration.
Ordering is not guaranteed across retries and concurrent events — if event A fires, fails, and retries while event B (which happened after A) succeeds on its first attempt, the customer may receive B before A. A system that needs strict ordering has to say so explicitly in its documentation and either serialize delivery per customer (at a real throughput cost) or make the event payload itself carry a sequence number the customer can use to reorder on their end — silently assuming ordering "usually" holds is not an honest answer.
A single slow customer endpoint must not block delivery to every other customer — delivery attempts need to be isolated per customer (separate queues or worker pools, not one shared queue processed strictly in order), the same isolation principle as this track's API-gateway exercise's per-backend circuit breakers, here applied to outbound delivery instead of inbound routing.
How to say it in the interview
"Every assumption that made HTTP APIs manageable elsewhere in this track breaks here, because the server on the other end of this outbound call is owned by a customer, might be down for their own unrelated reasons, might take 30 seconds to respond, and the only lever this system has is retry policy and clear failure signaling back to them — never a fix on their end."
Checklist
- Reframed the exercise around retry effort owed to an endpoint outside this system's control, rather than assuming reliability
- Used exponential backoff with a real time ceiling, not fixed-interval or unlimited retries
- Distinguished retryable failures (5xx, timeout) from non-retryable ones (4xx payload rejection)
- Surfaced a per-event delivery log with attempt outcomes and response codes to the customer
- Supported manual redelivery of permanently failed events within a retention window
- Signed payloads with a per-customer HMAC secret so receivers can verify authenticity
- Named duplicate delivery as a near-certainty at volume, not a rare edge case, and pushed idempotency handling to the receiver
- Stated explicitly that delivery ordering is not guaranteed across retries and concurrent events
- Isolated delivery per customer so one slow or down endpoint doesn't block delivery to every other customer