Cruxly › System design › Design Exercises

Notification System (HLD)

one event, five channels, one user who must never be paged twice

Prompt: design a notification system. An event ("your order shipped", "someone commented on your post") should reach a user by push, email, SMS or in-app — per-user, per-event-type preference — without duplicate or out-of-order delivery, and without waking someone at 3am for something that isn't urgent.

40 minutes first. The interesting part is not "send an email".

Applying the four moves: Scope — event in, delivery by push/email/SMS/in-app per user preference, no duplicates, no out-of-order, quiet hours respected; template authoring out. Estimate — peak event rate and fan-out per event (one comment can notify many users), plus the provider rate limits each channel imposes, which cap you more than your own capacity does. Sketch — event intake, a preference-and-routing step, per-channel queues and senders. Deep dive — duplicates come from retries, so idempotency sits above every channel; ordering is per user, not global; and "don't wake someone at 3am" is a scheduling rule that has to live in the pipeline, not in each sender.

Scope and numbers

In scope: accept an event, resolve which users should be notified and on which channels, render a message per channel, deliver it, track delivery status, respect user preferences and quiet hours, de-duplicate. Out of scope: the business logic that decides whether something is notification-worthy (that lives upstream, in the order service or the social-graph service) and marketing/bulk campaign sends, which have a different traffic shape and different consent rules. Naming that split early stops the interview from wandering into whichever upstream service happened to be mentioned.

Estimate. Say 500 million daily active users, an average of 8 notification-worthy events per user per day, each fanning out to up to 3 channels (push + in-app always, email/SMS conditionally).

Events in: 4 x 10^9 / 10^5 s ≈ 40,000 events/s average, ~120,000 peak.
Deliveries out: with ~1.8 channels per event on average, ~70,000 deliveries/s average, ~210,000 peak.
The channels do not share a rate limit. Push is nearly free at this volume; SMS has a real per-message cost and a much lower provider-side rate limit — this asymmetry is worth stating up front, because it is what forces per-channel queues rather than one shared send pipeline.

What the numbers say. This is a fan-out-and-render problem at moderate absolute volume, dominated not by raw throughput but by the fact that every channel has a different cost, a different failure mode, and a different definition of "sent."

The per-channel asymmetry isn't a detail to handle later — it's the reason the pipeline has to fork into separate queues at all.

The pipeline, and why each stage is separate

Fan-out happens once, immediately after preferences and dedupe are resolved; everything downstream of that point is per-channel, because push, email and SMS do not fail the same way or cost the same amount.

Ingest. The event lands in a durable log (not a direct call into the notification service) so a burst of order-shipped events doesn't back-pressure the order service itself, and so a downstream outage doesn't lose events — they wait in the log.

Fan-out. One worker tier consumes the log, resolves the user(s) and, per user, checks: is this event type enabled, is it currently quiet hours for this user's timezone, has this exact notification already been sent (dedupe, below). What survives becomes one row per (user, channel) — this is the only place fan-out happens, and it is the only place preferences are checked, deliberately, so a bug in one channel's sender can never cause a preference to be silently ignored.

Render. Per channel, because push wants a short title and body under a strict byte limit, email wants a full HTML template, SMS wants plain text under 160 characters with a shortened link. Rendering is templated and versioned so a template bug does not require redeploying the sender.

Send. One queue and one sender per channel, each with its own rate limit and retry policy, because a provider outage or rate-limit on one channel must not touch the others. This is the direct payoff of keeping fan-out and send as separate stages — the queues sit between them precisely so each channel's pace of consumption is independent.

De-duplication and idempotency

Two failure sources push toward duplicate sends, and they need different fixes.

The upstream event fires twice. A retry, a replay, an at-least-once queue redelivering. Fix: every event carries a unique event_id, and the fan-out stage records (event_id, user_id, channel) before enqueueing the send — a second delivery of the same event finds the row already present and no-ops. This has to be a single atomic insert-or-check, or two workers processing the same redelivered event race each other into sending twice; a unique constraint on (event_id, user_id, channel) in the datastore backing this check is what makes it atomic without extra coordination.

The same underlying fact produces two different events. Someone comments, then edits the comment within a second — is that one notification or two? This is a business-logic dedupe, not a delivery dedupe, and it can't be solved by an event ID because the two events are genuinely different events. The fix is a debounce window per (user, notification-type, subject): hold a low-priority notification for a few seconds before rendering, and if a superseding event for the same subject arrives inside that window, coalesce into one. High-priority events (security alerts, payment failures) skip the debounce entirely — the choice of what debounces is a product decision, and saying so out loud is the signal, not just implementing a fixed delay.

Sends themselves must be idempotent against the provider. A send call that times out might have actually succeeded on the provider's side. Pass an idempotency key to providers that support one (SES, most push gateways); for ones that don't, prefer at-least-once with the user-facing dedupe above absorbing the rare double-send rather than building idempotency the provider can't guarantee anyway.

An event ID can only ever catch the left case — the right one requires recognizing that two genuinely distinct events describe one fact the user only needs to hear about once.

Delivery status, retries, and per-channel differences

"Delivered" means something different per channel, and the design has to represent that honestly rather than one boolean:

Channel	What you actually learn	When
Push	Accepted by FCM/APNs; device may be offline, uninstalled, token stale	Immediately, from the gateway response
Email	Accepted by the provider, later a bounce/complaint webhook	Send is immediate; bounce can arrive minutes later
SMS	Provider accepted; carrier delivery receipt, if any, is unreliable	Similarly delayed and often absent
In-app	Actually delivered the moment it's written, since the client polls or is pushed the row directly	At write time

Model status as an append-only event log per (notification, channel) — queued → sent → bounced/failed → (retried) — rather than a single mutable status field, because a bounce arriving after a retry has already fired needs to be attributable to the right attempt, not overwrite it.

Retries are per-channel and bounded. A push token that's stale should not retry — it should trigger token cleanup. An SMS provider rate-limit (429) should back off and retry. A permanent email bounce should suppress future sends to that address until the user re-confirms it, or the sending domain's reputation degrades and every email starts landing in spam — this is the failure mode worth naming unprompted, because it's the one that silently takes down a channel for every user, not just the one who bounced.

Quiet hours and urgency are a per-user, per-timezone schedule check at the fan-out stage, with an explicit urgency override: a low-priority notification queues until quiet hours end (in the durable log, so nothing is lost); a security alert bypasses the check entirely. This has to be a stated field on the event (urgency: normal | critical), decided by the producer of the event, not inferred by the notification system — the notification system doesn't know if "someone logged into your account" is routine or alarming; the auth service does.

Overwriting status looks harmless until a late, asynchronous signal (a bounce, a complaint) needs to point at exactly one attempt among several — which an append-only log can do and a single field cannot.

Preferences, and the fan-out cost of getting them wrong

Preferences live in their own service, read (not owned) by the fan-out stage, because they change independently of any event and are queried far more than they're written — a classic case for the caching-and-read-replica treatment covered elsewhere in this track, not a novel problem.

The fan-out stage must fail closed on a preferences-service outage for anything above best-effort, and fail open (default to "send") only for genuinely low-stakes channels if the product explicitly wants that — but the default should be named and defended, not assumed. Silently sending everything during a preferences outage is a privacy and trust problem far worse than a delayed notification.

A user disabling a channel must take effect for in-flight fan-out, not just new events. If fan-out reads preferences once at the top of a large batch, a user who unsubscribes mid-batch still gets the tail of it — worth flagging as an edge case even if the fix (check preferences again immediately before send, not just at fan-out) costs an extra read per delivery, because the alternative is a user who unsubscribed and got notified anyway, which is the kind of bug that generates support tickets and press.

Reading preferences once per batch is the efficient design and the wrong one — the cost of correctness here is one read, paid every single time, not a special case for the rare mid-batch unsubscribe.

How to say it in the interview

"This looks like a fan-out problem and it is one, but the actual hard part is that five different downstream channels have five different failure behaviours and five different senses of what 'delivered' means — the design is mostly about making those differences not leak into the rest of the system."

Checklist

Separated the business decision to notify from the notification system itself
Named that channels have different cost and rate limits, and used that to justify per-channel queues
Put fan-out, preference-checking and dedupe in exactly one stage
Distinguished duplicate-event dedupe from duplicate-fact dedupe, with a different fix for each
Made the dedupe check a single atomic insert-or-check on (event_id, user_id, channel)
Modelled delivery status as an append-only per-attempt log, not one mutable field
Gave each channel its own retry policy tied to what the failure actually means
Named sender-domain reputation collapse as the failure that silently takes down a whole channel
Made urgency a field the event producer sets, not something the notification system infers
Chose fail-closed on a preferences-service outage for anything above best-effort, and defended the choice
Flagged that a mid-batch unsubscribe can leak into the tail of an in-flight fan-out

Practice this interactively