Cruxly › System design › Design Exercises

Feature Flag Platform

changing behavior for one user in production, right now, with no deploy

Prompt: design a feature flag / experimentation platform — engineers toggle features on or off, or roll them out to a percentage of users, without a code deploy, and the toggle takes effect within seconds across every running server.

40 minutes first. The hard part isn't storing a boolean. It's getting that boolean to thousands of already-running server processes fast enough, without every flag check becoming a network call.

Applying the four moves: Scope — toggle features or roll out to a percentage of users without a deploy, effective within seconds everywhere; full experiment analytics out. Estimate — flag evaluations per second across the fleet, which is every request times every flag it checks — far too many to make a network call for each. Sketch — a flag config store, servers that hold flag rules in memory and evaluate locally, and a push or streaming channel for updates. Deep dive — getting a change to thousands of running processes fast, deterministic percentage rollouts (hash the user ID so a user doesn't flip in and out), and what a server does when it can't reach the config store — serve its last known rules, never fail closed.

Why a flag check can't be a network call

Scope. In scope: flag definition and storage, propagation to running application servers, percentage-based rollout and consistent per-user bucketing, basic targeting rules. Out of scope: statistical experiment analysis (computing whether a rollout's effect is significant — a downstream data-science concern, not part of the flag-serving system itself).

A flag can be checked many times per request, and a request can involve many services — if is_enabled("new_checkout_flow", user) triggers a network call to a central flag service on every check, flag checks themselves become a meaningful source of latency and a new way for the whole system to fail when that central service has a bad day. The design constraint that shapes everything else: flag evaluation must be a local, in-memory operation on every check, not a remote call — which means flag definitions have to be pushed out to every application server ahead of time, and evaluation happens against a local cached copy.

Propagation: push over a stream, not poll over HTTP

Push-based propagation over a stream every server subscribes to is what keeps flag checks local while still updating within seconds of a change.

A change in the control plane is published to a lightweight pub/sub stream that every application server subscribes to on startup (this could be a dedicated pub/sub system, or reuse of the same message-queue infrastructure covered in this track's queueing material). Every server receives the update and refreshes its local in-memory flag cache — this propagates a change to every running process within seconds, without a single flag check ever leaving the process it's evaluated in.

A server that just started, or one that missed an update due to a transient disconnect, needs a full-state fetch on connect/reconnect, not just incremental updates — a server relying purely on "listen for changes from now on" would silently run with stale or entirely missing flag state if it started up between changes or reconnected after a gap. Fetch full current state on connect, then apply incremental updates from the stream afterward — the same "snapshot plus tail the log" pattern used elsewhere in this track's workflow-orchestration and collaborative- document exercises.

Consistent bucketing for percentage rollouts

A 10% rollout must put the same users in the "on" group on every single request, not re-roll the dice each time — a user seeing a feature flicker on and off between page loads is a broken experience and makes any experiment's results meaningless (its effect can't be attributed to a stable treatment group). The fix: derive the bucketing decision deterministically from a hash of the user's stable ID and the flag's name — hash(user_id + flag_name) % 100 < rollout_percentage — so the same user always lands in the same bucket for the same flag, on every server, without any server needing to remember or coordinate individual per-user assignments. Including the flag name in the hash input (not just the user ID) matters: it ensures a user's bucket assignment for one flag is independent of their assignment for a different flag, so a user isn't systematically always "in" or always "out" of every rollout, which would confound any experiment that runs concurrently with others.

Targeting rules (show this to users in Canada, on the beta program, on mobile) layer on top of percentage bucketing, evaluated in a defined order — a specific user override should win over a percentage rollout, which should win over a global default, and that precedence needs to be explicit and documented, not implicit in whatever order rules happen to be checked in code.

Failure modes and the honest edges

The flag service itself going down should never take down flag-checking application code — because evaluation reads a local cache, not a live service call, a flag service outage means flags simply stop updating, not that they stop working. Every application server keeps serving its last-known flag state. This graceful-degradation property is a direct consequence of the push-based, locally-cached design and worth stating as the payoff for that architectural choice.

A flag left on indefinitely after its rollout completes becomes unmanaged technical debt — a flag that was meant to be temporary (a kill switch during a risky launch) quietly becomes permanent infrastructure nobody remembers to remove, and the codebase accumulates branches for flags that will never again be turned off. A real system tracks flag age and usage, and surfaces stale, fully-rolled-out flags for cleanup — a process concern as much as a technical one, worth naming because it's a well-known operational failure of these systems in practice.

Flag evaluation order and interaction between multiple flags affecting the same code path — two flags that each independently seem safe can interact in an untested combination once both happen to be on for the same user simultaneously. This isn't fully solvable at the platform level, but the platform can at least make active flag combinations visible/auditable per user, rather than leaving engineers to reconstruct which flags were active after the fact during an incident.

How to say it in the interview

"Every single request path in the whole system might check a flag, so a flag lookup has to be as cheap as reading a local variable, which means the real design problem is getting flag state to already-running processes ahead of time, not looking it up when it's needed."

Checklist

Made local, in-memory evaluation the hard constraint, not an optimization added later
Used push-based propagation over a subscribed stream, not per-check or polling HTTP calls
Required a full-state fetch on server startup/reconnect, not just incremental stream updates
Derived percentage-rollout bucketing deterministically from a hash of user ID and flag name
Included the flag name in the bucketing hash input, not just the user ID
Defined an explicit precedence order for user overrides, targeting rules, and percentage rollouts
Named graceful degradation as a direct payoff of the local-cache design, not an incidental benefit
Flagged stale, fully-rolled-out flags as accumulating technical debt requiring active cleanup
Named untested interaction between multiple simultaneously-active flags as a real, only-partially-solvable risk

Practice this interactively