Cruxly › System design › Design Exercises

News Feed Ranking

scoring happens offline in batch, but the feed must feel instant and current

Prompt: design the ranking system behind a personalized news/social feed — given a candidate set of posts from people/pages a user follows, rank them by predicted relevance rather than reverse-chronological order, and serve the ranked feed with low latency.

40 minutes first. This exercise is different from the activity-feed fan-out exercise elsewhere in this track — that one solves how posts get to a user's feed; this one solves what order they appear in once they're there.

Applying the four moves: Scope — rank a candidate set by predicted relevance and serve it fast; the candidate generation (fan-out) is the activity-feed exercise, not this one. Estimate — candidates per request multiplied by feature-computation cost per candidate, against a latency budget — which is why you can't run a heavy model on every candidate. Sketch — multi-stage ranking: a cheap model narrows thousands of candidates to hundreds, then a heavier model ranks those. Deep dive — feature freshness versus latency (precomputed features go stale, online ones are slow), and the feedback loop where the ranking shapes the engagement data the next model is trained on.

Why ranking is a different problem from fan-out, and the two-stage shape that follows

Scope. In scope: candidate scoring/ranking given an already-assembled candidate set, the offline/online split, feature freshness. Out of scope: how candidates are assembled in the first place (that's the activity-feed exercise's fan-out problem — this exercise explicitly starts from "assume you already have 500 candidate posts for this user").

Naming the fan-out/ranking split explicitly is the strongest opening move — treating them as one undifferentiated "feed problem" misses that they have almost opposite constraints: fan-out cares about write amplification and delivery latency across a huge follower graph; ranking cares about model inference latency and feature freshness for a single user's request, against a candidate set that's already small (hundreds, not millions) by the time ranking sees it.

Real ranking systems are two-stage, not one model doing everything: a cheap candidate generation/lightweight scoring pass narrows thousands of possible posts down to a few hundred using simple, fast signals (recency, basic engagement counts, an approximate embedding similarity lookup), followed by an expensive, accurate ranking model that only has to score that much smaller shortlist. Running the expensive model against every possible candidate would be far too slow; running only the cheap pass would be too inaccurate. The two-stage funnel is the standard answer specifically because it's the only way to afford an accurate model at all.

Offline training, online serving — and why the split exists

The feature store is the seam between a pipeline that can afford to be slow and a request path that cannot.

Model training happens offline, on a schedule (hourly or daily, not per- request), over historical engagement data — this is a batch job that can take as long as it needs, because it never sits in a live request's critical path. Online serving reads the already-trained model's weights and precomputed features from a fast store, and only has to run inference — a comparatively cheap operation — within the request's latency budget. This offline/online split exists because training a model (an iterative, data-intensive process) and using an already-trained model to score a handful of candidates are fundamentally different computational costs, and conflating them would force every feed load to wait on work that belongs in a batch pipeline.

A feature store is the seam between the two halves — precomputed features (a user's historical engagement rate, a post's current like count, an embedding similarity score) are written by the offline pipeline and read, not recomputed, by the online path. This is the same precompute-ahead-of-read-time principle as this track's search- autocomplete exercise, applied to ranking features instead of a prefix trie.

Feature freshness and the staleness this split accepts

The offline/online split necessarily means some features are stale by the time they're used — a post's like count used for ranking was correct as of the last feature-computation run, not the literal current moment. This staleness has to be bounded and acknowledged, not treated as a bug: a like count that's 5 minutes stale barely changes a ranking decision; a like count that's 24 hours stale on a fast-moving viral post meaningfully could. Real systems handle the highest-value, fastest-moving signals (very recent engagement on very recent posts) via a separate, much more frequent (near-real-time) feature update path, while accepting slower, batch-computed staleness for less time-sensitive features (a user's long-term topic-affinity profile, which genuinely doesn't change meaningfully minute to minute) — naming which features need which freshness tier, rather than treating "freshness" as one uniform property of the whole system, is the sign of real understanding here.

A/B testing the ranking model itself needs to be designed in from the start, not bolted on — a new model version is rarely deployed to 100% of users immediately; a percentage of traffic is routed to the candidate model, and its effect on real engagement metrics is measured before full rollout. This requires the serving layer to support multiple concurrent model versions and consistent per-user assignment (the same user should see the same model version across their session, the same sticky- assignment idea as this track's rate-limiter and A/B-adjacent exercises touch on elsewhere), not a coin flip on every single request.

Failure modes and the honest edges

Feedback loops between ranking and user behavior can amplify existing biases — if the model ranks a certain content type higher, users engage with it more (because it's what they're shown), which the model then learns as "this content type is popular," reinforcing the original ranking decision regardless of whether it was actually the best one. This is a real, well-documented failure mode of any engagement-optimizing ranking system, worth naming explicitly rather than presenting engagement optimization as an unambiguous good.

Cold-start for new users or new posts — a brand-new user has no engagement history to compute personalization features from, and a brand-new post has no engagement signal yet either. Both need an explicit fallback (population-level default features for new users, recency- weighted scoring for new posts until enough real engagement accumulates) rather than the ranking model silently producing a low-confidence, effectively-arbitrary score for either case.

The two-stage funnel's cheap first pass can systematically exclude candidates the expensive model would have ranked highly — if the lightweight candidate-generation stage's simple signals correlate poorly with the accurate model's actual scoring criteria for some content type, that content type gets filtered out before the accurate model ever sees it, and no amount of improving the second-stage model fixes a problem introduced at the first stage. Periodically evaluating the first stage's recall against the second stage's actual preferences (not just optimizing each stage's own metric in isolation) is what catches this.

How to say it in the interview

"Fan-out solves delivery — getting candidate posts into a feed at all — and ranking is a completely separate problem layered on top of it: scoring a few hundred candidates by predicted relevance in the tens of milliseconds a page load can afford, using a model that was actually trained hours earlier on a much bigger, slower batch pipeline."

Checklist

Explicitly separated ranking from fan-out as two different problems with almost opposite constraints
Used a two-stage funnel — cheap candidate scoring, then expensive accurate ranking — rather than one model doing everything
Split model training (offline, scheduled) from inference (online, per-request) rather than doing both live
Used a feature store to write features once offline and read, not recompute, them online
Acknowledged feature staleness as a bounded, accepted property rather than treating it as a bug
Assigned different freshness tiers to different features based on how fast their value actually changes
Designed for sticky per-user model-version assignment to support A/B testing new rankers
Named engagement-feedback-loop bias as a real, well-documented failure mode of ranking systems
Gave new users and new posts explicit cold-start fallbacks rather than letting the model score them arbitrarily
Evaluated the cheap first-stage funnel's recall against the accurate model's actual preferences, not just its own metric in isolation

Practice this interactively