Cruxly

Cruxly › AI engineering › AI System Design Case Studies

Case Study: Adding LLMs to a Recommendation System Without Melting the Budget

200 million item views a day, a $60,000 bill you do not pay, and the three places an LLM actually earns its tokens

Open the interactive lesson

Prompt: "You run recommendations for a marketplace with 10 million daily users, about 20 items shown to each, and a catalogue of 2 million items. Leadership wants LLMs in the recommender. Where do they go? Can you call one for every impression? How do you evaluate it, handle new items, and keep the latency budget of 150 ms?"

40 minutes first. Count impressions, price the naive design, then place the LLM where the arithmetic allows. The trap is to answer "let the LLM rank everything" without noticing that the bill and the latency both rule it out, or to dismiss LLMs entirely when offline enrichment is cheap and useful.

Where LLMs fit, and the arithmetic that rules out the obvious design

Count item views before choosing anything, because the naive design is eliminated by one multiplication. A recommender has 10,000,000 daily users and shows 20 items to each: 10M x 20 = 200,000,000 item views a day, about 2,300 per second on average and several times that at peak.

Price "ask an LLM about every impression". Give each call 3,000 tokens (user history summary, candidate details, instructions). 200M x 3,000 = 600 billion tokens a day. At an assumed small-model price of $0.10 per million input tokens, that is 600,000 x $0.10 = $60,000 a day, $1.8 million a month, before output tokens. And it cannot meet latency: a ranking stage usually has on the order of 100-150 ms for the whole request, which an LLM call with hundreds of milliseconds of time to first token cannot fit (see TTFT, TPOT, and the Latency Budget). The idea fails on both counts, and you can say so in 30 seconds.

So where do they fit? Three places, in order of how easily they pay for themselves.

  1. Offline item understanding. Read the title, description, images and reviews of each item once, and emit clean attributes, a category, a short description, "who is this for" tags and a normalised text for embedding. It runs on the catalogue (2 million items), not on traffic, and is batchable.
  2. User-intent summaries. Condense a user's recent behaviour into a short natural-language profile ("planning a trek; compares waterproof jackets"), refreshed asynchronously when something changes and cached. It feeds the ranker as a feature or the reranker as context.
  3. Reranking and explanations on high-value surfaces. Take the 20-50 candidates a classic ranker already produced and reorder or explain them, only where a better top slot is worth the cost: a homepage hero, a "you may also like" email, a conversational discovery box.

Where they do not fit: candidate generation over millions of items. Picking 500 candidates from 2 million is a nearest-neighbour search, handled by two-tower embeddings and an ANN index (ANN Indexes in Depth covers the memory and latency), which takes milliseconds and costs nothing per query beyond the index. An LLM cannot scan 2 million items in a prompt.

Be honest about the evidence. A June 2026 preprint, "Diagnosing and Mitigating Retrieval Bottlenecks in LLM-Based Cold-Start Recommendation" (arXiv 2606.29947), reports that calibrated LLM rerankers did not consistently outperform strong collaborative and content baselines even when the right item was guaranteed to be in the candidate set. The authors of "LLM Reasoning for Cold-Start Item Recommendation" (WWW 2026, arXiv 2511.18261) report that their fine-tuned reasoning models outperform Netflix's production ranking model by up to 8 percent in certain cold-start cases. Both are research results on their own data, one from a preprint by a small group and one from the authors' own evaluation, and they point in different directions. I found no primary production write-up of LLM reranking at large scale, so this lesson is a design under stated assumptions, not a description of how any company does it.

The classic stack in one page, and the three insertion points

The LLM augments a stack that already works; it does not replace it. A recommender in one page:

  1. Retrieval (candidate generation). A two-tower model embeds users and items into one space; an ANN index returns the 500 items nearest to the user. Often several retrievers run in parallel (collaborative, content-based, popularity, recent-session) and their candidates are merged.
  2. Ranking. A learned model (gradient-boosted trees or a neural network) scores the 500 candidates with rich features (user history, item stats, context) and predicts clicks, purchases or watch time. This runs in tens of milliseconds.
  3. Re-ranking and business rules. Diversity (not 10 nearly identical items), freshness, inventory, legal filters, sponsored placements, and caps per seller.
  4. Logging. Impressions, clicks and conversions, joined to the exact features and scores shown, which becomes next week's training data.

Where each LLM use plugs in.

The budget for the reranker is the ranking budget. If the whole request has 150 ms and retrieval plus ranking take 90, the LLM cannot be called synchronously: a call of 3,000 prompt tokens takes hundreds of milliseconds to the first token. Three patterns work:

Feedback and training. Log which items the LLM moved, so you can compare moved items to unmoved ones, and train the ranker on LLM-derived features. Keep the ranker as the production default so a model outage or price spike changes quality at the margin, not availability.

Cost arithmetic: enrich once, summarise rarely, rerank selectively

Each LLM use is priced by how many times it runs, so the design is a choice of cheap run counts. Assumed small-model prices: $0.10 per million input tokens and $0.50 per million output tokens, with a 50 percent discount for batch processing.

:::key Verified October 2026 From Anthropic's pricing and batch documentation (retrieved October 2026): Claude Haiku 5.5 lists at $0.10 input and $0.50 output per million tokens for prompts up to 100,000 tokens, and the Message Batches API is documented as 50 percent cheaper than standard prices for asynchronous work that can wait up to 24 hours (most batches finish within an hour). These numbers are used as a worked example; prices change, so re-check before quoting. :::

1. Catalogue enrichment (once). 2,000,000 items x 500 input tokens (title, description, review snippets, attribute list) = 1.0 billion input tokens; x 100 output tokens (JSON attributes) = 200 million output tokens. With batch at half price: input 1,000 M x $0.10 x 0.5 = $50, output 200 M x $0.50 x 0.5 = $50: about $100 for the whole catalogue. New and edited items are a trickle: at an assumed 20,000 new items a month, about 667 a day, it is cents per day. Even doing it with a mid-size model at 20 times the price is $2,000, once. Do this first; it is the best return in the lesson.

2. User-intent summaries (daily, for changed users). Assume 2,000 input tokens (recent events) and 150 output tokens per summary. For all 10M users every day on batch: input 20 B tokens x $0.05 per M = $1,000, output 1.5 B x $0.25 per M = $375: $1,375 a day. Refresh only users whose behaviour changed materially (assume 10 percent): about $138 a day. A summary is read by many requests, so its cost per use is far below a cent.

3. Reranking the hero slot (per session, on one surface). Assume 5 percent of 10M daily users, 500,000 sessions, see the hero slot, each reranked once with 30 candidates: 3,000 tokens in and 60 tokens out (an ordered list of IDs). Input 500,000 x 3,000 = 1.5 billion tokens x $0.10 per million = $150 a day; output 500,000 x 60 = 30 M tokens x $0.50 per million = $15. About $165 a day, $4,950 a month. That is 0.3 percent of the naive design. With a 40 percent cache hit rate on coarse contexts, it falls to 0.6 x 165 = $99 a day. Request rate: 500,000 / 86,400 = 5.8 per second on average, about 29 at a 5x peak, which a small model fleet handles.

Compare with the value. If the reranked slot lifts hero-slot conversion by a relative 1 percent on a surface that drives, say, ₹2 crore of daily revenue (an assumed figure), the gain is ₹2 lakh a day against about ₹14,500 of model cost (at an assumed ₹88 to the dollar). Whether that is real is an experiment, not an assumption: see the next section. If the lift is zero, the right cost is zero and you turn it off.

Prompt caching helps the reranker too: instructions and the ranking rubric go first as a cached prefix (Prompt Caching: Paying for the Same Tokens Once), with the user summary and candidates last. At 3,000 tokens per call with a 1,000-token static prefix, the cached part is read at a fraction of list price; check the minimum cacheable length for your model.

Cold start and the long tail: enrichment helps, but retrieval is the bottleneck

New and rarely seen items have no interaction history, so a collaborative model cannot recommend them, and the content an LLM can read is the only signal. This is the strongest argument for LLMs in recommendation, with a caveat the evidence makes clear.

How enrichment helps. At listing time, an LLM turns the raw description into a clean, consistent record: category path, attributes (material, fit, occasion), a one-line summary, and a text embedding of that summary. A new item then enters the content-based retriever immediately, as a neighbour of similar items, and the ranker has attributes to work with while it waits for clicks. For long-tail items with sparse or messy text, the normalisation alone is useful: "blk cotton kurta M" and "Men's Black Cotton Kurta, Medium" become the same attributes. Choosing, Evaluating and Fine-Tuning an Embedding Model covers choosing and checking the embedding used for the text.

Evaluate the enrichment itself, not only the downstream metrics. Sample 300 items stratified by category and label the extracted attributes against ground truth. Report per-attribute precision; if a "material" field is 94 percent correct, 1 in 17 items carries a wrong material, and filters on it will hide good items. Check hallucinated attributes (a "waterproof" tag not in the source) with a rule that every attribute must be supported by a quoted span from the input, and drop attributes that fail.

The bottleneck is usually retrieval coverage, not reranking. The June 2026 preprint above (arXiv 2606.29947) found that with standard single retrievers the true target item appeared in a 200-item candidate pool only 4.6 to 22.9 percent of the time across its five domains, because between 32 and 91 percent of cold-start targets were brand-new items with no interactions. If the right item is not in the pool, no reranker, however clever, can promote it. Its proposed fix was a learned fusion of several retrievers. That result is from a single preprint on public-style benchmarks, so treat the percentages as illustrative of a mechanism: measure candidate recall for new items first, then decide whether reranking is where the problem lives. For our marketplace: with about 667 new items a day among 2 million, new items are about 0.03 percent of the catalogue per day but may be the most commercially important things to show, so reserve exploration slots for it, a business-rule decision, not a model one.

Exploration and feedback loops. A model trained on clicks recommends what was already clicked, so popular items get more impressions and more clicks (popularity bias) and new items starve. Counter with explicit exploration (a small share of impressions for new items, with a cap on cost per exposure) and log the propensity of each shown item so training can correct for it. LLM-derived scores must not be fed back as if they were user feedback: a reranker that favours long, flattering descriptions will push sellers to write them, and the loop will amplify it.

Evaluation: offline metrics, online experiments, and the guardrails

Offline metrics tell you whether to run an experiment; the experiment tells you whether to ship. The two disagree often enough that you plan for both from the start (Offline Versus Online Evaluation explains the general split).

Offline: nDCG and recall on logged data. nDCG@k rewards putting relevant items high. A hand example with graded relevance (3 = bought, 2 = clicked, 1 = viewed, 0 = ignored) for the top 5 shown: ranking A has relevance [3, 2, 3, 0, 1]; the ideal order is [3, 3, 2, 1, 0]. With gain 2^r - 1 and discount 1 / log2(position + 1):

Offline evaluation has a built-in bias: logged data only contains items the old system chose to show, so a new model that surfaces different items is scored on feedback that mostly does not exist. Use it to catch regressions and rank candidate models, not to prove a lift. Also measure candidate recall@500 separately from ranking quality.

Online: an A/B test on outcomes. Randomise by user (not impression), because one user's impressions are not independent. Primary metric: conversion or revenue per session on the surface; secondary: click-through, retention at 7 and 28 days, and diversity. The sample size for a small effect is large: for a baseline 4 percent click-through and a target relative lift of 2 percent (0.08 points), the rough rule n per arm = 16 x p(1-p) / delta^2 gives 16 x 0.04 x 0.96 / 0.0008^2 = 960,000 impressions per arm (a 5 percent relative lift needs about 154,000). With 500,000 sessions a day on the hero slot this is a few days, but clustered users and weekly seasonality mean you run at least a full week, and watch novelty effects fade.

Guardrail metrics that can stop a launch: p95 latency of the surface, the fallback rate (how often the LLM path timed out and the classic order was served), cost per session, diversity (the number of distinct sellers or categories in the top 10), and complaint or return rates. A lift in clicks that comes with falling diversity and rising returns is not a win.

LLM-specific tests. Run the reranker twice with the candidate list in a different order and measure how often the top item changes: a high flip rate means the model is reacting to prompt order, not relevance. Position effects in long contexts are documented in "Lost in the Middle" (Liu et al., arXiv 2307.03172), which found accuracy highest when relevant information is at the start or end of the input and lower in the middle; that paper studied question answering and key-value retrieval, so apply it here as a reason to test, not as a measured effect on rerankers. Mitigate by shuffling candidates and averaging a few runs when the budget allows, or by shortening the list.

Risks, and what to say when asked to halve the cost

The failure modes are specific to putting a language model in a ranking loop.

Make the system degrade safely. The LLM path has a deadline, a fallback and a kill switch; the ranker never depends on it for availability; and cost per session has an alert, so a bug that triggers a rerank per impression shows up in an hour, not at month end (Cost Attribution and Runaway Prevention).

If asked to cut the cost in half. The expensive piece is the reranker at $165 a day, so: (1) raise the cache hit rate with coarser context keys, since 40 percent hits already saves 40 percent; (2) rerank the top 20, not 30: 2,000 tokens instead of 3,000 cuts input by a third, if nDCG holds; (3) run the rerank only for sessions whose predicted uplift is high (new users, ambiguous intent) and skip it for users with a confident ranker score; (4) cache the instruction prefix. For the batch lines, use the batch endpoint (already assumed) and refresh summaries less often. What you would not cut: the A/B test, the fallback and the grounding check. And if the experiment shows no lift, the cheapest design is the one without the reranker, keeping the $100 enrichment.

How to say it in the interview

"I would not put an LLM in the request path for every impression: 10 million users times 20 items is 200 million item views a day, and even at 3,000 tokens and ten cents per million tokens that is 60 thousand dollars a day and far beyond a 150 millisecond budget. The classic stack, two-tower retrieval with an ANN index, a ranking model and business rules, stays. The LLM goes in three places where the arithmetic works. First, offline enrichment of the catalogue, batched once: 2 million items is about 100 dollars. Second, per-user intent summaries refreshed asynchronously and cached. Third, reranking the top 30 on one high-value surface, 500 thousand sessions a day, about 165 dollars a day, run async with a cache so latency stays out of the hot path. I evaluate offline with nDCG and recall, then online with an A/B test on conversion and retention with latency, cost and diversity as guardrails. The evidence that LLM rerankers beat strong baselines is mixed, so I would prove it on our data before scaling."

Checklist

Practice this interactively
Share this lessonWhatsAppLinkedInX

Related lessons