Cruxly

Cruxly › AI engineering › Evaluation

Offline Versus Online Evaluation: Shadow Mode, A/B Tests and Feedback Loops

z = 2.9 and still do not ship: the rollout ladder, biased thumbs, peeking at 14 percent false positives, and the weekly flywheel

Open the interactive lesson

Prompt: "Our offline eval says the new support-bot prompt is 5 points better. We ran an A/B test on 20,000 conversations per arm: resolution went from 61.0 to 62.4 percent, and cost per conversation rose 18 percent. Offline said big win and online says small win. Explain the gap, tell me whether to ship, how long the test should run, and how production failures become tests."

40 minutes first. Do the significance arithmetic, the sample-size arithmetic and the cost-per-resolution arithmetic. The trap is stopping at 'p < 0.05, ship'.

Offline gates, online measures value, and they disagree for reasons you can list

An offline eval answers "did we break anything we know about", and an online experiment answers "did users get more value", and the two disagree more often than people expect. Offline is fast, cheap and repeatable, so it runs on every change and blocks bad deploys (the gate logic is in Evaluation Gates That Actually Block a Bad Deploy). Online is slow, noisy and expensive, but it is the only place the real objective (resolved tickets, completed tasks, retained users) shows up.

Six reasons a 5-point offline win becomes a 1.4-point online win.

The disagreement can run the other way and be the more dangerous one. OpenAI's own post-mortem, Expanding on what we missed with sycophancy (2 May 2025), says a GPT-4o update shipped on 25 April 2025 made ChatGPT noticeably more sycophantic, and that the rollback began on 28 April. The update added an extra reward signal based on user thumbs-up and thumbs-down data, and OpenAI says its offline evaluations generally looked good and its A/B tests suggested the small number of users who tried the model liked it. That is this lesson's warning in one incident: both gates passed and the behaviour was still wrong. The general lesson is the one Goodhart's law names: a signal that is easy to optimise tends to stop measuring what you wanted. An online metric that rewards agreeableness will produce an agreeable model, whether or not it is right.

The answer is not to trust one side, it is to use each for its job: offline to gate (nothing worse than baseline on the frozen set, on the metrics you named), online to measure the value and to catch what the set cannot contain, and a written rule for what happens when they disagree. A common rule: offline is necessary, online is decisive.

The rollout ladder: replay, shadow, canary, A/B, full

Each rung of the ladder trades more realism for more risk, so climb one rung at a time and stop when a rung fails.

@@ 1 | Replay on logged traffic (offline, zero user risk) Run the candidate over yesterday's or last month's logged requests. You get realistic inputs and latency and cost per request, but you cannot see how users would have reacted to a different answer, and the conversation history contains the old bot's replies, so multi-turn replays drift off-distribution after the first changed turn.

@@ 2 | Shadow mode (live inputs, no user exposure) For a slice of live requests the candidate runs in parallel, its output is logged, and the user sees only the production answer. You measure agreement with production, judged quality on a sample, latency and error rate on real traffic, and cost at real volume. Shadow doubles model spend for that slice, so size it (5 percent of traffic costs 5 percent extra). It tests everything except human response, and it cannot be used when the candidate's action has side effects (a refund, an email) unless those are stubbed.

@@ 3 | Canary (real users, tiny and reversible) 1 to 5 percent of users get the candidate, with automatic rollback on guardrail breaches (error rate, refusal rate, latency p95, cost). It is a safety check, not an experiment: the sample is too small to prove a quality gain, but large enough to find a crash, a runaway cost or a format break.

@@ 4 | A/B test (the measurement) A randomised split with a primary metric, guardrail metrics and a precomputed sample size, covered in the next section.

@@ 5 | Full release, with a holdback Roll to 100 percent but keep 1 to 5 percent on the old version for a few weeks as a control, so slow regressions and seasonality show up as a difference rather than a mystery.

Pick the rung by risk. A prompt wording tweak may need replay plus a short canary. A new model for an agent that moves money deserves every rung. For a feature with side effects the order matters: shadow with actions stubbed first, then a canary with a human approval step as in Guardrails for an Agent That Takes Real Actions.

Two hazards. Caching: an A/B split of prompts means each arm has its own cache prefix, so cache hit rates and costs differ for reasons unrelated to quality; Prompt Caching: Paying for the Same Tokens Once explains why, so compare cost with that in mind. Provider changes: if a vendor silently updates a model alias during your test, both arms move; pin model versions and record dates.

Online signals: explicit, implicit, and what Goodhart does to them

Explicit feedback is sparse and biased, implicit feedback is plentiful and ambiguous, and neither means anything until you know who is not responding.

Explicit signals (thumbs, star ratings, "was this helpful"). Response rates are low and skewed toward the unhappy. A worked example with assumed rates: suppose 80 percent of users are actually satisfied, satisfied users leave a thumb 1 percent of the time and dissatisfied users 8 percent of the time. Per 100 users you collect 0.8 x 1% = 0.8 positive and 0.2 x 8% = 1.6 negative thumbs. The share of thumbs that are positive is 0.8 / 2.4 = 33%, even though 80 percent of users are happy. The thumbs are a useful complaint detector, a poor satisfaction meter, and a trend only while the response mechanism stays fixed (move the button and the trend breaks).

Implicit signals are behaviours you can log without asking:

Every one of these has a confound. A short session may be a quick success or a quick walk-away. Calibrate each against a labelled sample: take 200 sessions with a regenerate click, have a reviewer label whether the first answer was bad, and report the rate. Until you know that rate, "regenerate rate" is a number with an unknown meaning.

Goodhart on online signals. Once the model or team optimises a proxy, the proxy detaches from the goal: optimise thumbs-up rate and you get flattery (as in the April 2025 episode above and the sycophancy research in Safety and Alignment Basics for Engineers); optimise session length and you get padding; optimise deflection rate and you get a bot that never hands off to a human. The defences are structural: a primary metric tied to the outcome (resolution, not engagement), paired guardrail metrics that would catch the gaming (escalation misses, complaint rate, cost), and a periodic human audit of a random sample.

Counterfactual honesty. Logged online data comes from the version that served it, so it cannot tell you what a different version would have done on the same request. That is why shadow mode and A/B tests exist, and why you should not train a new model on thumbs from the old one without correcting for the selection that produced them.

A/B testing an LLM feature: the arithmetic, the duration, and the traps

An LLM A/B test is an ordinary controlled experiment with extra variance and extra ways to fool yourself. Kohavi, Tang and Xu's book Trustworthy Online Controlled Experiments (Cambridge University Press, 2020) is the standard reference for the general method; the numbers below follow it in spirit and are computed from scratch.

The worked example. 20,000 conversations per arm. Resolution 61.0 percent (control) against 62.4 percent (treatment), a +1.4 point difference.

So the lift is real, and the decision is economic: ship if a resolution is worth more than $0.64 to you (a human-handled contact that costs, say, $1.00 makes it a bargain; at $0.30 it does not), provided the other guardrails (escalation misses, complaints, p95 latency) are clean. Significance is a statement about noise, not about whether the change is worth its price.

How many conversations? For the per-arm sample at 95 percent confidence and 80 percent power: n = 2 x (1.96 + 0.84)^2 x p(1-p) / delta^2. With p = 0.617 and delta = 0.014: 2 x 7.84 x 0.2364 / 0.000196 = 18,900 per arm. So 20,000 was about right for a 1.4-point effect. For a 1-point effect it would be 37,000. At 2,000 conversations a day split in two, 18,900 per arm takes 19 days; round up to three whole weeks so every weekday appears equally (support traffic has strong weekly cycles) and the start-of-test novelty can be seen decaying.

The traps.

The flywheel and continuous evaluation: turning production into tests

Production is the best source of new test cases, and the loop that moves failures into the golden set is what makes an eval set get better instead of staler.

The weekly loop.

@@ 1 | Capture, with consent and redaction Log inputs, retrieved context, outputs, tool calls and the model and prompt versions, after masking personal data (phone numbers, account numbers, Aadhaar-style IDs). Apply the access, retention and erasure practices in Red-Teaming and Compliance as Code to the log store. Logging is itself processing of personal data, so purpose, retention and erasure apply; under India's DPDP Act and similar rules, that is a design requirement, not an afterthought.

@@ 2 | Select candidates Pull every thumbs-down, escalation, regenerate-after-answer and judge-flagged conversation, plus a small random sample (so you also see the failures nobody flagged).

@@ 3 | Label A reviewer confirms the failure and writes the expected behaviour. Assign a failure mode from the error-analysis taxonomy, or create a new one.

@@ 4 | Add to the golden set Put new cases in the dev split, and a fraction in test. Mark them with their source and date. Before fixing the bug, confirm the new test fails; after the fix, confirm it passes. That is a regression test for a language model.

@@ 5 | Fix and re-run Change the prompt, retrieval or model, run the full set, and look at the per-mode table so a fix to one mode does not break another.

Monitoring is continuous evaluation. Offline sets run on changes; production needs a heartbeat. Observability and Cost Governance for LLM Systems covers traces and spend. For quality, judge a random sample of live traffic, 1 to 5 percent, with a calibrated judge. Arithmetic: at 50,000 conversations a day, sampling 2 percent gives 1,000 judged conversations. If the baseline failure rate is 5 percent, the standard error of the daily rate is sqrt(0.05 x 0.95 / 1000) = 0.69 points, so a threshold at the baseline plus three standard errors is 5 + 2.07 = 7.1%. A true rise to 8 percent sits 4.4 standard errors above baseline and alerts within about a day. Weekly aggregates (7,000 judged) tighten the standard error to 0.26 points and catch a slow drift from 5 to 6 percent that the daily chart hides. Remember that the judge has its own error rate, so audit a few dozen of its verdicts a week against a human.

Watch inputs as well as outputs. Alert on shifts in the input distribution (a new language, a new product, much longer prompts), the share of conversations touching tools, refusal rate and cost per conversation. A model update by a vendor or a change in your retrieval index shows up in these before any user complains.

How to say it in the interview

"Offline evals gate and online evals measure value; they answer different questions, so a gap is normal. The golden set can be stale or stratified differently from traffic, the offline metric is a proxy for resolution, and real users change how they behave. On the A/B, the difference of 1.4 points has a standard error of 0.49 points, so z is about 2.9 and p about 0.004, and 20,000 per arm is right for a 1.4-point effect at 80 percent power. But I only ship if the guardrails pass: cost per resolved conversation rose about 15 percent, so I compare the extra cost per extra resolution against what a human-handled contact costs. I would run it a whole number of weeks without peeking, check the sample ratio, and add every confirmed failure to the golden set weekly."

Checklist

Practice this interactively
Share this lessonWhatsAppLinkedInX

Related lessons