Cruxly

Cruxly › AI engineering › Evaluation

Evaluation Foundations: Golden Sets, Error Analysis and Reading Benchmarks

week one on a feature with no evals, a golden set you can defend, 1,090 versus 625 items, and why 92 percent on MMLU says little

Open the interactive lesson

Prompt: "We shipped a support assistant for our UPI app three months ago. Every prompt change is judged by whoever tried it last, and the vendor just released a new model that 'scores 92 percent on MMLU'. You join on Monday with no evals at all. Walk me through your first week, tell me how big the eval set must be, and tell me whether we should switch models."

40 minutes first. Write the five-day plan, size the set for a 5-point regression with the formula, and decide what you would do with the vendor's number. The trap is answering with a tool name; the interviewer wants a method and three numbers.

Week one: read the data before you write a metric

Without an evaluation set, the product is steered by the last bug report, and every model upgrade is a bet you cannot score. The failure is not that people lack opinions; it is that two engineers who each tried five prompts on Friday will disagree on Monday, and nobody can say whether a change made 1,000 other conversations better or worse. The unit of progress for an LLM feature is a number you trust, produced by a procedure that someone else can rerun.

The consistent advice from practitioners starts with looking at data, not with tooling. Hamel Husain's essay Your AI Product Needs Evals (March 2024) says to read as many traces as you can and to keep reading until you stop learning something new, and to start with binary good/bad judgements because graded scales are harder to keep consistent. Anthropic's engineering post Demystifying evals for AI agents (January 2026) says that 20 to 50 simple tasks drawn from real failures is a great start. Anthropic's test-and-evaluate guidance, on the other hand, leans toward volume: many automatically graded items with a weaker signal beat a few hand-graded ones. These agree once you separate the stages: start tiny and read everything, then grow the set as the graders become trustworthy.

The five-day plan, with a deliverable for each day.

@@ 1 | Read 150 to 200 real conversations Sample from logs across segments (intents, languages, new versus old users). Write one line per conversation: what the user wanted and whether the answer was acceptable. No metrics yet.

@@ 2 | Write the pass criteria with the product owner Three to six yes/no statements such as "does not invent a refund policy", "quotes the correct fee", "hands off when the user mentions a fraud". If the owner and you cannot agree on a case, that case is a labelling-guide entry, not a metric.

@@ 3 | Label a golden set Two people label 150 to 400 stratified cases independently. Record their disagreement, resolve it in a call, and freeze a test split that nobody tunes against.

@@ 4 | Build the cheapest valid graders and a baseline Code assertions first, a judge only where code cannot decide. Run the current system and write down the number with its error bar.

@@ 5 | Compare candidates and publish the table Three prompts or five models on the same set, paired comparison, with cost and latency in the same table.

:::key Verified October 2026 OpenAI's evals guide, retrieved October 2026, says the hosted Evals platform is deprecated: existing evals are read-only from October 31, 2026 and the platform shuts down on November 30, 2026 (the page says to check OpenAI's deprecations page for the current timeline); the page points newcomers to its Datasets feature for building evals iteratively. The data model it taught (an item schema, graders called testing criteria, runs over a JSONL file) is a sensible template, but own your golden set in files you control; a vendor's eval product is a tool, not the asset. :::

Building a golden set you can defend

A golden set is a sample of real work, stratified so that rare but expensive cases are not drowned out, labelled by two people, and frozen before anyone tunes against it. Each clause removes a specific way the number can lie.

Sample real traffic, then stratify. A random sample of 300 conversations from a UPI support bot will be 70 percent "where is my refund", and your fraud-escalation cases, which carry the real risk, may be 3 of 300. Stratify by intent, language (English, Hindi, code-mixed), channel and user type, and then add the hard cases deliberately: last quarter's failures, adversarial prompts, the case a regulator would ask about. Keep the natural mix in a second set so you can still report a traffic-weighted number; the stratified set tells you what breaks, the natural set tells you how often users meet it.

Label with two people and measure how often they disagree. Two annotators label the same 100 items pass/fail. They agree on 85, which looks good. But if each says "pass" about three quarters of the time, chance alone produces a lot of agreement. Cohen's kappa corrects for it: here annotator A says pass on 78 and B on 77, so the chance agreement is 0.78 x 0.77 + 0.22 x 0.23 = 0.651, and kappa = (0.85 - 0.651) / (1 - 0.651) = 0.57. That is moderate agreement, not a ground truth. The right response is to read the 15 disagreements, find the unwritten rule (is a correct answer with the wrong tone a pass?), put it in the labelling guide, and relabel. If two careful people cannot agree, a judge model will not agree with either of them, and LLM-as-a-Judge: Evaluation That Scales Without Lying to You shows how to measure that.

Freeze a test split. Split into a dev part you look at, tune prompts on and read failures from, and a test part used rarely, such as before a release. An eval you tune against for three weeks stops measuring generalisation, in the same way as a training set stops being a test set. Version the dataset, the grader code and the labelling guide together, because changing a grader changes what every old number meant.

How big? Two different questions hide here. To estimate one rate, the margin of error for a proportion near 50 percent is 1.96 x sqrt(0.25 / n): that is plus or minus 10 points at n = 96, plus or minus 5 at n = 384 and plus or minus 3 at n = 1,067. For a pass rate of 80 percent the interval is narrower (n = 100 gives plus or minus 7.8 points, n = 400 gives plus or minus 3.9). To compare two versions you need more than to estimate one, and the sizing section below does that arithmetic. A starter set of 100 to 200 is fine to find failure modes; it is too small to settle a 2-point argument.

Error analysis: name the failures, count them, then choose metrics

Read the failures first, name them in your own words, count them, and only then decide what to measure. Generic metrics such as "helpfulness" or "coherence" are tempting because they come off the shelf, but they measure what a library author imagined, not how your product breaks. The method is borrowed from qualitative research: open coding, where you write a short free-text note per failing case and merge the notes into categories afterwards.

A worked pass. Take 100 conversations where the assistant failed (found by thumbs-down, escalations and an initial reviewer sweep). Read each, write one note, and group. A plausible count, illustrative numbers, not a measurement:

The counts sum to 100, and the top three modes cover 71 percent of failures. Those three, not "helpfulness", become your metrics: retrieval hit rate, instruction-and-handoff compliance, and policy freshness. Notice what the exercise tells you that no metric would: 34 percent of failures live upstream of the model, so a model swap would barely help. That is the answer to "should we upgrade the model?" in one number. Weigh counts by cost too: 9 tone failures matter less than 3 mis-quoted fees that become complaints, so add a severity column.

Stop when notes repeat. Hamel Husain's advice is to continue until you stop learning new failure types. With 100 failures you usually reach that, and a second reader recoding 20 of them checks that your categories are not private jargon. Then re-run the analysis every few weeks: once you fix the top mode, the second mode is the new frontier, and new modes appear after every launch.

:::warn Do not let the model that produced the failures name the failure categories for you and then call the result error analysis. An LLM can help cluster your own notes, but read at least 50 raw traces yourself first. The categories you did not see in the data are the ones you will never measure. :::

Pick the cheapest grader that is valid, and keep the hygiene

There are three kinds of grader, and the rule is to use the cheapest one whose verdicts you can defend.

Anthropic's agent-evals post makes two points that apply to all three. Read the transcripts, because "you won't know if your graders are working well unless you read the transcripts"; and a 0 percent pass rate across many trials usually signals a broken task or grader, not an incapable model. Their example is a model that scored 42 percent on a benchmark and 95 percent after grading and scaffold bugs were fixed, which is a reminder that the grader is part of the experiment, so audit it like code.

Hygiene that stops silent drift.

The sizing arithmetic: unpaired needs 1,090, paired needs 625 or 310

Detecting a 5-point difference between two versions takes about 1,090 items per version if you score them on different samples, and 625 or fewer if you score both on the same items. The worked example: current system 80 percent, candidate 75 percent (or the reverse), 95 percent confidence (z = 1.96), 80 percent power (z = 0.84).

Unpaired. The formula is n = (z_alpha + z_beta)^2 x (p1(1-p1) + p2(1-p2)) / delta^2. With p1 = 0.80, p2 = 0.75, delta = 0.05: (1.96 + 0.84)^2 = 7.84, the variance term is 0.16 + 0.1875 = 0.3475, so n = 7.84 x 0.3475 / 0.0025 = 1,090 items per arm.

Paired. Run both versions on the same items. Items both get right or both get wrong carry no information about the difference; only the discordant items, where exactly one version wins, do. If a fraction d of items are discordant, the McNemar-style sample size is n = (z_alpha x sqrt(d) + z_beta x sqrt(d - delta^2))^2 / delta^2. With d = 0.20: (1.96 x 0.4472 + 0.84 x 0.4444)^2 / 0.0025 = (0.8765 + 0.3733)^2 / 0.0025 = 625 items. With d = 0.10: (0.6198 + 0.2623)^2 / 0.0025 = 311. With d = 0.30 it is 938. Paired is cheaper because the item-to-item difficulty, which is most of the variance, cancels. And versions of one system differ by a prompt tweak, so they usually agree on most items, which puts you in the 10 to 20 percent range. The discordance rate is something you measure in a pilot run, not something you assume.

Why paired is the default. Besides needing fewer items, a paired analysis answers the actual question, which is "on the same cases, which version wins more often?", and it is what you report. The simplest version is a sign test on discordant items: if the candidate wins 14 and loses 5, the exact two-sided p-value is 0.064 (Metrics That Mean Something: Task Metrics, Ranking Metrics, pass@k and Significance uses this case and shows where common tests disagree at the margin). A paired bootstrap resamples the items with replacement, recomputes the difference each time, and reads the 2.5th and 97.5th percentiles as an interval. It also handles metrics that are not simple proportions, such as nDCG or an average rubric score.

:::warn Powering for 5 points means a 2-point change is invisible. At n = 100 with 15 percent of items discordant, the standard error of the paired difference is sqrt(0.15 / 100) = 3.9 points, so the 95 percent interval is about plus or minus 7.6. Do not ship, or abandon an idea, on a 2-point move on 100 items, and say out loud which minimum difference your set can detect. :::

Benchmark literacy: what a leaderboard rank does and does not tell you

A public benchmark number is evidence about that benchmark, and it carries three discounts before it says anything about your task: saturation, contamination and construct mismatch.

So what does "92 percent on MMLU" tell you? That the model can answer broad multiple-choice questions at a level where the benchmark has little remaining resolution, probably with some leakage, in English, with no information about your latency, cost, languages, tools or tone. It earns the model a place on a shortlist and nothing more.

The answer to "which model?" is a private leaderboard. Take 5 to 10 candidates that pass your constraints (price, residency, latency), run each on your frozen test split with the identical prompt and grader, and report quality with an interval, cost per 1,000 requests and p95 latency. Pairwise comparison against your incumbent is the sensible test. Re-run it when a vendor releases a new version, since that is a model upgrade you can now score instead of fear. Retire models that lose on your set whatever their ranking, and revisit the golden set itself each quarter, as your traffic drifts.

How to say it in the interview

"I would not touch the prompt in week one. I would pull a couple of hundred real conversations, read them, and write down in plain words what a good and a bad answer is. Then I would label about 150 to 400 stratified cases with two people, measure how often they disagree, freeze a test split, and count the failures by type before choosing any metric, because the top three failure modes become the metrics. For sizing, detecting 80 versus 75 percent unpaired needs about 1,090 items per version, but scoring both versions on the same items needs roughly 625 if a fifth of the items flip, so paired is the default. A public benchmark only builds my shortlist; the decision is a private leaderboard of five models on our own set, with error bars."

Checklist

Practice this interactively
Share this lessonWhatsAppLinkedInX

Related lessons