Cruxly › AI engineering › Evaluation

Metrics That Mean Something: Task Metrics, Ranking Metrics, pass@k and Significance

nDCG@3 by hand, MRR, an unbiased pass@k, a 14-to-5 split that three tests grade differently, and cost per success

Prompt: "Our answer pipeline went from 71 to 73 percent on a 150-item eval and the PM wants to ship. Separately, the retriever returned relevant, not, relevant, not, not for one query: compute nDCG@3 and, for three queries whose first relevant result sat at ranks 1, 3 and 2, the MRR. An agent solves a task in 9 of 10 tries when we allow ten attempts. What does that tell you? And why did someone tell me BLEU is useless for our chatbot?"

40 minutes first. Do both ranking calculations on paper, write the pass@k formula, and decide what you would need to see before believing the 2-point gain. The trap is quoting one number without saying how it was aggregated or how uncertain it is.

Match the metric to the task, and know what each one cannot see

A metric is a claim about what matters, so choose it from the shape of the output, not from what the library offers.

• Extraction and classification: exact match and token F1. For a field like a refund window, exact match after normalising case and whitespace is strict and honest. Token F1 gives partial credit. Take the prediction "refund within 7 days" against the reference "refund in 7 working days". Three tokens overlap (refund, 7, days), so precision is 3/4 = 0.75, recall is 3/5 = 0.60 and F1 is 2 x 0.75 x 0.60 / 1.35 = 0.667, while exact match is 0. Whether 0.667 deserves credit depends on the use: for a displayed hint it does; for a field that drives a payment it does not. Evaluation Beyond Accuracy covers precision, recall and cost thresholds for classifiers, so do not stop at accuracy there.
• Generation against a reference: ROUGE and BLEU. Both count n-gram overlap with reference text. BLEU came from machine translation, where the valid outputs are few; ROUGE from summarisation. They reward surface similarity and cannot see that "you can cancel before dispatch" and "cancellation is possible until the order ships" are the same answer. For dialogue the evidence is old and consistent: Liu et al.'s How NOT To Evaluate Your Dialogue System (EMNLP 2016) found that such reference-overlap metrics correlated only very weakly with human judgement on Twitter data and not at all on the technical Ubuntu data. A chat answer has too many valid forms for one reference to cover. They stay useful when the reference is the real constraint (translation, tightly specified summaries) and as a cheap smoke test.
• Open-ended answers: a rubric. Break "good" into the yes/no criteria from the previous lesson, Evaluation Foundations: Golden Sets, Error Analysis and Reading Benchmarks, and score each, using a calibrated judge as in LLM-as-a-Judge: Evaluation That Scales Without Lying to You. A weighted average of binary criteria is more repeatable than a single 1-to-10 score.
• Agents: task success. Did the end state satisfy the goal (refund issued, ticket closed with the right tag), checked by code against the environment, not by reading the model's claim about what it did.
• RAG: the metric set is in Evaluation Gates That Actually Block a Bad Deploy.

Choose the metric whose consumer matches your consumer. If a human skims the output, partial credit may be right. If a downstream program parses it, anything short of exact is a failure. State which one you picked and why, in one sentence, next to the number.

Ranking metrics by hand: precision, recall, MRR and nDCG

Retrieval and recommendation return an ordered list, so the metric must care about where the relevant items sit, not just whether they are present. Use one ranking throughout: relevance of the top five results is [R, N, R, N, N] (relevant, not, relevant, not, not) and the collection holds exactly two relevant items.

• Precision@k is the fraction of the top k that are relevant. P@3 = 2/3, P@5 = 2/5. It ignores order inside the top k.
• Recall@k is the fraction of all relevant items that appear in the top k. R@3 = 2/2 = 1.0. It rewards finding everything and ignores order.
• MRR (mean reciprocal rank) looks only at the first relevant result: reciprocal rank is 1 / rank. For three queries whose first relevant item sat at ranks 1, 3 and 2: (1 + 1/3 + 1/2) / 3 = 1.833 / 3 = 0.611. Use it when one good hit is enough ("find the policy page").
• nDCG@k rewards putting the best items early, with a logarithmic discount. With binary gains, DCG@k = sum of rel_i / log2(i + 1). Here DCG@3 = 1/log2(2) + 0/log2(3) + 1/log2(4) = 1 + 0 + 0.5 = 1.5. The ideal ordering, relevant items first, is [R, R, N], which gives IDCG@3 = 1 + 1/log2(3) = 1 + 0.631 = 1.631. So nDCG@3 = 1.5 / 1.631 = 0.92. A ranking that puts both relevant items first scores 1.0; one that puts the first relevant item at rank 3 scores much less.

Graded relevance and the gain variant. If judges give 0 to 3 stars, ranking gains [2, 3, 0] against an ideal [3, 2, 0] with linear gain gives DCG = 2 + 3/1.585 = 3.893, IDCG = 3 + 2/1.585 = 4.262, nDCG = 0.913. Many libraries use the exponential gain 2^rel - 1 instead, which weights top grades more heavily and gives a different number for the same list. When you quote an nDCG, say which gain and which discount you used, or two teams will fail to reproduce each other. The BEIR benchmark (Thakur et al., 2021, 18 datasets) reports nDCG@10, which is why you see it quoted for embedding models; Choosing, Evaluating and Fine-Tuning an Embedding Model uses the same quantities.

They disagree on purpose. If an LLM reads the top 3 chunks, recall@3 is the quantity that matters, because order inside the window changes little. If a person reads down a result list, nDCG or MRR matters. Pick the metric that matches the consumer, and report two of them so the disagreement is visible.

pass@k and pass^k: one success versus every success

pass@k asks "did at least one of k attempts succeed", pass^k asks "did all k succeed", and they answer opposite business questions. Anthropic's Demystifying evals for AI agents (January 2026) states it plainly: pass@k suits a tool where one success is enough (generate several candidate patches, run the tests, keep a passing one), and pass^k suits a customer-facing agent that must work every time. Its example: a 75 percent per-trial success rate gives 0.75^3 = 42% for three trials in a row. The tau-bench paper (Yao et al., June 2024) introduced pass^k for tool-using agents and reported that even state-of-the-art function-calling agents succeeded on fewer than half of tasks, with pass^8 below 25 percent in the retail domain. The definition is reused from Guardrails for an Agent That Takes Real Actions.

The unbiased pass@k estimator. Do not estimate pass@k by running exactly k attempts once; it is noisy. The Codex paper (Chen et al., 2021, Evaluating Large Language Models Trained on Code) generates n >= k samples per task, counts the c that pass, and uses pass@k = 1 - C(n - c, k) / C(n, k), the probability that a random subset of k samples contains at least one pass. The matching estimator for reliability is pass^k = C(c, k) / C(n, k), the chance that a random k-subset is all passes. Worked example: n = 10 attempts, c = 3 pass.

• pass@1 = 1 - C(7,1)/C(10,1) = 1 - 0.7 = 0.30
• pass@3 = 1 - C(7,3)/C(10,3) = 1 - 35/120 = 0.708
• pass@5 = 1 - C(7,5)/C(10,5) = 1 - 21/252 = 0.917
• pass^3 = C(3,3)/C(10,3) = 1/120 = 0.008

The naive plug-in values (use p = 0.3) are 1 - 0.7^3 = 0.657 for pass@3 and 0.3^3 = 0.027 for pass^3. They are biased: the first is about 5 points too low and the second is more than three times too high. The combinatorial forms use the sample without replacement and are unbiased.

What does pass@10 = 90 percent tell you about reliability? Almost nothing. If attempts were independent with the same per-try rate p, 1 - (1 - p)^10 = 0.9 gives p = 1 - 0.1^(1/10) = 0.206. A system that succeeds one try in five has a pass^3 of 0.206^3 = 0.9%. Real tasks are heterogeneous, with some easy and some hopeless, so the independent-attempt model is only a rough guide; but the direction is firm. Retrying until success needs an oracle (a test, a verifier) that knows when it has succeeded. Without one, pass@10 is a ceiling on capability, not a promise to a customer.

:::key Verified October 2026 Definitions above are from the Codex paper (arXiv 2107.03374, July 2021), the tau-bench abstract (arXiv 2406.12045, June 2024) and Anthropic's engineering post of 9 January 2026. The tau-bench figures are for the agents and domains it tested in 2024; do not quote them as current model performance. :::

Aggregation traps: macro versus micro, averaged ratios, and Simpson's paradox

Most wrong conclusions from a correct metric come from how the scores were rolled up. Four traps, each with arithmetic.

Micro versus macro. A multilingual eval has 900 English items scoring 90 percent and 100 Hindi items scoring 60 percent. The micro average (pool all items) is (810 + 60) / 1000 = 87%. The macro average (average per group) is (90 + 60) / 2 = 75%. Neither is wrong; they answer different questions. Micro says "how will an average request go", macro says "how well do we serve each group". Report both, plus the per-group table, as Multi-Tenant, Multilingual RAG at Scale argues. The blended 87 percent is the number that hides a 30-point gap.

Averaging ratios. Conversation A has 1 claim and 1 is correct (100 percent). Conversation B has 10 claims and 2 are correct (20 percent). The mean of the two ratios is 60 percent; pooled, it is 3 / 11 = 27%. The first treats a one-claim reply and a ten-claim reply as equally important, the second weights by volume. Decide which you mean before writing the formula.

Simpson's paradox. A new version can be better on every segment and worse overall if the mix changed. Suppose the old eval had 200 easy items and 50 hard; the refreshed eval has 40 easy and 160 hard (a deliberate toughening after a launch). Scores:

The new version beats the old in both segments (90 against 80, 18.75 against 10) and the overall score halves, since the set is mostly hard now. The lesson: when the evaluation set changes, the history is broken; re-score old versions on the new set, or compare per segment.

Reporting a rate with its denominator. "Hallucination fell to 2 percent" needs "of 412 claims" or "of 96 answered questions". Percentages without counts hide small samples and shifting denominators, such as abstentions that quietly leave the base. The same point appears in Hallucination: Taxonomy, Detection and Mitigation in Production reports both per-claim and per-answer rates for exactly this reason.

Is it real? Intervals, paired tests and the 14-to-5 split

A score is an estimate, so every comparison needs an interval or a test, and the honest answer for small moves is often "not yet". Start with the size of the noise.

One rate. At 80 percent on n items the 95 percent interval is 1.96 x sqrt(0.8 x 0.2 / n): plus or minus 7.8 points at n = 100, 5.5 at n = 200, 3.9 at n = 400 and 2.5 at n = 1,000. Two versions each at 80 percent on 100 items each are indistinguishable unless they differ by more than about 11 points (sqrt(2) x 7.8), which is why unpaired comparisons waste data.

Paired comparison. Score both versions on the same items and count the discordant ones. The standard error of the paired difference is about sqrt(d / n) where d is the discordant fraction. The smallest difference you can reliably detect (95 percent confidence, 80 percent power) is roughly 2.8 x sqrt(d / n). With 15 percent discordance: 10.8 points at n = 100, 4.8 at n = 500, 3.4 at n = 1,000. So the PM's 71 to 73 on 150 items (a net of 3 items out of 150) is not detectable; ask for the discordance count first.

The worked case: 200 items, the candidate wins 14, loses 5, ties 181. The net gain is 9 / 200 = 4.5 points. Three reasonable tests give three answers, which is the point:

• Exact sign test (binomial on 14 of 19 discordant): two-sided p = 0.064. Not significant at 0.05. (A one-sided test would give 0.032, but picking one side after seeing which way the data leaned is exactly the flexibility to avoid; use two-sided unless a regression-only question was fixed in advance.)
• McNemar chi-square, uncorrected: (14 - 5)^2 / (14 + 5) = 4.26, p = 0.039. Significant. With the continuity correction, (9 - 1)^2 / 19 = 3.37, p = 0.066. Not significant.
• Paired bootstrap (10,000 resamples): 95 percent interval of roughly 0.5 to 9.0 points. The simple normal approximation, SE = 2.16 points, gives about 0.3 to 8.7. Both exclude zero, narrowly.

The conservative reading: the exact test is the one that does not rely on large-sample approximations with 19 discordant items, and it says 0.064. The case is borderline, and the right action is neither "ship, p < 0.05" nor "no effect". It is to look at which 19 items flipped, and to extend the set. If the split had been 15 to 4, the exact p is 0.019. Choose the test before you look, report the interval, and treat a result within a hair of 0.05 as a prompt for more data. And when you run many comparisons, expect some to look significant by chance; compare against a pre-agreed minimum difference that matters.

Bootstrap in one paragraph. Resample the items with replacement, recompute the metric, repeat 1,000 to 10,000 times, and read the 2.5th and 97.5th percentiles. It works on any metric (nDCG, an average rubric, a ratio of sums), and you should resample whole conversations or whole users, not individual turns, when items are not independent.

Cost and latency are metrics too: quality at a fixed cost

A quality score without a price and a tail latency next to it is half a result, because the best model on quality is rarely the best model to ship. Report one table with quality (and its interval), cost per request, cost per success, and p95 latency.

Illustrative table (assumed numbers, not measurements of any real model):

Cost per success is cost per request / success rate (0.012 / 0.82 = 0.0146). It charges failures to the successes that must pay for them, and it is the number to compare when failed requests are retried or escalated. Moving from A to B buys 4 points of success for $0.019 more per request, which is $0.019 / 0.04 = $0.475 per extra success. Whether that is a good deal depends on what one success is worth: cheap for a high-value claim, ridiculous for a chit-chat reply. Model C may beat both when a router sends only the hard 20 percent of traffic upward, as Model Selection: Cost, Latency, and Capability Trade-offs and Observability and Cost Governance for LLM Systems work out.

Use percentiles for latency, never the mean. The user waiting on the worst 5 percent of requests is the user who files the complaint; TTFT, TPOT, and the Latency Budget shows how to budget it. In an agent loop also record steps and tool calls per task, because a model that succeeds in 12 steps costs more than one that succeeds in 5.

Set the bar before the run. Write the release rule down in advance: "success must not drop by more than 1 point (paired, lower bound of the interval), cost per success must not rise more than 10 percent, p95 must stay under 2.5 seconds." Then a result either clears it or not, and nobody argues afterwards about which metric to look at. The lesson Offline Versus Online Evaluation: Shadow Mode, A/B Tests and Feedback Loops shows what to do when offline numbers clear and the online ones do not.

How to say it in the interview

"I match the metric to the task: exact match and token F1 for extraction, rank-aware metrics for retrieval, a rubric or task-success check for open answers, and not BLEU for chat because one reference cannot cover the valid replies. For the ranking, DCG at 3 is 1 plus 0.5, which is 1.5, the ideal is 1.631, so nDCG is 0.92; MRR is 0.611. Ten attempts with a 90 percent success says little about reliability: if attempts were independent that is a per-try rate near 21 percent, so success on three in a row would be under 1 percent. And 71 to 73 on 150 items is inside the noise, since the paired standard error is a few points; I would report an interval, a paired test, cost and latency together before shipping."

Checklist

• Chose each metric from the output shape and wrote down who consumes it
• Report two ranking metrics (recall@k and nDCG or MRR) and state the gain and discount convention
• Used the unbiased pass@k and pass^k estimators with n at least k samples per task
• Never present pass@k as reliability; pair it with pass^k or a verifier story
• Report both micro and macro averages and the per-segment table, with denominators
• Re-scored old versions whenever the evaluation set or its mix changed
• Compared versions on the same items with a paired test and an interval, and picked the test before looking
• Computed the minimum detectable difference for my set size and discordance rate
• Resample whole conversations or users in the bootstrap when items are not independent
• Put quality, cost per success and p95 latency in one table with a release rule written before the run

Related lessons

• Evaluation Foundations: Golden Sets, Error Analysis and Reading Benchmarks
• LLM-as-a-Judge: Evaluation That Scales Without Lying to You
• Hallucination: Taxonomy, Detection and Mitigation in Production
• Offline Versus Online Evaluation: Shadow Mode, A/B Tests and Feedback Loops