Cruxly › AI engineering › NLP and Text Classification

Evaluation Beyond Accuracy

per-class metrics, calibration, cost-optimal thresholds, and drift that actually matters

Prompt: a content-moderation classifier reports 96.2% accuracy and 0.91 F1 on the evaluation set. It shipped last quarter. Moderators say it is "missing obvious stuff," and one high-profile false removal made it to social media. Both complaints are true simultaneously. Explain how, then design the evaluation and the threshold you should have had — including what you would alert on in production.

40 minutes first. The two numbers in the prompt are not wrong, they are irrelevant, and being able to say precisely why is the entire exercise.

Why 96.2% accuracy and 0.91 F1 are compatible with a broken system

Accuracy on an imbalanced problem is a statement about the class distribution, not about the model. If violating content is 2% of moderation traffic, the constant classifier that approves everything scores 98% accuracy — better than the model in the prompt — while catching zero violations. Any accuracy figure quoted without the majority-class base rate next to it is unreadable, and the first question to ask about a 96.2% is "what does predicting the majority class get?"

A single F1 number hides which class is failing, and F1 on a balanced set hides it twice. Work a concrete case. Production is 2% violating; the evaluation set was built 50/50 because that was easy to assemble from a moderation queue.

Recall did not move — it is a within-class quantity and the balance of the set does not affect it. Precision collapsed, from 0.93 to 0.21, because at the same false-positive rate there are now 49 times more negatives to generate false positives from.

FP rate from the eval set:  0.89 × (1/0.93 − 1)          ≈ 0.067
At 2% prevalence:           TP = 0.02 × 0.89             = 0.0178
                            FP = 0.98 × 0.067            ≈ 0.0657
Production precision:       0.0178 / (0.0178 + 0.0657)   ≈ 0.21
Production F1:              2 × 0.21 × 0.89 / (0.21 + 0.89) ≈ 0.34

So the model that scores 0.91 on the eval set is, in production, wrong about roughly four out of every five things it flags. That is the high-profile false removal in the prompt, and it was fully visible in arithmetic before launch.

The "missing obvious stuff" complaint is a different failure and needs the confusion matrix to see. 0.89 recall is not the same as 0.89 recall on every subcategory. Decomposed, it is routinely 0.95 on the high-volume, lexically-obvious violation type and 0.55 on a low-volume one — and the low-volume one is what moderators notice, because they see the misses, not the base rates. An aggregate recall of 0.89 is an average over subpopulations that reports the behaviour of none of them.

So state the rule plainly: "F1 on a balanced set ships theater." Every evaluation number must be computed on a test set stratified to the production class frequency, and reported per class, or it is not measuring the system anyone will actually operate.

The metric set that is actually worth reporting

Report all four of these, and refuse to summarize them into one number.

Per-class precision and recall, on the production distribution. Not accuracy, not one F1. A twelve-row table where each row is a class and the columns are precision, recall, support, and the count of errors. It is less quotable than "0.91" and it is the only version that tells an operator what will happen.

Macro F1 alongside weighted F1, always both. Macro F1 averages per-class F1 with every class weighted equally, so a rare class failing completely drags it down hard. Weighted F1 weights by support, so the same failure is nearly invisible. Reporting only weighted F1 is how a class with 200 items a month silently stops working; reporting only macro F1 overstates the impact of a class nobody sends. The gap between them is itself the diagnostic: weighted 0.89 against macro 0.61 says, unambiguously, "the rare classes are broken and the common ones are fine."

A calibration curve per class. Bucket predictions by confidence and plot predicted confidence against observed accuracy within each bucket. A calibrated model puts items it called 0.80 at roughly 80% correct. This is a separate property from accuracy — a model can rank perfectly and still be badly calibrated — and it is the property every downstream threshold decision silently assumes.

A confusion matrix, read off-diagonal. The diagonal is what you got right and is the boring part. The off-diagonal cells are the model's actual error structure, and they are almost never uniform: one or two cells hold most of the mass, and they are usually the same class pairs that had a low kappa in the labeling lesson. Same underlying cause — a taxonomy boundary the guidelines never drew — showing up twice.

Then size the test set, because this is where a lot of evaluation quietly fails. 100-200 examples per class, stratified to production frequency. Below 100 per class the per-class confidence interval is too wide to be useful: at n=50 and an observed recall of 0.85, the 95% interval runs roughly 0.72-0.93, so a genuine 10-point drop is statistically invisible and drift detection on that class cannot work. Plus a second, separately-maintained minority-oversampled set for rare classes — hundreds of examples of a class that appears 200 times a month, collected over a longer window — because stratifying to production frequency gives a 0.5%-prevalence class almost no test items at all. Two test sets with two jobs: the stratified one estimates production behaviour, the oversampled one has the statistical power to measure the rare classes at all. Reporting rare-class metrics off the stratified set is reporting noise.

Calibration before thresholds, and why LLM scores are not calibrated

The order matters and it is the single most common process error here: calibrate per class, then set the threshold. A threshold is a statement about probability — "act when the probability of violation exceeds 0.85" — and it means nothing if the number labeled 0.85 does not correspond to 85%. Tuning a threshold on uncalibrated scores produces a number that works on today's score distribution and silently breaks when the next model version shifts it, even if that version is more accurate.

How to check, concretely. Ten confidence buckets, observed accuracy in each, plot against the diagonal. Summarize with expected calibration error — the support-weighted mean gap between bucket confidence and bucket accuracy. Under 0.05 is good; over 0.15 means the confidence scores should not be used for any decision. Do this per class, because a model is routinely well-calibrated on its high-support classes and wildly overconfident on the rare ones, and a global ECE averages that away.

Two standard fixes, and when each applies. Temperature scaling divides the pre-softmax logits by a single learned scalar fit on a held-out set — one parameter, preserves the ranking exactly so accuracy is unchanged, and it is the right default because neural classifiers are systematically overconfident in a way one scalar largely corrects. Isotonic regression fits a monotone mapping per class, which is strictly more flexible and needs meaningfully more held-out data; below a few thousand held-out items it overfits and you should stay with temperature scaling. Fit either on a calibration split that is not the test set — calibrating on test and then reporting test calibration is circular.

Now the part that matters most for anyone doing this with an LLM: LLM classifiers are usually miscalibrated out of the box, whether you read token log-probs or ask for a judge-style 1-5 score. Part of the reason is post-training itself: pretrained base models tend to be reasonably calibrated, and RLHF and instruction tuning measurably worsen calibration, typically toward overconfidence — so the chat model you actually call is the less calibrated one. Log-probs reflect next-token likelihood under the model's generation distribution, which is not a calibrated posterior over your label set, and it is sensitive to prompt wording, label ordering and the surface form of the label string — rename a class from "spam" to "unsolicited bulk" and the "probability" moves without the underlying judgement changing. Judge-style numeric scores are worse: they clump heavily on round numbers and on the upper half of whatever range you offered, so a "1-5 confidence" in practice is mostly 4s and 5s. The consequence to say out loud: a threshold on raw LLM confidence is not a probability threshold, it is a threshold on an uncalibrated score. If you must use one, fit an explicit calibration map from LLM score to empirical accuracy on a labeled held-out set, treat it as a monotone ranking signal at best, and re-fit it after every prompt or model change — because both invalidate it and neither is guaranteed to announce itself.

The threshold is where you encode business cost

Choosing a threshold by maximizing F1 asserts that a false positive and a false negative cost the same amount. That assertion is almost never true, and it is never examined, which is why F1-optimal thresholds ship operating points nobody would have approved if the trade had been stated in rupees.

Work the example. Moderation at 1,000,000 items/month, 2% violating (20,000 violating, 980,000 clean). Costs, from the business rather than from the model: a false negative — violating content left up — costs ₹400 in expected harm, trust damage and eventual escalation. A false positive — legitimate content removed — costs ₹120: an appeal, a moderator's review time, and user annoyance. So the ratio is FN:FP ≈ 3.3 : 1, meaning you should accept roughly three false positives to prevent one false negative.

Sweep the threshold on the model's raw, uncalibrated score — the usual situation before anyone has fit a calibration map:

F1 peaks at 0.80 — precision 0.78 and recall 0.74 is the most balanced row and the one an F1 sweep selects. Expected cost is minimized at 0.65. The gap is about ₹0.8 lakh a month, which is the price of using the wrong objective, and the direction of the gap is the more instructive part: the cost-optimal threshold is lower than the F1-optimal one, because false negatives cost 3.3× more, so you should deliberately tolerate worse precision than F1 considers balanced. An F1-optimized moderation classifier is systematically too permissive, every time, and nobody notices because the metric looks good.

Calibrate first, and the sweep collapses into one line of arithmetic. If the score is a calibrated probability p that an item is violating, then removing it costs ₹120 × (1 − p) in expected false-positive cost, and leaving it up costs ₹400 × p in expected false-negative cost. Remove whenever the second exceeds the first:

remove if  p > C_FP / (C_FP + C_FN) = 120 / (120 + 400) ≈ 0.23

• That's the main payoff of calibrating — the cost-optimal threshold becomes a formula you can recompute the moment the business changes a cost, instead of a re-sweep.
• Notice how low it is. With false negatives 3.3× as expensive, you should act on anything more than about 23% likely to be violating. The 0.65 found by sweeping raw scores is on a different, uncalibrated scale, which is why the two numbers aren't comparable.
• The formula assumes calibration actually holds. Check it on a held-out calibration split first (the previous section), or the closed form gives you a confident wrong answer.

Then say the two things that make this operational rather than a nice table. The cost ratio is a product and legal decision, not an ML decision — your job is to force someone to name it, because if they will not, the threshold encodes whatever ratio F1 implies and that number was chosen by a formula rather than by the business. And per-class thresholds, not one global threshold: a 12-category model has twelve different cost ratios, since mislabeling a billing ticket costs a re-route while missing a security incident costs something else entirely. One threshold across all classes is the same averaging mistake as one F1.

A third operating mode worth naming unprompted: three-way routing instead of a binary decision. Above a high threshold, act automatically. Below a low threshold, take no action. In between, route to a human queue. Now the optimization is over two thresholds and a review-capacity constraint — you can often push automated precision above 0.95 and recall above 0.9 in combination, provided the middle band fits the number of reviewers you actually have. Any threshold design that ignores review capacity is a design for a system with infinite moderators.

Drift detection that does not burn on-call

Instrument every production trace with two things: the input embedding and, wherever a label eventually arrives, an eval score. The embedding gives you a distribution you can compare against a reference window; the eval score tells you whether any change in that distribution actually damaged anything. Logging one without the other is why most drift monitoring gets muted.

A practical detector that beats staring at distance metrics: train a classifier to distinguish reference samples from production samples. Take embeddings from your reference window and from the last week of production, label them by origin, train a small model, and read its AUC. At 0.5 the two sets are indistinguishable — no drift. AUC above roughly 0.55-0.60 is a typical drift signal, meaning production text is now separable from your reference, and the classifier's feature importances or its highest-confidence production examples tell you what changed, which a single KS-statistic or PSI number never does.

Then the rule that keeps the alert trustworthy: alert on the joint condition of input drift AND a measurable eval drop, never on drift alone. Input distributions move constantly for benign reasons — a marketing campaign, a new customer segment, a product launch that changes the vocabulary, seasonality. If the model's per-class metrics are holding, that drift is a fact about your traffic, not an incident. Drift without eval impact is a false alarm and burns on-call, and a drift alert that has paged three times with nothing to fix is a drift alert that gets silenced right before the fourth time, which is the real one.

Which eval metrics, given labels are delayed. Ground truth for moderation arrives hours or days later via appeals and human review, so run two tiers. Immediately available proxies: per-class prediction rate (the share of traffic predicted as each class), mean confidence per class, the share of predictions falling in the low-confidence middle band, and the abstention rate — a class whose prediction rate halves overnight is a strong signal before any label exists. Then lagged true metrics from whatever labeled stream you have, computed on the same cadence as the labels arrive.

And the refresh loop, which is the actual remediation. Monthly, promote borderline production traces — those near the decision threshold, and those the drift classifier flagged as most production-like — re-label them, and add them to training and to the test set. Focus specifically on the confusion matrix's off-diagonal cells: the pair with the most error mass is where the next 200 labels pay for themselves several times over compared with 200 random ones. That closes the loop back to the labeling lesson, and it is also the mechanism that keeps the test set from aging into irrelevance while the traffic moves.

One trap worth naming: do not refresh the test set and then compare the new score with the old score. Two numbers from two different test sets are not comparable, and a "regression" after a test-set refresh is usually just a harder test set. Keep a frozen test set for longitudinal comparison and a rolling one for current behaviour, report both, and say which is which.

How to say it in the interview

"Those two numbers are compatible with the system being badly broken, because both are aggregates over a class distribution that is nothing like production. If violating content is 2% of traffic, a model that flags nothing at all scores 98% accuracy — so 96.2% is not evidence of anything. And a single F1 number has already silently decided that a false positive and a false negative cost the same, which in moderation is flatly untrue. I want per-class precision and recall on the production distribution, a calibration curve, a confusion matrix, and a threshold set from the cost ratio rather than from F1."

Checklist

• Asked what predicting the majority class scores before reading the accuracy figure
• Explained that precision collapses on the production distribution while recall does not
• Reported per-class precision and recall and refused to collapse them into one number
• Reported macro F1 next to weighted F1 and read the gap as a diagnostic
• Sized the test set at 100-200 per class and justified the floor statistically
• Kept a second minority-oversampled set for the rare classes
• Calibrated per class before setting any threshold, and said why that order matters
• Stated that LLM log-probs and judge-style scores are miscalibrated out of the box
• Chose the threshold by minimizing expected cost, with a named FN:FP ratio — and used the closed form once calibrated
• Forced the business to name the cost ratio instead of supplying one
• Alerted on the joint condition of input drift AND an eval drop
• Used a reference-vs-production classifier's AUC as the drift detector
• Refreshed monthly by re-labeling borderline traces from the worst off-diagonal cells

Related lessons

• Text Classification: When a Fine-Tuned Encoder Beats an LLM
• Building the Labeled Dataset
• Multilingual and Domain-Specific Text