Cruxly › AI engineering › Evaluation

LLM-as-a-Judge: Evaluation That Scales Without Lying to You

a model grading a model is a measuring instrument, and instruments have biases

Prompt: your team ships prompt and model changes weekly, and human review of 500 sample answers per release takes three days. Someone proposes using a strong LLM to grade the answers instead. Design that evaluation so its scores can actually be trusted to block a bad release.

30 minutes first. The naive version asks the judge "which answer is better?" and believes it. The research says that judge will often pick whichever answer it saw first, can be swayed by padding, and may favour its own outputs.

Why it works at all, and the three ways to ask

The reference study is Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (NeurIPS 2023). Its headline result: on first-turn MT-bench questions, with tied votes excluded, a strong judge (GPT-4) agreed with human experts 85% of the time — slightly higher than humans agreed with each other (81%). Random guessing would score 50% on that setup. Counting ties as well, the figures are 66% and 63% (random would be 33%). Chatbot Arena data showed a similar pattern.

It compared three ways of asking:

:::warn Read the numbers as "possible", not "automatic" Those agreement figures came from two 2023 datasets (MT-bench and Chatbot Arena), with careful prompts. They show a judge can track human judgement — not that your judge, on your task, does. Later work (JUDGE-BENCH, Bavaresco et al. 2024, 20 datasets) found judge–human agreement varies widely from task to task. It has to be measured (last section). :::

The biases, with the numbers

The same paper measured specific biases. The models it tested are now old, so the exact percentages will differ for today's judges — but these bias types still show up, and the way to find them is the same experiments.

Position bias — the judge prefers whichever answer it reads first (or second). They swapped the order of the two answers and checked whether the verdict stayed the same:

Even the best judge changed its verdict on about a third of pairs just because the order changed.

Verbosity bias — the judge can be fooled by length. Their "repetitive list" attack padded an answer by restating its own points in different words — longer, but no better. On a small test of 23 answers, how often each judge preferred the padded version:

GPT-4 mostly resisted this particular attack — that doesn't show it has no preference for length in general. Later work found raw win rates on automatic benchmarks reward longer answers enough that AlpacaEval added a length-controlled version (Dubois et al., 2024).

Self-enhancement — possible, but not settled by this paper. Compared with human ratings, GPT-4 gave its own answers about a 10% higher win rate and Claude-v1 about 25% higher. But the judges also favoured some other models, GPT-3.5 did not favour itself, and the authors say plainly that with their limited data they cannot determine whether the bias exists. Later work (Panickssery et al., 2024) did find self-preference, linked to models recognising their own text — so treat it as a real risk to check for, not a measured constant.

Weak grading on maths and reasoning. On 10 maths questions, each judged in both orders (20 judgments), GPT-4 called a wrong answer correct 14 times with a plain prompt. The authors also show it being misled by the answer it was grading on an elementary problem it could solve correctly when asked on its own — so "the judge is smart enough to do the task" does not mean it grades the task well.

Correcting for them in practice

Each bias has a known fix. Build them into the evaluation, not into a note in the README.

• Position bias → run both orders. Ask with A first, then with B first. If the two verdicts disagree, count it as a tie. This is the paper's own recommendation, and it turns position bias from a hidden error into a visible "can't tell." It doubles the number of judge calls — a cost worth naming.
• Verbosity bias → judge against a rubric, not a gut feeling. Tell the judge what a good answer must contain and that length is not a criterion. Where you can, score one specific property per call ("is every claim supported by the sources?") instead of "which is better overall."
• Possible self-preference → use a judge from a different model family than the model being evaluated, or at least check whether scores shift when you swap judges. It's cheap insurance against a bias that later research did find.
• Weak reasoning → give it the right answer and let it think. In the paper, grading maths with a plain prompt failed 14/20 times; asking the judge to reason step by step first cut that to 6/20; giving it a reference answer cut it to 3/20. Reference-guided grading is the strongest tool you have for anything with a correct answer.
• Few-shot examples help, but cost. Adding three worked judging examples raised GPT-4's order-consistency from 65.0% to 77.5% — but the longer prompt made each call about 4× more expensive, consistency is not the same as accuracy, and examples can bring in their own bias. The authors kept the zero-shot prompt as their default.

Check the judge before you trust it

A judge is a measuring instrument. Calibrate it before it gates anything.

@@ 1 | Build a small human-labelled set A few hundred examples from your own traffic, graded by people who know the domain. This is the ground truth the judge is checked against.

@@ 2 | Measure agreement before relying on it — with the right statistic Run the judge on that set and compare. Report a chance-corrected score such as Cohen's kappa (or Krippendorff's alpha with several raters), not just percent agreement: raw agreement looks high whenever most answers are good, or when ties are dropped. Later work (Thakur et al., 2024, Judging the Judges) showed high percent agreement can hide low kappa. For a release gate, also measure the one number that matters most: how often the judge passes an answer your humans marked bad.

@@ 3 | Pin the judge model and the prompt A judge that changes under you changes every score at once. Verdicts also shift with rubric wording, the score scale and the output format, so the prompt is part of the instrument too. Pin both, and treat changing either as a change that needs re-checking against the human set.

@@ 4 | Keep humans on a sample, and watch for gaming Have people grade a small random slice of every run. That catches the judge drifting — and it catches the bigger risk: optimising against a fixed judge teaches the system to please the judge, not the user. This is well documented: researchers showed a "null model" that returns one carefully crafted constant response to every question — ignoring the question entirely — could reach an 86.5% length-controlled win rate on AlpacaEval 2.0 by manipulating the judge's prompt template (Zheng et al., Cheating Automatic LLM Benchmarks, ICLR 2025). Once a judge's score becomes the target, it stops being a good measure.

:::key How this connects to the rest of the track This is the engine behind the CI gates in Evaluation Gates That Actually Block a Bad Deploy, and behind outcome grading in Evaluating an agent. A gate is only as trustworthy as the judge underneath it. :::

How to say it in the interview

"An LLM judge can agree with human raters about as often as humans agree with each other, but it has measurable biases: it is swayed by answer order, can be fooled by length, and may favour its own outputs. So I would treat it like any measuring instrument: correct for the known biases, and check it against a small human-labelled set, with a chance-corrected agreement score, before I let it gate a release."

Checklist

• Chose the judging format for the task: pairwise for comparing versions, single grading for tracking, reference-guided for anything with a right answer
• Named the documented biases — position, verbosity, weak grading of reasoning — and possible self-preference
• Ran pairwise comparisons in both orders and counted disagreement as a tie
• Used an explicit rubric that excludes length, and scored one property per call where possible
• Used a judge from a different model family than the model under test
• Validated the judge against a human-labelled set with a chance-corrected score (e.g. Cohen's kappa) and its miss rate on bad answers
• Pinned the judge model and prompt, and kept humans grading a sample of every run
• Named the risk of optimising against the judge

Related lessons

• Evaluation Foundations: Golden Sets, Error Analysis and Reading Benchmarks
• Metrics That Mean Something: Task Metrics, Ranking Metrics, pass@k and Significance
• Hallucination: Taxonomy, Detection and Mitigation in Production
• Offline Versus Online Evaluation: Shadow Mode, A/B Tests and Feedback Loops