Cruxly

Cruxly › AI engineering › Portfolio Projects

Project: An Eval Harness and LLM-as-Judge Toolkit You Can Trust

deterministic graders first, a judge that must pass a calibration bar, paired comparisons with intervals, and a regression gate whose false-alarm rate you have measured

Open the interactive lesson

Project: "Every team says it has evals. Mine says a prompt change is 'better' and then users complain. Build the harness I would trust: graders that cannot be gamed, an LLM judge that is allowed to score only after it has been checked against human labels, comparisons that say how sure they are, and a gate engineers will not switch off."

Time-box: 20 hours minimum, about 30 typical, about 45 with the stretch goals. You will have a tested package (judgekit, standard library plus llmkit, 36 passing tests): a run cache and registry, six deterministic graders including a code grader that an early exit cannot fool, kappa and sample-size arithmetic, pointwise and order-swapped pairwise judges with versioned rubrics, a calibration card that refuses a judge below a bar, bias probes, a paired comparison with a three-state gate and a report, and a meta-evaluation that measures the harness itself. It is called judgekit because llmkit already contains a module named evalkit.

Why this project, the honest limits of offline judges, scope and the success metric

Evaluation is the most repeated differentiator in hiring-side guidance, and the part most people skip or get wrong. This project shows that you can design a grader, check a judge against humans, put an interval on a claim, and build a gate. The method behind it is the one the first lesson of this topic cites: read real failures before writing rubrics, prefer binary pass or fail over arbitrary scores, use code assertions wherever you can, and keep the broken cases in the CI set.

A limit to state first. There is no real model in this lesson. The judges under test are scripted: a FakeLLM whose accuracy and biases are inputs. That is stronger than a real model for one purpose, testing the harness, because the truth is known and you can ask whether the harness detects the planted flaw. It proves nothing about how well a real model judges, so every judge-quality number below is a property of the stand-in; calibrating a real judge needs a key and you report your own kappa. The fixture (Indian state capitals, English and Hinglish, 60 items) has labels true by construction; real humans disagree, so double-label a slice and report human-to-human kappa too.

Scope. Library, not a framework: dataset loader, run cache, graders, statistics, judges, calibration, bias probes, comparison, gate, report. Non-goals: a hosted UI, a prompt optimiser, multi-judge routing (a stretch goal). I did not build a command-line entry point or a parallel judge runner; the scripts under evals/ and examples/ are the interface, and a dry-run function prints expected calls and cost before any spend. What a reviewer will probe: can I trust the judge, how many items do I need, why paired, what the gate's false-alarm rate is, and when you would not use a judge.

Success metric, set before building (assumed targets). A judge scores only if kappa is at least 0.6 overall and on every slice of at least 10 items. The gate's false-positive rate on no-change reruns stays under 3 percent. A planted position bias must be detected and the swap must remove the confidently wrong verdicts. Statistics must match hand-computed values.

:::key Where the 0.6 bar comes from, and its limits (read 24 October 2026) The common reading of kappa uses the Landis and Koch (1977) bands, where 0.61 to 0.80 is 'substantial'. The Wikipedia article on Cohen's kappa notes those bands are not universally accepted, that the original supplied no evidence for them, and that kappa depends on how the categories are distributed. So 0.6 here is a stated convention I chose and defend, not a law; raise it for high-stakes gating. On the judge side, Zheng et al., 'Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena' (arXiv 2306.05685, v1 June 2023, revised December 2023) report position, verbosity and self-enhancement biases, and over 80 percent agreement between strong judges such as GPT-4 and human preferences on their benchmarks. That is the reason the probes below exist; it is not a number to quote for your own judge. :::

Builds on. Evaluation Foundations: Golden Sets, Error Analysis and Reading Benchmarks, Metrics That Mean Something: Task Metrics, Ranking Metrics, pass@k and Significance, LLM-as-a-Judge: Evaluation That Scales Without Lying to You, Evaluation Gates That Actually Block a Bad Deploy and The Prompt Lifecycle: Versioning, Regression Tests and Automatic Optimisation. Intervals (bootstrap_ci, paired_compare, wilson) come from llmkit.evalkit; this project adds agreement, power and the gate and does not redefine them.

:::warn What this lesson prints, and what it does not judgekit is eight modules plus tests and evals/meta/ scripts, roughly 1,200 lines of library code. The lesson prints the core of each idea and describes the rest by its test. It does not print data.py beyond RunCache (the Example and Grade types, Registry, load_jsonl, dataset_hash, run_system), Rubric and PointwiseJudge, most graders, bias.py, simulated.py (apart from pairwise_judge), the compare function, evals/meta/world.py (which builds the 60 labelled items), human_labels.jsonl, power.py, run_meta.py and examples/apply_to_rag.py, so you write those yourself. Consequence: the calibration, bias, gate and power tables below come from my complete copy, and your own build will reproduce them only if you generate the same 60 items and the same hash-seeded scripted judges; treat them as the shape to aim for, not as a result your first build must match. Because the package __init__.py fixes the names, here it is:

# -*- coding: utf-8 -*-
from .data import Example, Grade, RunCache, Registry, load_jsonl, dataset_hash, run_system  # noqa: F401
from . import graders  # noqa: F401
from .stats import cohen_kappa, confusion, n_for_halfwidth, detection_rate  # noqa: F401
from .judges import Rubric, PointwiseJudge, PairwiseJudge, Verdict  # noqa: F401
from .calibrate import calibrate, require, JudgeNotCalibrated  # noqa: F401
from .bias import position_probe, length_probe, self_preference_probe, flagged  # noqa: F401
from .compare import compare, gate, markdown, dry_run, Rule  # noqa: F401

The RAG comparison near the end also needs the previous project's repository (that lesson prints only part of it), so its numbers need your own build and golden set. :::

Architecture, the run cache and the deterministic graders

The cache key contains everything that can change an output. RunCache keys a system's output on the system version, the prompt version, the example id and the example input. Rerun the same system on the same set and every item is a hit; change the prompt version and nothing is served stale. Two tests prove both directions, plus one that proves a restart still hits. A Registry table records, per run, the dataset hash, system, judge, rubric hash, seed and cost, so any number in a report can be reproduced. Rubrics are plain files with a name and a version in the header, and Rubric.hash covers the name, version and text, so a silent edit to the rubric is a different hash.

class RunCache:
    """Outputs keyed by (system version, prompt version, example id, example input). Change any of
    them and the key changes, so a stale answer can never be served for a new prompt."""

    def __init__(self, path: str = ""):
        self.path, self.mem, self.hits, self.misses = path, {}, 0, 0
        if path:
            try:
                with open(path, encoding="utf-8") as f:
                    self.mem = {k: v for k, v in (json.loads(x) for x in f)}
            except FileNotFoundError:
                pass

    @staticmethod
    def key(system_version: str, prompt_version: str, ex: Example) -> str:
        return hashlib.sha256("|".join([system_version, prompt_version, ex.id, ex.input]).encode("utf-8")).hexdigest()

    def get_or_run(self, k: str, fn: Callable[[], Any]) -> Any:
        if k in self.mem:
            self.hits += 1
            return self.mem[k]
        self.misses += 1
        v = self.mem[k] = fn()
        if self.path:
            with open(self.path, "a", encoding="utf-8") as f:
                f.write(json.dumps([k, v], ensure_ascii=False) + "\n")
        return v

Use a deterministic grader whenever one exists. Exact and normalised match, a regex that must fullmatch (a search would pass '42 apples' for a pattern that means 'a number'), a numeric grader that accepts Indian digit grouping (1,00,000 equals 100000) within a tolerance, a small JSON-schema subset (type, enum, required, properties, items, minimum, maximum, no extra keys), and a code grader. The tests are adversarial on purpose: a boolean is not an integer even though Python says True is an int; a float is not an integer; an extra key fails; an empty string fails the regex.

The code grader shows what 'cannot be gamed' costs. It writes the candidate and your asserts to a temporary file, runs it with python -I under a timeout, and requires a one-time token printed after the asserts. A candidate that calls os._exit(0) before the asserts exits with code 0, so a grader that checked only the exit code would pass it; the token is why it does not. A candidate with while True times out. This is not a security sandbox: a hostile candidate can still read files or open sockets. Run untrusted code in a container with no network and limited resources.

def python_exec(tests: str, timeout_s: float = 5.0) -> Grader:
    """Run candidate code, then `tests` (asserts), in a fresh interpreter with a timeout.
    Pass needs a one-time token printed AFTER the asserts, so os._exit(0) or sys.exit(0) cannot fake it.
    This is NOT a security sandbox: for untrusted code run it in a container with no network."""
    def grade(output, ex):
        token = secrets.token_hex(8)
        script = "%s\n\n%s\nprint(%r)\n" % (output, tests, "PASS-" + token)
        with tempfile.TemporaryDirectory() as d:
            p = os.path.join(d, "cand.py")
            with open(p, "w", encoding="utf-8") as f:
                f.write(script)
            try:
                r = subprocess.run([sys.executable, "-I", p], capture_output=True, text=True, timeout=timeout_s, cwd=d)
            except subprocess.TimeoutExpired:
                return Grade(False, 0.0, "timeout after %gs" % timeout_s)
        ok = r.returncode == 0 and ("PASS-" + token) in r.stdout
        return Grade(ok, float(ok), "" if ok else (r.stderr.strip().splitlines() or ["no pass token"])[-1][:120])
    return grade
def test_python_exec_pass_fail_timeout_and_the_exit_trick(self):
    tests = "assert add(2, 3) == 5"
    g = graders.python_exec(tests, timeout_s=2.0)
    self.assertTrue(g("def add(a, b):\n    return a + b", ex(None)).passed)
    self.assertFalse(g("def add(a, b):\n    return a - b", ex(None)).passed)
    slow = g("def add(a, b):\n    while True:\n        pass", ex(None))
    self.assertFalse(slow.passed)
    self.assertIn("timeout", slow.reason)
    cheat = g("import os\ndef add(a, b):\n    return 0\nos._exit(0)", ex(None))      # exit code 0 before the asserts
    self.assertFalse(cheat.passed)

The statistics: kappa, intervals, pairing and how many items you need

Agreement is not accuracy, and accuracy against a lazy rater is a trap. Cohen's kappa corrects observed agreement for the agreement expected by chance given each rater's own label rates: kappa = (po - pe) / (1 - pe). The test works a case by hand: 10 items, rater A says yes 6 times and rater B 5 times, they agree on 7, so po = 0.7, pe = 0.6 x 0.5 + 0.4 x 0.5 = 0.5 and kappa is 0.4. A second test shows why you cannot use raw agreement: with human labels 95 percent 'pass', a judge that says 'pass' to everything agrees 95 percent of the time and has kappa 0.

def cohen_kappa(a: Sequence, b: Sequence) -> float:
    """Chance-corrected agreement. 0 = no better than chance given each rater's own label rates.
    If both raters use one label for everything, expected agreement is 1 and kappa is undefined; we return 0."""
    n = len(a)
    assert n == len(b) and n > 0
    po = sum(x == y for x, y in zip(a, b)) / n
    ca, cb = Counter(a), Counter(b)
    pe = sum(ca[k] * cb[k] for k in set(ca) | set(cb)) / (n * n)
    return 0.0 if pe >= 1.0 else (po - pe) / (1.0 - pe)

Report intervals, and pair the comparison. wilson gives an interval for a proportion; for 70 correct of 100 it is 0.604 to 0.781 (the test checks the formula's value). paired_compare bootstraps the per-item differences of two systems scored on the same items. Pairing matters because it removes the item difficulty both systems share. The test builds a system B that fixes 10 of A's misses and breaks 2: the two unpaired Wilson intervals overlap, while the paired interval excludes zero.

How many items do you need? The answer is larger than people expect. For a single score near 0.7, a 95 percent interval of plus or minus 7 points needs about 165 items (1.96^2 x 0.21 / 0.07^2 = 164.6), and the test pins that number. To detect a 5-point difference between two systems you need more, and how many depends on how much they disagree. simulate_pair makes B differ from A on a stated share of items, and detection_rate counts how often a test notices. Output of python evals/meta/power.py, a simulation under those assumptions (A passes 70 percent; 1,000 simulated experiments per row):

Chance to notice a true +5 point gain (A passes 70%), 1000 simulated experiments each
scenario                                                           n   unpaired   paired
B better on 8.5% of items, worse on 3.5% (12% disagree)           50         1%       5%
B better on 8.5% of items, worse on 3.5% (12% disagree)          100         2%      23%
B better on 8.5% of items, worse on 3.5% (12% disagree)          200         6%      47%
B better on 8.5% of items, worse on 3.5% (12% disagree)          400        26%      81%
B better on 17.5%, worse on 12.5% (30% disagree)                  50         6%       6%
B better on 17.5%, worse on 12.5% (30% disagree)                 100         8%      11%
B better on 17.5%, worse on 12.5% (30% disagree)                 200        17%      22%
B better on 17.5%, worse on 12.5% (30% disagree)                 400        34%      41%
False alarms when B is the same as A (7% better, 7% worse), n=100, 1000 experiments:
  unpaired 0.0%   paired 2.6%

Read it carefully. Take the first scenario, where B is better on 8.5 percent of items and worse on 3.5 percent, a true gain of 5 points. At 100 items, an exact paired test (a sign test on the items where they differ) notices it 23 percent of the time; at 200, 47 percent; it takes about 400 to reach 81 percent. The unpaired z-test that treats the two scores as independent is far weaker, 2 percent at 100 items, because it ignores that the same items were scored twice. In the second scenario the systems disagree on 30 percent of items, so the same 5-point gain is harder to see. The practical rules: a 5-point change on 40 items is noise; a bigger set beats a cleverer test; and when your golden set is small, say 'not distinguishable at n = 100' rather than 'no effect'.

The judge: versioned rubrics, pointwise verdicts, and pairwise with order swap

A judge is a function from (question, answer) to a verdict you can parse, and it must be allowed to fail. PointwiseJudge sends the rubric as the system message and the question and answer as the user message, asks for JSON with a verdict of pass or fail and a reason, strips code fences, and returns None for a reply it cannot parse, counted separately from a 'fail'. With samples greater than one it takes a majority vote and treats a tie as no verdict. A test checks that the judge sees only the rubric, the question and the answer, with the exact message text.

Pairwise judging asks twice. PairwiseJudge.compare asks 'which is better' with the answers in one order and then the other. If both orders pick the same answer, that answer wins; if the judge picks slot A both times, it was following position, not content, and the result is a tie. Ties are a feature: a judge that abstains is better than one that is confidently wrong.

class PairwiseJudge:
    """Which answer is better? Ask in both orders. Agreement is a verdict; disagreement is a tie,
    because a judge that changes its mind when you swap the slots was not judging the answers."""

    def __init__(self, llm: LLMClient, rubric: Rubric, swap: bool = True):
        self.llm, self.rubric, self.swap = llm, rubric, swap

    def ask(self, question: str, first: str, second: str) -> Optional[str]:
        """Raw single call: 'first', 'second', 'tie' or None (unparsable)."""
        msgs = [{"role": "system", "content": self.rubric.text + '\nReply with JSON only: {"winner": "A" or "B" or "tie", "reason": "..."}'},
                {"role": "user", "content": "QUESTION:\n%s\nANSWER A:\n%s\nANSWER B:\n%s" % (question, first, second)}]
        w = str(parse_json(self.llm.complete(msgs, temperature=0).text).get("winner", "")).strip().lower()
        return {"a": "first", "b": "second", "tie": "tie"}.get(w)

    def compare(self, question: str, a: str, b: str) -> str:
        """'a', 'b' or 'tie' (also returned when a reply is unparsable)."""
        one = self.ask(question, a, b)
        if not self.swap:
            return {"first": "a", "second": "b"}.get(one, "tie")
        two = self.ask(question, b, a)
        if one == "first" and two == "second":
            return "a"
        if one == "second" and two == "first":
            return "b"
        return "tie"

Planting a flaw to see if the harness finds it. The simulated pairwise judge prefers the higher-quality answer, except that with a stated probability it simply says 'A', and it can add a bonus per hundred characters (length bias) or for a marked 'family' (self-preference). Its randomness is a hash of the inputs, so every run is identical.

def pairwise_judge(quality: Callable[[str], float], position_bias: float = 0.0, length_bonus: float = 0.0,
                   self_bias: float = 0.0, family_marker: str = "", salt: str = "p1") -> FakeLLM:
    """Prefers the higher `quality`, except: with probability `position_bias` it just says A; it adds
    `length_bonus` per 100 characters; it adds `self_bias` to answers carrying `family_marker`."""
    def reply(messages):
        user = messages[-1]["content"]
        q = _between(user, "QUESTION:\n", "\nANSWER A:\n")
        a, b = _between(user, "\nANSWER A:\n", "\nANSWER B:\n"), _between(user, "\nANSWER B:\n", None)
        if u01(salt, "pos", q, a, b) < position_bias:
            return json.dumps({"winner": "A", "reason": "scripted position bias"})

        def score(x):
            return quality(x) + length_bonus * len(x) / 100.0 + (self_bias if family_marker and family_marker in x else 0.0)
        sa, sb = score(a), score(b)
        w = "tie" if abs(sa - sb) < 1e-9 else ("A" if sa > sb else "B")
        return json.dumps({"winner": w, "reason": "scripted"})
    return FakeLLM(default=reply, model="sim-judge")

The test puts the better answer in slot A on even pairs and slot B on odd ones, using a judge with a 50 percent position bias. Without the swap it picks the worse answer on some pairs. With the swap it is never confidently wrong, and the cases it cannot resolve become ties. The numbers from python evals/meta/run_meta.py (30 pairs, a 40 percent position bias):

== bias probes on 30 pairs
judge                                slot-A%     flip%   better%  pad-win%  flags
fair                                     50%        0%      100%       50%  -
position bias 40%                        70%       40%       60%       50%  ['position bias: slot A chosen 70% of the time']
length bonus                             50%        0%      100%      100%  ['length bias: padded answer wins 100% of decided calls']

== position-biased judge (40%), better answer placed in slot A / B alternately
  swap=False right 26  wrong  4  tie  0
  swap=True  right 18  wrong  0  tie 12

What the table says. The fair judge picks slot A half the time and the better answer every time. The position-biased one picks A 70 percent of the time and changes its mind on 40 percent of pairs when the slots swap. Under the swap protocol it is wrong on none of the 30 pairs and abstains on 12, trading coverage for correctness. The length-biased judge passes the position probe but prefers a padded copy of an answer on every decided call, and the probe flags it.

:::warn A failure I hit while building this My first length probe counted every call where the padded copy was chosen, in either order. Run on the position-biased judge, it reported 61 percent for the padded answer and flagged 'length bias' that was really position bias leaking in. The fix is to count only verdicts that survive the swap, so position noise becomes ties. Probes interact: remove position bias first, then test for length. :::

Judge calibration and the slice trap

No judge scores anything until it has been checked against human labels, and the check must be able to say no. calibrate takes the judge's pass or fail for each labelled item, the human label and a slice name, and returns a card: agreement, kappa with a bootstrap interval, the confusion counts, and kappa per slice. It records failures in words: overall kappa below the bar, or any slice with at least 10 items below it. require(card) raises JudgeNotCalibrated with those words, so a pipeline cannot read a score from an uncalibrated judge by accident.

def calibrate(judge: str, rubric_hash: str, pred: Dict[str, bool], human: Dict[str, bool], slice_of: Dict[str, str],
              min_kappa: float = 0.6, min_slice_n: int = 10) -> Card:
    """pred and human map example id -> pass/fail. Every id must be labelled by both."""
    ids = sorted(human)
    assert set(ids) == set(pred), "judge and human labels must cover the same items"
    j, h = [pred[i] for i in ids], [human[i] for i in ids]
    k, lo, hi = kappa_interval(j, h)
    card = Card(judge, rubric_hash, len(ids), sum(x == y for x, y in zip(j, h)) / len(ids), cohen_kappa(j, h),
                lo, hi, confusion(j, h), min_kappa=min_kappa)
    if card.kappa < min_kappa:
        card.failures.append("overall kappa %.2f < %.2f" % (card.kappa, min_kappa))
    for s in sorted(set(slice_of[i] for i in ids)):
        sid = [i for i in ids if slice_of[i] == s]
        sk = cohen_kappa([pred[i] for i in sid], [human[i] for i in sid])
        card.slices[s] = {"n": len(sid), "kappa": sk}
        if len(sid) >= min_slice_n and sk < min_kappa:
            card.failures.append("slice %r kappa %.2f < %.2f (n=%d)" % (s, sk, min_kappa, len(sid)))
    return card

Running three scripted judges against the 60-item fixture (20 English and 10 Hinglish questions, each with a right and a wrong answer; labels true by construction) gives, from python evals/meta/run_meta.py:

== judge calibration against 60 human labels (rubric correctness v1, hash ffd25a6e5f49), bar kappa >= 0.6
sim 90% accurate       n=60 agree=0.90 kappa=0.80 (95% CI 0.64 to 0.93)  english k=0.90 (n=40)  hinglish k=0.60 (n=20)  -> USABLE
sim 65% accurate       n=60 agree=0.65 kappa=0.30 (95% CI 0.07 to 0.53)  english k=0.50 (n=40)  hinglish k=-0.10 (n=20)  -> REFUSED: overall kappa 0.30 < 0.60; slice 'english' kappa 0.50 < 0.60 (n=40); slice 'hinglish' kappa -0.10 < 0.60 (n=20)
sim 95% en / 55% hinglish n=60 agree=0.75 kappa=0.50 (95% CI 0.25 to 0.70)  english k=0.85 (n=40)  hinglish k=-0.20 (n=20)  -> REFUSED: overall kappa 0.50 < 0.60; slice 'hinglish' kappa -0.20 < 0.60 (n=20)

Read the first row twice. The 90-percent-accurate stand-in earns kappa 0.80, and yet its 95 percent interval runs from 0.64 to 0.93, because 60 items is a small sample, and its Hinglish slice sits at exactly 0.60 on 20 items, passing by a hair. If you quote 'kappa 0.80' without the interval and the slice, you overstate what 60 labels can show. The second judge is refused outright. The third is refused overall here, so to demonstrate the slice rule alone the test constructs labels where English is perfect and half the Hinglish answers are wrong: overall kappa is at least 0.6, the card still fails, and the failure message names the hinglish slice. That is the whole argument for per-slice cards: an average over a strong majority hides a failing minority, and the minority is often the language or domain you care about.

Calibrate on data the rubric has not seen. If you tune the rubric text on the same 60 items, the kappa measures your tuning. Keep a second labelled set, split by id, for the final card, and re-run calibration whenever the rubric hash, the judge model or the judge's settings change. Put the judge's name, the rubric hash, the date and n on the card; a card without them cannot be reproduced.

Comparison, the gate and a false-alarm rate you have measured

A gate engineers will not disable has three properties: it rarely cries wolf, it explains itself, and it does not block on a hunch. compare takes two score maps over exactly the same example ids (it asserts they match, because comparing different samples is the classic mistake) and returns the paired difference with an interval and the sign test, a per-slice table, and the ids where B did worse. gate then returns one of three states:

def gate(c: Comparison, rule: Rule = Rule()) -> GateResult:
    """Block only when a drop is big enough to matter AND the bootstrap interval excludes zero AND the exact
    sign test agrees AND the set is not tiny. Anything short of that is a warning: too few items to say."""
    fail, warn = [], []
    o = c.overall
    if o["mean_diff"] <= -rule.min_effect:
        sure = o["ci_high"] < 0 and o["p_sign"] < rule.alpha and c.n >= rule.min_n
        (fail if sure else warn).append(
            "overall %+.3f (95%% CI %+.3f to %+.3f, n=%d)" % (o["mean_diff"], o["ci_low"], o["ci_high"], c.n))
    for s, r in c.slices.items():
        if r["n"] >= rule.min_slice_n and r["mean_diff"] <= -rule.slice_min_effect:
            (fail if r["ci_high"] < 0 and r["p_sign"] < rule.alpha else warn).append(
                "slice %r %+.3f (CI %+.3f to %+.3f, n=%d)" % (s, r["mean_diff"], r["ci_low"], r["ci_high"], r["n"]))
    return GateResult("fail" if fail else "warn" if warn else "pass",
                      ["BLOCK: " + m for m in fail] + ["warn: " + m for m in warn])

Block only when the drop is at least 3 points, the bootstrap interval excludes zero, the exact sign test on the disagreeing items gives p below 0.05, and the set has at least 30 items. A drop that fails any of those conditions is a warn, with the numbers printed. Slices use a higher bar (10 points) and a minimum of 15 items, because small slices are noisy.

:::warn A failure I hit while building this My first gate blocked on the bootstrap interval alone. A test with A at 8 of 10 and B at 5 of 10 (three items worse, none better) came back as a block, and it should not have: with only three disagreeing items the exact sign test gives p = 0.25. A percentile bootstrap on a handful of items is anti-conservative, because most resamples contain at least one of the three bad items. The fix was the sign-test condition and the minimum size. The test now asserts a warning. :::

def test_a_slice_regression_is_found_even_when_the_overall_number_is_flat(self):
    sl = ["easy"] * 90 + ["hinglish"] * 30
    a = [0.0] * 12 + [1.0] * 78 + [1.0] * 30              # A misses 12 easy items
    b = [1.0] * 12 + [1.0] * 78 + [1.0] * 18 + [0.0] * 12  # B fixes them but breaks 12 of 30 Hinglish items
    c = compare(*scores(a, b, sl), n_boot=500)
    self.assertAlmostEqual(c.overall["mean_diff"], 0.0)   # flat overall: +12 and -12
    g = gate(c)
    self.assertEqual(g.status, "fail")
    self.assertIn("hinglish", g.messages[0])

Measuring the gate itself. run_meta.py simulates 200 reruns each of three situations on 150 items where A passes 70 percent, and reports how the gate treated them:

== regression gate on 200 simulated reruns, n=150, A passes 70%
  no real change (6% better, 6% worse)           blocked   0.5%   warned  11.0%   passed  88.5%
  real 5-point drop (3% better, 8% worse)        blocked  38.0%   warned  37.0%   passed  25.0%
  real 10-point drop (2% better, 12% worse)      blocked  89.5%   warned   7.5%   passed   3.0%
  (28 s on a re-run in review; 61 s in my first run)

What you may claim. On pure noise (the second system is the first system rerun, better on 6 percent of items and worse on 6 percent) the gate blocked 0.5 percent of reruns, one in 200, and warned on 11 percent; the target was under 3 percent. A real 10-point drop was blocked 89.5 percent of the time (a re-run in review reproduced every percentage). A real 5-point drop was blocked only 38 percent of the time, and the rest warned or passed. That is the honest power of a 150-item set against a 5-point regression, consistent with the power table: the gate protects against big regressions and is nearly blind to small ones. If 5 points matters to you, the fix is more items or a better metric, not a looser threshold, which would raise the false-alarm rate. These are simulated rates under stated assumptions; measure yours on your own set with your own seed.

Applying it to the RAG assistant, and what the harness caught

The previous project's golden set is the first real customer. examples/apply_to_rag.py takes the shipped retriever configuration (hybrid with rerank and split) as system A and a dense-only retriever as candidate B, scores each of the 42 answerable questions, groups by question type, and runs compare and gate. For recall at 5 the output is:

## hybrid+rerank+split (A, shipped) vs dense (B, candidate) (n=42, paired)

| | mean |
|---|---|
| hybrid+rerank+split (A, shipped) | 0.921 |
| dense (B, candidate) | 0.651 |

Difference (B - A): **-0.270**, 95% CI -0.401 to -0.143; B better on 0, A better on 13, tied on 29; sign test p = 0.0002

| slice | n | diff | 95% CI |
|---|---|---|---|
| exact | 8 | +0.000 | +0.000 to +0.000 |
| lookup | 20 | -0.300 | -0.500 to -0.100 |
| multi | 6 | -0.556 | -0.833 to -0.278 |
| paraphrase | 8 | -0.250 | -0.625 to +0.000 |

Gate: **FAIL**
- BLOCK: overall -0.270 (95% CI -0.401 to -0.143, n=42)
- BLOCK: slice 'lookup' -0.300 (CI -0.500 to -0.100, n=20)

The gate blocks, as it should: B is worse on 13 questions and better on none, the paired interval is entirely below zero, and the sign test agrees. The multi slice shows the biggest drop, minus 0.556 on 6 questions, but at n = 6 the slice is below the minimum for a slice verdict, so the report shows the number without gating on it.

The finding: my first answer-correct comparison reported 0.000 for the dense candidate. I had passed one abstention threshold, 0.15, to both systems. That threshold was chosen for the shipped system's reranker scores; the dense-only system's scores live on a different scale, so every dense answer was below the threshold and it abstained on all 50 questions. The harness printed a candidate score of exactly zero with 25 losses and 17 ties; a mean of exactly 0.000 on a system that clearly retrieves something is a bug signal, and the worst-case list pointed straight at it.

   (hybrid+rerank+split: tau=0.15, abstained on 2 of 50 questions)
   (dense: tau=0.15, abstained on 50 of 50 questions)
## hybrid+rerank+split (A, shipped) vs dense (B, candidate) (n=42, paired)

| | mean |
|---|---|
| hybrid+rerank+split (A, shipped) | 0.595 |
| dense (B, candidate) | 0.000 |

This was a setup error in my comparison, not a defect in the previous project's repository. The abstention threshold is part of the system, so it must be chosen per system, on the dev split only. With that fixed (python examples/apply_to_rag.py):

   (hybrid+rerank+split: tau=0.15, abstained on 2 of 50 questions)
   (dense: tau=0.00, abstained on 11 of 50 questions)
## hybrid+rerank+split (A, shipped) vs dense (B, candidate) (n=42, paired)

| | mean |
|---|---|
| hybrid+rerank+split (A, shipped) | 0.595 |
| dense (B, candidate) | 0.452 |

Difference (B - A): **-0.143**, 95% CI -0.238 to -0.048; B better on 0, A better on 6, tied on 36; sign test p = 0.0312

| slice | n | diff | 95% CI |
|---|---|---|---|
| exact | 8 | +0.000 | +0.000 to +0.000 |
| lookup | 20 | -0.150 | -0.300 to +0.000 |
| multi | 6 | +0.000 | +0.000 to +0.000 |
| paraphrase | 8 | -0.375 | -0.750 to -0.125 |

Gate: **FAIL**
- BLOCK: overall -0.143 (95% CI -0.238 to -0.048, n=42)
- warn: slice 'lookup' -0.150 (CI -0.300 to +0.000, n=20)

Answer-correct is 0.595 for the shipped configuration and 0.452 for dense, a paired difference of minus 0.143 with an interval of minus 0.238 to minus 0.048, A better on 6 questions and B on none, and the gate blocks. The lookup slice (20 questions) is reported as a warning because its interval touches zero. I did not apply the toolkit to the extraction project's golden set in this pass; it needs only a score per document id and a slice, and it is a good exercise.

Cost, latency, deployment and the observability of the evaluator

A judged run costs two calls per item for pairwise, so print the bill first. dry_run(n_items, calls_per_item, tokens_in, tokens_out, usd_in_per_m, usd_out_per_m) returns calls, tokens and dollars without calling anything. For 150 items, 2 calls each, 1,000 input and 200 output tokens per call at assumed prices of $0.15 and $0.60 per million (not a vendor quote), it returns 300 calls, 360,000 tokens and $0.081, and a test pins that arithmetic. Replace the prices with your judge model's current ones and date them in the report. I did not build parallel execution, so I make no latency claim; a thread pool with a small limit is a straightforward stretch goal, and it must go through the cache so reruns cost nothing.

Deployment: a pip-installable package, a reusable CI workflow, and a static report.

:::key Verified 24 October 2026 (check current limits before you rely on any of this) Reusable workflows. GitHub's documentation says a reusable workflow must include workflow_call under on, lives in .github/workflows (subdirectories are not supported), and is called with uses: {owner}/{repo}/.github/workflows/{file}@{ref}; the ref can be a SHA, tag or branch, and the page recommends a SHA as the safest. Chains are limited to ten levels, and secrets are passed explicitly or with secrets: inherit. GitHub Pages. For accounts on GitHub Free the repository must be public to publish Pages; published sites may be no larger than 1 GB, with a soft 100 GB per month bandwidth limit and a soft limit of 10 builds per hour that does not apply when you publish with a custom GitHub Actions workflow. Sources: docs.github.com pages on reusing workflows, creating a Pages site and Pages limits, read 24 October 2026. A static HTML version of the markdown report on Pages is a free live demo; it holds no secrets and no live model. :::

The registry is the observability. Each run writes the dataset hash, system, judge, rubric hash, seed and cost. When a number in a report is questioned, those fields tell you which items, which prompt and which judge produced it, and the cache means the rerun is free. Treat a change in rubric hash as a break in the history: scores from different hashes are not comparable until the judge is recalibrated.

Failure analysis, pitfalls and stretch goals

Failures for docs/failures.md, each with a guarding test. The simulator's quality function matched the substring 'Delhi' inside 'New Delhi', so a fair judge scored 90 percent accuracy instead of 100 until the test caught it and the answer was parsed as a sentence. The length probe confused position bias with length bias, fixed by counting only swap-consistent verdicts. The gate blocked on three bad items out of ten, fixed with the sign test and a minimum size. The RAG comparison used one abstention threshold for two systems and reported an exact zero.

Pitfalls. Using one model as generator and judge without a self-preference probe. Calibrating on the data you tuned the rubric on. Reporting means without intervals. Comparing runs on different samples. Letting a prompt change silently invalidate history. Treating a 5-point change on 40 items as signal. Treating a scripted judge's numbers as a model's. Trusting a code grader to be a sandbox.

Stretch goals. A parallel judge runner with a request budget. Multi-judge ensembles that route disagreements to a human. Distilling the judge into a small classifier (the fine-tuning project in this topic). Synthetic test-case generation with human review. Agent trajectory graders for tool-call correctness. A command-line entry point and the reusable workflow above.

Present it: README numbers, interview questions, pitch and resume bullet

The README results table (mine; regenerate yours). Python 3.9.13, Windows 11.

itemvalue
tests36 passing, offline, no key
meta-evaluation set60 synthetic human-labelled items (labels true by construction), 30 pairs, scripted judges
judge card (90% stand-in)kappa 0.80 (95% CI 0.64 to 0.93), n = 60; Hinglish slice 0.60, n = 20
refused judges65% stand-in (kappa 0.30), 95/55 slice stand-in (kappa 0.50)
position bias with swap4 confidently wrong of 30 without; 0 wrong, 12 ties with
gate false-positive rate0.5% blocked, 11% warned, over 200 no-change reruns at n = 150
gate sensitivity38% blocked at a 5-point drop; 89.5% at a 10-point drop (n = 150)
power (simulated)paired test: 23% at n = 100, 81% at n = 400 for a 5-point gain with 12% disagreement
applied to the RAG setrecall@5 0.921 vs 0.651 (blocked); answer-correct 0.595 vs 0.452 (blocked), after fixing a threshold mistake

The five questions this project prepares you for, with outlines.

@@ 1 | Can I trust an LLM judge? Only after calibration against human labels with kappa and an interval, per slice; bias probes for position, length and self-preference; pairwise with swap; a judge from a different family than the generator; and recalibration when the rubric or model changes. Cite Zheng et al. for the documented biases, and say your scripted tests prove the harness, not a model.

@@ 2 | How many examples do I need to detect a 5-point improvement? Binomial variance: at 0.7 and 150 items a single score is about plus or minus 7 points. Pairing and a test on the disagreeing items help; show your power table, and say it depends on how much the systems disagree.

@@ 3 | Offline eval says better and users say worse. Why? Distribution mismatch, a stale golden set, a metric that is not what users value, judge bias. Fix with online signals, shadow mode and refreshing the set from production failures.

@@ 4 | Design a regression gate engineers will not disable. A measured low false-alarm rate, slice awareness, a minimum size and an exact test, clear output with the worst items, block versus warn. Quote your simulated 0.5 percent and 38 percent honestly, as simulated.

@@ 5 | When would you not use an LLM judge? Whenever a deterministic check exists: schema, exact value, code tests. Judges are for subjective or open-ended quality, they cost calls, and they are less reproducible.

The 90-second pitch (ILLUSTRATIVE). 'Teams quote eval scores nobody can trust. I built a small harness where a judge may score only after it clears a calibration bar against human labels, per slice. Pairwise judging asks in both orders and calls disagreement a tie. Comparisons are paired with an interval, and the gate blocks only when the drop is large, significant and the set is big enough. I measured the gate in simulation with scripted stand-in judges: on 200 no-change reruns it blocked 0.5 percent, and at 150 items it catches a 5-point drop only 38 percent of the time, which I say rather than hide. The failure I found was comparing two RAG variants with one abstention threshold, which produced an exact zero.' Hinglish opener: 'Maine ek eval toolkit banaya jisme LLM judge tabhi use hota hai jab woh insaani labels ke saath kappa 0.6 clear kare.'

The resume bullet (ILLUSTRATIVE shape; numbers are what this repo prints). Built judgekit (Python): deterministic graders, calibrated pairwise and pointwise LLM judges with position-swap debiasing, paired bootstrap and sign-test comparisons, and a regression gate with a measured 0.5 percent false-block rate on 200 simulated reruns; 36 offline tests.

An honest time estimate. About 30 hours: data and cache 3, graders 4, statistics 4, judges and simulator 5, calibration and probes 4, compare and gate 4, meta-evaluation 3, applying and write-up 3. The 20-hour minimum drops the probes and the report; stretch goals reach 45.

How to say it in the interview

"I do not trust a judge until it has been scored against human labels, so my harness refuses to use one below kappa 0.6 overall or on any slice with enough items. Pairwise judging asks in both orders and calls disagreement a tie. Comparisons are paired, with an interval and an exact sign test, and the gate blocks only when a drop is big, the interval excludes zero, the sign test agrees and the set is not tiny. I measured the gate on 200 simulated no-change reruns and it blocked 0.5 percent of them, and I know that at 150 items it catches a 5-point drop only about 38 percent of the time, so I say so. My judges here are scripted stand-ins with programmable accuracy and bias, which tests the harness against a known truth and says nothing about any real model."

Checklist

Practice this interactively
Share this lessonWhatsAppLinkedInX

Related lessons