Cruxly › AI engineering › Portfolio Projects

Project: Fine-Tune or Distill a Small Model, With a Proper Before/After Evaluation

Distil a scripted teacher into a small student, then prove with intervals, leakage checks and a break-even gate whether to ship it

Project: "We label about 17,000 support messages a day (roughly 500,000 a month) with a large model, and the bill and the delay are starting to hurt. Someone says we should fine-tune a small model. Before I agree, I want a before and after that I can defend: the same held-out messages, the same metrics, intervals on every number, the cost of getting there, and a rule that says when we ship it and when we do not."

Time-box: about 40 hours for the full plan (28 for a bare run, 58 with the stretch goals). What you build here is smaller on purpose and fully offline: a pure-Python student trained on labels from a scripted teacher, with a data pipeline that has a leakage check, 22 passing tests, a printed before/after table, a break-even calculator and a ship gate that, on this data, says no. The first thing to know is what is not here: no transformer, no LoRA, no GPU run and no real model's accuracy. The lesson says how the real run replaces the toy and which of its numbers you must measure yourself.

Why this project, and what it proves to an employer

Most fine-tuning projects on a portfolio prove that the author can run a notebook; this one is built to prove that the author can decide whether to run it. A reviewer who has seen forty "fine-tuned Llama on my dataset" repositories looks for four things: a baseline that a prompt alone could not beat, a test set the training data never touched, a number with an interval on it, and a cost line. The fifth thing, rarely present, is a willingness to write "do not ship this" when the numbers say so. This lesson ends with that sentence, produced by code.

The setting is a narrow task: route a support message into one of five intents (refund request, order status, cancel order, address change, product question), in English and Hinglish. A small model can plausibly match a large one on a narrow task, which is the claim you must test, not assume.

You practise the pipeline in the order a real project needs it. Write the decision memo first. Remove duplicates, split by a hash of the content and prove nothing in test resembles training. Label with a teacher and audit it. Train with early stopping. Compare with honest baselines on untouched data. Measure cost, compute the break-even and apply a gate. A team that splits before it deduplicates measures its own leakage and calls it accuracy.

Scope, metrics and non-goals (numbers first)

The scope is one task, five classes, 2,787 generated messages after dedup and a student you can read in 96 lines. The project prints accuracy and macro-F1 with 95% Wilson intervals on 560 test items; the paired student-minus-teacher difference with a bootstrap interval; five-seed variance; slices including a separate 200-item out-of-distribution set; teacher-versus-gold kappa; the count of test items that duplicate training (it must be 0); latency; cost per 1,000 requests; and the break-even month.

The gate is written before the result, and it has two halves. The student ships only if the lower bound of the interval on (student minus teacher) accuracy is no worse than minus two points, and only if it is at least five times cheaper per 1,000 requests at the volume you actually have. If either half fails, the decision memo recommends prompting. A gate defined after seeing the numbers is a rationalisation with a threshold in it.

Non-goals. No transformer is trained, no real language model labels anything (the teacher is a scripted function with two planted mistakes), no GPU is used, and no accuracy number is claimed for any real model. The data is synthetic, generated by data.py, so its licence is yours and so is its weakness: a student that does well here has shown the pipeline works, not that your tickets are easy.

Architecture and the seam: what is fake, what is real

Every step that touches a model goes through the seam from the first project, and every step that is arithmetic is plain code with a test. The flow has three checkpoints where leakage is stopped, drawn below.

What is fake. The teacher is PlantedNoiseTeacher behind llmkit's FakeLLM, wrapped in MeteredLLM so each label has a token count and a price. The price table is the first lesson's assumed one, not a quote from any vendor. The student is a softmax regression on hashed word and bigram features. The "small model prompted without examples" baseline is a keyword list I wrote. What is real: the dedup, the split, the metrics, the training loop with real weights and real early stopping, the intervals, the latency measurement on my machine, and the arithmetic of cost and break-even.

The offline-test statement. The 22 tests prove plumbing and invariants: stable splits, caught duplicates, a recovered noise rate, hand-checked metrics, a loop that lowers its loss, an endpoint contract, a two-halved gate. They do not prove a transformer fine-tune would work on your data.

What is printed and what is not. The project is 676 lines in nine modules; the lesson prints about 215 lines of Python. Not printed: __init__.py (empty), data.py (the template generator), serve.py (a 29-line WSGI endpoint), most of experiment.py, and tests/_paths.py, which puts src/ and the first project's llmkit on the import path. Some excerpts omit imports. Layout:

finetune/
  src/student/  data.py dedup.py splits.py teacher.py model.py metrics.py breakeven.py serve.py experiment.py
  evals/        run.py (the whole experiment)  leakage_probe.py  fresh_seed.py
  tests/        _paths.py test_splits_dedup.py test_teacher_noise.py test_metrics.py
                test_breakeven_gate.py test_train_smoke.py

Run python -m unittest discover -s tests and python evals/run.py from finetune/ on Python 3.9; the full run takes about 50 seconds.

Milestone 1: the decision memo and the break-even formula

The memo is one page and it contains one formula, because the formula is what turns "it will be cheaper" into something that can be wrong. Months to break even equal the upfront cost divided by the monthly saving. The upfront cost is labelling, training and the engineering hours. The monthly saving is what you pay the teacher per month minus what the student costs to run. If the saving is zero or negative, there is no break-even at all, and the function must say so instead of dividing.

# -*- coding: utf-8 -*-
"""student.breakeven: cost per 1,000 requests, the monthly bill, the break-even month, and the ship gate.
Every price and hour count is an input, never a constant: replace the numbers with yours."""
from typing import Optional


def cost_per_1000(avg_in_tokens: float, avg_out_tokens: float, usd_per_m_in: float, usd_per_m_out: float) -> float:
    return 1000 * (avg_in_tokens * usd_per_m_in + avg_out_tokens * usd_per_m_out) / 1e6


def monthly_api(requests: int, per_1000: float) -> float:
    return requests / 1000 * per_1000


def break_even_months(upfront: float, monthly_teacher: float, monthly_student: float) -> Optional[float]:
    """Months until the upfront cost (labelling + training + engineering hours) is paid back.
    None means the student is not cheaper to run, so there is no break-even at this volume."""
    saving = monthly_teacher - monthly_student
    return None if saving <= 0 else upfront / saving


def ship_gate(diff_ci_low: float, teacher_per_1000: float, student_per_1000: float,
              max_drop: float = 0.02, min_ratio: float = 5.0) -> dict:
    """Ship only if (a) the LOWER bound of (student - teacher) accuracy is no worse than -max_drop and
    (b) the student is at least min_ratio times cheaper per 1,000 requests."""
    ratio = teacher_per_1000 / student_per_1000 if student_per_1000 > 0 else float("inf")
    quality_ok, cost_ok = diff_ci_low >= -max_drop, ratio >= min_ratio
    return {"quality_ok": quality_ok, "cost_ok": cost_ok, "cost_ratio": ratio, "ship": quality_ok and cost_ok}

The test pins the arithmetic by hand. 100 input and 10 output tokens at the assumed 2.50 and 10.00 dollars per million is 350 micro-dollars, 0.35 dollars per 1,000; 800 upfront against a 200 monthly saving is four months; a student that is not cheaper returns None. Every price is an argument, so the memo can be refilled when a price changes.

Milestone 2: dedup, the split, and the contamination check

Duplicates are removed before the split, and the split is keyed on content, because an id-keyed split lets the same text sit on both sides. Normalisation lowercases, replaces order ids and drops punctuation. Exact matches go first; near duplicates are found with MinHash banded into eight bands of four rows and verified by the Jaccard similarity of 4-character shingles at 0.75.

# -*- coding: utf-8 -*-
"""student.dedup: exact and near-duplicate detection (MinHash with banding), and the test-overlap check."""
import hashlib
import re
from typing import Dict, List, Sequence, Tuple

NEAR = 0.75          # character 4-gram Jaccard at or above this is "near-duplicate"
PERMS, BAND = 32, 4  # 32 hash functions in 8 bands of 4 rows: a pair at Jaccard 0.8 shares a band almost surely
_P = (1 << 61) - 1
_AB = [(int.from_bytes(hashlib.blake2b(b"a%d" % i, digest_size=8).digest(), "big") | 1,
        int.from_bytes(hashlib.blake2b(b"b%d" % i, digest_size=8).digest(), "big")) for i in range(PERMS)]


def normalise(text: str) -> str:
    """Lower-case, order ids become #, punctuation and repeated spaces go."""
    t = re.sub(r"o-\d{4}", "#", text.lower())
    return " ".join(re.sub(r"[^\w# ]", " ", t).split())


def shingles(text: str, k: int = 4) -> set:
    t = normalise(text)
    return {t[i:i + k] for i in range(max(1, len(t) - k + 1))}


def jaccard(a: set, b: set) -> float:
    return len(a & b) / len(a | b) if a | b else 1.0


def signature(sh: set) -> Tuple[int, ...]:
    xs = [int.from_bytes(hashlib.blake2b(s.encode("utf-8"), digest_size=8).digest(), "big") for s in sh]
    return tuple(min((a * x + b) % _P for x in xs) for a, b in _AB)


def find_duplicates(items: Sequence[dict]) -> Dict[str, Tuple[str, str]]:
    """Returns {dropped_id: (kept_id, 'exact'|'near')}. The first occurrence is kept."""
    kept, dropped, seen = [], {}, {}
    sh, buckets = {}, {}
    for it in items:
        key = normalise(it["text"])
        if key in seen:
            dropped[it["id"]] = (seen[key], "exact")
            continue
        s = shingles(it["text"])
        sig = signature(s)
        cands = set()
        for b in range(PERMS // BAND):
            cands |= set(buckets.get((b, sig[b * BAND:(b + 1) * BAND]), ()))
        hit = next((c for c in sorted(cands) if jaccard(s, sh[c]) >= NEAR), None)
        if hit is not None:
            dropped[it["id"]] = (hit, "near")
            continue
        seen[key], sh[it["id"]] = it["id"], s
        for b in range(PERMS // BAND):
            buckets.setdefault((b, sig[b * BAND:(b + 1) * BAND]), []).append(it["id"])
        kept.append(it["id"])
    return dropped


def contamination(train: Sequence[dict], test: Sequence[dict]) -> List[Tuple[str, str, float]]:
    """Test items whose normalised text equals, or is near-duplicate of, a training item. Report must say 0."""
    tr = {}
    for it in train:
        tr.setdefault(normalise(it["text"]), it["id"])
    tsh = {it["id"]: shingles(it["text"]) for it in train}
    out = []
    for it in test:
        key = normalise(it["text"])
        if key in tr:
            out.append((it["id"], tr[key], 1.0))
            continue
        s = shingles(it["text"])
        best = max(((jaccard(s, v), k) for k, v in tsh.items()), default=(0.0, ""))
        if best[0] >= NEAR:
            out.append((it["id"], best[1], round(best[0], 3)))
    return out

# -*- coding: utf-8 -*-
"""student.splits: a deterministic split keyed on the content, so a text can never sit in two splits."""
import hashlib
from collections import Counter
from typing import Dict, List

from .dedup import normalise

CUTS = (70, 80)      # buckets 0-69 train, 70-79 validation, 80-99 test


def split_of(text: str, salt: str = "v1") -> str:
    b = int(hashlib.sha256((salt + normalise(text)).encode("utf-8")).hexdigest()[:8], 16) % 100
    return "train" if b < CUTS[0] else ("val" if b < CUTS[1] else "test")


def split(items: List[dict], salt: str = "v1") -> Dict[str, List[dict]]:
    out: Dict[str, List[dict]] = {"train": [], "val": [], "test": []}
    for it in items:
        out[split_of(it["text"], salt)].append(it)
    return out


def balance(items: List[dict]) -> Dict[str, int]:
    return dict(sorted(Counter(i["label"] for i in items).items()))

Results on the 3,000-item pool: 213 items removed (19 exact, 194 near), leaving 2,787 split into 1,947 train, 280 validation and 560 test, with 105 to 120 test items per class. A test asserts no merged pair has different labels.

The planted-duplicate test measures the detector's recall, and the honest number is not 100%. I add 50 exact and 50 near copies (one letter swap or one filler word) to a clean pool. Over five seeds the detector caught all 250 exact and 245 of 250 near copies; on a very short message one edit can push Jaccard under 0.75. The plan asked for "all caught", the test asserts at least 45 of 50 and says why. An exhaustive overlap check is the second instrument: it found 0 test items duplicated in training, and 2 validation items the banded detector had missed. I left them in and report them.

What leakage does to a number, on this toy. evals/leakage_probe.py splits the same pool three ways with 100 planted duplicates and trains the same student each time.

split by id, no dedup                train 2162 test  633  test items duplicated in train:  89   test accuracy 0.9210
split by content hash, no dedup      train 2166 test  629  test items duplicated in train:  76   test accuracy 0.9364
dedup first, then split by content   train 1947 test  560  test items duplicated in train:   0   test accuracy 0.9339

The counts are the evidence: 89 and 76 test items had a twin in training, 0 after the fix. Accuracy moves less than the noise on 560 items, because a bag-of-words model generalises to a template variant. A larger model that memorises would show inflation; the check is justified by the count, not by an accuracy jump.

Milestone 3: the teacher, its audit, and its cost log

A teacher has an error rate, and the student's ceiling is the teacher's quality, so build the instrument that measures it first. The scripted teacher knows the gold label and makes two planted mistakes: about 6% random wrong labels (chosen by a hash, so reruns match), and half of the Hinglish address-change messages always labelled order status. The second kind is systematic and does not average out.

class PlantedNoiseTeacher:
    def __init__(self, gold: Dict[str, dict], p_rand: float = 0.06, salt: str = "T1"):
        self.gold, self.p_rand, self.salt = gold, p_rand, salt

    def __call__(self, messages: List[dict]) -> str:
        text = messages[-1]["content"].split("Message: ", 1)[1]
        rec = self.gold[text]
        label, conf = rec["label"], 0.93
        if _h(text, self.salt + "r") < self.p_rand:
            others = [c for c in CLASSES if c != label]
            label, conf = others[int(_h(text, self.salt + "o") * len(others))], 0.55 + 0.3 * _h(text, self.salt + "c")
        elif rec["lang"] == "hi" and rec["label"] == "address_change" and _h(text, self.salt + "s") < 0.5:
            label, conf = "order_status", 0.8
        return json.dumps({"label": label, "confidence": round(conf, 2)})

The audit measures it against humans. In the toy, the validation split's gold labels play the part of a 280-item human-audited sample, and Cohen's kappa between teacher and gold is 0.8654. Kappa is agreement beyond what the class frequencies give by chance. The code is short enough to check by hand, and so is the calibration error used later.

def kappa(a: Sequence[str], b: Sequence[str]) -> float:
    """Cohen's kappa between two labelings: agreement beyond what the class frequencies give by chance."""
    n = len(a)
    po = sum(x == y for x, y in zip(a, b)) / n
    ca, cb = Counter(a), Counter(b)
    pe = sum(ca[c] * cb[c] for c in set(ca) | set(cb)) / (n * n)
    return (po - pe) / (1 - pe) if pe < 1 else 1.0


def ece(conf: Sequence[float], correct: Sequence[bool], bins: int = 10) -> float:
    """Expected calibration error: bin by confidence, average |accuracy - confidence| weighted by bin size."""
    n, total = len(conf), 0.0
    for b in range(bins):
        idx = [i for i, c in enumerate(conf) if min(int(c * bins), bins - 1) == b]
        if idx:
            acc = sum(correct[i] for i in idx) / len(idx)
            total += len(idx) / n * abs(acc - sum(conf[i] for i in idx) / len(idx))
    return total

The test plants a known noise rate and asks whether the pipeline recovers it. With random noise at 0% and then 10%, measured error on a 600-item sample equals the planted rate plus the systematic share, within 2.5 points. In the full run the teacher is wrong on 8.8% of training items, which the toy knows only because it has gold; a real project has the audit sample alone. A second test checks the cost log: 50 items make 50 metered calls and the dollars equal tokens times the assumed price. Labelling 2,227 train and validation items cost 0.99 dollars at the assumed "large" price, at 128 input and 12 output tokens per call (tokens estimated as characters over four).

Milestone 4: the student, its training loop, and how LoRA differs

The student is a softmax regression on hashed word and bigram features: the smallest thing with a real training loop. It has 5 by 4,096 weights, L2 regularisation, a seeded shuffle and early stopping on the best validation loss.

def features(text: str, dim: int = DIM) -> Dict[int, float]:
    toks = re.findall(r"[a-z0-9#]+", re.sub(r"o-\d{4}", "#", text.lower()))
    grams = toks + [a + "_" + b for a, b in zip(toks, toks[1:])]
    idx = {zlib.crc32(g.encode("utf-8")) % dim for g in grams}
    v = 1.0 / math.sqrt(len(idx)) if idx else 0.0
    return {i: v for i in idx}

    def fit(self, texts: Sequence[str], labels: Sequence[str], seed: int, epochs: int = 30, lr: float = 0.5,
            l2: float = 1e-5, val: Optional[Tuple[Sequence[str], Sequence[str]]] = None, patience: int = 3) -> dict:
        """Per-example SGD. With a validation set, keep the weights of the best validation loss and stop
        after `patience` epochs without improvement. Returns the training history."""
        rng = random.Random(seed)
        xs = [features(t, self.dim) for t in texts]
        ys = [self.classes.index(l) for l in labels]
        vx = [features(t, self.dim) for t in val[0]] if val else None
        vy = [self.classes.index(l) for l in val[1]] if val else None
        order, hist, best, best_w, bad = list(range(len(xs))), [], 1e9, None, 0
        for ep in range(epochs):
            rng.shuffle(order)
            for n in order:
                p = self.proba_f(xs[n])
                for k in range(len(self.classes)):
                    g = p[k] - (1.0 if k == ys[n] else 0.0)
                    if g:
                        w = self.W[k]
                        for i, x in xs[n].items():
                            w[i] -= lr * (g * x + l2 * w[i])
                        self.b[k] -= lr * g
            rec = {"epoch": ep + 1, "train_loss": self.loss(xs, ys)}
            if vx is not None:
                rec["val_loss"] = self.loss(vx, vy)
                if rec["val_loss"] < best - 1e-4:
                    best, bad = rec["val_loss"], 0
                    best_w = ([r[:] for r in self.W], self.b[:])
                else:
                    bad += 1
            hist.append(rec)
            if vx is not None and bad >= patience:
                break
        if best_w is not None:
            self.W, self.b = best_w
        return {"history": hist, "stopped_at": len(hist)}

Early stopping uses validation only. The loop restores the best weights and stops after three epochs without improvement; the five seeds stopped after 12, 11, 15, 11 and 12 epochs. Tests give it wrong validation labels and assert it stops before epoch 30, check that one epoch lowers the loss below ln 5 (1.6094), that a seed reproduces weights, and that a saved model round-trips.

    def test_one_epoch_on_twenty_examples_lowers_the_loss(self):
        m = SoftmaxStudent(data.CLASSES)
        h = m.fit(self.texts, self.labels, seed=0, epochs=1)
        self.assertLess(h["history"][0]["train_loss"], 1.6094)          # ln(5): the loss of knowing nothing

How a real LoRA run differs. Replace model.py with a pretrained transformer and train adapters. A LoRA adapter on a d by d matrix adds two thin rank-r matrices, 2·d·r numbers instead of d squared: at d = 1,024 and r = 8, 16,384 against 1,048,576, about 1.6%. The base weights stay frozen and in memory (0.5 billion parameters at 2 bytes is about 1 GB), with activations and optimizer state on top. Everything outside model.py stays. LoRA and QLoRA: What Actually Changes During Fine-Tuning explains the mechanism.

:::key Verified 28 October 2026 PyPI, read 31 October 2026, shows Hugging Face TRL at 1.17.1 (uploaded 30 October 2026, Python 3.11 or newer) and PEFT at 0.21.2 (Python 3.10 or newer, uploaded 1 October 2026). Both move fast, so pin the versions you tested with. TRL's documentation shows SFTTrainer("Qwen/Qwen3-0.6B", train_dataset=dataset, peft_config=LoraConfig()), computes loss on the completion only by default for prompt-completion data, and suggests a learning rate near 1e-4 for adapters. My machine runs Python 3.9.13, so I installed neither and ran no LoRA training. Check the pages and versions yourself. :::

:::key Checked 28 October 2026 (hedged) I did not open Colab's or Kaggle's current terms, so I state no GPU model, session length or quota for their free tiers. Check them on the day, and save adapter checkpoints in case a session ends early. :::

Milestone 5: the before and after evaluation

The comparison is one table on the same 560 test items, with an interval on every row, and the rows include the "before" a reviewer wants. The test split is read in one place in experiment.py, after training, early stopping and the abstention threshold have all been fixed on validation.

The majority class and the untrained student sit at the 20% chance floor; a keyword-list stand-in for an untuned prompted small model reaches 0.66. The teacher stand-in reaches 0.92, the student 0.90, and the same student on all 1,947 gold labels (which no real project has) 0.93, the ceiling. A student on only 280 human labels reaches 0.87, the honest rival to paying a teacher to label: the extra teacher labels are worth about three points.

The interval supports a narrower claim than the table invites. Student minus teacher is -0.0161 with a bootstrap interval of -0.0500 to +0.0161; the student won 38 items and lost 47, with 475 ties (sign test p = 0.3857). The data does not show the student is worse, and does not show it is within two points: the lower bound is outside the gate's allowance.

    st, tc = preds["student, teacher labels (after)"], preds["teacher (stand-in)"]
    pc = paired_compare([float(a == b) for a, b in zip(y, tc)], [float(a == b) for a, b in zip(y, st)])
    report["student_minus_teacher"] = {k: round(v, 4) if isinstance(v, float) else v for k, v in pc.items()}

Seeds, calibration, abstention. Five-seed macro-F1 is 0.9011, 0.9084, 0.9030, 0.9119 and 0.8979 (mean 0.9045, range 0.014), so a one-point difference between seeds is noise. Expected calibration error is 0.0427. An abstention threshold of 0.73, chosen on validation, makes the student answer 79.3% of test messages at 96.6% accuracy and pass the rest on.

    # abstention: the threshold is chosen on VALIDATION, then applied to test
    vsc = [(students[0][0].predict(t), g) for t, g in zip(*va)]
    thr = next((t / 100 for t in range(50, 100) if (lambda acc: acc and sum(acc) / len(acc) >= abstain_target)(
        [p == g for (p, c), g in vsc if c >= t / 100])), 0.99)
    kept = [(a == b) for (_, c), a, b in zip(sc, y, st) if c >= thr]
    report["abstain"] = {"threshold_from_val": thr, "test_coverage": round(len(kept) / len(y), 4),
                         "test_acc_on_answered": round(sum(kept) / max(1, len(kept)), 4)}

The gate says no. The quality half fails (lower bound -0.05). The cost half passes at 500,000 requests a month against a large-tier teacher: 22 times cheaper. The decision is not to ship the student alone and to test the abstain-and-fall-back design. Against a small-tier teacher the ratio is 1.3 and both halves fail.

Milestone 6: slices, out-of-distribution data, and where it breaks

An average accuracy hides the slices where the student fails, and this student fails in two places that the average does not show.

Hinglish and typo slices sit a few points under the teacher, within the noise of 199 and 83 items. On messages with two intents the student gets 65% against the teacher's 89%; a bag of words cannot see which clause came first, and my stand-in teacher is not a bag of words, so read this as where an order-blind model breaks. On the 200 out-of-distribution messages ("kindly process a reimbursement") the student falls to 0.505, and to 0.38 on the Hinglish part.

The consequence is a monitoring plan, not a patch: abstain below the validation threshold, log the abstention share as a drift signal, and send a sample of low-confidence messages to a person.

Cost, latency and the break-even table

Student latency is about 0.04 to 0.06 ms at p50 and 0.06 to 0.11 ms at p95, in process, on one Windows laptop, over two sessions of three repeats of the 560 test items (the last session: 0.052 to 0.056 and 0.091 to 0.105). That is a Python function, not a service benchmark; teacher latency here is a fake clock, so I report none.

The break-even table uses assumed prices, and the answer flips with the teacher. Inputs: 128.1 input and 12.2 output tokens per call (measured); the first lesson's assumed price table ("large" 2.50 and 10.00 dollars per million tokens, "small" 0.15 and 0.60); a CPU container assumed at 10 dollars a month; an upfront cost of 806 dollars (the measured 0.99 labelling cost, 40 engineer-hours at an assumed 20 dollars, 5 dollars of assumed compute). Replace all of them.

Against a large-tier teacher the student pays back in 66.6 months at 50,000 requests a month, 3.8 at 500,000 and 0.4 at 5 million. Against a small-tier teacher there is no break-even at 50,000 (its 1.33-dollar bill is below the 10-dollar container), 247 months at 500,000 and 6.6 at 5 million. The memo's sentence: fine-tuning pays at high volume against an expensive comparison model, and not otherwise.

Deployment, with limits read on the day. The student is a JSON file and a 29-line WSGI app; it needs no GPU.

:::key Verified 28 October 2026 Hugging Face's Spaces documentation says Gradio and Docker Spaces "run on compute and require a paid plan to create", Static Spaces are free, default CPU Basic is 2 vCPU and 16 GB RAM, and free hardware sleeps when unused. Render's free-tier page says a Free web service spins down after 15 minutes without traffic (about a minute to spin up), gives 750 free instance hours a month per workspace, loses local files on restart, and is not for production. I read both on this date and deployed nothing; check current limits. :::

Observability, probes and the failure analysis

In serving, log the confidence distribution and the abstention share per day, keep a sample of low-confidence messages for relabelling, and record a run registry entry per student (config, seed, data hash, test metrics).

What I tried to break, and what broke. The leakage probe showed overlaps of 89 and 76 that accuracy would not reveal. The dedup detector's recall was 245 of 250, and 2 validation near-duplicates got through. My first generator was so repetitive that 1,208 of 3,000 items were removed (from my run notes; that version is gone, so you cannot rerun it); I added variety until 213 were. And one thing a reviewer should be told: I changed the generator twice after seeing test numbers. The first version was trivially separable (a gold-trained student scored 100%), so I made 25% of messages mixed-intent. That makes the task harder rather than favouring the student, and the final numbers come from one run of the final generator, but the test split did influence the design, so treat the table above as possibly optimistic. To check, evals/fresh_seed.py (26 lines, not printed) draws a new pool with seed 4242 from the frozen generator and reruns everything once: teacher 0.9099, student 0.8991, student minus teacher -0.0108 with a bootstrap interval of -0.0432 to +0.0216, the gate says no again, kappa 0.8590, mixed-intent 0.66. The same templates generate both pools, so this tests the sampling, not the generator. Make such changes on validation.

Three failure cases for the README. The student inherits the Hinglish confusion: address-change recall 0.857, 9 of 105 sent to order status; fix by a slice audit and relabelling. Mixed intent sits at 65%; fix with order-aware modelling or routing two-intent messages to a person. Out-of-distribution sits at 50.5%; fix with abstention and drift logging.

How to present it: README numbers, the pitch, the resume bullet, five questions

The README opens with the decision and the gate. The table shows kappa 0.8654, test-to-train overlaps 0, student 0.9018 against the teacher stand-in's 0.9179 with the paired interval, the five-seed range, latency, cost per 1,000 requests with the price date, break-even at three volumes, and the gate result: not shipped.

The 90-second pitch (ILLUSTRATIVE; use only numbers your repository prints): the question (can a small model replace a large one for routing?), the method (content-keyed split, audited teacher, a 280-label human baseline), the result (indistinguishable on 560 items but an interval that cannot rule out a five-point loss), the decision (the gate said no; abstain-and-fall-back is next) and the limit (a toy student and scripted teacher, so a LoRA run comes next with the same code).

An ILLUSTRATIVE resume bullet: built a distillation-evaluation harness (Python): content-keyed splits with a contamination check (0 test overlaps), teacher audit (kappa 0.87), paired-bootstrap comparison and a cost gate that rejected shipping.

The five interview questions, with outlines:

• When would you fine-tune instead of prompting or RAG? Stable narrow task, high volume, latency or privacy limits, labelled data; changing knowledge goes to RAG; show the break-even and say it may never pay against a cheap teacher.
• How do you know the model is actually better? Untouched test, contamination count, a small human-labelled baseline, paired intervals, slices, seeds, cost beside accuracy.
• What can go wrong with teacher labels? The teacher's errors are the ceiling; systematic noise does not average out; audit with kappa; score against gold, never teacher labels.
• LoRA: what changes and what does it cost? Adapters on chosen matrices, 2·d·r numbers instead of d squared, frozen weights still in memory, learning rate near 1e-4 per TRL's docs.
• Better on the task, worse elsewhere? Forgetting: add a general-ability slice, mix general data, lower rank or learning rate, or route.

Pitfalls and stretch goals

Pitfalls: near-duplicates in test; an untuned prompt or the dearest teacher as a strawman; a break-even without labelling and engineering cost; one seed; a teacher that also grades; a win inside the interval; a design changed after seeing the test score.

Stretch goals, none done: a real LoRA run with TRL and PEFT on the same splits and report; a quantised student; DPO on judge-ranked pairs; this student as the capstone's triage step.

Time: about 40 hours offline; a real LoRA run adds 12 to 18.

How to say it in the interview

I did not fine-tune a language model in this project, and I will say so in the first sentence of the README. I trained a small classifier on labels from a scripted teacher, because that is the version I could run offline and verify end to end. The point is the evaluation around it: a split nobody can contaminate, a teacher whose mistakes I measured, baselines that are not strawmen, intervals on the claims, and a gate that said no. When I swap the toy for a LoRA run on a GPU, every file except one stays the same.

Checklist

• Before you put this on your resume, the README's first lines say that the student is a toy classifier and the teacher a scripted stand-in, and that no real model's accuracy is claimed.
• Before you put this on your resume, the decision memo and the ship gate were written before the first evaluation run, and the gate has a quality half and a cost half.
• Before you put this on your resume, duplicates are removed before the split, the split is keyed on content, and an exhaustive check reports how many test items duplicate a training item.
• Before you put this on your resume, the dedup detector's recall was measured on planted duplicates, and the misses are reported.
• Before you put this on your resume, the teacher was audited against a human-labelled sample with kappa, and the student is scored against gold labels, not teacher labels.
• Before you put this on your resume, the baselines include majority, an untuned prompted model, the teacher and a student trained on a small human-labelled set.
• Before you put this on your resume, every comparison between systems is paired and carries an interval, and the README states only what the interval supports.
• Before you put this on your resume, results are reported over several seeds, and the choice of threshold and stopping point was made on validation.
• Before you put this on your resume, the slices (language, typos, mixed intent, out of distribution) are in the report with the failures named.
• Before you put this on your resume, the break-even table shows both an expensive and a cheap teacher, includes labelling and engineering cost, and every price and hour is labelled as an assumption.
• Before you put this on your resume, latency is stated as a one-machine sample with its observed range, and nothing about a real model's speed is claimed without a real measurement.
• Before you put this on your resume, anything you changed after seeing the test score is written down in the README.

Related lessons

• Build Projects That Get You Hired: The Process, the Shared Test Seam and How to Show It
• Project: A Document Extraction Pipeline With Validation and a Field-Level Scorecard
• Project: A RAG Assistant Over Real Documents With an Eval Gate in CI
• Project: A Tracing and Cost Dashboard for LLM Apps (Build Your Own Mini-Langfuse)
• Project: An Eval Harness and LLM-as-Judge Toolkit You Can Trust
• Project: A Text-to-SQL Analyst With Read-Only Guardrails and Execution-Based Evals