Cruxly

Cruxly › AI engineering › Fine-Tuning and Model Adaptation

Distillation and Synthetic Data: Teachers, Students and the Quality Loop

a $400 data bill, a 25-day payback, 10 TB of logits you will not store, and the golden set that decides whether it worked

Open the interactive lesson

Prompt: "Our product sends 100 million tokens a month to a frontier model and the invoice is the biggest line in the budget. Someone says we should distil it into a small model. Tell me when that is the right call, how you would generate the training data without poisoning the student, what you are allowed to use as a teacher, and how you would know the student is good enough to ship."

40 minutes first. Do the break-even arithmetic with every assumption labelled, size the data pipeline, and write the acceptance test before you generate a single example. The trap is treating distillation as "call the big model a lot and fine-tune on the answers" with no filter, no held-out gold and no look at the licence.

When distillation pays: the business case before the pipeline

Distillation is a cost and latency play on a narrow task, and you can price it before writing any code. Fine-Tune, RAG, or Prompt: The Decision Nobody Wants to Make Twice names it as a legitimate reason to fine-tune; this lesson gives the economics and the pipeline. The classic idea is older than LLMs: Hinton, Vinyals and Dean's Distilling the Knowledge in a Neural Network (March 2015) transfers what an expensive model, or an ensemble, knows into a single cheaper model.

The worked example, with every assumption labelled. Use assumed round prices, not any vendor's: teacher output at $10 per million tokens, the self-hosted or small student at $0.20 per million tokens.

That payback has two conditions attached, and the second is the one that fails. First, volume must persist: at 10M tokens a month the saving is $98 and payback stretches to about eight months. Second, the student must reach the agreed quality. If it only reaches the teacher's level on 70 percent of cases, you either route the rest back to the teacher (a router, which Model Selection: Cost, Latency, and Capability Trade-offs covers) or give up the saving. Write the acceptance criterion before generation: for example, "within 2 points of the teacher on a 400-example golden set, and no worse than the teacher on the safety slice".

When not to distil. The task changes weekly (the data will be stale before the student ships); volume is low; the work needs broad capability rather than one narrow task; or your provider's terms forbid it (section 4). Also compare the cheaper options first: a better prompt on a smaller model, prompt caching (Prompt Caching: Paying for the Same Tokens Once), or a router.

What the student can and cannot inherit. It inherits the teacher's behaviour on the distribution you sampled. It does not inherit capabilities you never exercised, and it inherits the teacher's mistakes and biases as confidently as its successes. A student trained on 50,000 support replies learns the support style; asked a coding question, it falls back to what the small base model can do.

Variants: sequence-level, logit-level, on-policy, and reasoning traces

The variants differ in what you must be able to read from the teacher and how much signal each example carries. Fine-Tune, RAG, or Prompt lists them; here is the mechanism.

Sequence-level distillation (SFT on teacher outputs). Sample the teacher's text, then fine-tune the student on it as ordinary supervised data. It needs only an API that returns text, which is why it is the default. Each example carries one sampled answer, a single path through the teacher's distribution.

Logit-level distillation. The student is trained to match the teacher's whole next-token distribution, not only the sampled token. Hinton et al. soften both distributions with a temperature T so that small probabilities show up. On logits [4, 2, 0], T = 1 gives probabilities [0.867, 0.117, 0.016] and T = 4 gives [0.506, 0.307, 0.186]: at T = 4 the student sees that the second option is far more plausible than the third, information that the sampled token alone discards. The paper multiplies the soft-target loss by T^2 to keep gradient magnitudes comparable across temperatures. The catch is access: you need the teacher's logits, which hosted APIs rarely return in full, and the storage is large. For 40M tokens and an assumed 128,000-token vocabulary at 2 bytes, full distributions would be 40M x 128,000 x 2 B = 10.24 TB; storing only the top 20 token IDs and log-probabilities (4 plus 2 bytes each) is 40M x 20 x 6 B = 4.8 GB. Top-k storage is the practical compromise, and it needs the teacher and student to share a tokenizer, or a mapping step.

On-policy distillation. The student generates, and the teacher scores or corrects the student's own outputs. Sequence-level training teaches the student on teacher-shaped text, but at inference the student conditions on its own earlier tokens, which can differ; on-policy data trains on the states the student actually visits. It costs more per example, since the student must generate and the teacher must grade, and it is worth the trouble when the student's errors compound over long outputs.

Reasoning-trace distillation. For reasoning tasks the teacher's chain of thought is the training target. The DeepSeek-R1 report states that reasoning trajectories from its RL-trained model were distilled and used to train other models, and that the capability transfers to smaller models; treat the size of the gain as that lab's claim on its benchmarks. Two cautions. The trace is only as good as the filter: keep traces whose final answer verifies, not all traces. And length costs money twice, at generation and again at student serving, so measure answer quality per output token, not only accuracy.

Rejection sampling and best-of-n are the cheapest quality lever for synthetic data. Sample n answers per prompt, keep the best by a verifier or judge. If the teacher's per-sample pass rate is an assumed 30 percent and you draw n = 8, then the chance that a prompt has at least one passing sample is 1 - 0.7^8 = 94.2 percent, with 2.4 passing samples expected. Generation cost per kept prompt is 8 x 800 = 6,400 tokens, which at the assumed $10 per million is $0.064 / 0.942 = $0.068. The filter multiplies the generation bill by about eight, so it belongs in the price from the start; the earlier $400 figure assumed one sample per example, no filtering.

The synthetic-data pipeline and the failure it must avoid

A synthetic-data pipeline has six stages, and each one is a place where a bad assumption leaks into the student. Training Data: Dedup, Contamination, and Knowing It Actually Worked covers dedup and contamination on real data; this is the generation-side version.

@@ 1 | Seeds from real traffic Start from real prompts (sampled and anonymised) and a stratified list of intents, including the hard and rare ones. Seeds written by the same model that will answer them produce a narrow, self-similar set.

@@ 2 | Generate with variation Vary temperature, system prompts and personas; ask for several styles. Record the teacher model name, version, date and sampling settings for every example, so you can redo or remove a slice later.

@@ 3 | Filter by checks, cheapest first Programmatic checks (schema, tests, exact answers) first; a calibrated judge for what remains (LLM-as-a-Judge: Evaluation That Scales Without Lying to You); then a human sample. A judge from the teacher's own family favours the teacher's habits.

@@ 4 | Deduplicate and measure diversity Near-duplicate removal (n-gram or embedding) plus coverage counts per intent. Fifty thousand examples that are 5,000 distinct ones repeated ten ways is a 5,000-example dataset.

@@ 5 | Keep human gold, never trained on A few hundred human-labelled examples, stratified by intent and difficulty, split before any generation. This is the only measurement the pipeline cannot corrupt.

@@ 6 | Train, evaluate, iterate Look at the slices that regress and generate more for those, rather than scaling the whole set.

Model collapse is the failure the pipeline must design against. Shumailov et al., The Curse of Recursion: Training on Generated Data Makes Models Forget (May 2023), report that training on model-generated content makes the tails of the original distribution disappear, over repeated generations. A toy calculation shows why tails go first. If a rare intent is 1 percent of the true data and a generation draws 200 examples, the chance it never appears is 0.99^200 = 13.4 percent; at 100 examples it is 0.99^100 = 36.6 percent. Once a generation has none, the next model has little chance of producing it, and it tends to stay gone. The practical lessons: stratify so rare cases get a quota, and never train only on synthetic data.

The research is more nuanced than "synthetic data ruins models". Gerstgrasser et al., Is Model Collapse Inevitable? (April 2024), find that collapse appears when each generation replaces the real data with its predecessor's output, but is avoided when synthetic data is added to the original real data, and prove a bounded test error for a linear model under accumulation. These are controlled settings, not your production pipeline, so treat them as a reason to keep and mix real data, and still measure diversity yourself.

Contamination runs in both directions. If the teacher has seen your eval set, or if your seed prompts were drawn from it, the student learns the answers and the golden set stops measuring generalisation. Hold out the eval set before generating, and n-gram or embedding-match the synthetic data against it. And if the student is later judged by a model of the teacher's family, the judge and student share habits, so keep at least some of the evaluation human.

Terms, contamination, and knowing the student is good enough

Check what you are allowed to do before you generate a dollar of data. Provider terms differ and change.

:::key Verified October 2026 Anthropic's Commercial Terms of Service (effective 17 June 2025), section D.4 (use restrictions), bar using the services to build a competing product or service, including to train competing AI models, except as expressly approved by Anthropic. OpenAI's terms of use have prohibited using output to develop models that compete with OpenAI since at least March 2023; I confirmed the current wording only through secondary copies, so read the live terms for the account and region you use. Open-weight model licences differ from one another and may place conditions on using outputs; read the one for your teacher. None of this is legal advice; have counsel read the terms for your use. :::

In practice: distilling from your own fine-tuned models, from open-weight teachers whose licences allow it, or from a vendor that has agreed in writing is clean. Distilling from a competitor's API into a model you sell is the case the restrictions are written for. Record the teacher, version, date and terms basis in your data manifest, since a licence question months later needs an answer per example slice.

How do you know the student is good enough? Not by a held-out loss, and not by the student's score on the teacher's judge. Run the acceptance test written at the start.

The interview answer. "I would distil when volume, task stability and a clear metric line up, and I would say the number: at my assumed prices about $800 fixed against $980 a month. I would generate from real seeds, filter with checks and a calibrated judge, keep real data in the mix, hold out human gold before generating, check the teacher's terms, and ship only if the student is within the agreed margin on the golden set with an interval, on every slice that matters."

How to say it in the interview

"Distillation makes sense when one narrow task has high volume, a stable definition of good, and an expensive model doing it. I would generate teacher outputs on real seed prompts, filter them with verifiers, a calibrated judge and best-of-n where a checker exists, deduplicate for diversity, and keep a human-labelled golden set that is never used for training. I would price it first: at my assumed numbers a $400 generation bill plus judging and training pays back in under a month at 100 million tokens, but only if the student lands within the agreed margin of the teacher on the golden set, with the error bar reported. I would check the teacher provider's terms before training on its outputs, and keep real data in the mix so the student does not narrow around the teacher's habits."

Checklist

Practice this interactively
Share this lessonWhatsAppLinkedInX

Related lessons