Cruxly › AI engineering › LLM Foundations

From Pretraining to Chat Model: The Pipeline and the Scaling Arithmetic

what each training stage changes, and training cost from 6ND

Prompt: "Explain pretraining, SFT and RLHF in two minutes. Then estimate what it costs to train a 70B model from scratch. Then tell me why a vendor would train an 8B model on 15 trillion tokens when it is 'past the optimal point', and why that model has never heard of yesterday's news."

40 minutes first. Do one calculation on paper: FLOPs from 6 N D, then GPU-hours, then dollars. The trap is quoting stage names without knowing what each stage can and cannot change, or giving a cost with no stated assumptions.

The stages, and what each one changes

Each stage changes a different thing, and almost all of the model's knowledge comes from the first.

@@ 1 | Pretraining Next-token prediction on trillions of tokens of web text, code, books and papers. This is where the model acquires language, facts, code and reasoning patterns. It costs the vast majority of the compute. The result is a base model: it continues text, so asked a question it may produce more questions or a plausible web page.

@@ 2 | Mid-training or annealing Late in pretraining, the learning rate is decayed while the mix shifts toward higher-quality data (code, maths, curated text, longer documents). It is the cheap way to lift capability and extend context. Labs describe this differently; treat the name as loose.

@@ 3 | Supervised fine-tuning (SFT) Train on conversations rendered into the chat template (see Tokenization), with the loss computed only on the assistant's tokens. Suppose an example is 10 tokens, 6 of them the user's prompt and 4 the assistant's answer. With loss masking, the average is taken over those 4 tokens; the 6 prompt tokens contribute nothing. Without it, the model is also trained to imitate users and templates, and 60 percent of the gradient signal goes to learning to write prompts. SFT teaches format, tone and instruction-following; a few thousand good examples can change behaviour a lot. It does not teach much new knowledge, and examples that rely on facts the model lacks teach it to answer confidently anyway.

@@ 4 | Preference tuning and reinforcement learning Show the model pairs of answers where one is preferred (by humans, or by a model following written principles, as in Constitutional AI) and train it to favour the preferred one: RLHF with a reward model, DPO directly on the pairs, or RL against verifiable rewards like unit tests and maths answers for reasoning. The InstructGPT paper's headline is the size of the effect: on the authors' prompt distribution, human raters preferred outputs of a 1.3B-parameter model tuned this way over those of the 175B GPT-3. Preference tuning changes which answers the model prefers, not what it knows. The detailed mechanics are in the post-training lessons later in this track; here, know the shape.

@@ 5 | Safety tuning and evals Refusal behaviour, red-team data and classifiers. It trades some helpfulness for fewer harmful outputs, and no stage guarantees either.

What a stage cannot do is the interview-worthy part. SFT cannot reliably add facts. Preference tuning cannot add capabilities that pretraining lacked, though it can elicit ones the model already has. Neither can undo a knowledge cutoff. Prompt, retrieval and tools are how you supply what training did not.

Scaling arithmetic: compute is about 6 N D

Training compute is approximately 6 x N x D FLOPs, where N is the parameter count and D is the number of training tokens. The 6 comes from the forward pass (about 2 N FLOPs per token, one multiply and one add per weight) plus the backward pass (about 4 N, twice the forward). It ignores attention's quadratic term, which is small at moderate context, so it is a good first-order estimate and one to state with that caveat.

The Chinchilla result (Hoffmann et al., arXiv 2203.15556) trained over 400 models to find the best split of a fixed compute budget between parameters and tokens, and concluded they should scale together; the common summary is about 20 tokens per parameter. They tested it by training a 70B model on 1.4 trillion tokens with the same compute as the 280B Gopher but 4x more data, and the smaller model outperformed it on a broad range of tasks.

Worked estimate for a 70B model at 20 tokens per parameter:

D          = 20 * 70e9                       = 1.4e12 tokens
FLOPs      = 6 * 70e9 * 1.4e12               = 5.88e23
H100 bf16 dense peak ~ 989 TFLOP/s, 40% utilisation = 3.956e14 FLOP/s
GPU-seconds = 5.88e23 / 3.956e14             = 1.486e9
GPU-hours   = 1.486e9 / 3600                 = 412,875   (about 413,000)
cost at $2 / GPU-hour                        = $826,000   (about $0.83M)
wall clock on 2,048 GPUs                     = 412,875 / 2,048 = 202 hours = 8.4 days

Each assumption moves the answer: at 30 percent utilisation the GPU-hours rise to about 550,000; at $3 per GPU-hour the cost is $1.24M. That figure is compute for one successful run. Real projects pay several times more for failed runs, ablation experiments, data processing, evaluation, staff, and the post-training stages. Quote it as a floor: "about $0.8M of GPU time at stated assumptions, a lower bound on the project cost".

:::key Verified October 2026 NVIDIA's H100 SXM datasheet lists bf16 at 1,979 TFLOP/s with structured sparsity and states that the dense figure is half of that, so about 989 TFLOP/s dense; the 40 percent utilisation and $2 per GPU-hour are assumptions, not quotes. Rental prices change month to month and by contract; replace them with your current quote. As a cross-check, 6ND for a 405B model on 15.6T tokens is 3.8e25 FLOPs, which matches the 3.8e25 FLOPs and 15.6T tokens that Meta's Llama 3 paper (arXiv 2407.21783) reports for its 405B model. :::

Why vendors train far past Chinchilla-optimal

Chinchilla-optimal minimises training compute for a given quality; it ignores that most of a popular model's life is spent serving. Take Llama 3 8B, which Meta reports was trained on over 15 trillion tokens:

tokens per parameter = 15e12 / 8e9 = 1,875     (about 94x the Chinchilla ratio of 20)
training FLOPs       = 6 * 8e9 * 15e12 = 7.2e23   (22% more than the 70B Chinchilla run's 5.88e23)

So the "overtrained" 8B model cost more to train than the compute-optimal 70B, and it will not match that 70B model. Why do it? Because serving cost scales with parameters: each generated token costs about 2 N FLOPs. If the 70B model serves 10 trillion tokens over its life, inference is 2 x 70e9 x 10e12 = 1.4e24 FLOPs, 2.4x its entire training run. A model with one eighth of the parameters is roughly eight times cheaper per token and fits on one GPU (see GPU Memory Math: Weights, KV Cache, and What Actually Fits). Spending extra training compute to get a smaller model with similar quality is a good deal whenever lifetime inference dominates, which it does for any widely used model. The original scaling-law analysis assumed one-off training and ignored inference; the Chinchilla ratio is a minimum-training-cost recipe, not a law about the best model for a product.

Mixture-of-experts adds a second axis: total versus active parameters. An MoE model routes each token through a few expert MLPs, so only a fraction of its weights do work per token. Mixtral 8x7B (arXiv 2401.04088) has 47B total parameters with 13B active per token (two of eight experts per layer). DeepSeek-V3 (arXiv 2412.19437) reports 671B total, 37B active, trained on 14.8 trillion tokens, so about 5.5 percent of parameters are active per token. Two rules follow, and the second catches people out:

• Training and per-token compute scale with active parameters: 6 x 37e9 x 14.8e12 = 3.29e24 FLOPs for the DeepSeek-V3 arithmetic at active size.
• Memory scales with total parameters: all 671B must be resident to be routed to. A model that is cheap per token can still need a large GPU cluster just to hold weights. Serving consequences are in the later MoE lesson.

Data, contamination, and why the model does not know yesterday's news

Data decides more than architecture, and most of it is filtering. A pretraining corpus starts as a huge web crawl and is cut down by language identification, quality classifiers, heuristics (length, symbol ratios), toxicity and personal-data filters, and deduplication at several levels: exact, near-duplicate (MinHash-style), and repeated boilerplate. Duplicates matter because repeated text is memorised and wastes tokens. The final mixture sets proportions between web, code, books, papers and multilingual text; changing it changes capabilities (more code gives better reasoning on structured tasks), which is why mixes are tuned empirically. Synthetic data (model-written explanations, solutions, instruction data) now carries part of the load, especially in post-training, with the risk that errors and style are amplified when models train on their own output.

Contamination is when benchmark questions leak into training data, so a score measures memory rather than skill. Defences are n-gram overlap checks against test sets and private held-out evals. When you read an eval table, ask how the test sets were protected and whether the numbers are the lab's own runs; treat vendor-reported scores as that vendor's evaluation, not an independent result.

The knowledge cutoff follows from the pipeline: the corpus is frozen at a date, then months of filtering, training and post-training pass before release, and the model is then used for a year or more. Facts after the cutoff are simply absent. Worse, the model may answer confidently from older patterns. Mitigations are in your hands, not the model's: put the current date in the system prompt, retrieve fresh documents, call tools for live data. The retrieval lessons in this track cover how.

Read a model card with a checklist. Find: total versus active parameters; tokens trained; context length and how it was extended; licence (commercial use, user-count or output restrictions); the cutoff date; which evals and whose runs; safety and red-team section; hardware needed. Check the licence before you build, because switching later is costly.

Base, instruct and reasoning checkpoints, and what you can influence

A base model is a text continuer, an instruct model is a base model after SFT and preference tuning, and a reasoning model is further trained to produce long internal chains of thought before answering. Which one you use follows the job.

• Base model: good for continuation, few-shot pattern completion, scoring text, and as the starting point for your own fine-tuning when you control the whole format. Poor at following instructions; it needs careful prompts.
• Instruct / chat model: the default for products; follows instructions, uses tools, stops cleanly.
• Reasoning model: spends many hidden tokens thinking, which raises accuracy on maths, code and multi-step tasks and raises latency and cost (see TTFT, TPOT, and the Latency Budget). It is not better at everything; for extraction and simple classification it is slower and costlier for no gain.

If you fine-tune, start from the instruct checkpoint when your data is a small set of examples that adjust behaviour, and from the base only when you have enough data and want full control of the format. The choice between prompt, retrieval and fine-tuning is covered in Fine-Tune, RAG, or Prompt: The Decision Nobody Wants to Make Twice, and model choice in Model Selection: Cost, Latency, and Capability Trade-offs.

What a hosted-API engineer can and cannot influence is the senior framing:

Two things follow. Put your effort where the left column is: an evaluated prompt and retrieval pipeline beats guessing about weights. And plan for the right column: pin versions, run your eval suite on every provider model change, and keep a fallback.

The two-minute answer to "explain pretraining, SFT and RLHF" is therefore: "Pretraining predicts the next token over trillions of tokens and is where knowledge comes from; SFT trains on curated conversations with loss only on the assistant turn so the model learns the chat format; preference tuning, RLHF or DPO, teaches it which of two answers people prefer. Each later stage is cheap compared to the first and changes behaviour, not knowledge."

How to say it in the interview

"Pretraining teaches the model language and world knowledge by predicting the next token over trillions of tokens; supervised fine-tuning teaches it the format of being an assistant on a small set of curated conversations; preference tuning and reinforcement learning then shape which answers it prefers. Training compute is about 6 times parameters times tokens, so a 70B model on 1.4 trillion tokens is 5.9e23 FLOPs, roughly 413,000 H100-hours at 40 percent utilisation, about $0.8 million at $2 an hour before experiments and failures."

Checklist

• Stated what each stage changes: pretraining knowledge, SFT format, preference tuning which answers
• Explained loss masking on assistant tokens with a worked token count
• Estimated training compute as 6ND and explained the 2N forward plus 4N backward split
• Converted FLOPs to GPU-hours with stated peak FLOPs and utilisation
• Quoted dollar cost as a floor and listed what it excludes
• Explained overtraining by lifetime inference cost, with a number
• Separated active parameters (compute) from total parameters (memory) for MoE
• Chose base, instruct or reasoning checkpoint by task and cost
• Handled the knowledge cutoff with the current date, retrieval or tools
• Read a model card for active parameters, tokens, licence, cutoff, and whose evals

Related lessons

• Transformers and Attention, From the Whiteboard
• Tokenization: BPE, Vocabulary Size, and Why Tokens Are the Unit of Everything
• Embeddings and Similarity: Cosine, Dot Product, and What a Vector Encodes
• Sampling and Decoding: Temperature, Top-p, Beam Search, and Determinism