Cruxly › AI engineering › Fine-Tuning and Model Adaptation

Fine-Tune, RAG, or Prompt: The Decision Nobody Wants to Make Twice

the data-changes test, the citation test, and the default-to-RAG pattern

Prompt: a mid-size e-commerce company's support bot keeps giving wrong answers about product specs and pricing. An engineer proposes fine-tuning the model on the full product catalog "so it actually knows our products." Decide whether that is the right fix, design what you would do instead if not, and say precisely what will happen six weeks after either path ships.

40 minutes first. The proposal sounds reasonable and is the majority answer in the room. It is also the answer that creates a second, worse incident a few weeks after the first one is declared fixed.

Why the catalog proposal fails on its own terms, not just in theory

State the six-week outcome before debating the six-month one. The engineer's proposed fix ships, the demo questions score well because they resemble training examples, and the team moves on. Six weeks later: (1) the catalog has changed — new products added, three SKUs discontinued, prices updated for a sale — and every answer touching those items is now confidently wrong instead of merely outdated, because nothing in the system knows the weights are stale; and (2) a customer asks about a product that appeared only once or twice in the training data, in an unusual phrasing, and gets a fluent, specific, wrong answer, because fine-tuning never gave the model a lookup table — it gave the model a distribution over how catalog questions get answered, and that distribution generalizes past what it was actually shown.

Name the mechanism, because "it doesn't know facts" undersells the problem. A model fine-tuned on catalog Q&A pairs learns the shape of a good catalog answer — the phrasing, the level of detail, which fields tend to get mentioned — far more reliably than it learns the specific numbers inside any one answer. That is why the failure mode is not "the model says it doesn't know" (which would be safe) but "the model states a plausible, wrong spec with full confidence" (which is the actually dangerous outcome, and the one support bots ship with alarming regularity).

Apply the test. Does the answer depend on data that changes? Prices, stock levels, specs for new SKUs, discontinued items — yes, constantly, on a timescale of days. That single question is enough to disqualify fine-tuning as the fix for this complaint, independent of any other consideration.

The decision framework: three tests, in order

Test 1 — does the answer depend on data that changes? If yes, retrieve it; don't bake it in. This is the test that resolves the large majority of "should we fine-tune" questions in production systems, because most of what people want a model to "know" is exactly the kind of thing that has an owner, a last-updated timestamp, and a change log elsewhere in the company — a catalog, a pricing table, a policy document, an inventory count. RAG keeps the model's knowledge current by construction: updating a document is a write to an index, not a multi-hour training run, and the model never has to "forget" a stale fact because it never memorized the fact in the first place.

Test 2 — do you need provenance? If a user or an auditor needs to know which document justified an answer — a compliance question, a legal citation, "show me where it says that" — RAG wins outright and the discussion is over. A fine-tuned model can produce an answer that resembles what it was trained on; it cannot point at the specific document, because the document's identity was never part of what got trained — only its influence on the weights survived, and influence is not a citation.

Test 3 — are you trying to change behavior that should not change? Style, tone, output format or schema adherence, a domain-specific reasoning or triage pattern, or distilling a frontier model's behavior into a smaller, cheaper model — these are the cases fine-tuning is actually for, because they are properties of how the model answers, not what it needs to look up, and they are stable across the kind of day-to-day data changes that break test 1.

The worked example: what actually ships

The real fix for the support bot, priced out. Put the catalog behind retrieval: index every product row (spec, price, stock, description) with embeddings refreshed on every catalog update — a write path that already exists in most e-commerce stacks, since the catalog already has an update event somewhere. Every support answer retrieves the current rows for the products mentioned and grounds its response in them, with the retrieved row's last-updated timestamp available for the response to reference if needed. A price change propagates to every future answer within the refresh window (usually seconds to minutes), not the days or weeks a retraining cycle would take.

What fine-tuning still earns a place for, in the same system. The support bot's tone — apologetic but not groveling, specific next steps instead of vague reassurance — and its escalation pattern — when to offer a refund path unprompted versus when to require the customer to ask — are stable company policy, not catalog data. They don't change when a product price changes, and prompting alone tends to drift under load (a long system prompt encoding tone and escalation rules competes with the retrieved context for the model's attention and gets inconsistently followed as the retrieved context grows). That is a legitimate, narrow fine-tuning target: a few hundred to a few thousand examples of well-handled support conversations, fine-tuned onto the base model, producing consistent tone and escalation behavior regardless of which product the retrieved context happens to be about.

State the composed system out loud, because this is the actual answer in the room. "I would not fine-tune on the catalog — it's data that changes weekly, and fine-tuning would make the bot confidently wrong instead of honestly outdated. I'd put the catalog behind retrieval so every answer is grounded in the current row. Separately, I'd fine-tune on a few hundred examples of well-handled support conversations to lock in tone and escalation behavior, because that's a stable pattern a long system prompt handles inconsistently under load. RAG for what changes, a small fine-tune for what shouldn't." That sentence is the senior answer, and it uses both tools for what each is actually good at instead of picking a side.

The distillation case, because it's a real and different use case worth naming even though it doesn't apply here. If the goal instead were "we currently call a frontier model for every request and it's too expensive and too slow at our volume," fine-tuning a smaller model on that frontier model's outputs — distillation — is a legitimate, current, and distinct reason to fine-tune that has nothing to do with teaching facts. It is teaching a cheap model to imitate an expensive model's behavior on a task, which is exactly the kind of stable pattern fine-tuning is built to capture.

One more option the catalog case usually deserves: don't embed the catalog at all. Prices, stock and specs are structured fields, and an embedding search over product rows is poor at exactly the things customers ask — exact SKUs, numeric comparisons ("under ₹2,000"), filters ("in stock, size 9"). Give the model a tool — a SQL or catalog-API lookup — and let it query the live table. Use retrieval for the unstructured part (manuals, reviews, policy pages). And for a small, stable corpus that fits in the context window, a fourth option is simply putting it all in the prompt with prompt caching, which avoids building retrieval at all.

Beyond supervised fine-tuning: preference tuning and RL

Supervised fine-tuning (SFT) teaches the model to imitate examples. Two other families change what it prefers, and a 2026 interview will ask when you'd reach for each.

• DPO is the practical default for preference tuning. It optimises directly on preference pairs with a simple classification-style loss — no separate reward model, no RL loop — which is why it largely replaced PPO outside frontier labs.
• GRPO is what made reasoning-style training accessible. It samples a group of answers per prompt, scores each with the verifier, and pushes toward the above-average ones — no value network needed. Its weak spot is the verifier: a reward that can be gamed will be (passing tests by special- casing them), so the checker is the thing to engineer carefully.
• Providers expose these as products — preference fine-tuning and reinforcement fine-tuning APIs that take your graders — so this is no longer only a research-cluster decision.

The usual escalation path is SFT → preference tuning → RL, and you stop as early as the eval allows. SFT first gets the format and baseline behaviour; add DPO when you can see specific failures and can say which of two answers is better; reach for RL only when there's a reliable automatic checker and the task genuinely needs the model to search for answers rather than copy them. LoRA works with all three — and is sufficient even for RL, where the signal per episode is small.

Distillation has variants too, worth naming precisely:

• Sequence-level (SFT on teacher outputs) — the simplest and most common; the student only sees the teacher's chosen text.
• Logit-level — the student matches the teacher's full token distribution, which carries more information per example but needs access to the teacher's logits (impossible through most closed APIs).
• On-policy distillation — the student generates, and the teacher scores or corrects the student's own outputs, which fixes the mismatch between what the student saw in training and what it produces at inference.

What goes wrong even when you pick correctly

RAG is not automatically safe just because it's the right default. If the retriever returns stale or wrong rows — a caching layer that didn't invalidate, a duplicate SKU with an old price still indexed — the model will ground its confident answer in confidently wrong retrieved context, which looks identical to a fine-tuning failure from the user's side. Retrieval freshness is now the thing that has to be correct; you have not eliminated the staleness problem, you have moved it to a place where fixing it is a document update instead of a retrain, which is the entire point, but it still has to actually be operated correctly.

A fine-tune for behavior still needs a regression check against the base model. Fine-tuning on support conversations to fix tone can degrade the model's product reasoning if the training examples are narrow enough that the model starts pattern-matching on conversation structure instead of reasoning about the retrieved content — this is the same catastrophic- forgetting risk covered in the training-data lesson, and "I only changed tone" is not a guarantee that only tone changed.

Naming the condition that would flip the decision is what separates a memorized framework from actual judgment. If the catalog were genuinely frozen — a fixed reference product line that doesn't change for a year — the data-changes test would no longer disqualify fine-tuning, and the calculus would shift toward whichever approach is cheaper to operate at the company's actual query volume. Stating that condition, even though it doesn't hold here, is what shows you're applying a test rather than reciting a rule.

How to say it in the interview

"Fine-tuning on the catalog is the wrong fix even though it will look like it works at first. A catalog is data that changes — prices, stock, specs, new SKUs — and fine-tuning bakes a snapshot of it into the weights, so the model goes stale the moment the catalog updates, which for e-commerce is constantly. It also does not actually solve the stated problem: fine-tuning teaches the model a style of answering about products, not a lookup table of product facts, so it will still confidently invent specs for items it saw rarely in training. I would put the catalog behind retrieval so every answer is grounded in the current row, and reserve fine-tuning for the one thing RAG does not fix here: making the bot consistently apply the company's tone and escalation policy."

Checklist

• Applied the 'does the answer depend on data that changes' test before proposing a fix
• Named the specific failure mode: fine-tuning teaches a style of answering, not a lookup table
• Stated the six-week outcome, not just the day-one demo result
• Used the provenance/citation test as an independent, decisive check
• Identified a legitimate fine-tuning target in the same system, not just the wrong one
• Named distillation as a distinct, current reason to fine-tune
• Proposed the composed system — RAG plus a narrow fine-tune — instead of picking one tool for everything
• Named a condition that would flip the decision
• Proposed a SQL/tool lookup for structured catalog fields instead of embedding search
• Chose between SFT, DPO, and RL with verifiable rewards by the training signal available

Related lessons

• LoRA and QLoRA: What Actually Changes During Fine-Tuning
• Training Data: Dedup, Contamination, and Knowing It Actually Worked
• RLHF, DPO and RL with Verifiable Rewards: How Post-Training Actually Works
• Distillation and Synthetic Data: Teachers, Students and the Quality Loop