Cruxly › AI engineering › Prompting, Structured Output & Tool Calling

The Prompt Lifecycle: Versioning, Regression Tests and Automatic Optimisation

prompts as versioned artifacts, a regression suite, a canary you can actually read, and what an optimiser needs before you trust it

Open the interactive lesson

Prompt: "Our support prompt has 14 rules and 1,900 tokens, edited by six people over a year. One customer says answers got worse last week. How do you manage prompts in production, find out what changed, roll back, and slim the prompt down without guessing?"

40 minutes first. Define what a prompt version is, design the regression suite, work out whether you can detect the damage at one customer's traffic, and price the clean-up. The trap is treating the prompt as a string in a config file that anyone can edit.

A prompt is a production artifact

If a prompt can change without a commit, a review and a recorded eval result, you cannot answer "what changed?" when a customer complains. Treat each prompt like a deployable service config.

Identity. Every prompt has an id (support-reply), a version (a monotonically increasing number or a content hash), and an owner who gets the page.
Storage. Either in the repo as files (reviewed in pull requests, diffed, tied to the release) or in a registry with the same properties: immutable versions, an audit trail, an alias such as production that points at one version. The failure to avoid is an admin UI where anyone edits the live text.
A pinned model. A version is the pair of prompt text and model (plus parameters such as temperature and effort), because the same text behaves differently on another model, as Reasoning Models and Test-Time Compute: When to Think and How to Prompt Them shows. The model identifier is part of the version record.
An attached eval. The record stores the regression-suite result for that exact pair, so "is v14 better than v13?" is a lookup, not a debate.
A template, not a string. The prompt has named slots ({customer_name}, {ticket}, {retrieved_docs}), and the slot list is part of the interface. Trusted text (policies, examples) and untrusted text (ticket body, retrieved pages, tool results) go in different, fenced slots, and the template test fails if untrusted text appears outside its slot. Untrusted text is data, never instructions, per Prompt Engineering as Engineering: Structure, Examples and What Actually Moves Quality.
A log field. Every model call logs prompt_id, prompt_version, model, and a hash of the rendered prompt. Observability and Cost Governance for LLM Systems explains the rest of the telemetry; this field is what makes a bad answer attributable.

Do not store prompts only in application code strings scattered across services. Duplicated copies drift: the retry path uses the old wording, the batch job uses a third. One registry or one directory, loaded by id, and a lint that fails the build on an inline prompt that is not registered.

Treating prompts as code also means treating changes as deploys: a one-word edit passes through the same review and test as a code change, because a one-word edit can flip a behaviour. In an interview defend the properties (immutable versions, alias, audit, attached eval), not a product name.

The regression suite: assertions, judged properties and what triggers a run

A prompt regression suite is a golden set with checks that run on every prompt edit and every model change, and it fails the build. It extends the golden set from Evaluation Foundations: Golden Sets, Error Analysis and Reading Benchmarks with checks suited to prompts, and it is the gate described in Evaluation Gates That Actually Block a Bad Deploy.

Three kinds of check, cheapest first:

Deterministic assertions. The output parses against the schema; contains the refund policy id when the ticket mentions a refund; does not contain a competitor's name, a phone number pattern or the string "As an AI"; stays under 150 words. These are exact, fast and free of judge noise. Most regressions are caught here.
Reference checks. For tasks with a right answer (a category, an amount), compare to the label.
Judged properties. Tone, completeness and groundedness need a rubric-driven judge model. Pin the judge, calibrate it against human labels and never let it grade its own family without a check, as LLM-as-a-Judge: Evaluation That Scales Without Lying to You explains. Judged scores are noisy, so use them as a trend with a wider threshold.

Build the set from real inputs: include the easy majority, the previous incidents (every bug becomes a permanent case), the adversarial inputs, and the tail. Keep it versioned like the prompt. The run triggers are an edit to the prompt, an edit to the model or its parameters, a change in the retrieved-context format, and a scheduled run, since hosted models can change under a stable name (use dated model ids where the vendor offers them, and a scheduled run for those that do not).

Prompt slimming uses the suite as an instrument: ablate one rule at a time. Worked example, with assumed numbers. The support prompt has 14 rules and 1,900 tokens (about 136 tokens per rule on average). Run the 150-case suite with each rule removed in turn, 14 ablations plus the baseline: 15 x 150 = 2,250 calls. At an assumed 1,900 input tokens at $2 per million plus 200 output tokens at $8 per million, a call costs $0.0038 + $0.0016 = $0.0054 and the whole ablation costs about $12.15. Suppose it finds: 5 rules whose removal changes nothing (flips balanced, say 3 up and 3 down), 2 rules whose removal improves the score (say 9 cases up and 1 down, so the rule was hurting), 1 rule whose removal clearly hurts (12 cases down, 3 up: load-bearing), and 6 that matter a little.

Treat that result as a screen, not a verdict. A sign test on 9 versus 1 gives p of about 0.02 and on 12 versus 3 about 0.035, but you ran 14 tests. At a 0.05 threshold the chance of at least one false alarm across 14 independent tests is 1 - 0.95^14 = 0.51, and you expect 0.7 false positives by chance. A Bonferroni-corrected threshold would be 0.05 / 14 = 0.0036, which neither finding meets. So the right claim is "these three rules are candidates", then confirm: remove the 5 inert and 2 harmful rules together, giving a 1,100-token prompt (assumed, 800 tokens fewer; the average of 136 tokens per rule would predict about 950, so the removed rules are assumed to be shorter than average), and run that single change on a fresh set not used in the ablation. If the score holds within a paired test, ship it. At 40,000 calls a day and an assumed $2 per million input tokens, 800 x 40,000 x 2 / 1e6 = $64 a day before caching, about $23,360 a year (800 x 40,000 = 32 million tokens a day). Prompt Caching: Paying for the Same Tokens Once reduces that, which is why the saving is a bonus and not the justification: fewer rules means fewer contradictions.

Rolling out a change: canary, per-customer slices and rollback by id

A canary is only useful if the slice is large enough to see the harm you care about, so do the sample-size arithmetic before you promise one. Route 5 percent of the 40,000 daily calls, 2,000 calls, to the new version and the rest to the old, assigning by a stable hash of the conversation or customer so one conversation never mixes versions. Compare a quality signal that you can measure online: a thumbs-down rate, an escalation-to-human rate, a judged sample, or a rule-based check. The offline-versus-online trade-offs are in Offline Versus Online Evaluation: Shadow Mode, A/B Tests and Feedback Loops.

How much traffic detects a drop? For a success rate near 80 percent (assumed) and a 3-point drop to detect with 80 percent power at a 5 percent two-sided significance level, the standard two-proportion formula n = 2 x (1.96 + 0.84)^2 x p(1-p) / d^2 = 2 x 7.84 x 0.16 / 0.03^2 = about 2,788 calls per arm. A 2,000-call-a-day canary needs about 1.4 days to reach that. For a 5-point drop it needs 1,004 per arm, half a day. So a 5 percent canary for 48 hours can see a 3-point drop; it cannot see a 1-point drop (the requirement scales with 1/d^2, so roughly 25,000 per arm), and you should say so rather than declare a quiet canary "safe". Also decide the stop rule in advance, because peeking at a running test and stopping when it looks good inflates false positives.

The single unhappy customer is a slice problem. Customer X sends 300 calls a day. To detect a drop from 90 percent to 84 percent (assumed) in their slice, with a success rate near 87 percent, n = 2 x 7.84 x 0.87 x 0.13 / 0.06^2 = about 493 per arm. Even if X's 300 calls a day were split 50/50 (150 per arm), that is 493 / 150 = about 3.3 days; at a 5 percent canary share X would see about 15 new-version calls a day and never get there. If X was not split at all, a live comparison is impossible and you need a replay of their stored inputs through both versions. That is the answer to "how do we find out": (1) filter logs by customer and prompt_version and compare before and after for that customer, (2) replay their last few hundred stored inputs through v14 and v15 offline and compare, a paired design that needs far fewer cases than two independent groups, (3) read the failures, because a systematic cause (a rule about regional pricing they alone trigger) usually shows in a dozen transcripts. Then add those inputs to the golden set so the suite catches that class next time.

Rollback is moving the alias to the previous version. If the version is immutable and the alias is the only mutable thing, rollback takes seconds and needs no deploy. Define the trigger up front (guardrail metrics such as escalation rate or complaint rate crossing a threshold), automate it for the severe cases, and keep the previous version warm. The same discipline appears in incident handling in Red-Teaming and Compliance as Code.

Automatic prompt optimisation, and why you still read the diff

An optimiser searches prompt space using your metric, so it is only as good as the metric and the data split you give it. Three families:

Compile-style (DSPy). The DSPy paper (Khattab et al., 2023, "DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines") treats a pipeline as modules with signatures, and an optimiser tunes instructions and few-shot demonstrations against a metric on training examples. You write the program and the metric; the system writes the wording.
Reflective evolution (GEPA). "GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning" (Agrawal et al., arXiv 2507.19457) has a model read execution traces, write a natural-language diagnosis, propose a prompt edit, and keep a Pareto set of candidates. The authors report it beating a reinforcement-learning baseline with far fewer rollouts; the abstract's headline figures differ between the paper's versions (it was revised in February 2026), so quote the claim as "reported by its authors, on their benchmarks" and read the current version before citing a number.
LLM-written rewrites. Ask a strong model to improve the prompt given failing cases. It is cheap, needs no framework, and is the same loop as a human editor, with the same overfitting risk.

What every optimiser needs, and how each goes wrong.

A metric that means something. The optimiser maximises the number you give it. If the metric is "judge score", it will find phrasing the judge likes (length, confident tone) which humans may not. This is Goodhart's law, and LLM-as-a-Judge: Evaluation That Scales Without Lying to You lists the judge biases it will exploit. Prefer deterministic metrics, or a judge calibrated against human labels.
Three splits: train, dev, test. The optimiser reads train and uses dev to choose among candidates; the test set is touched once, at the end. With 150 cases, an optimiser can memorise a few quirks: an example, a specific phrase, a clause tuned to three tickets. Report the test number, not the dev number, and expect it to be lower. If you cannot spare a test set, you do not have enough labelled data for an optimiser.
A budget. Every candidate evaluation is calls over the training set. 200 candidates on 100 cases at the earlier $0.0054 a call is 20,000 calls, about $108, which is cheap against an engineer-week but not free, and the figure grows with prompt length and judge calls.
A human reading the diff. The output is a prompt, so review it like a pull request: does it add a rule that contradicts policy? Does it hard-code a customer name from the training data? Does it smuggle an instruction that moves untrusted text into a privileged slot? An optimised prompt that scores 3 points higher and contains a leaked example is a regression.

Optimisation is a lifecycle step, not a one-off. A prompt tuned for one model is tuned to its habits, so on an upgrade re-run the optimiser with the new model instead of assuming the result transfers. And it is no substitute for error analysis: if the failures come from missing context (retrieval did not return the policy), no wording will fix them.

Prompt debt and the migration audit

Prompt debt is the pile of rules whose reason nobody remembers, and a model upgrade is when you pay it. Typical contents of an old prompt: a rule for a bug fixed two models ago; two rules that contradict ("always be concise" and "explain your reasoning in detail"); emphatic wording added after one bad incident; examples for a label that no longer exists; a hard-coded date; a clause copied from another team's prompt. Each costs tokens on every call and adds a chance of conflict, and a model that follows instructions more literally than its predecessor turns a harmless old rule into an active one.

A migration audit when the model changes:

Freeze the baseline. Run the regression suite on the old prompt and old model, and store the result.
Run the old prompt on the new model, unchanged. This isolates what the model change did. Some failures will be real regressions; some will be the new model correctly refusing something the suite wrongly rewarded.
Check the new model's defaults. Effort, temperature handling, token limits, structured-output support and tool-choice rules can differ per model, and several of the changes in this topic (forced tool choice rejected on some current models, thinking budgets removed) surface as errors or silent behaviour changes; see Function Calling Basics: The Loop, Parallel Calls, Errors and Forced Choice.
Ablate. Remove rules one at a time as in the earlier section, starting with emphatic and step-by-step ones, and keep what the data supports.
Re-optimise if you use an optimiser, then review the diff and run the held-out set once.
Canary, then promote, with the old model/prompt pair kept as the rollback target.

Give every rule a reason and a test. The practice that keeps debt down is a comment beside each rule in the source: why it exists, which incident or case id it came from, and which suite case would fail without it. A rule with no linked case is a candidate for deletion at the next audit. Rules that exist for safety (never reveal another customer's data) get a case in the suite and, as Guardrails for an Agent That Takes Real Actions argues, are enforced in code too.

:::warn Template injection is a lifecycle bug as well as an attack. A prompt that interpolates untrusted text into a trusted slot (the ticket body into the system message, a retrieved page into the instruction block) is vulnerable no matter how good the wording is. Review every template change for new slots, and test with adversarial inputs, as in MCP Servers, Prompt Injection, and Tamper-Evident Audit. :::

How to say it in the interview

"A prompt is a production artifact: it lives in version control or a registry with an id, an owner, a pinned model and the eval result of the version that shipped, and every call logs the prompt id so any answer can be traced back. Changes go through a regression suite of golden inputs with deterministic assertions plus judged properties, run on every prompt edit and every model change, then a canary on a slice of traffic, and rollback is switching the id. For the one unhappy customer I compare their traffic by prompt version, knowing their volume may be too small for a firm conclusion. To slim the prompt I ablate rules one at a time on a fixed set, treat the result as a screen because fourteen tests will throw up false alarms, and confirm the combined cut on fresh cases. If I automate improvement with an optimiser, it needs a metric and a train, dev and test split, and I read its diff because it overfits."

Checklist

Every prompt has an id, an immutable version, an owner, a pinned model and an attached eval result
Prompts live in version control or a registry with an alias, not in scattered strings or a live-edit UI
Every model call logs prompt_id, version, model and the rendered-prompt hash
Untrusted text is only allowed in named, fenced slots and a template test enforces it
A regression suite with deterministic assertions, reference checks and judged properties runs on every prompt and model change
Every production incident becomes a permanent golden case
Ablation results are treated as a screen and confirmed on fresh cases
Canary size was computed from the effect you need to detect, with the stop rule fixed in advance
Rollback is an alias move with automated triggers on guardrail metrics
Optimiser runs use a trusted metric, train/dev/test splits, a budget, and a human-reviewed diff
Each rule carries a reason and a linked case, and model migrations follow the audit steps

Practice this interactively