Cruxly › AI engineering › Prompting, Structured Output & Tool Calling

Structured Outputs: Schemas, Validation Loops and What 'Guaranteed' Means

what schema-constrained decoding guarantees, the five ways a valid-looking answer still breaks, and a validation loop with a cap

Open the interactive lesson

Prompt: "We extract the invoice total, currency and due date from supplier PDFs into JSON that feeds our payments system. How do you get reliable JSON out of an LLM, and what does the vendor's strict mode not protect you from?"

40 minutes first. Pick the level of enforcement, design the schema, and write the validation loop with its retry cap. The trap is saying "we use strict mode, so the output is guaranteed": strict mode guarantees a shape, and a payments system needs the values to be right.

Three levels of enforcement, and what each one guarantees

There are three levels of getting JSON from a model, and only the third makes a promise about structure.

Prompt only. "Respond in JSON with keys total, currency, due_date." The model usually complies. It sometimes wraps the JSON in prose or a markdown fence, drops a key, or returns a string where you wanted a number. Every consumer needs a tolerant parser and a retry.
JSON mode. The API constrains the output to be syntactically valid JSON. OpenAI's structured-outputs guide states plainly that JSON mode does not enforce your schema, and recommends Structured Outputs over it where available. You get a parseable object whose keys and types are still the model's choice.
Schema-constrained decoding ("strict" / structured outputs). You supply a JSON Schema and the decoder is restricted so that the output matches it. The mechanism, masking tokens that would break the grammar during generation, and its quality cost are covered in Cost Attribution and Runaway Prevention and the sampling lesson Sampling and Decoding: Temperature, Top-p, Beam Search, and Determinism; this lesson is the application-side contract.

:::key Verified October 2026 Anthropic (structured-outputs docs, retrieved October 2026): put the schema in output_config.format with type: "json_schema"; no beta header is needed; the older output_format parameter is deprecated. The docs state that the result is schema-compliant "through constrained decoding", and list exceptions: stop_reason: "refusal" (the refusal message takes precedence over the schema) and stop_reason: "max_tokens" (output may be incomplete). Not supported: recursive schemas, numerical constraints (minimum, maximum, multipleOf), string length constraints, array constraints beyond minItems of 0 or 1, and additionalProperties other than false; unsupported features return a 400. The first request with a schema pays grammar-compilation latency, and compiled grammars are cached for 24 hours from last use. Enum values may differ from the schema only in capitalisation. Limits listed: 20 strict tools per request, 24 optional parameters and 16 union-typed parameters across strict schemas. OpenAI (structured-outputs guide, retrieved October 2026): set text.format to type: "json_schema" with strict: true; every property listed in required and additionalProperties: false; a refusal can appear instead of schema-conformant content and must be checked first; reaching max_output_tokens yields status: "incomplete"; the first request with a new schema has extra latency. Parameter names change between API versions; confirm on the current pages before copying. :::

"Guaranteed" means the parser will accept it, within a supported subset, when generation finishes normally. That sentence has three qualifiers, and each is a production incident. Within a supported subset: if you write "minimum": 0 on an amount and the vendor does not enforce numeric constraints, a negative total is a perfectly valid output. The Anthropic SDK helpers (parse() with a Pydantic model) move unsupported constraints into the field descriptions and validate the response locally against the original model, which is a good default; a raw schema sent directly gets no such transformation. When generation finishes normally: a refusal or a truncation both produce a 200 response whose content does not match your schema. Syntax, not truth: the decoder cannot know that the invoice says 4,120.00 and the model wrote 4,210.00.

Designing the schema for the model: legal honesty, field order, descriptions

Design the schema so that the honest answer is a legal answer. Extract the invoice total with "total": {"type": "number"} as a required field and you have told the decoder that the output must contain a number. A document with no total leaves the model two moves: refuse the whole call, or invent a number. It will often invent one; zero is the favourite, because zero is a safe-looking number. OpenAI's guide makes the same point in its own words: a model told to follow the schema "will always try to adhere" to it, so give it a way out. The fix is a nullable field plus a stated confidence:

{
  "reasoning": "string  - where on the page the total appears, or why it is absent",
  "total": {"type": ["number", "null"]},   // null when the document states no total
  "currency": {"enum": ["INR", "USD", "EUR", "unknown"]},
  "total_confidence": {"enum": ["stated", "computed_from_lines", "unclear"]}
}

Worked example (assumed numbers, not a measured run). Suppose a test set of 100 invoices whose total is genuinely blank (a credit note, a cut-off scan, a pro-forma). Assume that with the required-number schema the model returns a made-up total on 31 of the 100, and with the nullable-plus-confidence schema on 4 of the 100. The change removes 27 of 31 invented values, an 87 percent reduction (27 / 31 = 0.87). Treat both counts as assumptions: your rate depends on the model, the documents and the prompt, and you replace them by running the set. What carries over is the procedure: build the blank-total set first, because a test set made only of good invoices cannot show this failure at all. The four remaining inventions are the ones for the semantic checks in the next section.

Five rules for the schema, each with a reason:

Reasoning before answer. Tokens are generated in schema order, so a reasoning field placed before total lets the answer condition on it. Anthropic's docs note that required properties appear first in generation, then optional ones, so make the reasoning field required and ordered where you want it. On a model that already thinks, drop this field; see Reasoning Models and Test-Time Compute: When to Think and How to Prompt Them.
Enums over free text. A category as a string returns "Billing", "billing dept" and "BILLING"; an enum returns one of four. Include an explicit other or unknown member so the model is not forced into a wrong one. Keep enums short; a 200-value enum is a retrieval problem, not a schema one.
Descriptions are instructions. The field description is part of the prompt for that field: "ISO 8601 date, the due date not the invoice date; null if absent". Terse names like d1 waste the model's best clue.
Flat beats clever. Deep nesting, many unions and many optional fields run into the complexity limits (24 optional parameters, 16 union types in Anthropic's documented limits) and inflate the compiled grammar, which also lengthens the first-call latency.
Mirror your domain model. If the downstream table has amount_minor as integer paise, ask for that, not for a string like "Rs 4,120.50" that you must parse.

Validate twice: syntax, then meaning, with a capped retry

Strict mode replaces the syntax check; it does not replace the semantic check. Validation has two layers, and the second is yours.

Layer 1, syntax and shape. The JSON parses, the required keys exist, enums are members, types are right. With constrained decoding this almost always passes when generation finishes normally; run the check anyway (a Pydantic or JSON Schema validator costs microseconds) because of refusals, truncation and the unsupported constraints above.
Layer 2, semantics. Business rules that the schema cannot express, or the decoder does not enforce. Does total equal the sum of line items plus tax, within a rounding tolerance? Is the due date on or after the invoice date? Is the total between 0 and your sanity cap? Does the supplier_id exist in the vendor table (a referential check)? Does the quoted text actually appear in the source document (a grounding check, as in Hallucination: Taxonomy, Detection and Mitigation in Production)? For an invoice with lines 1,250.00 and tax 225.00, the stated total must be 1,475.00; 1,457.00 passes layer 1 and fails layer 2.

On failure, retry with the error, not the same prompt, and cap the retries. Append the model's invalid output and a precise message ("total 1457.00 does not equal lines 1250.00 + tax 225.00 = 1475.00; re-read the totals row") and call again. A cap of two is typical. The economics, with assumed rates: suppose 8 percent of first attempts fail layer 2, and each retry fixes 60 percent of the remaining failures. After one retry 0.08 x 0.4 = 3.2 percent remain; after two, 0.08 x 0.4 x 0.4 = 1.28 percent. Expected calls per document with a cap of two: 1 + 0.08 + 0.08 x 0.4 = 1.112, an 11 percent cost increase for removing about 84 percent of the failures (from 8 to 1.28). The two rates are assumptions and the arithmetic treats retries as independent, which is optimistic: a document the model misreads once is often misread again, which is why the retry should carry the specific error and why the residual still goes to a human queue, not to another loop.

for attempt in 0..2:
    resp = call_model(prompt, schema, error_feedback)
    if resp.stop_reason == "max_tokens": raise Truncated   # raise max_tokens, do not retry blindly
    if resp.stop_reason == "refusal":   return Refused     # not a schema error; route it
    data = parse(resp)                                       # layer 1
    errors = business_rules(data, source_doc)                # layer 2
    if not errors: return data
    error_feedback = describe(errors)
return NeedsHuman(last=data, errors=errors)

Never loop without a cap and never silently coerce. An uncapped loop on a document the model cannot read is a runaway bill, the same hazard Cost Attribution and Runaway Prevention describes. And rounding a bad total to "make it pass" hides the signal that your extraction is failing.

When the output is valid and still wrong: truncation, refusals, empties and versions

Treat every structured call as having five outcomes, not two. Success and failure is the wrong model. The outcomes are: complete and correct, complete and wrong (the semantic failures above), truncated, refused, and valid-but-empty.

Truncation. The output hit max_tokens; the JSON is cut, or has closed early. The size of the answer is under your control: 40 line items at about 25 tokens each is 1,000 tokens, so a limit of 512 truncates every large invoice and none of the small ones, which means the failure looks random in aggregate and is perfectly predictable by size. Size max_tokens from the largest document you accept, alert on any max_tokens stop, and fall back to chunked extraction for outsized inputs.
Refusals. A refusal is a response, not a schema error, and both vendors document that it can appear in place of schema-conformant output. Handle it before parsing. Anthropic bills the refusal response; do not retry it blindly with the same input.
Valid but empty. An object of all nulls and "unknown" values is schema-valid and a legitimate answer for a blank page, so count the rate; a jump from 3 percent to 15 percent after a deploy is a regression (or a scanner fault) that no validator flags.
Plausible but wrong. Counter with grounding (require a quote from the document and check it appears), cross-field checks, and a small sampled human audit with an error budget, per Evaluation Foundations: Golden Sets, Error Analysis and Reading Benchmarks.
A possible quality tax. Cost Attribution and Runaway Prevention discusses the measured cost of constraining decoding; whether it hits your task is empirical, so if quality moves after turning strict mode on, compare the same prompt with and without it on your eval set before concluding anything.

Version the schema like an API, because it is one. Downstream consumers (the payments service, the analytics table) depend on the field names, so: adding an optional field is backward compatible; renaming or removing a field, tightening an enum, or changing a type is breaking. Put a schema_version in every stored result and in every log line along with the prompt id and model, so a bad batch can be traced to the exact combination that produced it. Run old and new schema side by side on a replay set before switching, migrate consumers first when you remove anything, and keep the old version readable for as long as stored results reference it. Changing the schema also invalidates the compiled grammar cache (the first call pays compilation latency again; Anthropic documents that changing only a name or description does not) and, with Anthropic, changing output_config.format invalidates the prompt cache for the conversation, so schedule schema changes rather than shipping them mid-peak. Prompt Caching: Paying for the Same Tokens Once explains why.

How to say it in the interview

"There are three levels: asking for JSON in the prompt, JSON mode which guarantees only that it parses, and schema-constrained decoding which guarantees the output matches the schema. I use the third, and I still do not trust the content. Strict mode can be cut short by max_tokens, can be replaced by a refusal, supports only part of JSON Schema so my numeric ranges are not enforced by it, and says nothing about whether the total is the right number. So I design the schema so that an honest answer is legal, with a nullable total and a confidence enum, put any reasoning field before the answer, then validate twice: the parser for syntax, and my own business rules for meaning, such as line items summing to the total. On a failure I retry with the error message fed back, at most twice, log the schema version on every call, and route what is still wrong to a human."

Checklist

Use schema-constrained output (strict) rather than prompt-only or JSON mode wherever the provider offers it
Read the provider's list of unsupported schema features and enforce those constraints in your own validator
Check stop_reason (or status) for refusal and max_tokens before parsing
Make absence legal: nullable fields, an unknown enum member and a confidence field
Put a required reasoning field before the answer on non-thinking models, and drop it on thinking models
Add semantic checks: sums, date order, ranges, referential lookups, and quote grounding
Retry with the specific validation error, a hard cap of two, and a human queue for what remains
Size max_tokens from the largest accepted input and alert on every max_tokens stop
Test on a set that includes blank, malformed and adversarial documents, not only good ones
Log schema_version, prompt id and model with every result and monitor the all-null rate
Treat schema changes like API changes: additive only, versioned, replayed, consumers migrated first

Practice this interactively