Cruxly › AI engineering › Prompting, Structured Output & Tool Calling

Reasoning Models and Test-Time Compute: When to Think and How to Prompt Them

thinking tokens are billed output, effort is the dial, and the right level comes from a paired sweep, not a default

Prompt: "We moved a planning feature to a reasoning model. Quality is up but the p95 request now takes 90 seconds and the bill tripled. When should we use a reasoning model, how do we control its cost, why did our old step-by-step prompt get worse, and how do you show progress for a 90-second request?"

40 minutes first. Run the cost-per-correct-answer arithmetic for three effort levels, say which one you would ship and what evidence would change your mind, and rewrite one old prompt line. The trap is treating "reasoning" as a quality switch: it is a spend dial with diminishing returns.

What a reasoning model is, and what its tokens cost

A reasoning model produces a long private chain of thought before its visible answer, and you pay for that chain as output. Models in this class are trained, largely with reinforcement learning on tasks with checkable answers, to write out intermediate work, try approaches, check them and backtrack before committing. DeepSeek's R1 paper (2025) is the open description of the recipe, and the vendor APIs expose the same behaviour as a feature.

Thinking tokens are output tokens. Anthropic's thinking documentation says the tokens spent reasoning are billed as output tokens even when the thinking text is not returned, and that they count toward max_tokens. OpenAI's reasoning guide says the same: reasoning tokens are not visible through the API, still occupy context, and are billed as output; usage reports them as reasoning_tokens.
You usually do not see the raw chain. Anthropic returns a summary of the reasoning when you ask for it, and charges for the full thinking tokens generated, not the summary tokens, so the billed output count does not match the count you can see. OpenAI exposes optional reasoning summaries.
A cap can be consumed before any answer appears. OpenAI's guide notes that when max_output_tokens is reached the response can come back incomplete before any visible output, and recommends reserving generous room for reasoning and output when starting out. If max_tokens is small, the model can spend all of it thinking.

More thinking helps, with diminishing returns, and it is a different axis from a bigger model. Snell et al. ("Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters", 2024) studied spending extra compute at inference rather than in training, and found that the best strategy depends on the difficulty of the prompt, which motivates allocating compute per prompt. In their FLOPs-matched comparison, extra test-time compute let a smaller model beat a much larger one on problems where the smaller model already had a non-trivial success rate, but not on the hardest ones, within the models and tasks they studied. That per-difficulty dependence is the reason to route by task.

There are two ways to spend inference compute. Longer chains (higher effort) let a single attempt check and repair itself. Sample and vote runs several independent attempts and takes the majority. With an assumed single-attempt accuracy of 71 percent and independent errors, a majority of five is correct with probability P(at least 3 of 5) = 0.85. That costs five times the tokens: 5 x 1,200 = 6,000 output tokens, $0.06 at an assumed $10 per million output tokens, against 0.71 for $0.012. Independence is optimistic, since a hard problem is hard for every sample, and voting needs answers that can be compared (a number or a label, not an essay). Voting fits classification and maths with a single answer; longer chains fit open-ended planning and code. The inference mechanics behind these costs are in TTFT, TPOT, and the Latency Budget.

The control surface: effort, token caps and routing

Effort is the primary dial, max_tokens is the safety cap, and the old "thinking budget" is gone from current Claude models. The vendor controls are named differently and change often, so keep the names in one dated place.

:::key Verified October 2026 Anthropic effort docs (retrieved October 2026): set output_config.effort to low, medium, high, xhigh or max (not every model offers every level). The default is high on most models and medium on Claude Opus 5.5 and Claude Haiku 5.5. Effort affects all output tokens, including text, tool calls and thinking, and is "a behavioral signal, not a strict token budget"; lower effort also means fewer, terser tool calls. Anthropic's thinking page: adaptive thinking is on by default on the listed current models, and the older thinking: {type: "enabled", budget_tokens: ...} form returns a 400 on Opus 5.5, Sonnet 5.5, Haiku 5.5, Fable 5.1 and Mythos 5.1 (it remains, deprecated, on some older models). Thinking text is returned only if you opt in with display: "summarized"; the default on the newest models is "omitted", which Anthropic says gives a faster time to first text token when streaming. Anthropic recommends a fresh effort sweep on your own evals when changing models and a large max_tokens at high settings. OpenAI's reasoning guide (retrieved October 2026): reasoning.effort takes model-dependent values from none to max; defaults vary by model; max_output_tokens bounds reasoning plus visible output; OpenAI suggests reserving at least 25,000 tokens when starting out. Names, defaults and model lists change with every release; re-read these pages before relying on them. :::

Three practical rules follow.

Lower effort first when you want less cost or latency. Anthropic's guidance for a thinking-enabled workload is to lower effort first, since it scales the whole response down, thinking included. Because effort is a signal and not a budget, a hard question can still think a lot at low; do not use it as a hard cap.
Use max_tokens as a ceiling, not a target. Set it high enough that thinking plus the answer fit, and alert on any max_tokens or incomplete stop, which means the answer was cut or never started. A cap that is too low produces the worst outcome: you pay for the thinking and get no answer. The same stop-reason hygiene as in Structured Outputs: Schemas, Validation Loops and What 'Guaranteed' Means applies.
Route per request, not per product. Classification, extraction, chat and any fixed pipeline need little or no thinking; planning, multi-step code and hard analysis need more. A cheap first-pass router (a rule, or a small model) chooses the level, and the choice is logged with the result. With the sweep numbers in the next section, a mix of 70 percent low, 25 percent medium and 5 percent high costs 0.7 x $0.012 + 0.25 x $0.035 + 0.05 x $0.09 = $0.02165 per task, about $21.65 per 1,000 tasks, against $35 for all-medium and $90 for all-high, provided the router does not misroute hard items to low. Model Selection: Cost, Latency, and Capability Trade-offs treats effort as one of the dials; this lesson owns the measurement.

Prompting a model that thinks: goals, constraints and checks, not scripts

Prompts that script the reasoning were written for models that could not reason, and on a model that can they are at best wasted tokens. Anthropic's guidance for its current models says to prefer general instructions over prescriptive steps ("think thoroughly" often beats a hand-written plan), because the model's reasoning frequently exceeds what a person would prescribe, and to rely on thinking, not manual chain-of-thought prompting, on its newest models. OpenAI's guide likewise asks for the task, constraints and desired output (and, for agentic work, what counts as done and how to verify), and warns against prescribing every intermediate step. Three changes to an old prompt:

Delete "think step by step" and numbered procedures unless a step is a real business rule (apply the discount before tax, because that is the law). Keep procedure that is policy; drop procedure that is cognition.
State the goal and the constraints once, clearly. "Produce a staffing plan for 40 agents over 3 shifts. Hard constraints: no agent works more than 9 hours or two shifts; every shift has at least 8 agents. Optimise for the lowest overtime." The model decides the search strategy.
Add verification criteria. "Before answering, check that every constraint holds and report any you could not satisfy." A model that can think can use a checklist; without one it often stops at the first plausible plan.

Why did our old prompt get worse on the new model? There are four candidate causes, in the order I would test them: the step list now constrains a better process; emphatic wording ("ALWAYS", "NEVER") now over-triggers because the model follows instructions more closely; examples teach a format or an approach the new model would improve on; and the effort default differs (a model that defaults to medium instead of high thinks less for the same prompt, which can look like a regression). Run the regression set at two effort levels before editing the text, and ablate rules one at a time, the method in The Prompt Lifecycle: Versioning, Regression Tests and Automatic Optimisation.

Keep format examples, and keep the reasoning out of the schema. Worked examples still shape how a thinking model approaches similar problems, according to Anthropic's guide, so a few diverse examples remain useful. But a reasoning field in your output schema now asks the model to write its thinking a second time, as billed output; remove it on a thinking model. One further caution from Anthropic's tool-definition guidance: a tool parameter that asks the model for its step-by-step reasoning can trigger a refusal, so ask for a short explanation or evidence field instead.

Think step by step. First list all agents. Then list all shifts. Then for each agent
consider every shift... ALWAYS double-check. Output your reasoning in "reasoning".

Produce a 3-shift staffing plan for 40 agents.
Hard constraints: <= 9 h per agent per day, <= 2 shifts, >= 8 agents per shift.
Goal: minimise total overtime hours.
Before you answer, verify every constraint and list any you could not meet.
Return JSON: {"plan": [...], "unmet_constraints": [...], "overtime_hours": number}

The worked example: an effort sweep, cost per correct answer, and where the knee is

Choose the effort level from a sweep on your own eval, and judge each step up by what it costs per extra correct answer. Run the same 200-item set at three effort levels. The results below are an assumed example (round numbers I chose to show the arithmetic, not a measured sweep); output price is an assumed $10 per million tokens, and input cost is ignored to keep the comparison to thinking spend.

Effort	Accuracy	Correct of 200	Avg output tokens	Cost per task	Cost per correct answer
low	71%	142	1,200	$0.012	$0.0169
medium	79%	158	3,500	$0.035	$0.0443
high	82%	164	9,000	$0.090	$0.1098

The cost per task is tokens x $10 / 1e6 (1,200 tokens = $0.012). Cost per correct answer is cost per task divided by accuracy: 0.012 / 0.71 = $0.0169, 0.035 / 0.79 = $0.0443, 0.090 / 0.82 = $0.1098. That ratio rises steadily, but the better question is the marginal cost of each extra correct answer. Going from low to medium adds $0.023 per task and 8 points of accuracy, so each additional correct answer costs 0.023 / 0.08 = $0.29. Going from medium to high adds $0.055 per task for 3 points, so each additional correct answer costs 0.055 / 0.03 = $1.83, about six times as much. On 1,000 tasks a day, high over medium is an extra $55 a day for about 30 more right answers.

Test whether the gains are real before pricing them. With 200 items, a 3-point gap is 6 more correct answers. Pair the runs: suppose that between medium and high, 12 items flip to right and 6 to wrong (net +6). The exact two-sided sign test on 12 versus 6 gives p of about 0.24, so the data do not distinguish high from medium; the 3 points may be noise. Between low and medium, suppose 20 flip up and 4 down (net +16): p of about 0.0015, which is strong evidence for a real gain. The Wilson 95 percent intervals tell the same story, roughly 73 to 84 percent for medium and 76 to 87 percent for high, heavily overlapping. These flip counts are assumptions for the example; with your data, compute them. The medium level is the knee in this example: the last real gain is bought there, and the next step costs more than it can be shown to return. If high-stakes items matter (a wrong plan costs a customer), you may still pay for high on just those tasks, via routing.

Latency follows tokens. At an assumed decode speed of 80 tokens per second, 1,200 tokens takes 15 s, 3,500 takes 43.75 s and 9,000 takes 112.5 s of generation, and a 90-second request is roughly 7,200 thinking-plus-answer tokens. The cost view should include the whole task: if a higher effort halves the number of failed attempts a retry loop would need, it can be cheaper per completed task even though it costs more per request. Evaluation Foundations: Golden Sets, Error Analysis and Reading Benchmarks and Metrics That Mean Something: Task Metrics, Ranking Metrics, pass@k and Significance cover the set and the test.

Latency, progress and when not to use a reasoning model

A reasoning request has a long silent start, so design the interface for it. Time to first visible token now includes the thinking time. A 90-second request with nothing on the screen is indistinguishable from a hang, and users retry, which doubles the spend. Four options, in rising effort:

Stream whatever arrives. The visible answer starts when thinking ends; on Anthropic's API you can request summarised thinking and stream it as it is produced (display: "summarized"), and Anthropic documents a beta display: "updates" mode that returns short progress updates between tool calls while the reasoning blocks stay empty (it requires the header thinking-display-updates-2026-08-18 and may change). Summaries are produced by a separate model and their behaviour is documented as subject to change, so show them as "progress", not as the model's literal reasoning. Setting display: "omitted" drops thinking text from the stream, and Anthropic says that makes the first text token arrive sooner.
Show your own progress. In an agent, your loop knows what is happening ("searching 3 sources", "checking constraints"); Streaming Interfaces That Show Their Work has the patterns.
Move it off the request path. Run long jobs asynchronously and notify; a user waiting 90 seconds for a plan is a bad UI, and a user told "ready in about two minutes, we will message you" is fine.
Cap the wait. Set an application timeout shorter than the infrastructure's, and fall back to a lower effort level with a visible note when it trips.

Measure the tail, not the mean: difficulty varies, so p95 at high effort is far above the average, and the TTFT, TPOT, and the Latency Budget arithmetic applies with thinking tokens added to the output length.

When not to use a reasoning model. The cost and the latency both scale with thinking you may not need. Skip it, or set the lowest level, for: extraction with a fixed schema, classification into a handful of labels, formatting and rewriting, retrieval-grounded answers that need one lookup, and any step in a pipeline whose logic you have already coded. These tasks are bound by the information in the input, not by reasoning depth. The test is empirical and cheap: run the same eval at two effort levels, and if the scores are equal within a paired test, the lower level wins on cost and speed. Anthropic's effort page, in its guidance for Claude Opus 4.7, also warns that max can lead to overthinking on some structured-output tasks, another reason to measure instead of defaulting up.

:::warn Do not read the visible reasoning text as an explanation of the answer. The summaries are generated by a separate process, can be omitted, and may not reflect every step that produced the result. For audit, log inputs, tool calls, outputs and checks, not the thinking. :::

How to say it in the interview

"A reasoning model spends output tokens thinking before it answers, and those tokens are billed as output and add latency whether or not I can see them. So I treat effort as a dial and pick the level from a sweep on my own eval: same set, three levels, accuracy and tokens at each, then cost per correct answer and the marginal cost of each extra correct answer, with a paired test on the flips so I do not buy three points of noise. I route by task: low for classification, extraction and chat, higher for planning and code. I prompt for the goal, the constraints and how to check the result, and I delete my old step lists and think-step-by-step lines, because they were written for models that could not think. For a 90-second request I stream progress and a summary instead of a silent wait, and I set max_tokens high enough that thinking does not eat the answer."

Checklist

Cost is computed from output tokens including thinking, and max_tokens leaves room for thinking plus the answer
Effort level is chosen from a sweep on our own eval, not copied from the previous model or a blog
Each step up in effort is judged by marginal cost per extra correct answer, with a paired test on the flips
Requests are routed by task type and the level is logged with the result
Old prompts have their step lists, think-step-by-step lines and emphatic rules ablated on the new model
Prompts state goal, hard constraints and how to verify, and the schema has no duplicate reasoning field
Alerts fire on max_tokens or incomplete stops
Interfaces for long requests stream progress or run asynchronously, with a timeout and a lower-effort fallback
Visible thinking summaries are not stored or shown as the model's literal reasoning
Vendor parameter names and defaults live in one dated place in the code and docs

Practice this interactively