Cruxly › AI engineering › LLM Backend Infrastructure

Cost Attribution and Runaway Prevention

tag before you send, cap before you regret it

Prompt: design cost tracking for an API that calls foundation models on behalf of multiple tenants and features, such that a runaway cost event is caught within minutes, not discovered on next month's invoice — and such that when costs spike, you can answer exactly which tenant, feature, and code path caused it.

40 minutes first. The naive version tracks total spend. That number alone answers nothing when it doubles overnight.

Attribute before you send, not after

Every outbound LLM call carries structured metadata attached at the call site: tenant ID, feature/endpoint name, and a request purpose (user- facing query, background batch job, internal tooling). This metadata has to be attached before the call is made — a provider's response and billing data have no knowledge of your internal tenant or feature structure, so if you don't tag it going out, you cannot reconstruct it coming back.

Token counts, not just dollar cost, belong in the log. Providers change pricing; token counts don't retroactively change. Logging raw token counts alongside the cost-at-time-of-request lets you recompute historical spend under a new pricing model and separate "we used more tokens" from "the price per token went up." And "input vs. output" is no longer enough buckets — log each one the provider bills differently:

• Uncached input tokens — full input price.
• Cache-write input tokens — often priced above normal input.
• Cache-read input tokens — often a small fraction of normal input.
• Output tokens.
• Reasoning/thinking tokens — billed as output but invisible in the response text, and frequently the majority of output on reasoning models.
• Pricing tier — batch (often about half price) vs. synchronous.

Cost attribution built on input/output totals alone can be wrong by multiples on cache-heavy or reasoning-heavy traffic.

A dashboard that can answer "which tenant, which feature, which hour" in one query is the actual deliverable — not a total spend number on a dashboard. Structure the logging schema (tenant, feature, timestamp, token counts, cost, model, provider) so that query is a straightforward filter, not a data archaeology project.

Budget caps that degrade gracefully instead of failing hard

A hard cutoff ("tenant exceeds budget, all requests fail") is usually the wrong default, because it turns a cost problem into a total outage for that tenant at the worst possible moment. Graceful degradation — route to a cheaper model, reduce max output tokens, or queue non-urgent requests instead of serving them immediately — preserves basic functionality while cost pressure is addressed, and is almost always the better product decision than an outright failure.

Set budget alerts at multiple thresholds (50%, 80%, 100% of a period's allocation), each triggering a different response. 50% might be silent logging; 80% might page an on-call engineer to investigate the trend; 100% triggers the graceful-degradation path. A single "you're over budget" alert at 100% gives no advance warning to intervene before the degradation kicks in.

Before degrading, pull the two cost levers that cost nothing in quality.

• Prompt caching. Most requests repeat a long static prefix — system prompt, tool definitions, shared documents. Provider prompt caching bills those cached tokens at a fraction of the normal input price. Put static content first so the prefix is actually reusable.
• Batch APIs for anything asynchronous. Nightly summaries, bulk classification and backfills don't need an immediate answer; major providers' batch endpoints run them at roughly half price. This is the "async by default" principle with a price tag attached.

Distinguish a runaway from expected growth. A tenant's costs doubling because their user base doubled is success; a tenant's costs doubling because a retry loop is silently re-sending the same failed request is an incident. Track cost-per-successful-request as a separate metric from total cost — a spike in total cost with a flat or falling cost-per-request usually means legitimate growth; a spike in cost-per-request means something is retrying, looping, or sending unnecessarily large prompts.

Finding the runaway: the debugging path that actually works

A retry loop is the most common runaway cause, and it's invisible in aggregate metrics until it's expensive. A request that fails validation, gets retried with the same malformed input, fails again, and retries again looks like normal traffic in a total-requests-per-minute graph. Log a retry-attempt-number field on every request, and alert specifically on a rising rate of high-attempt-number requests — that pattern is what a retry loop's fingerprint actually looks like.

Prompt bloat is the second most common cause, and it grows silently. A prompt template that concatenates growing conversation history, retrieved context, and system instructions can grow token count per request over time without any single code change being the obvious cause — it's death by a thousand small additions. Track prompt token count as its own time-series metric, not just total cost, so a slow upward creep is visible before it becomes a step-change in the monthly bill.

A single bad actor (one endpoint, one code path, one buggy integration) usually accounts for a disproportionate share of an unexpected spike, and the fastest path to root cause is always "sort by cost, group by endpoint," not staring at a total-spend line graph. Build that specific query as a first-class, one-click dashboard view — the query you'll run during every actual incident should not require writing SQL under pressure.

The trade-offs worth naming out loud

Attribution overhead is real but small, and worth stating precisely. Attaching metadata to every request adds negligible latency but real engineering discipline — every call site needs to remember to tag correctly, which is why a shared client wrapper that forces tagging (the metadata fields are required parameters, not optional ones) beats a convention that's easy to forget in a new code path.

Graceful degradation has a UX cost that needs product buy-in, not just engineering approval. Silently routing a tenant to a cheaper, possibly lower-quality model when they hit a budget threshold is a product decision with a real user-facing impact, and it should be visible to the tenant (a status indicator, a notification) rather than a silent quality downgrade discovered by the user noticing worse answers.

Real-time cost tracking versus batch reconciliation is a genuine trade-off. Real-time tracking (computing cost the moment a response comes back) enables fast alerting but can be wrong if a provider's actual billing differs from the estimate (rounding, discounts, committed-use pricing). Batch reconciliation against the provider's actual invoice, run daily, is the source of truth for finance even while real-time estimates drive operational alerting — running both, and knowing which one to trust for which purpose, is the mature answer.

Structured output: constrained decoding is enforcement, not parsing-and-retry

The naive mental model — ask the model for JSON, parse the response, retry if parsing fails — is not what a well-integrated stack does anymore, and knowing why matters for anyone about to design a backend around structured output. Constrained (grammar-based) decoding enforces schema validity at the token level: a JSON Schema or grammar is compiled into an automaton before generation starts, and at every single decode step, every token that would violate the schema — closing a string that was never opened, emitting a key not in the schema, a type mismatch — has its logit masked to negative infinity before sampling even happens. (Recursive JSON Schema needs a stack, so engines such as XGrammar compile a byte-level pushdown automaton, not a plain finite-state machine.) The model is mechanically prevented from producing an invalid token.

Know exactly what that guarantees, and what it doesn't:

• It guarantees every token emitted so far is a valid prefix — no "almost valid JSON" mid-stream.
• It does not guarantee a complete document. If generation hits max_tokens mid-object, you get a valid prefix that isn't parseable JSON. Check the stop reason and handle truncation.
• It never guarantees semantic validity. The refund_amount field will be a number; whether it's the right number is still your validation layer's job.

XGrammar is the default structured-generation backend for most LLM inference engines (per its README, read 2026-10-22), including vLLM, SGLang, TensorRT-LLM and MLC-LLM — worth cross-referencing against the serving-stack material elsewhere in this track, since this is the same inference stack, wearing a different hat. Its README describes the overhead for JSON generation as "near-zero"; the per-token mask cost is on the order of tens of microseconds and depends on the engine, the grammar and the batch size, so measure it on your own stack rather than quoting a figure. The follow-up XGrammar-2 (arXiv 2601.04426, January 2026, aimed at agentic workloads with dynamic schemas) reports over 6× faster grammar compilation than prior engines. All of which means the old objection — "structured output is too slow to use in production" — is largely outdated for a stack that integrates it properly. The grammar-compile step happens once per schema (and is cached), not once per token, which is what makes the per-token check cheap enough to not matter at typical batch sizes.

The real trade-off: a quality tax that isn't uniform, and a latency cost at high batch size

Constrained decoding measurably taxes model quality, not just latency — sometimes. The evidence is genuinely contested: some studies (e.g., "Let Me Speak Freely?", 2024) reported reasoning-quality drops under strict format constraints, and follow-up work attributed much of that to poor prompts and schema design rather than to constrained decoding itself. Don't quote a single universal percentage; the effect is task-dependent and largely mitigable. Masking schema-invalid tokens does narrow the sampling distribution, so measure it on your own task.

The mitigations are schema design, and they're worth naming:

• Put a reasoning field before the answer fields. Keys are generated in order, so a "reasoning" key first lets the model think before it commits to "label".
• Make honest answers schema-legal. Nullable fields or explicit "unknown" enum values remove the failure mode in the next paragraph.
• Order keys deliberately — fields the model should decide later go later.

The tax is not uniform across model capability, and the failure mode on a weak model is worse than "slightly lower quality." A strong model absorbs the constraint with a small hit — it can still find a schema-legal continuation that's also a genuinely good answer. A weaker model, forced into the same schema-legal subspace, can collapse toward whichever schema-legal continuation has the lowest perplexity — which means it will confidently fill a required field with plausible-looking nonsense rather than leave it empty, because leaving it empty isn't schema-legal. This is a specific, nameable failure mode distinct from ordinary hallucination: the model isn't guessing because it lacks information, it's guessing because the schema forbids the honest answer ("I don't know" or an empty field) and something schema-shaped has to come out.

There can be a real latency cost at high batch sizes, so benchmark it on your own stack rather than quoting a number. The mask computation is per-sequence CPU-side bookkeeping — every sequence in the batch needs its own automaton state advanced and its own logits masked — and that doesn't amortize across the batch the way matrix-multiply-heavy work does. Engines overlap it with GPU compute to hide it, with varying success depending on the engine version, schema complexity and batch size. This is the detail worth having ready for anyone about to put structured output behind a high-throughput serving endpoint: the "XGrammar is basically free" framing from the previous section holds at typical request-level latency, and stops holding once you're optimizing for maximum throughput at large batch sizes.

State the actual recommendation, because "always use structured output" and "never use it" are both wrong. Use constrained decoding by default for anything a downstream system parses — the syntactic guarantee is real and valuable, and the per-token cost is negligible at normal request volume. Watch the quality tax specifically when the task is open-ended content generation dressed up in a schema, not pure data extraction, and benchmark the high-batch latency cost explicitly if the endpoint is being sized for high-throughput serving rather than typical request-level latency.

How to say it in the interview

"An aggregate spend number is useless for debugging - it tells you something's wrong but not what. Every request needs to be tagged with tenant, feature, and endpoint BEFORE it's sent, because reconstructing that context from provider logs afterward is often impossible - the provider doesn't know or care about your internal structure."

Checklist

• Tagged every outbound request with tenant, feature, and purpose before sending
• Logged raw token counts per billing bucket — uncached, cache-write, cache-read, output, reasoning — separately from computed cost
• Used prompt caching and batch APIs before degrading quality
• Chose graceful degradation over a hard cutoff as the default budget response
• Set alerts at multiple budget thresholds with different responses at each
• Tracked cost-per-successful-request separately from total cost
• Logged retry-attempt-number and alerted on its rising rate
• Tracked prompt token count as its own metric, not folded into total cost
• Built a one-click 'sort by cost, group by endpoint' view as a first-class dashboard
• Explained constrained decoding as token-level logit masking against a compiled grammar, and its limits
• Named XGrammar's sub-40-microsecond per-token overhead in vLLM/SGLang/TensorRT-LLM as why the speed objection is outdated
• Treated the quality effect of format constraints as task-dependent, and mitigated it with schema design
• Identified that a weaker model under constrained decoding fills required fields with confident nonsense rather than leaving them empty
• Benchmarked constrained-decoding latency at the target batch size before sizing a high-throughput endpoint

Related lessons

• Multi-Provider Failover That Actually Fails Over
• Prompt Caching: Paying for the Same Tokens Once