Cruxly

Cruxly › AI engineering › Reliability, Security & Governance

Observability and Cost Governance for LLM Systems

Langfuse-style tracing, semantic caching, and the alert that catches drift

Open the interactive lesson

Prompt: design the observability and cost-control layer for a production LLM-backed system: full request tracing, semantic caching for repetitive queries, and intelligent model routing that degrades gracefully under budget pressure rather than failing outright.

40 minutes first. The traditional metrics (latency, error rate) are necessary here but nowhere near sufficient.

Tracing a probabilistic system: what a normal APM trace is missing

A full trace needs the prompt, the retrieved context (if any), the raw model response, and the final processed output — not just a span duration. Standard application performance monitoring traces show you that a call happened and how long it took; debugging why an AI system produced a bad output requires seeing exactly what went into the model and what came out, end to end, for that specific request.

Sample and store full traces for a percentage of traffic, always, not only when an error is detected. A traditional system's happy path usually doesn't need deep tracing because deterministic code that passed its tests keeps behaving the same way. An LLM's "happy path" can quietly degrade in quality without throwing any error at all — the only way to catch that is having traces to review, including from requests that technically succeeded.

Group and analyze traces by prompt template version, model version, and retrieval strategy version — not just by endpoint. When quality changes, the actual question is "did it change after the last prompt edit, the last model upgrade, or the last retrieval change," and answering that requires every trace to carry those version tags, not just a timestamp to eyeball against a deploy log.

Use the emerging standard instead of a homegrown trace format. OpenTelemetry's GenAI semantic conventions define standard attributes for model calls — model name, token counts, operation, provider — so traces from different frameworks and vendors land in the same backend and can be queried the same way.

Pin dated model snapshots, not floating aliases. A model name like "latest" can be repointed by the provider without notice, which turns "the model changed under us" into an undebuggable incident. Call a dated snapshot, record it on every trace, and upgrade deliberately behind an eval gate.

Semantic caching: real savings, real correctness risk

A semantic cache serves a stored response when a new query is similar enough (by embedding distance) to a previously answered one — and "similar enough" is a threshold that trades cost savings against correctness risk directly. A loose threshold catches more cache hits (more savings) and also more false matches (queries that are similar in wording but different in intent, served a wrong cached answer). A tight threshold is safer and saves less. This trade-off should be tuned deliberately per use case, not left at a library's default.

Cache invalidation needs a real trigger, not just a TTL. If the underlying data a cached answer depends on changes (an order's status updates, a policy changes), a time-based expiry alone means the cache can serve a stale answer for the entire TTL window even though the correct answer changed the instant the underlying data did. Where possible, tie cache invalidation to the actual data-change event, not just to elapsed time.

Never semantically cache anything where a wrong-but-plausible answer has real consequences. A semantic cache for "what are your business hours" queries is low-risk — a near-miss is harmless. A semantic cache for "what's the status of my specific order" is high-risk — a similar-looking but different order number returning a cached answer for the wrong order is a real, specific, harmful error. Scope semantic caching to genuinely low-stakes, repetitive query classes, and use exact-match caching (or no caching) elsewhere.

Scope the cache key by tenant and, where answers are personal, by user. A semantic cache shared across tenants can hand one customer's cached answer — built from their private documents — to another customer whose question happened to look similar. That's a data leak, which is worse than a near-miss answer. Include tenant (and user, where relevant) in the cache key, so similarity is only ever computed within one isolation boundary.

And check the bigger lever first: provider prompt caching. Caching the long static prefix (system prompt, instructions, retrieved documents shared across requests) is supported natively by major providers, carries no correctness risk because the full request is still processed, and on long-prompt workloads usually saves more than a semantic cache does.

Intelligent routing and graceful degradation under cost pressure

Route requests to different models based on complexity, not uniformly to the most capable (and most expensive) model for everything. A classification step — even a cheap, fast one — that routes simple queries to a smaller/cheaper model and reserves the most capable model for genuinely complex requests can cut costs substantially without a proportional quality loss, because most real traffic is not uniformly difficult.

Design the routing decision to be inspectable and correctable, not a black box. If a routing model consistently misclassifies a specific query pattern as "simple" when it needs the more capable model, that needs to be discoverable from monitoring (quality metrics broken down by which model handled the request) and fixable without redeploying the entire system.

Budget-driven degradation should be layered, not binary. Under increasing cost pressure: first reduce non-critical background processing, then route more aggressively to cheaper models, then cap response length, and only as a last resort, queue or delay non-urgent requests. A single binary "on budget / over budget" switch skips the layered response that actually preserves the most user value per rupee of remaining budget.

The alert that catches drift, and the one that catches nothing useful

Alert on trend changes in quality-proxy metrics (faithfulness scores from continuous evaluation, user-reported issue rate, response length distribution), not only on hard failures. A model behaving worse but still returning syntactically valid, non-error responses produces zero traditional alerts while genuinely degrading the product — quality-proxy metrics are the only signal that catches this class of drift at all.

A cost-per-request alert threshold set once and never revisited becomes useless as the system evolves. As legitimate usage patterns shift (more complex queries, a new feature with different token needs), a static threshold either fires constantly on legitimate growth or fails to catch a genuine anomaly once the "normal" baseline has moved. Recalibrate thresholds against a rolling baseline, not a fixed historical number.

The single most valuable alert to build, and the one most often skipped: a sudden change in the distribution of any key metric, not just its average. A cost-per-request average that stays flat while its variance triples means something has changed — a subset of requests getting dramatically more expensive while most stay normal — that an average alone completely hides.

How to say it in the interview

"Latency and error rate tell me if the system is up. They tell me nothing about whether the model started hallucinating more after a silent provider-side update, or whether a semantic cache started serving wrong-but-similar answers. Those need their own signals, and building only the traditional ones means being blind to the failure mode that actually matters most here."

Checklist

Practice this interactively
Share this lessonWhatsAppLinkedInX

Related lessons