Cruxly

Cruxly › AI engineering › Agentic Systems

Guardrails for an Agent That Takes Real Actions

tool contracts, approval gates, and the line between automated and autonomous

Open the interactive lesson

Prompt: design an agent that can resolve customer support tickets by querying order status, issuing refunds up to a limit, and escalating to a human — using LangGraph-style stateful orchestration, with tool access scoped tightly enough that a reasoning error can't become a financial incident.

40 minutes first. The reasoning loop is the part everyone designs first and the part that matters least here.

Tools are the actual security boundary

Every tool the agent can call needs the same rigor as a public API endpoint, because from the agent's reasoning process, that's exactly what it is — an interface it invokes based on a probabilistic judgment that can be wrong. A refund(order_id, amount) tool needs input validation, an amount ceiling, idempotency (the same refund request, retried by a confused agent loop, must not double-refund), and its own audit log entry — independent of whatever logging the agent's overall reasoning trace has.

Scope each tool to the minimum capability that accomplishes its purpose. A tool that queries order status should not, as a side effect, also be capable of modifying that order — even if the same backend function technically supports both, expose them as two separate tools with separate permission levels. The narrower each tool's capability, the smaller the blast radius of the agent invoking the wrong one.

A tool's failure mode matters as much as its success path. What happens when refund() times out — does the agent retry (risking a double refund if the first attempt actually succeeded server-side), or does it report uncertainty upward? Design every tool's contract to make the failure case safe by default, the same idempotency-key discipline used for payment APIs elsewhere in production system design, applied here because an agent retrying a failed tool call is a very real and very common failure mode.

Derive the idempotency key outside the model. If the LLM is asked to generate the key, a retry after a timeout can easily mint a fresh key — and the guard sees two different requests. Build it in code from stable facts: hash(ticket_id, "refund", order_id, amount). Then the same logical action always carries the same key, however many times the agent retries.

The tool definition is the interface the model actually reads, so design it like one.

Approval gates: enforced in code, not in the prompt

"Ask for approval before refunds over ₹1,000" as a system-prompt instruction is a suggestion the model can fail to follow, not a guarantee. The gate has to be enforced at the tool layer: the refund() tool itself checks the amount and, above the threshold, does not execute — it instead returns a "pending approval" state and creates an approval request, no matter what the agent's reasoning concluded. The model can be persuaded, confused, or simply wrong; the code path enforcing the gate cannot be talked out of its job.

An approval gate needs a real human-facing interface with enough context to actually decide, not just a bare "approve/deny" button. The approver needs to see what the agent is trying to do and the evidence that bears on it — the order, the customer's history, the delivery record, the exact ticket text. Without that, an approval gate becomes rubber-stamping, which provides the appearance of oversight without the substance of it.

Escalation timeouts need a defined default, and it should fail toward the safer option. If a human doesn't respond to an approval request within a set window, does the action proceed, or does it stay blocked? For anything involving money leaving the business, the default should almost always be "stays blocked" — an unapproved action that doesn't happen is recoverable; one that happens without approval, even by timeout default, may not be.

State, memory, and the checkpoint that lets you stop an agent mid-task

An agent's state needs to be externally inspectable and interruptible, not held only in an in-process reasoning loop. If an agent is mid-way through a multi-step task (queried the order, decided to refund, about to call the refund tool) and needs to be paused — because a human flagged a concern, because the system needs to restart — the state machine's current position needs to be a real, persisted checkpoint, not something lost the moment the process restarts.

Checkpointing every state transition also gives you a debugging trace for free — the same mechanism that lets you pause and resume an agent lets you reconstruct, after the fact, the sequence of states it moved through.

Memory across a conversation needs an explicit boundary on what persists and for how long. An agent remembering context from a previous turn in the same conversation is expected; an agent's memory silently leaking context across different customers' conversations, or retaining sensitive information (a customer's card details mentioned in passing) indefinitely, is a data governance failure hiding inside a feature that looks purely like good UX.

Multi-agent coordination: the supervisor pattern, and why it beats a free-for-all

A single agent trying to do intent classification, order lookup, refund processing, and escalation all in one reasoning loop accumulates tool access and responsibility that's hard to scope safely. The supervisor pattern splits this: a lightweight supervisor agent classifies intent and routes to a specialized sub-agent (a refund agent, an order-status agent, an escalation agent), each with only the tools its specific job requires. This is the same principle as least-privilege access control, applied to agent architecture rather than to a user permission system.

Before splitting into agents at all, ask whether you need an agent. The pattern that most often wins in production is the simplest one that works (Anthropic's "Building Effective Agents"): a fixed workflow — classify, then call a known sequence of tools — where the steps are predictable, and an open-ended agent loop only where they genuinely aren't. Multi-agent is a further step up in cost and failure modes, and the counter-argument (Cognition, 2025) is that splitting a task across agents loses shared context — especially harmful when sub-agents make decisions that must stay consistent with each other.

And be honest about how much security the split buys. The supervisor's routing is itself an LLM decision that can be wrong or manipulated, so the refund agent's tool gates are still the real boundary — a split architecture with weak tool gates is not safe, and a single agent with strong tool gates often is. The split is mainly about keeping each agent's context and tool list small and focused.

Sub-agents should not be able to invoke each other directly, only through the supervisor. Direct agent-to-agent calls without a coordinating layer make the overall system's behavior much harder to trace and gate — every action needs to be attributable to a decision the supervisor made, not an emergent interaction between two sub-agents neither of which the supervisor explicitly authorized.

The supervisor itself needs the tightest guardrails of anything in the system, since it's the single point that decides which sub-agent — and therefore which tool surface — gets invoked at all. A supervisor that misclassifies intent and routes a refund request to an agent scoped for read-only order lookups fails safely (nothing happens); a supervisor that misclassifies in the other direction and routes a simple status query to the refund agent is a much more dangerous failure mode, because it puts a low-stakes interaction in front of a high-stakes tool.

What multi-agent coordination actually costs, and the failure mode that eats the savings

Multi-agent systems cost several times what a single agent costs, and you should quote a sourced ratio, not a made-up dollar figure. The citable numbers come from Anthropic's 2025 write-up of its multi-agent research system:

Then derive the dollar figure live instead of memorising one. A monthly cost without request volume is meaningless:

monthly cost = tasks/month × tokens/task × price per token
             = 50,000 tasks × 200K tokens × $2 per 1M tokens = $20,000

:::key Verified October 2026 The $2 per million input tokens is Claude Sonnet 5.5's list price on Anthropic's pricing page, read 31 October 2026, and the example counts every token at that input rate for simplicity (output tokens cost more: $10 per million). It is an illustration of the formula, not a quote; swap in the current rate for your model. :::

Change any input — volume, tokens per task, model tier, cache hit rate — and the answer moves by multiples. Say the formula, plug in the interviewer's numbers, and name the ratio (multi-agent ≈ 3-4× single-agent tokens) as the multiplier on top.

PatternToken cost vs. single agentShape
Orchestrator-worker (bounded)Lower end — workers return short summariesOne delegates; workers report back
Broad parallel fan-outHigher end — every worker carries a full contextMany agents explore in parallel

The infinite handoff loop is a classic coordination failure, and it is a coordination bug, not a reasoning bug. Agent A hands a task to B, B hands it to C, and C hands it back to A — and because nobody was ever assigned actual ownership of the task, each agent re-plans from scratch on receipt rather than continuing the others' work. Context degrades further with every hop (each agent sees a shrinking, re-summarized slice of what came before), so the loop doesn't just fail to terminate — it gets less capable of resolving itself the longer it runs. The fix is architectural, not a smarter prompt: a designated owner for the task that handoffs report back to rather than through, exactly the supervisor shape from the previous section, is what a handoff chain without an owner is missing.

The MAST taxonomy and why reliability compounds against a long chain

MAST (Cemri et al., "Why Do Multi-Agent LLM Systems Fail?", 2025) classifies failures from traces of several open-source multi-agent frameworks into three categories. Use the paper's own names — a paraphrase is the tell that you only read a summary:

Name the fine-grained modes, not just the buckets. The frequent ones — step repetition, disobeying the task specification, premature termination, incorrect verification — are more useful in a design discussion than the category labels, because each one maps to a concrete guard: a step budget, a spec checker, a completion criterion, an independent verifier. And the distribution is not even, so don't claim "roughly equal measure" — the specification/design category dominates.

Reliability compounds multiplicatively across a chain, which is the concrete argument for keeping chains short and adding verification checkpoints rather than trusting a chain of any length. A chain of five steps, each individually 95% reliable, is not 95% reliable end to end — it's 0.95^5, roughly 77%. That gap between "each step looks fine" and "the chain looks fine" is exactly what a verification-gap failure hides: without an independent check at some point in the chain, the system has no way to notice it's already operating at 77% actual reliability while every individual agent reports success.

Evaluating an agent: outcomes, trajectories, and pass^k

An agent eval is not a chat eval with more steps. The agent acts in an environment, so the eval has to run it in one — a sandboxed copy of the order system, a fake payments API, a scripted customer — and check what actually changed, not just what the agent said.

Grade two things, separately:

Reliability is the metric that matters for an agent, and pass@k measures the wrong thing.

Build the eval set from real tasks and real failures, then gate on it. Script a few hundred representative tickets with known correct end states, including the edge cases — partial refunds, duplicate requests, a customer who changes their mind mid-conversation, an injection attempt in the ticket text. Run each several times (agents are stochastic), report pass^k and the trajectory-violation rate, and add every production incident as a new case. If any grading uses an LLM judge — "was the reply polite and accurate?" — calibrate it against human labels before trusting it, as in the RAG evaluation lesson.

How to say it in the interview

"The agent's reasoning can be wrong sometimes - that's expected of any system built on an LLM. The design question is: when it's wrong, what's the blast radius? That question is answered entirely by the tools it has access to and the limits enforced on each one, not by how good the reasoning is."

Checklist

Practice this interactively
Share this lessonWhatsAppLinkedInX

Related lessons