Cruxly

Cruxly › AI engineering › Agentic Systems

Coding Agents and Computer-Use Agents: Harness, Sandbox and Verification

a 40-step session costs $9.84 uncached and about $1.07 cached; the sandbox, the oracle and the screenshot loop that decide whether it ships

Open the interactive lesson

Prompt: "Design a coding agent for our monorepo of 3 million lines. Engineers will assign it tickets; it should open pull requests. Tell me what tools it has, how it finds the right files, where it runs, how you know its change is correct, and what a long session costs. Then tell me how a browser agent that fills in vendor portals differs."

40 minutes first. Price a 40-step session where the context grows by 3,000 tokens a step, with and without prompt caching, and price a screenshot. The trap is describing the model and ignoring the harness: the tools, the sandbox and the oracle are what make a coding agent safe, and the token arithmetic is what makes it affordable.

The coding-agent harness: a few sharp tools and just-in-time context

A coding agent is a model in a loop with tools, and the quality of the tools matters more than the cleverness of the prompt. Look at the tool set that Claude Code exposes through the Claude Agent SDK, because it is a good reference design: file Read, Edit and Write; search with Glob (find files by pattern) and Grep (search contents by regex); Bash to run commands, tests and git; web fetch and search; and orchestration tools to spawn sub-agents, ask the user a question, and track a task list. Read-only tools can run in parallel, while tools that change state run one at a time to avoid conflicts.

Edits are diffs, not whole files. The edit tool replaces an exact old string with a new one, and fails if the old string is not unique or not found. That failure is a feature: it forces the model to have read the current file, keeps the change small enough to review, and avoids the silent corruption of regenerating 800 lines from memory. Design your own edit tool the same way: precondition on the current text, return a clear error, never "best effort".

Context is found just in time, not preloaded. A 3-million-line monorepo is perhaps 30 million tokens or more, far beyond any window and too expensive to resend even if it fit. The pattern described in Anthropic's context-engineering write-up, and used by Claude Code, is to load a small, high-signal guide up front (a project file such as CLAUDE.md with layout, commands and conventions) and let the agent explore with glob and grep, reading only what it needs. Compare the two strategies in tokens: grepping for a function name returns perhaps 20 lines (a few hundred tokens), then reading two files of 300 lines is about 6,000 tokens; the guide is a one-off 1,500. That beats any embedding index you would have to keep fresh against 500 commits a day, and it cannot return a stale chunk, because it reads the current file. It is weaker when a concept is not named in the code (the "where do we handle retries?" question), which is where an optional semantic search tool helps.

Sub-agents protect the main context. A sub-agent explores in its own window and returns only its final summary, so the main agent's context grows by a few hundred tokens instead of the whole exploration. The Agent SDK documents this directly as a way to keep a long session lean. The cost is that the summary is lossy: tell the sub-agent exactly what to return (file paths and line numbers, not prose).

Compaction handles the rest. When the window nears its limit the harness summarises older turns and keeps recent ones and key decisions. Anything the agent must not forget (acceptance criteria, files already changed, test results) belongs in the project file or a scratchpad, because summaries drop detail. The Agent SDK also lets a pre-compaction hook archive the full transcript.

:::key Verified October 2026 Tool names, parallel-versus-sequential execution, sub-agents, compaction and turn and budget caps are from the Agent SDK "How the agent loop works" page (read October 2026). The just-in-time retrieval pattern is from Anthropic's "Effective context engineering for AI agents" article. Anthropic's engineering posts on the Agent SDK and on Claude Code best practices exist but are usage guidance; I did not find a dedicated architecture write-up, so statements about internals are limited to the documentation pages above. :::

The bill: a 40-step session, uncached and cached

Each step re-sends the whole conversation, so a long session's input cost is a triangular number. Say the context grows by 3,000 tokens a step (a file read, a test output, the model's reply) over 40 steps. Step n sends about 3,000 x n input tokens.

Now add prompt caching (the mechanism is in Prompt Caching: Paying for the Same Tokens Once). The prefix of step n is the whole of step n-1, so each step writes only its 3,000 new tokens to the cache and reads everything before them.

:::key Verified October 2026 From the Anthropic prompt-caching documentation and pricing table, read October 2026: five-minute cache writes cost 1.25 times the base input price and one-hour writes 2 times. Cache reads cost 0.1 times base by default, with exceptions: 0.05 times for Claude Opus 5.5 and Sonnet 5.5, and 0.025 times for Fable 5.1 and Mythos 5.1. The worked example uses Opus 5.5 at $4 per million input ($5 write, $0.20 read). With a model at 0.1 times the arithmetic changes: for the same session at a $5 base price, uncached is 2.46 x $5 = $12.30 and cached is 0.12 x $6.25 + 2.34 x $0.50 = $0.75 + $1.17 = $1.92. Minimum cacheable prompt sizes vary by model (512 to 4,096 tokens). Prices change often; re-check before quoting. :::

Three caveats keep you honest. First, output tokens are not in this figure; they are billed separately and are linear in steps. Second, the cache has a time-to-live (five minutes by default), and each hit refreshes it; a tool that takes ten minutes to run, such as a slow test suite, can let the prefix expire so the next step re-writes it at full cost. Third, anything that changes early in the prompt, such as a timestamp in the system prompt or a re-ordered tool list, breaks the prefix for every later token, which is why the MCP revision asks for deterministic tool order.

The business consequence. If a resolved ticket needs one such session, input costs about $1.07 against $9.84. At a 60 percent resolution rate, the input cost per resolved ticket is 1.07 / 0.6 = $1.78 cached versus 9.84 / 0.6 = $16.40 uncached. Whether either is acceptable depends on what an engineer-hour costs you, which is the comparison to make aloud.

The sandbox: what a prompt cannot enforce

Anything you write in the system prompt is a request; only the environment is a rule. A coding agent runs generated shell commands on your behalf, reads files that may contain hostile text, and can be steered by a dependency README or a pull request comment. "Never run destructive commands" in the prompt lowers the odds. A container that has nothing worth destroying removes the problem.

@@ 1 | Isolation Run each task in a fresh container or microVM with a throwaway filesystem cloned from the repo at a commit. Destroy it at the end. The agent can wreck its own copy, not yours. A container shares the host kernel; if tasks run code from untrusted sources, a microVM or a gVisor-style sandbox is a stronger boundary.

@@ 2 | No standing credentials The sandbox holds no cloud keys, no deploy tokens and no write access to the main branch. It pushes to a branch via a narrow, short-lived token, or hands the diff to a service that opens the pull request. Secrets for tests are fake.

@@ 3 | Network egress allowlist Allow the package mirror and the git host; deny everything else. This stops both exfiltration (an injected instruction to curl your source to a server) and the agent downloading arbitrary tools.

@@ 4 | Resource limits CPU, memory, disk, process count and wall-clock caps. An infinite loop or a fork bomb should kill the sandbox, not a shared build machine. These pair with the turn and spend budgets from The Agent Loop in Depth.

@@ 5 | Permission tiers inside the agent Reads are free, edits inside the working tree are allowed, commands matching a deny-list or touching paths outside the tree need approval. The Agent SDK gives this as permission modes, allow and deny lists and hooks that run before each tool call, and hooks run in your process, outside the model's context, so the model cannot talk its way past them.

The unifying idea is the same as in Guardrails for an Agent That Takes Real Actions: put the control in the part of the system the model cannot edit. MCP Servers, Prompt Injection, and Tamper-Evident Audit covers the injection path; here, assume injection will sometimes work, and make sure it works inside a box with nothing valuable in it.

:::warn The sandbox is not the repo's dependency install step npm install and pip install run arbitrary code from the internet inside your sandbox. If the sandbox has credentials or open egress, a malicious package inherits them. This is the most common way "no standing credentials" gets quietly broken. :::

Verification, reward hacking and how to read a SWE-bench number

The oracle for a coding agent is a test run that the agent cannot rig. Tests give the loop ground truth, which is why coding is the most successful agent domain. But "make the tests pass" is a goal the agent can satisfy dishonestly, and it will if the honest path is hard. Reward hacking in coding looks like this: special-casing the test's inputs, editing the test to expect the buggy output, deleting a failing test, adding skip, weakening an assertion, modifying the CI config or a grader script, or catching exceptions so a failure no longer surfaces.

Defences, in order of strength.

@@ 1 | Protect the oracle Mount test directories and grader scripts read-only, or restore them from the base commit before the final run. The agent can add new tests, but it cannot alter existing ones.

@@ 2 | Re-run independently The harness runs the full suite in a clean checkout of the agent's diff. The model's report "all tests pass" is never the evidence.

@@ 3 | Review the diff for what it touched Flag any change to test files, CI configuration, lint rules, dependency pins or .skip markers, and send those to a human by default.

@@ 4 | Hold out checks the agent never saw A hidden test set or a property-based check catches solutions that fit the visible tests only.

Reading SWE-bench-style numbers. SWE-bench (Jimenez et al., arXiv 2310.06770, SWE-bench: Can Language Models Resolve Real-World GitHub Issues?) builds 2,294 tasks from real GitHub issues and pull requests in 12 popular Python repositories; given a codebase and an issue, the system must edit the code to address it. The paper's headline result was that the best model then, Claude 2, resolved 1.96 percent of issues. Public scores have risen enormously since. Anthropic's article on evaluating AI agents notes that SWE-bench Verified scores are now above 80 percent and approaching saturation, so differences at the top stop discriminating. When you see a resolved-rate number, ask:

The same article distinguishes pass@k (succeeds at least once in k tries) from pass^k (succeeds every time): a task with a 75 percent success rate per trial has a pass^3 of about 42 percent (0.75^3 = 0.42). For an agent that merges code, the second one is what the on-call engineer experiences. Build your own eval from your own repo's resolved tickets, run each three or more times, and report cost per resolved ticket beside the rate.

Computer-use and browser agents: the screenshot loop and its failure modes

A computer-use agent is the same loop with a different observation: a picture of the screen instead of a text result. The model sees a screenshot, emits an action (click at coordinates, type, scroll, press a key), your code performs it, and you return the next screenshot. It reaches software with no API, such as a vendor portal, and pays for that in tokens, latency and reliability.

:::key Verified October 2026 From the Claude computer-use tool documentation, read October 2026: the current client toolset is computer_toolset_20260801 (generally available on the Claude API, no beta header), with member tools such as screenshot, left_click, type and zoom. A screenshot costs roughly 1,000 to 1,800 input tokens. Recommended resolutions are 1024x768 to 1280x800, and the docs advise avoiding anything above 1920x1080. Cap the loop with a maximum iteration count (the reference loop uses 10). Security guidance: dedicated VM or container with minimal privileges, no sensitive data or logins available to the model, an allowlist of domains, and human confirmation for consequential actions. Instructions on web pages or in images can override yours; Anthropic's classifiers scan tool results for likely injections, and the docs still say to apply the precautions. Check the page for your model, since older toolset versions differ. :::

The arithmetic. Using the image formula from the next lesson, a 1456 x 819 screenshot costs ceil(1456/28) x ceil(819/28) = 52 x 30 = 1,560 visual tokens, inside the documented 1,000 to 1,800 range. A 40-step browser task that keeps every screenshot re-sends them: 1,560 x (1 + 2 + ... + 40) = 1,560 x 820 = 1,279,200 tokens, which at $4 per million is $5.12 uncached for the images alone. Keeping only the last three screenshots caps the resend at 3 x 1,560 = 4,680 tokens per step, for a total of about 182,520 tokens ($0.73). Pruning old screenshots, while keeping a short text log of what happened, is the standard cost control, and it also avoids the stricter per-image size limit that applies once a request carries more than 20 images.

How it fails.

Product design: suggestion, autonomy, review and rollback

The most important coding-agent design choice is how much autonomy you give, and it should rise only as measured trust rises. Think of four rungs.

Move up a rung only when your eval shows it deserves that: resolved rate on your own tickets, pass^k over repeats, the rate of diffs that touched tests, and reviewer acceptance rate. Say that last metric aloud; the number an engineering manager cares about is "what share of the agent's PRs get merged without rework."

Review UX decides whether the agent saves time. A 900-line diff with no explanation costs more review time than writing the change. Require a short description of intent, the files touched grouped by reason, the tests added, any assumption the agent made, and what it could not verify. Make the diff small by design: one ticket, one concern.

Rollback and blast radius. Every agent change must be revertible with one command: work in a branch, never push to the default branch directly, label agent commits, and keep a kill switch that stops new agent runs without touching the humans' pipeline. Scope permissions per repository: an agent assigned to a docs repo has no business reading the payments repo, and the credentials for one should not open the other.

Cost per accepted change is the unit metric. Using the earlier numbers: $1.07 of cached input per session, add output tokens, and divide by the 60 percent resolved and the share of those merged, then compare with the engineer-time saved. With assumed figures (a developer-hour of Rs 800 to Rs 2,000 fully loaded, and a session of about Rs 150 all-in, roughly $1.8 at an assumed Rs 83 per dollar, covering input plus output), a session that saves 30 minutes is an easy yes; the same session that produces a PR needing two hours of rework is a clear no. State your own assumptions.

How to say it in the interview

"The agent is a loop around five or six tools: read, search, edit as a targeted diff, run commands, and ask for approval. In a monorepo it must not stuff the repo into context; it keeps a short project guide up front and finds files just in time with glob and grep, and it hands self-contained searches to sub-agents that return a short summary. It runs in a throwaway container or microVM with no standing credentials and an egress allowlist, because a prompt cannot enforce any of that. The oracle is the test suite, run by the harness and not by the model, with test and grader files protected and the diff reviewed for edits to them, since an agent will take the shortcut if one exists. A 40-step session that grows 3,000 tokens a step re-sends 2.46 million input tokens, about 10 dollars uncached and about one dollar with caching, so caching is a launch requirement. A browser agent uses the same loop but its observation is a screenshot of about 1,500 tokens, so I cap iterations, keep only the last few screenshots, run it in a VM with no credentials, and require a human to confirm anything irreversible."

Checklist

Practice this interactively
Share this lessonWhatsAppLinkedInX

Related lessons