4×
cheaper than tool-calling
Technical report · code is the tool call
Codemode gives an agent a persistent Python kernel and compiles every tool into an importable module. Instead of emitting one JSON tool-call per turn and receiving the whole result back into its context, the agent writes a program: it loops, filters, keeps state in variables, and prints only what matters. On matched tasks it holds parity with Claude Code at a fraction of the cost, context, and latency — and when a task involves work that runs longer than a single turn, like a build or a long job, it can start that work, let go of the machine while it runs, and collect the result on a later turn.
Part of a harness's job is not visible in any API surface: it decides which region of the model's ability a task lands in. The same problem can be posed to a model in a way it has practiced millions of times, or in a way it has barely seen. A good harness shifts the task toward the first — into the model's local distribution, where it is fluent.
Code is that region. Models are trained on code continuously — through pre-training and all the way through post-training — so writing a loop, composing a helper, keeping a value in a variable are among the most deeply practiced things a model does. Tool-calling, by contrast, is a comparatively thin, recent post-training veneer: a narrow JSON dialect layered late. Asking a model to solve a task as code plays to its deepest training; asking it to solve the same task as a sequence of tool-call blocks asks it to work in the dialect it saw least.†
A tool call is a single function invocation with a flat bag of arguments, and its result returns to the model whole. A program can fetch two collections, intersect them, keep the intersection in a variable, and surface only the answer — deciding, at run time, what is worth its own attention. That control over its own context is not a feature bolted onto the model; it is what writing code is.
And there is a second reason, about the interface itself. An API is not a thinner tool schema — it is a richer one. Typed clients, SDKs, OpenAPI specifications: describing a service as an API is how machines have talked to machines for decades — a composable, expressive convention the model has read millions of times. A tool-call schema is a narrow, recent re-description of that same idea. Codemode simply hands the model the mature medium — the API, as code — instead of a flattened restatement of it.
Three separate things get cheaper when tools become code.
You load less up front. A tool can only be called if its description is already in the prompt, so a tool-calling harness has to hold the whole catalogue in context — a few hundred tools is roughly 100,000 tokens, sitting there on turn one and re-read on every turn after, whether or not the task ever touches most of them. Codemode loads almost nothing; it opens a tool's documentation only when it is about to use it. This is a flat cost every task pays, large or small.
You keep less of what comes back. When a tool returns, its result drops into the conversation whole — the tool's author fixed that shape long ago. When code calls the same tool, the agent decides in the moment what to keep: fetch a ninety-row list and keep only the fourteen rows it needs, leaving the rest in a variable the model never reads. This is a large effect — force a codemode agent to print every response in full and its context more than doubles, its cost up about 74%.
You write less to act. A tool-calling agent emits a separate structured block for every call and takes each whole result back; over a task that is dozens of bulky blocks. Code does the same work in one compact cell per turn. This saving is independent of the other two: even a codemode agent forbidden to filter anything was still about 3.8× cheaper than tool-calling — the difference was never only about what it kept, but about how little it had to write. The last two savings grow with how much the task does, so the bigger the job, the wider the gap.
Where this sits. The second and third savings are not unique to a full runtime. Programmatic tool calling — a feature, available today, where the model orchestrates ordinary tools from inside code and keeps the intermediate results in the code environment instead of the conversation — already recovers much of them. Codemode is the far end of the same idea: the tools are not called from code, they are code — typed API modules the model imports — the kernel persists across turns rather than one orchestration, and long-running work becomes durable, resumable jobs. It is a spectrum from tool-calling, to code that orchestrates tools, to a full API-and-kernel runtime — not a binary.
A codemode session is a long-lived kernel. Before the agent's first cell, the runtime compiles the API surface into importable, typed, documented modules on disk. The source can be anything that describes an API: our convention today compiles each MCP server into its own file under a shared api/ directory — but point the same agent at an OpenAPI spec run through a code generator, or at an existing typed client, and it works just as well. The only requirement is that the tools arrive as code the model can read and call. Below is one tool through the pipeline we run today: the MCP the harness receives, the Python it generates, and the code the agent then writes.
1 · the source — an MCP tool (or an OpenAPI spec, an SDK…)
{
"name": "spotify__show_liked_songs",
"description": "Get a list of songs you have liked.",
"input_schema": {
"type": "object",
"properties": {
"access_token": {"type": "string", "description": "Access token from spotify login."},
"page_index": {"type": "integer", "minimum": 0, "default": 0},
"page_limit": {"type": "integer", "minimum": 1, "maximum": 20, "default": 5},
"sort_by": {"type": "string", "description": "+/- one of liked_at, play_count, title"}
},
"required": ["access_token"]
}
}
2 · the compile — generated api/spotify.py
"""spotify — generated from the MCP schema. Do not edit.""" async def show_liked_songs( access_token: str, page_index: int | None = None, page_limit: int | None = None, sort_by: str | None = None, ) -> Any: """Get a list of songs you have liked. Args: access_token: Access token obtained from spotify app login. page_index: The index of the page to return. sort_by: +/- one of liked_at, play_count, title. """ return await _call("spotify", "show_liked_songs", {"access_token": access_token, "page_index": page_index, ...})
The schema becomes a real Python signature: types, the union for optionals, the docstring the model reads before calling. The body forwards through one transport, so every arm reaches the world the same logged way. The agent finds this by reading api/spotify.py — never by holding 469 schemas in context.
3 · the use — the agent writes code
import api.spotify async def all_pages(fn, **kw): # reused for every paginated call, in any app out, i = [], 0 while True: page = json.loads(await fn(page_index=i, page_limit=20, **kw)) out += page if len(page) < 20: return out i += 1 liked = await all_pages(api.spotify.show_liked_songs, access_token=st) lib = await all_pages(api.spotify.show_song_library, access_token=st) print(len(liked), len(lib)) → 92 41
libids = {s['song_id'] for s in lib} # set → O(1) membership
common = [s for s in liked if s['song_id'] in libids]
titles = [s['title'] for s in common]
print(titles) → ['Midnight City', 'Retrograde', ... ] (14 titles)
These are real cells from a passing run. The task's entire logic — "liked songs ∩ library" — is a set intersection, one line of code. A tool-calling agent has no ∩: it must pull both full lists into its transcript and reason over them token by token, or fan out a call per song. Here the two 90-item lists live in kernel variables and never touch the model's context — only the fourteen titles it will send do. That is the pipeline paying out on an ordinary errand.
A tool-calling agent is not helpless with long work. It can launch a process in the background — nohup … & through a bash tool — and poll it with later tool calls. On a machine you already hold open, that can be enough. The hard part was never starting the work; it is reliably coming back to it. On a real 25–40 minute compile, Claude Code did background the build, polled it, and even reached for a monitor tool — and then ended its turn with "OCaml is still compiling. I'll continue when it lands." There was no next turn. The build never finished; the task failed. The mechanism to background was right there, and it still didn't bridge the wait.
Codemode turns "come back to it" into a first-class primitive instead of a process the agent has to remember to poll. Its one deliberate break from ordinary Python is a wait that ends the turn rather than blocking. Work is a durable, named, file-backed job; the agent yields, the host sleeps, and a later turn collects it by name. A thirty-minute build becomes ordinary control flow:
# the whole build, backgrounded — the last thing this turn does jobs.submit("cd /tmp/CompCert && make -j$(nproc) all", name="build", max_runtime="4h") jobs.wait(wake_after=900) → submitted build (pid 41134); log /tmp/harness/jobs/build.log → waiting 900s outside the kernel; nothing is held.
j = jobs.get("build") print(j.done, j.ok) → True True → ... OCAMLOPT driver/Frontend.ml · ccomp built
submit and wait are the last statements of one turn; get is the first of the next. Between them the agent does not exist — the kernel is idle, the host is released, and the build runs unattended on disk. wake_after is a deadline, not a fixed sleep: the agent is re-invoked the instant the job finishes, or when the deadline elapses — whichever comes first — so a build that takes six minutes costs six, not the fifteen it was willing to wait. The full run installs apt, opam, and coq the same way — submit, yield, collect next turn — four named jobs, and the kernel is never held.
The real trace is not a happy path — the agent hit a 404 download mirror, hunted GitHub tags, resolved a Coq version conflict, and debugged a linker error (undefined reference to `sqrt` → add -lm) — all while four named jobs (apt, opam, coq16, build) ran in the background. Zero blocking sleeps. The kernel was never held. It is the same task Claude Code walked away from.
-p tool-calling loop has no resume seam, not as "codemode wins long tasks N% of the time." The same harness in plain tool-calling mode also finished, by blocking on a build that fit its budget — so the point here is not cost (this task's cost story belongs to the full-suite numbers, not one build); it is that codemode finishes without holding anything, and keeps finishing when a wait exceeds the budget, where blocking breaks too. The jobs runtime is the seam that makes the wait survivable.Claude Code has the easy version of this problem. It is a local harness — it runs on your machine and can hold a process open on your own computer for as long as it likes; the compute is already there and already yours. A harness you host — one remote runtime serving many users — cannot. It can't run unbounded compute, and it can't keep a VM pinned open for thirty minutes waiting on a build. Held-open sessions are precisely what make hosting an agent expensive.
The jobs framework was designed for that constraint. Because wait ends the turn instead of blocking, the agent releases its compute the instant it begins waiting. A job is a file-backed process with a bounded lifetime, and it is awaitable and resumable — so a scheduler can put the whole session to sleep and wake it, on a different machine, when the job signals done. Nothing is pinned to one long-lived host.
That is what lets the runtime live on small, short-lived micro-VMs instead of one giant always-on box: the VM can be torn down while a job runs on managed compute, and rebuilt only when there is a result to collect. And because the kernel's state is recoverable — jobs persist on disk, and the agent can re-fetch anything a fresh kernel has lost — you can evict the REPL under memory pressure and let the agent rebuild. That claim is now measured, not asserted: in a live injection test the kernel is SIGKILLed mid-task with two jobs running and state the agent had saved to disk; the fresh kernel's death notice names what survived (files intact, jobs still running, reach them by name), and the agent reads its state back and collects both jobs — each run exactly once, 10/10 checks, $0.07 a run. The scheduler is equally real: the full CompCert build ran under it end to end — six wakes (one of them on a failed job the agent then recovered from), about thirteen short-lived harness processes spanning a single task, $0.39.
There is a compounding effect here that is easy to miss: the representation choice makes the hosted architecture cheap. Decoupling the agent loop from its execution environment puts a boundary between them, and every action pays a trip across it. A tool-calling agent making ~40 calls per task pays that toll ~40 times — for it, colocating loop and compute is load-bearing. Codemode measures 1.00 actions per turn, ~9 turns per task: the same boundary costs it ~9 trips. And the stateless-harness move — rebuild the loop from the transcript on every wake — is nearly free when each turn's transcript is one code cell plus what the code chose to print, rather than a 100k-token catalogue re-read. The properties that make codemode cheap per task are the same properties that make it cheap to host.
The host side of this is deliberately small. Between turns, the only durable truth is what is on disk — the transcript, the jobs manifest, the artifacts — and everything else is rebuilt from it. That one invariant (tested by killing every process mid-flight and replaying from disk alone) replaces the recovery journals, hand-off manifests, and live-worker coordination that a stateful session server accretes: the wake scheduler that re-invokes a sleeping session when its job finishes or its deadline passes is ~200 lines, with single-writer leases and exactly-once wakes derived from files rather than coordinated across processes.
This is exactly where ad-hoc backgrounding falls down. A nohup … & process is pinned to the machine that spawned it — no durable handle, no way to resume it on another host — so it dies the moment a hosted harness reclaims the VM. To make backgrounding survive being hosted, you end up rebuilding precisely this: durable, named, awaitable, resumable jobs. And the payoff is not hypothetical — the remote setting is the harder one, and codemode still outperformed Claude Code, whose own background-and-monitor tooling was available and did not close the gap.
A hosted runtime executes code the model wrote, so the security question is not "is the model trustworthy" but "what can the code it writes reach." Codemode's answer is a hard split between two kinds of process. The brain — the agent loop — is stateless, lives harness-side, and holds the real credentials. The cells — everything the model authors — run in an OS sandbox with no network, a filesystem confined to the task, and an environment constructed from an explicit allow-list rather than inherited. Agent code has no execution surface on the harness side at all, so a credential in the harness's environment is unreachable the way a database password is unreachable from SQL.
The one network route out of the sandbox is the egress proxy: a route table of whatever APIs the deployment chooses to expose. Each route forwards to one upstream and injects that upstream's credentials at the boundary, so no credential of any kind exists sandbox-side; every request is audited — session, route, status, bytes — never a key, never a body. Notably, there is no route to the model provider, and the proxy refuses to configure one: intelligence is a capability, not an endpoint. Code that wants more intelligence calls spawn(), which is priced, capped, and leaves a transcript — a raw provider call from a cell would be unmetered spend that never appears in the trace.
Subagents follow the same rule. spawn() does not fork a child from the kernel — it is an API call on the proxy's control surface; a harness-side broker claims the request (an atomic rename two processes cannot both win) and starts the child's brain in the harness's world, with the child's own kernel back in the sandbox. Exactly-once is enforced from disk and tested by SIGKILLing the broker at every stage of the chain — a request is never lost and a model run is never duplicated.
The guiding rule for every primitive: write down what a competent engineer would type without the new thing, then get as close to that as the constraints allow. A bespoke API is a tax the model pays on every task, because it appears in zero pretraining tokens. So the surface is deliberately ordinary — imports, function calls, and one unavoidable deviation.
The compiled catalogue: one app.py per service, every tool a typed, documented async function. Signatures carry the schema — unions, enums as Literal[...] — and consequential defaults are promoted to required so a decision can't be made silently.
Durable background work. submit returns immediately and spools to a file that survives kernel death; wait ends the turn instead of blocking; get(name), manifest(), and kill(name) recover and manage jobs across turns and restarts.
Subagents, built on the jobs substrate — a child is a job. It returns immediately; a harness-side broker starts the child (the kernel never forks a brain), it runs with its own sandboxed kernel and its own tool grant, and its result is collected like any other job. Nine tool schemas in a tool-calling harness; one module here.
Shell and file navigation as native tools, in the language the model was trained to use — not hand-rolled through the kernel. Every long or blocking bash command is filtered by the same guard the kernel uses and redirected to a job, so the shell can never quietly hold the turn.
A persistent kernel that an agent writes arbitrary code into is a sharp instrument. Most of the engineering is the machinery that keeps it from cutting the wrong way — and that makes the good idioms the easy ones.
File-backed submit / wait / get. Jobs survive a kernel restart because they are processes on disk, not objects in memory; a fresh kernel is told what is still running and recovers it by name. The substrate under both background work and subagents.
durable · recoverable · eviction-tested liveThe in-process shapes a Python programmer reaches for — threading, subprocess, a long sleep — fail loudly at the point of use, each error naming the job primitive to use instead. A wall-clock watchdog catches anything that blocks past the ceiling and rewrites the interrupt into the same redirect.
Schemas become typed Python: unions, enums, required-ness. A one-time, schema-only enrichment pass promotes constraints stated in prose ("a number, a yes/no") into the type system — narrowing-only, every proposal logged and verified against the source. Runtime validation then checks each call at the call site, in-process, before the request leaves — where a tool-calling arm learns of a bad argument a provider round-trip later.
the contract, enforced in the languageAny API description compiles in — an MCP server, an OpenAPI spec, a typed client. Our convention lays each service out the way an SDK is, api/gmail.py rather than a schema per call, so one read delivers a whole service's contract, its conventions, and the sibling it should have called instead.
Modeled on the edit-tool contract: the first call to an API is refused until its file has been read, and the refusal names the exact lines. A verified precondition, not an advisory — there is no path through without the read.
verified precondition · edit semanticsWhen an optional argument's default resolves against runtime state — "otherwise, replies to the sender" — the model, feeling certain, will skip it even with the docstring in context. So the compiler makes those parameters required: the decision becomes a value the code must supply, and the model computes it correctly.
forced explicitness > requested readingTwo baselines, and they are different things. The first is the same harness in tool-calling mode — identical machinery, but the tools exposed as ordinary JSON tool-calls instead of code. Comparing against it isolates the one variable: code versus tool-calling. The second is Claude Code — a separate, real product — which answers whether codemode holds up against a strong agent people actually use. Score is reported as underpowered: frontier models saturate these suites, so an identical config can score 8/8 then 3/8 on a re-run. Cost, context, and latency are the endpoints that move.
This is the number no prompt tuning can close for a tool-calling arm: schemas must be resident to be callable, so the catalogue is paid on turn zero and re-paid every turn after. It is why cost and latency track context, and why a 100k-context arm can ignore a rule it is technically "looking at."
| Endpoint | codemode | tool-calling | claude-code | advantage |
|---|---|---|---|---|
| Cost / task | $0.21 | $0.83 | $1.02 | 4.0× / 2.3× |
| First-turn context | ~5k | ~101k | ~106k | ~22× |
| Accumulated context | 115k | 988k | 921k | ~8× |
| Wall-clock / task | 61s | — | 88s | 1.4× |
| Long build (30 min) | PASS | PASS | FAIL | only one |
Cost advantage is 4.0× vs the tool-calling harness (full suite, n=432) and 2.3× vs Claude Code (held-out). Latency is warm-kernel; see integration notes. The long-build row is a single illustrative task (n=1) — an existence proof, not a rate.
Given a REPL, the model reaches for what every Python corpus taught it — time.sleep, subprocess.Popen, threading.Thread. Fighting that prior is a losing battle; the win is to make the natural idiom durable underneath, and to force a deviation only where one is unavoidable. The single forced deviation is wait — because a wait needs wall-clock to pass, and a single-threaded kernel serving a turn-based agent cannot both hold and yield. Everything else is ordinary code.
When a guard fired and its message named the right primitive, the model adopted it on the very next cell — every time. When we instead advertised the rule in the system prompt, one run clamped to the exact threshold and never discovered the primitive at all. A rule in a prompt is a statement of intent; a verified error is a control. We stopped writing advisories and started making the wrong path fail at the point of use.
It is tempting to treat "keep the context small" as a scaffold trick any arm could adopt. It isn't. In tool-calling, every result lands in the transcript and management means summarizing a transcript the agent doesn't control. In a REPL, the caller chooses what survives each call and state lives outside the transcript entirely. These are different formulations of the same problem — and the REPL's is the one that scales.
The sharpest failure we found: an agent replying to its own sent email, omitting the recipient argument whose default is "the sender." It had the docstring resident, it had even printed the correct recipients moments before — and still skipped the slot, because "to whom" was not part of its concept of reply. Reading did not fix it; a visible-but-optional schema did not fix it. Making the parameter required did, on the first try — the model computes the right value when the language forces it to supply one.
What it takes. A kernel per session, a one-time compile step from your API schemas to the typed module tree, and the guard layer. The one operational cost to plan for is the kernel's environment. A warm kernel starts in about five seconds; the very first boot has to page a few hundred MB of interpreter and libraries in from disk — about 70s cold, and worse if many kernels boot at once and contend for I/O. But this is one-time and global: the pages stay warm in the OS cache, so every kernel after the first is fast. A prewarm step — boot one throwaway kernel to warm the cache — erases it, and with it codemode is faster end to end than tool-calling, because latency is dominated by prefill and prefill by context. A warm pool is a launch requirement for a hosted service, not an optimization.
Where it wins. The advantage scales with the work: tasks that make many calls, carry a large tool catalogue, or run long enough to need background work. On short, shell-dominated tasks the representation has little to grip and a competent bash-tool arm is a wash — codemode's edge there is only the jobs runtime, and only when a wait is real.
The honest trade. The advantage is not uniform. Codemode reads an API's contract before it calls it and takes more, smaller turns than a tool-calling agent — so on a simple errand that discipline can leave it no cheaper per task, even as it stays far ahead on context. The gap is decisive in aggregate and on the tasks that actually fail — where an unread contract or an unbounded wait is the failure — and thin on short, self-contained work. Adopt it for what its shape is good at, not as a universal drop-in.