Model routing & context economics

Building with agents

Two decisions dominate what an agent costs and how well it performs: which model runs, and what sits in its window. Both are usually made by default (the biggest model, the fullest window) and both defaults are wrong. The working rules: one model per run, and context is a budget, not a bucket. Everything in this page is those two rules with the receipts attached.

Route by task, not by habit

Frontier models exist for hard reasoning. Most agent work is triage, transformation, and bookkeeping. Sending that to your most expensive model is paying architect rates for data entry. Route by what the task actually demands:

Task type	Model tier	Why
Triage, classification, routing, yes/no gates	Cheap / small	The output is a label. A small model returns it faster, at a fraction of the cost, and the loop can afford to call it constantly
Bulk work: summarising, extraction, format conversion, first-draft code, test scaffolding	Standard	High volume, well-specified, verifiable downstream. The standard tier is where cost per unit of useful work is lowest
Hard reasoning: architecture calls, subtle debugging, adversarial review, ambiguous trade-offs	Frontier	A wrong answer here is expensive to detect and more expensive to unwind. This is the only tier where paying top rate pays back

The routing decision itself is a triage task, so it belongs on the cheap tier, or better, in deterministic code. A keyword match or file-count threshold that picks the tier is free and auditable.

One model per run

Route between runs, not within one. Once a run starts, it finishes on the model it started on. The reason is economic: switching models mid-run breaks the prompt cache.

The cache works on exact prefixes, per model. Every turn of an agent loop resends the entire accumulated context: system prompt, tool definitions, every prior message and tool result. The cache is what makes that affordable, because the unchanged prefix is read from cache instead of re-processed at full price. Switch models mid-run and none of that prefix exists on the new model. The whole accumulated context is re-billed as fresh input, at the write premium, on the very turns when the context is largest. If a task genuinely needs a different tier partway through, end the run and hand off: pass a compact result forward and start the next run clean on the right model.

Cache economics, in numbers

From Anthropic’s prompt-caching documentation: cache reads cost 0.1× the base input price, a 90% reduction. Cache writes cost 1.25× base input (a 1-hour TTL variant costs 2×). The default TTL is 5 minutes, refreshed every time the prefix is reused. An agent looping steadily keeps its cache warm for free; one that pauses six minutes pays the write premium again. The practical discipline: keep stable content (system prompt, tool definitions, standing instructions) at the front and byte-identical across turns. One edited character early in the prefix invalidates everything after it.

Context rot: a longer window is not a better one

The budget framing isn’t only about money. Chroma’s context-rot research measured 18 models, frontier included, and found that performance degrades as input length grows, even on tasks the models handle easily at short lengths. In one result, models answered questions better from a focused ~300-token context than from the full ~113k-token input containing the same information. The window’s advertised size tells you what the model accepts, not what it handles well. Every irrelevant token costs money and dilutes the model’s attention on the tokens that matter.

So spend the window like a budget: every token in it should be earning its place on this task. “It might be relevant” is how windows fill with rot.

Two regimes of retrieval

There are two regimes of getting context into the window, and mixing them up is where most context waste comes from:

Small unit → load the whole thing. If the task lives in one coherent unit (one module, one document, one config) load it completely and let it sit in the cache. Whole-unit context beats fragments: the model sees real structure instead of snippets, and after the first turn you’re paying cache-read prices for it. Don’t retrieve cleverly what you can load cheaply.

Cross-unit or large scope → deterministic lookup. When the answer could be anywhere in a large corpus, don’t pour the corpus into the window; the context-rot numbers above are exactly what that buys you. Use a deterministic step (a grep, an index, a database query, a generated map) that returns just the relevant slice, and load only that. This is the 12-factor-agents principle of owning your context window: what enters the window is an engineering decision you make deliberately, not a side effect of whatever retrieval happened to return.

The test between regimes: can you name the unit? If yes, load it whole. If you’d have to search to find it, search deterministically first and load what the search returns.

The budget in practice

Route each task to the cheapest tier that genuinely handles it. Keep each run on one model so the cache does its work. Load small units whole; look large scopes up. That is what lets agents run all day without the bill or the error rate telling you to stop. Where these per-run decisions become a team-level design is the next page.

Designing an agent operating model A reliable operating model is mostly deterministic software with strategically placed model steps, and no capability ships without a baseline.

Sources: Anthropic: prompt caching (0.1× cache reads, 1.25× writes, 5-minute default TTL), Chroma: context rot (degradation with input length across 18 models; focused ~300-token vs. full ~113k-token contexts), 12-factor-agents (own your context window).