Optimizing Token Efficiency for Agentic Inference Systems
With usage-based billing, every token in an agentic session hits three things at once: your credits, latency, and the context window the agent has left to finish the job. Worse, each model generation tends to burn more tokens per task than the last – so the harness, not the model, is where you claw efficiency back. None of this is one big win; it's a steady stream of small ones across two cost centers: the repeated prompt prefix, and the tool payload sent every turn.
How Agentic Requests Spend Tokens
Two costs sit at the heart of every request. The prompt prefix – system instructions, tool definitions, repo context, conversation history – repeats on nearly every turn; cached, it can be up to ~10× cheaper than recomputing it. Tool-definition overhead is the full JSON schema for every tool, historically loaded into context on every request whether or not the model uses it.
Lever 1: Prompt Caching
OpenAI models cache the prefix automatically; the lever is how long it survives. Setting prompt_cache_retention: "24h" moves state from fast GPU memory (dropped after ~5–10 min idle) to roomier GPU-local storage for up to 24h. Relative cache-hit-rate increase after a 40–60 min gap:
| Gap | GPT-5.2 | GPT-5.3-Codex | GPT-5.4 |
|---|---|---|---|
| 40–60 min | +338% | +279% | +919% |
Gains are largest after long pauses – exactly when the default cache would have expired and forced a full-price reprocess.
Anthropic models use explicit cache_control breakpoints instead of inferred prefixes. Spending all four deliberately – two on the most stable boundaries (end of tool defs, end of system prompt) plus two rolling anchors on recent messages – pushed the hit rate to ~94% for agentic workloads.
Lever 2: Tool Search
Defer tool definitions: the model sees only name + description upfront and loads the heavy schema on demand. Deferred tools sit at the end of context, so the cached prefix stays intact.
OpenAI (native defer_loading, GPT-5.4+), 4-day experiment:
| Metric | GPT-5.4 | GPT-5.5 |
|---|---|---|
| P50 tokens/turn | −9.81% | −8.61% |
| P50 TTFT (time to first token) | −6.88% | −7.34% |
| P50 TTC (time to complete) | −5.31% | −5.42% |
Per session, median total tokens fell −8.97% (GPT-5.4) and −10.92% (GPT-5.5).
Anthropic (defer_loading: true, core tools kept hot), 7-day experiment:
| Metric | Scope | Delta |
|---|---|---|
| Total prompt tokens | p50/turn | −11.30% |
| Total prompt tokens | p95/turn | −8.85% |
| Total prompt tokens | p50/user | −18.32% |
| Total tokens | p50/user | −18.03% |
≈18% fewer tokens per session for the median user. Moving search client-side (embedding-guided, intent-matched) added latency wins on top:
| Metric | Model | Delta |
|---|---|---|
| TTFT (p50) | Claude Opus 4.6 | −1.91% |
| TTC (p95) | Claude Opus 4.6 | −2.57% |
| TTC (p95) | Claude Sonnet 4.6 | −3.35% |
| User error rate | Claude Sonnet 4.6 | −4.01% |
Lever 3: WebSocket Transport
An agentic turn fires many sequential requests. WebSocket mode in the Responses API holds one connection open and reuses recent response state, cutting continuation overhead (OpenAI models):
| Metric | Pctile | GPT-5.3-Codex | GPT-5.4 |
|---|---|---|---|
| TTFT | p50 | −19.46% | −16.37% |
| TTFT | p95 | −12.92% | −15.78% |
| TTC (by turn) | p50 | −13.55% | −11.74% |
Engagement rose too – active users +1.27% / +2.17%, two-day engagement +1.90% / +3.14% – enough to make WebSockets the default for GPT-5.2+.
Takeaway
Caching keeps the prefix cheap, tool search keeps the payload lean, and WebSockets keep the transport tight. Each is a few percent; together, double-digit token and latency reductions per session. The next frontier is offloading narrow work – workspace search, command-running, summarizing – to specialized subagents running on the cheapest model that can do the job.