Most teams running AI agents burn tokens like they’re free. They don’t need to. The gap between the most expensive and cheapest viable model for a given subtask often exceeds 10x. The problem isn’t the model — it’s the assumption that one model fits every step of an agent’s execution.
You already know the basics: pick a cheaper provider, reduce context length, use caching. That’s table stakes. The real savings come from restructuring how your agent works — and measuring cost per successful task, not per token.
A cheap model that produces garbage costs more than an expensive model that works the first time.
The Real Cost Isn’t Per Token — It’s Per Successful Task
Run the same task twice. First try uses GPT-4o at $15 per million input tokens. It hallucinates a number, fails verification, re-runs. Total cost: $30 per task, 2x latency. Second try uses a fine-tuned Qwen2.5 32B at $2 per million input tokens, gets it right the first time. Total cost: $2.
The expensive model lost by every metric. But most teams never measure cost per outcome. They measure cost per API call and assume the cheapest call wins. That assumption breaks when cheaper models fail more often, forcing retries that consume more tokens than the original attempt.
Track failure rates per model per task type. Then calculate the break-even. For many classification and extraction tasks, a $5 model that fails 15% of the time still beats a $20 model that fails 5% of the time — because the retry overhead is less than the per-call premium.
Where Most Agent Spending Leaks
Three patterns inflate agent costs more than anything else.
Over-engineering: You prompt the agent with five paragraphs of context when two would do. Every unnecessary token costs money and adds noise. The agent wastes reasoning steps parsing background information it doesn’t need. Trim system prompts to the minimum viable instruction. Anything that isn’t a rule, a constraint, or an example belongs in a reference document the agent can read — not something it must read.
Redundant verification loops: You make the agent check its own work, then check the check, then check the check’s check. Each loop doubles the token spend. A single verification pass by a small model catches 80% of errors. A second pass by the same model catches another 5%. The third pass catches nothing. Use a single pass with a cheap model unless the task carries legal or financial risk.
Context window waste: Every agent call includes the entire conversation history. That history grows with each step. By the tenth step, you’re paying for 50,000 tokens of “I already said that.” Implement sliding windows: only pass the last 3-5 interactions plus a compressed summary of everything before. Tools like mem0codecodecodecodecode or simple summary chains cut context costs by 60-70%.
Model Tiering: Match the Model to the Task’s Determinism
Not every step in an agent’s workflow needs the same reasoning power.
Deterministic tasks — data extraction, format conversion, simple lookups — need a model that follows instructions reliably. They don’t need creativity. A fine-tuned Llama 3.1 8B handles these for cents on the dollar. Test it once. If it passes, lock it in.
Non-deterministic tasks — strategy synthesis, complex code generation, multi-step planning — need the reasoning depth of a frontier model. That’s where you spend your budget. The ratio should be 80:20 or higher in favor of cheap models for the grunt work.
The mistake is running everything through a single “smart” model because it’s easier to implement. It’s not easier on your wallet.
The Verification Paradox
You want to reduce costs. You also need to trust the output. Verification seems like an extra expense. Done wrong, it is.
Done right, verification is the cheapest insurance you can buy. Use a smaller model as the judge. Feed it the original task, the output, and a rubric. Ask it to pass or fail. Do not ask it to rewrite — that doubles the spend. A simple pass/fail with a confidence score lets you route failures back to the heavy model while letting successes through.
This pattern — heavy model for generation, light model for verification — cuts total cost by 30-50% in practice. The heavy model only fires when the light model says “I’m unsure.”
Reduce Model Size Iteratively
This is the single highest-leverage action you can take. Start with your most expensive approved model. Run a batch of test tasks. Check whether a cheaper model from the same family produces equivalent results. If yes, test the next tier down. Keep going until quality drops below your threshold.
In Claude, that means starting with Opus, testing Sonnet, testing Haiku. In Google, Gemini Ultra -> Pro -> Flash. In open-source, Qwen 2.5 72B -> 32B -> 7B. Each step down cuts cost by 3-10x.
You will find tasks where the cheap model works perfectly. Write a routing rule. Never call the expensive model for those tasks again.
Batch and Cache for Repetitive Subtasks
Many agent tasks are variations on a theme: “Summarize this email,” “Extract the invoice number,” “Check if this support ticket matches a known pattern.” Each call currently pays full price for context and reasoning it already performed.
Cache the results of deterministic subtasks. If the agent already extracted the invoice number from a document, don’t re-extract it on the next step — store it in a short-lived key-value store and reference it. For summarization, use a two-stage pipeline: generate once, then for related documents, pass only the diff against the cached summary.
Batching multiple independent calls into a single API request reduces per-token overhead from the provider. OpenAI and Anthropic both offer batch APIs at 50% discount. If your agent can tolerate a few minutes of latency, queue up 20 extraction tasks and fire them together. The savings add up.
Practical Checklist for Senior Practitioners
- Profile every subtask in your agent’s workflow. Measure token spend per subtask.
- Set a quality threshold (e.g., 95% accuracy on a held-out test set) and find the cheapest model that meets it.
- Implement a two-tier model system: cheap for generation, cheap for verification, expensive only for exceptions.
- Add a context window budget. Hard-limit the number of tokens passed to the model per call.
- Cache deterministic outputs at the task level, not just the prompt level.
- Route tasks by complexity. A simple lookup should never even reach the agent — handle it with a lookup table or a regex.
The Cost Metric That Gets Ignored
Teams optimize for per-token cost because it’s the number on the bill. But the metric that determines whether your agent system is sustainable is cost per useful action — the total spend divided by the number of tasks that actually produce value.
That metric forces you to account for retries, failed tasks that consume tokens and produce nothing, and manual review overhead when the agent’s output isn’t trustworthy. A cheap model that produces garbage costs more than an expensive model that works the first time.
The best teams build a feedback loop: log every task, its cost, its outcome, and which model handled it. They review those logs weekly. They demote models that show high failure rates and promote models that deliver consistent results — regardless of the per-token price.
That’s how you reduce costs without breaking your workflow.
