OpenAI cut GPT-4o-class inference pricing by more than 60% over the past year. Anthropic matched. Google went lower. DeepSeek made headlines by pricing at fractions of a cent.
So here's the question almost no one's asking loudly enough: if AI inference is getting so cheap, why are most companies' AI bills still climbing?
The answer isn't complicated. It's that almost everyone is doing their AI API pricing comparison using the wrong metric. You're looking at token prices. You should be looking at cost per useful output — and the gap between those two numbers is where your budget disappears.
The Headline Price Is a Lie
Every major AI provider lists a per-million-token rate. It's the first thing you see on their pricing page. It's also the least useful number for making an actual cost decision.
Token pricing assumes all tokens are created equal. They're not. A 1 million-token context window doesn't cost the same as a 200K window at the same per-token rate — because in practice, you'll use more of it. A model that's cheap per token but requires three prompt retries to produce a usable response is more expensive than a pricier model that nails it on the first try.
The most deceptive comparison happens between providers with different context window sizes. Google offers Gemini models with a 1 million token window at rates that look competitive on a per-token basis. But if your workflow actually fills that context, you're paying for a million tokens of input every single request — most of which may be redundant padding because the model's attention isn't precise enough to leverage the right 50K tokens on its own. The per-token rate looks identical. The bill isn't.
Five Costs That Don't Show Up on a Pricing Page
When teams build an AI API pricing comparison, the spreadsheet usually has columns for input tokens, output tokens, and maybe caching. That spreadsheet is missing the costs that actually determine whether you're overspending.
- Caching hit rates: Most providers now offer prompt caching — Anthropic, OpenAI, and Google all do. But they implement it differently, and cache eviction policies vary significantly. A provider with a lower list price but aggressive cache eviction can end up more expensive for repeated-query workloads than one with a slightly higher per-token rate and generous caching behavior.
- Batch pricing and latency trade-offs: OpenAI and Anthropic both offer batch APIs at roughly 50% discounts. But batch jobs have minimum queue times — often measured in hours — and no streaming support. If your product needs real-time responses, batch pricing is irrelevant. If it doesn't, you're leaving money on the table. The real question isn't "what's the cheapest per-token rate?" It's "what's the cheapest rate for my actual latency requirements?"
- Retry and failure costs: No model is perfectly reliable. Rate limits, server errors, and timeouts happen in production. When a request fails and you retry — potentially with a fallback provider — you're paying twice for the same output. Cheaper models and newer providers often have stricter rate limits and less mature infrastructure, which means more retries in practice.
- Prompt engineering overhead: Some models need significantly more prompt crafting to produce consistent, high-quality outputs. That's not a direct API cost, but it's a real one: more engineering hours, more testing iterations, more fine-tuning of system prompts. A model that works reliably with minimal coaxing has a lower total cost of ownership than a technically cheaper model that demands constant attention.
- Quality-adjusted cost per task: This is the one that matters most. If Model A costs $0.50 per task and succeeds 95% of the time, and Model B costs $0.30 but only succeeds 80% of the time, Model A is cheaper. The human review, correction, and rework costs on Model B's 20% failure rate swamp the per-task savings. Most teams never measure this, and it's the single biggest source of hidden AI spending.
A Framework That Actually Works
Stop comparing token prices. Start comparing cost per useful completion for your specific workload. Here's how to do it in an afternoon:
- Define "useful" first: For your actual production task — not a synthetic benchmark — define what counts as a successful output. It might be "extracts all 15 required fields with 99% accuracy" or "generates a response the end user doesn't revise." Be specific. Generic benchmarks like MMLU or HumanEval won't tell you anything about your use case.
- Run a 500-completion test: Take your real production prompts and run them across every provider you're evaluating. Log every single completion: tokens consumed, latency, whether it met your quality bar, and how many retries it required. This takes a few hours and a few dollars. It's the best-invested time this quarter.
- Calculate real cost: Total API spend — including retries and failed requests — divided by the number of useful completions. That's the number that matters. Compare that across providers.
- Add penalty weights: Factor in the things your spreadsheet can't capture: rate limit friction, integration complexity, vendor lock-in risk, and model stability across version updates. These are real costs even when they don't appear on an invoice.
Where the Real Savings Actually Hide
Once you measure cost per useful completion instead of cost per token, some counterintuitive findings emerge — and they're where the actual money gets saved.
Prompt caching is usually the single biggest lever
For workflows with repeated system prompts or shared context — RAG pipelines, customer support bots, code review tools, document processors — caching can reduce effective input cost by 80–90%. The provider with the best caching behavior for your specific access pattern will almost always beat the provider with the lowest list price. This alone can flip a comparison.
Model routing is dramatically underused
Most teams pick one model and route everything through it. That's like using a sledgehammer to hang every picture in your house. Routing straightforward tasks to smaller, cheaper models — Claude Haiku, Gemini Flash, GPT-4o-mini — while reserving frontier models for tasks that genuinely need deeper reasoning can cut total costs by 40–60% with no quality degradation. Tools like LiteLLM and Portkey make this routing trivial to implement.
Self-hosting has a narrower sweet spot than you think
Running Llama or Mistral on your own GPU cluster eliminates per-token costs entirely — but introduces infrastructure overhead, scaling complexity, and operational risk. The break-even point depends heavily on your volume and your team's ML ops maturity. For most teams processing under roughly 50 million tokens per month, managed APIs remain cheaper when engineering time is factored in.
What to Do This Week
If you're building or revising a cost comparison right now, here's the actionable checklist:
- Run the 500-completion test across your shortlisted providers using your actual production prompts — no synthetic benchmarks.
- Enable prompt caching everywhere it's available and measure real hit rates against your traffic pattern.
- Implement at least two-tier model routing — a cheap model for structured, well-defined tasks and a frontier model for ambiguous or creative ones.
- Revisit monthly. Provider pricing shifts fast. The cheapest option last quarter may not be this quarter.
The price war is real, and inference costs are genuinely dropping. But you only capture those savings if you're measuring the right thing. Token prices make for good press releases and competitive marketing. Cost per useful completion makes for good engineering.