Inference cost is a product decision

Here is the strangest fact in AI product economics: the price of intelligence collapsed by orders of magnitude, and margins barely moved. GPT-4-class output that cost around $20 per million tokens in late 2022 costs around $0.40 in 2026 - a 50x drop in three years, and equivalent-capability comparisons put the full decline as high as 1,000x, one of the fastest cost collapses in computing history. Meanwhile AI application companies still report gross margins of 50 to 60 percent against the 80 to 90 percent SaaS benchmark, and inference's share of total spend rises as products mature.

The resolution of the paradox is the thesis of this post: inference cost is not a procurement line that falls with market prices. It is the sum of dozens of product decisions - which model answers which request, what gets cached, how much reasoning a query deserves, what latency the user actually needs. Cheap tokens multiplied by wasteful product architecture equals an expensive product. The teams with healthy margins made the bill a design input. Here is how they do it.

Table of contents
1. Why falling prices didn't fix margins
2. The unit that matters: cost per successful outcome
3. The four product levers
4. Routing: the biggest lever, and the sharpest edge
5. Caching: margin you can switch on
6. Latency is a budget, spend it deliberately
7. The honest tradeoffs
8. A worked example
9. Takeaways

1. Why falling prices didn't fix margins

Three mechanisms eat the price decline before it reaches your P&L:

• Consumption grows into the price cut. Cheaper tokens make richer features viable - longer context, more retrieval, agentic loops - so usage per user action climbs. ICONIQ's benchmark data shows inference rising from 20 to 23 percent of total spend as products mature. Success makes the meter run faster.
• Reasoning models invert the pricing story. A reasoning-mode request can consume up to 100x more internal tokens than it outputs. Per-token price falls; per-answer cost quietly climbs. If your product defaults every query to a reasoning model, your unit economics are set by that default, not by the rate card.
• Waste is relative, and waste compounds. As one analysis puts it, the absolute cost of tokens keeps falling, but a product consuming 4x the tokens it needs pays 4x market rate forever - at $5 per million or at $0.50. Deflation does not forgive architecture.

And the deflation itself is slowing: the consensus expectation is 3 to 5x annual declines through 2027, tapering to 1.5 to 2x - not the 10x-per-year cliff of 2021 to 2025. The free ride is ending; what remains is engineering.

2. The unit that matters: cost per successful outcome

Token accounting is where cost work goes to die, because nobody upstairs thinks in tokens - they think in outcomes. The metric we instrument first on every engagement is:

cost_per_successful_outcome =
    total_inference_spend / count(outcomes_that_needed_no_redo)

# "Successful" is doing the work in that formula.
# A $0.002 ticket-classification that a human re-does is not
# a $0.002 outcome; it is a failed outcome plus a human minute.
# If half your outputs need cleanup, your real unit cost is
# more than double the dashboard number.

This one metric reframes every optimization that follows. Downgrading a model that drops success from 96 to 88 percent is not a cost cut; it is a cost increase wearing one. Conversely, a prompt change that lifts first-pass success by five points is a margin project as real as any infrastructure work. Cost and quality stop being separate meetings.

Supporting instrumentation, per the standard FinOps playbooks: tokens per user action (p50 and p95), calls per action, cache hit rate, and cost per request percentiles. An afternoon of tagging requests with feature and tenant IDs pays for itself in the first week of looking at the breakdown.

3. The four product levers

Every inference bill decomposes into the same four factors:

bill = requests
     x tokens_per_request
     x price_per_token(model)
     x (1 - cache_hit_rate)

# Lever 1: requests        - does this action need a model call at all?
# Lever 2: tokens/request  - context discipline, output caps, fewer shots
# Lever 3: model choice    - routing; the 5x-200x lever
# Lever 4: cache hits      - prompt caching, semantic caching

Notice that all four are product decisions. Whether a keystroke triggers a completion (lever 1), how much retrieved context gets stuffed into the prompt (lever 2), whether a password-reset question deserves a frontier reasoning model (lever 3), and whether the same question answered twice costs twice (lever 4) - none of these are settable by procurement. They are settable in design reviews, which is why the design review is where we put the cost conversation.

4. Routing: the biggest lever, and the sharpest edge

Model choice dominates the arithmetic. Within a single vendor lineup, small models run at roughly a fifth the price of flagships; across tiers, the spread is wider - one published support-workload example shows a 190x cost difference between a frontier model and a small model at 10,000 tickets a day. If even 60 percent of your traffic is genuinely simple, routing it to a small model is the single largest COGS lever you own. A distilled or fine-tuned small model trained on your domain frequently beats the general flagship on your task while costing a fraction to run.

Now the sharp edge. A Towards Data Science postmortem describes a team that built exactly this - a fast classifier trained on 200,000 historical queries, labeling each "simple" or "complex" in under 30 milliseconds. The bill fell. Then customer satisfaction fell, quietly, with a lag: the router misjudged edge cases, and users bailed to human support before ever filing the complaints that would have shown up in the eval set. By week sixteen the team had rolled the router back to conservative settings, and satisfaction did not recover until week twenty.

The lesson is not "don't route." It is that a router is a model, and it needs what every model needs: ground truth. Before the router ships, you need an eval pipeline with known-good answers per traffic class, a shadow period where routing decisions are logged but not enforced, and an escalation path (small model answers, judge checks, frontier model retries on failure). Cascades convert routing mistakes from silent quality loss into a measurable retry rate - which is the difference between a lever and a landmine.

5. Caching: margin you can switch on

Two caches, two different mechanics, both underused:

• Provider prompt caching. OpenAI, Anthropic, and Google all discount cached input tokens by 50 to 90 percent. If your requests share a long system prompt, tool definitions, or few-shot examples - and in production they almost always do - structuring prompts so the shared prefix comes first is close to free margin. This is an afternoon of prompt reordering, not a project.
• Semantic caching. For FAQ-shaped traffic, an embedding-similarity cache in front of the model eliminates 20 to 40 percent of inference entirely. The engineering is real (similarity thresholds, invalidation on content updates, per-tenant isolation) but the hit-rate math usually clears the bar within a quarter.

For self-hosted serving, the same logic extends down the stack: continuous batching (the production default in vLLM and TensorRT-LLM) has published throughput gains up to 23x over naive serving, quantization cuts model memory by up to 75 percent, and KV-cache compression research now reaches 16x compression at around 1 bit per channel. Stacked, teams report 60 to 80 percent bill reductions - but each layer needs its quality eval, per the same principle as routing: an optimization without a regression harness is a bug you scheduled.

6. Latency is a budget, spend it deliberately

Latency and cost are the same decision viewed from two sides, because the expensive configurations are usually the slow ones (big models, long reasoning chains) and the cheap accelerants (batching, queueing) trade latency the other way. So set latency budgets the way you set cost budgets - per feature, from user need:

• Interactive chat needs fast first tokens, which argues for small models with streaming, and escalation only when needed.
• Autocomplete needs sub-second everything, which rules out frontier models entirely - and that constraint, embraced early, saves you from designing a feature whose unit economics never had a chance.
• Batch summarization needs no interactivity at all, which means it belongs on the batch APIs at a standing 50 percent discount, running overnight. Any non-realtime workload paying realtime prices is a design oversight with a monthly invoice.

The anti-pattern is a single global configuration - one model, one timeout, one quality bar - because it forces every feature to pay for the most demanding feature's requirements. Per-feature budgets are the product expression of the routing lever.

7. The honest tradeoffs

What the cost-optimization playbooks tend to soft-pedal:

• Every lever adds a failure mode. Routers misclassify, caches serve stale answers, quantized models drift on edge cases, cascades add tail latency on escalation. You are trading a simple expensive system for a complex cheap one. The complexity is affordable exactly when you have evals; without them it is not.
• Optimization has a payroll cost. The 60 to 80 percent reductions come from engineers who could be building features. Below roughly $5k a month of inference spend, most of this post is premature - take the provider prompt cache, set output caps, and go build product. The levers earn attention when the bill rivals a salary.
• Model upgrades reset your tuning. Provider model refreshes change latency, quality, and cache behavior under your feet. Budget a re-evaluation pass per major model release; teams that skip it discover their router's "simple" class silently degraded.
• Margin structure is partly strategic, not tactical. The ICONIQ data shows application-layer companies reselling a model at 45 percent margins while product-plus-model innovators reach 53. If your product is a thin proxy on someone else's model, no cache fixes that; the fix is up the value chain.

8. A worked example

Composite numbers from a support-automation product of the shape we see most often - 1M requests a month, naive architecture: every request to a frontier model, no caching, no routing:

Baseline:   1M req x 3k tok  x frontier rate          = $8,800/mo

Route 65% to small model (evals + shadow period first) = -$4,900
Provider prompt cache (shared 1.2k-token prefix)       = -$1,100
Semantic cache, 25% hit rate on FAQ traffic            = -$  900
Batch tier for overnight digest jobs                   = -$  400
                                                  ------------
Result:                                          ~ $1,500/mo
Success-rate change (cascade with judge):    96.1% -> 95.8%
Cost per successful outcome:              $0.0092 -> $0.0016

An 83 percent bill reduction with a quality change inside noise - consistent with the 60 to 80 percent reductions reported across the industry playbooks. The order matters: evals existed before the router, the shadow period ran before enforcement, and the success-rate column sat next to the savings column in every review. Same levers, wrong order, and this table becomes the Towards Data Science postmortem from section 4.

9. Takeaways

• Token prices fell by orders of magnitude; margins didn't follow. Consumption growth, reasoning-token multipliers, and architectural waste ate the difference - and the deflation is slowing.
• Instrument cost per successful outcome first. It merges the cost and quality conversations into one honest number.
• The bill is four product levers: request necessity, tokens per request, model choice, cache hits. All of them are set in design reviews.
• Routing is the biggest lever (5x to 190x) and the sharpest: no router without ground-truth evals, a shadow period, and an escalation cascade.
• Take the free money: provider prompt caching, output caps, batch APIs for non-realtime work.
• Below ~$5k a month, skip most of this. Above it, treat inference cost as a product KPI with an owner, a target, and a place on the roadmap.

References

1. AI inference economics: the 1,000x cost collapse (GPUnex)
2. AI inference cost economics 2026: GPU FinOps playbook (Spheron)
3. Unit economics of AI products (Institute PM)
4. AI unit economics: pricing and margins framework (Digital Applied / ICONIQ data)
5. We built a routing layer to cut our AI costs. It broke the product. (Towards Data Science)
6. Reduce AI inference costs: caching, batching, quantization (All Days Tech)
7. LLM inference optimization techniques (Runpod)
8. KV cache is 1 bit per channel: coupled quantization (arXiv)

Hero image: Igor Omilaev on Unsplash.

We help teams ship AI products whose margins survive success. Explore our other insights or get in touch if you would like to talk it through.