Integrating AI spend into your FinOps Strategy

FinOps has been all about reigning in EC2 sprawl, right sizing S3 storage classes, and finding EBS volume orphans for ten years now. These areas are still very relevant but there is a new area that has emerged at the top of most cloud invoices and for which many finance and platform teams have not yet developed the discipline to manage: tokens.

The shift has happened fast. According to the FinOps Foundation’s sixth annual State of FinOps report based on 1,192 practitioners representing more than $83 billion in annual cloud spend the share of FinOps teams managing AI spend jumped from 31% two years ago to 98% in 2026. AI cost management is now the top forward-looking priority in the discipline and the single most-requested new skillset among practitioners. But 73% of organizations still overspent on AI relative to their cost expectations in the past year.

In this post, we will look at why AI cost management is different from regular cloud spend, what you learn from the data about how you spend your AI dollars, and how you can include AI spend as part of your FinOps approach.

Key Takeaways

• Despite the fact that 73% of organizations continue to overspend on AI, 98% of FinOps teams now manage AI costs versus just 31% two years ago (FinOps Foundation, State of FinOps 2026).
• It is believed that Inference, not Model Training, constitutes 80–90% of AI spend and that the use of GPUs during that Inference process happens 15–30% of the time (FinOps Foundation).
• Prompt caching is the highest-leverage architectural lever available: up to a 90% discount on cached input tokens with Anthropic’s API, and roughly 50% with OpenAI’s automatic caching.
• AI spend needs the same tagging, budgeting, and alerting discipline as compute and storage plus a few LLM-specific metrics that generic cloud cost tools don’t track.
• Ownership of AI cost governance is often unclear; FinOps teams need to explicitly bring engineering, product, and data science into the loop.

What “Tokenmaxxing” Actually Means

“Tokenmaxxing” is the pattern of maximizing model usage such as bigger context windows, more agentic tool calls, higher-tier models by default without matching cost discipline. It shows up as large system prompts resent on every turn, defaulting every workload to the most expensive model, unbounded agentic tool-call chains, and retrieval pipelines that over-stuff context “just in case.” None of this is unreasonable during a prototype; the problem is it rarely gets revisited once a feature scales. Unlike a forgotten EC2 instance, an unoptimized LLM call gets more expensive every time it runs — and it runs on every user interaction.

Why AI Spend Breaks Traditional Cloud FinOps

Classic FinOps assumes predictable workloads, capacity-tied spend, and commitment-based discounts. AI spend breaks all three. It’s metered per-request rather than per instance-hour, so cost volatility has no reserved-capacity lever to smooth it. Cost also scales with content, not just traffic the same feature gets pricier over time simply because conversation histories or retrieved context grow. And per the FinOps Foundation, 80–90% of AI spend sits in inference (not training), while GPU utilization during that inference commonly runs only 15–30% a large share of the bill pays for idle, provisioned capacity. Finally, model choice is itself a cost lever with no infrastructure equivalent: swapping to a smaller model can cut per-request cost sharply with little quality loss on well-scoped tasks.

The Highest-Leverage Fix: Prompt Caching

Source: Anthropic API pricing, mid-2026. OpenAI’s automatic caching yields roughly 50% off cached prefixes.

A prompt’s stable portion is system instructions, tool definitions, static reference docs can be cached after the first call, so reused prefixes bill at a steep discount. Anthropic bills cache reads at roughly 10% of the standard rate (a 90% discount), with a modest premium on the initial write. OpenAI’s caching is automatic and yields about 50% with zero code changes. The gap between theoretical and realized savings is usually a hit-rate problem: security firm ProjectDiscovery raised its cache hit rate from 7% to 84% and cut total LLM spend by 59–70% the pricing was there the whole time, but the prompt structure wasn’t using it. Caching also stacks with Anthropic’s Batch API, which discounts every token 50% for asynchronous jobs like summarization or bulk classification.

Building AI Cost Governance into FinOps

• Tag and attribute every call : route request metadata back to team, feature, and environment, the way cloud resources get cost-allocation tags.
• Set budgets and rate limits per workload: with 73% of organizations exceeding AI cost projections last year, waiting for the monthly invoice isn’t a strategy.
• Route by task complexity: UC Berkeley’s RouteLLM (ICLR 2025) cut costs over 85% on MT-Bench while keeping ~95% of frontier quality; Stanford’s FrugalGPT showed cascade routing cutting costs up to 98% in some settings. Production write-ups consistently land in a 40–85% range.
• Cache deliberately: order prompts with the stable system/tool-definition portion first and the variable user content last, since one changed token at the top invalidates the whole cache.
• Batch what doesn’t need to be real time: good candidates are nightly jobs and bulk classification; poor candidates are anything latency-sensitive, like interactive chat.
• Trim context deliberately: treat context size as a tunable cost variable: retrieve less, summarize, and truncate history rather than resending it in full.

Practical Levers to Cut Token Spend

None of these require switching vendors or renegotiating pricing. They’re the AI-era equivalent of right-sizing an instance or moving cold data to a cheaper storage tier, but where most of the documented savings actually live.

Metrics and Dashboards to Track

Traditional cloud cost dashboards track spend by service, region, and tag. An AI-aware FinOps dashboard needs a few additional signals layered on top:

• Cost per feature / per user session: not just total monthly spend, but spend normalized against usage, so a growing bill can be distinguished from a growing and inefficient bill.
• Average tokens per request (input vs. output): a rising trend line here is often the earliest warning sign of prompt bloat or unbounded context growth, well before it shows up as a budget overrun.
• Cache hit rate: the single most direct measure of whether caching investments are actually paying off; ProjectDiscovery’s move from 7% to 84% is the benchmark to aim toward, not the norm to expect immediately.
• Model mix over time: the share of calls going to each model tier, to catch quiet drift toward higher-cost models as defaults change or new engineers join a project.
• Cost per successful outcome: for agentic workflows, tracking cost against completed tasks (not just calls made) surfaces waste hidden in retries and failed tool-call chains.

Who Should Own AI Cost Governance

Source: FinOps Foundation, State of FinOps 2026 Report.

AI Cost Management is an area that combines the expertise of engineering, product management, data science, and finance. The move from 31% to 98% of FinOps teams owning AI Cost Management in two years has been faster than most organizations can figure out which group owns it. One possible solution: finance is responsible for visibility (tags, budgeting, reporting), engineering is responsible for levers via regular code reviews, and product is responsible for trade-offs on quality vs. cost.

Frequently Asked Questions

Prompt caching realistically saves up to 90% on Anthropic and ~50% on OpenAI but realized savings depend heavily on cache hit rate, as ProjectDiscovery’s 7%-to-84% jump shows. The fastest, lowest-risk wins are enabling caching on stable prompts and routing simple, high-volume tasks to smaller models. AI spend is worth budgeting as its own category, even inside the same overall cloud budget, because its growth drivers, usage, context length, model choice differ enough from compute and storage that lumping them together hides what’s actually driving cost.

The Bottom Line

Tokenmaxxing is not the bad guy; it is merely an outcome of optimizing for speedy shipping, which is correct in the beginning. The real issue lies in allowing this default behavior to continue once the particular functionality starts scaling, especially considering that 73% of organizations have already exceeded their AI budget estimates. The solution is not to slow down on AI adoption but to adopt the exact FinOps methodology that was used to control cloud spending a decade ago.