NOTES · NOV 9, 2026 · BY TECH BENDERS
Keeping the cost of AI under control
Tokens, caching, model routing and budgets, explained for decision makers. Mechanisms documented by the providers, with no invented prices.
AI costs rarely explode on day one. They creep. A feature that cost almost nothing in testing reaches thousands of users, a prompt grows a few paragraphs at a time, an agent starts reading whole documents instead of the parts it needs, and one month the invoice is a line someone has to explain.
The good news is that the mechanisms to keep that cost under control are well documented and not complicated. This post explains them for decision makers, without code and without prices: prices change, mechanisms last. For each one we link to the provider's official documentation.
Where the money goes: tokens
Language models are billed by tokens, small pieces of text. Providers publish prices per million tokens, and they price two things separately:
Input tokens: everything you send the model. The instructions, the conversation so far, the documents, the results of any tools it called.
Output tokens: everything the model writes back.
In Anthropic's price table, output tokens cost several times more than input tokens for every model listed. Two consequences follow. First, a long answer is more expensive than a long question. Second, input adds up silently: in an agent or a chat, every new turn resends the history, so the same context is paid for again and again unless you do something about it.
That is why most cost control is really context control: what you send, how often you send it, and to which model.
Lever 1: cache what repeats
Most AI applications send the same thing over and over: a long system prompt, a set of tool definitions, a policy document, the earlier part of a conversation. Prompt caching lets the provider store that repeated prefix and reuse it.
How it works, according to the official documentation:
The cache matches a prefix: the beginning of the request, in a fixed order (tools, then system instructions, then messages). Any change in the cached part breaks the match, so stable content goes first and changing content goes last.
By default a cache entry lives five minutes, and it is refreshed at no extra cost each time it is used. A one-hour option exists at additional cost.
Writing to the cache costs a little more than normal input (1.25 times the base input price for the five-minute cache, twice the base price for the one-hour cache). Reading from it costs a fraction (0.1 times the base input price on most models, and less on some newer ones).
The documentation spells out the break-even: caching pays off after one cache read for the five-minute duration, or after two for the one-hour duration.
Also, on the Claude API, tokens read from the cache do not count toward the input-tokens-per-minute rate limit. On AWS, Amazon Bedrock also supports prompt caching.
The question to ask your team: what share of our input tokens is read from the cache? The Claude Console shows cache rate on its Usage page, and every API response reports how many tokens were written to and read from the cache.
Lever 2: batch what can wait
Not every AI task needs an answer in seconds. Classifying last month's support tickets, summarizing a document archive, generating product descriptions overnight: these can wait.
For that kind of work, Anthropic's Batch API processes requests asynchronously "with a 50 percent discount on both input and output tokens." The official pricing page also states that the prompt caching multipliers stack with the Batch API discount, so the two can be combined.
The question to ask your team: which of our AI workloads are interactive, and which only look interactive because nobody separated them?
Lever 3: send each task to the right model
Using the most capable model for everything is the simplest setup and often the most expensive one. Anthropic's own pricing guidance for agents is to "choose Haiku for simple tasks, Sonnet for most production workloads, and Opus for the most complex reasoning." Routing is the practice of making that choice per task instead of once per application.
There are three ways to do it, from simplest to most automated:
Route by task type in your own code. Classification, extraction and short replies go to a smaller model; complex reasoning goes to a larger one. This is explicit and easy to audit.
Tune effort within one model. Some models expose an effort parameter that, in the documentation's words, lets you "control how many tokens Claude spends when responding," trading thoroughness for token efficiency with a single model. For many routes this is simpler than maintaining several models.
Use a managed router. On AWS, Amazon Bedrock intelligent prompt routing routes requests between models in the same family by predicting response quality per request. AWS lists its limitations plainly: it is optimized for English prompts, it cannot adjust to application-specific performance data, and it may not route optimally for specialized use cases.
Whichever you choose, measure the right thing: cost per completed task, not cost per request. A cheaper model that needs three attempts, or a human correction, is not cheaper.
Lever 4: budgets and limits that actually stop spending
Monitoring tells you what happened. Limits decide what is allowed to happen. Both matter.
Spend limits on the provider. In the Claude Console, an organization can set its own monthly spend limit below the cap of its usage tier. When usage reaches that limit, requests stop with an error until the limit is raised or the month resets.
Limits per workspace. The same documentation describes custom spend and rate limits for each workspace, so one project or team cannot consume the budget of the others. (You cannot set a spend limit on the default workspace, so put real projects in their own.)
Budgets on the cloud account. If AI runs on AWS, AWS Budgets can alert on actual and forecasted spend and trigger actions. AWS warns that there can be a delay between a charge and its notification, so a budget alert is a safety net, not a real-time brake.
The question to ask your team: if a bug made our agent loop all weekend, what would stop it, and how much would it cost before it stopped?
Architecture decides more than any discount
Discounts reduce the price of tokens. Architecture decides how many tokens exist in the first place, and who pays for them.
When we built the Open Data Agent, the main cost decision was not a discount. It was where the reasoning lives: in the account of the person using the assistant, with our server only translating requests to public data and returning JSON. The infrastructure bill and the intelligence bill never mix. Then, inside that design, the tools aggregate on the source and return small results, because every row returned becomes someone's input tokens.
The same thinking applies to any AI feature:
Send the part of the document that matters, not the whole document.
Summarize or trim long histories instead of resending them forever.
Do counting, filtering and grouping in the database, not in the model.
Cap output length where a short answer is enough.
A one-page checklist
Before an AI feature reaches production, a decision maker should be able to get a clear answer to each of these:
Which model handles each task, and why that one?
What is our cache hit rate, and what content is meant to be cached?
Which workloads run in batch?
What does one active user cost per month, and how is that measured?
What spend limits exist, at which level (organization, workspace, cloud account), and who gets alerted?
What happens to cost if usage grows ten times?
If the team cannot answer, the cost is not under control yet. It is just small for now.
Go further
The path Generative AI on AWS covers running generative AI in production with security guardrails and cost controls. The path AI for leaders is for people who make the decisions without writing the code. And the workshop Automate a business process end to end ends with an estimate of what the automation costs to run.
If you want a second opinion on the cost of an AI system you already run, talk to us.