Running a successful proof of concept with AI is cheap. Uploading a document to a public API, getting a good response, and moving on costs nothing or a few pounds. But running AI at scale — reliably, with acceptable latency, handling millions of requests — costs significantly more, and the hidden costs often exceed the obvious ones.
This article maps where the money goes so you can estimate before you start.
The layers of cost
There are four distinct cost layers, and not understanding the difference between them is why many projects exceed budget.
Token costs: the actual cost of running the model. A million tokens might cost £2 to £20 depending on the model. This is usually the smallest cost.
Infrastructure: servers, storage, and networking to serve the model. Often larger than token cost for high-traffic systems.
Operations and support: monitoring, logging, alerting, debugging, dealing with failures, keeping things running. Easily larger than infrastructure for mature systems.
Accuracy and liability costs: the cost of catching errors, reviewing outputs, fixing mistakes, and dealing with the consequences of incorrect AI output. Often the largest cost and usually not budgeted.
Token costs are not the budget
If you have a task that processes one million tokens a month, the token cost is perhaps £5-10. That feels trivial and leads to the assumption that AI is inexpensive.
But that is not the cost of the system. A system serving one million tokens per month might need to make requests to multiple models, retry failed requests, maintain fallbacks for when the primary system is overloaded, and cache results to avoid redundant calls. All of that increases effective token consumption.
More significantly, a system serving users at scale usually needs a mixture of models. You use a fast cheap model for routine queries, a more capable model for complex ones, and perhaps a smallest-cost model for classification and routing. Token cost scaling is sublinear; the cost to run at ten times the volume is usually three to five times the token cost.
Budget for 2-3 times the baseline token estimate, and more if your use case requires high quality or fallback paths.
Infrastructure scales with traffic, not with model capability
If your system processes 1,000 requests per minute, you need servers that can handle that throughput. The cost of those servers is largely independent of whether you are running a small model or a large one, and much larger than the token cost.
A rough framework: a single modern server can handle 100-500 API requests per minute depending on response latency. A system handling 1,000 requests per minute needs 3-5 servers, plus load balancing, storage for request history, and redundancy. That infrastructure costs £2,000-5,000 per month.
Token costs for 1,000 requests per minute across a full month might be £100-500. The infrastructure cost is ten to fifty times higher.
This dominates once you get past proof-of-concept scale. Many teams assume they are paying for tokens and discover too late that they are really paying for servers.
Operations is the hidden multiplier
A system that works correctly is not the same as a system that works in production.
An AI system in production needs:
Logging and monitoring: what happened in each request? Which failed? What was the latency? Were there errors? This logging often costs more to store and query than the compute itself. Expect to store 1-10MB per 1,000 requests, which costs £20-200 per month for a modest traffic system, scaling from there.
Alerting: if the system starts failing, who knows? If latency spikes, what is the alert? These need to be configured, tuned to avoid false positives, and monitored. The work is not heroic but it is ongoing.
On-call rotation: when the system fails at 2am, someone has to page out. Even if failures are rare, the on-call obligation has a cost. Two or three engineers covering on-call for a production system is at least £50k/year.
Incident response: when something goes wrong, someone has to investigate, understand the cause, fix it, and ensure it does not happen again. This is usually 50% of infrastructure team time for mature systems.
Observability tooling: beyond logging, you need distributed tracing, error tracking, and analytics. Tools like Datadog, New Relic or Honeycomb cost £500-2,000 per month depending on data volume.
In aggregate, operations is usually £3,000-8,000 per month for a modest system. The cost scales with traffic and with how much you care about reliability.
Accuracy and error costs are the largest hidden expense
Every AI system produces errors. The question is what those errors cost you and who bears the cost.
If your system uses AI to classify inbound emails, every classification error means an important message goes to the wrong folder. Someone has to eventually notice and fix it. The cost per error is small but the aggregate is noticeable. With 100,000 inbound emails per month and a 2% error rate, that is 2,000 miscategorisations per month — and someone (human or a cleanup system) has to handle each.
More seriously, if your system uses AI to make decisions that affect customers — credit decisions, content moderation, recommendations — errors can cause real damage. A credit decision error costs reputation and liability. A content moderation error costs trust.
The standard approach is to have humans review outputs. This is expensive. A human reviewing outputs costs £10-30 per hour depending on complexity and location. Reviewing 10,000 decisions per month at 10 decisions per hour is 1,000 hours, or £10-30k per month.
Some systems sample rather than audit everything. A 5% sample of 10,000 decisions is 500 reviews per month, or £5-15k per month. This catches patterns and outliers but misses systematic errors.
No large production AI system runs without human review of outputs. The cost of that review is usually the largest cost, and it is rarely included in the initial budget.
A budgeting framework
For a small system processing 1,000 requests per month:
- Tokens: £2-5
- Infrastructure: £100-300
- Operations: £200-400
- Review (if needed): £0-500
For a system processing 1 million requests per month:
- Tokens: £100-500
- Infrastructure: £2,000-5,000
- Operations: £2,000-5,000
- Review (if needed): £5,000-30,000
For a system processing 100 million requests per month:
- Tokens: £10,000-50,000
- Infrastructure: £10,000-50,000
- Operations: £5,000-20,000
- Review (if needed): £50,000-300,000
The review cost is highly variable depending on whether errors can be automated and what the cost of an error is.
How to keep costs down
Use the cheapest capable model. Fast models cost much less and are fine for most tasks. Use expensive models only for tasks that genuinely need the capability.
Cache aggressively. If the same query might appear multiple times, caching saves both tokens and infrastructure. A good cache hit rate — 30-50% for many applications — cuts cost dramatically.
Batch process when possible. Rather than serving individual requests in real-time, batch them and process overnight. This lets you use cheaper batch APIs and avoid the infrastructure cost of real-time serving.
Automate review where possible. Rather than humans reviewing every output, use simple heuristics to flag only outputs that are likely wrong. Catch 90% of errors with 20% of the review effort.
Start small and measure. Build the system for 10,000 requests per month before building for 10 million. Measure where the money actually goes. The results usually surprise people.
The hard truth is that token cost — the thing everyone focuses on — is often less than 10% of the total cost of production AI. The rest is infrastructure, operations, and accuracy assurance.