What Is AI Inference?

Most explanations of AI inference start with a definition and end before the part that matters. This one goes the other way around.

Training and inference are different activities

Training is how a model is built. You take a very large amount of text, run it through an architecture with billions of adjustable parameters, and repeatedly correct those parameters until the model predicts text well. It happens once, it takes months, it needs a large cluster, and when it finishes you have a file. A set of weights.

Inference is what happens every time anyone uses that file. The weights do not change. The model is not learning from the question. It is doing a fixed computation: given this input, produce the most probable continuation.

The distinction matters commercially because the costs behave in opposite directions. Training is a large one-time expense that a handful of organisations in the world take on. Inference is a small recurring expense that happens on every single request, forever, and that is the one almost every company is actually paying.

What happens when a model answers

Walk through one request.

Tokenisation. Your text is split into tokens, which are chunks of characters, roughly three quarters of a word on average. "Infrastructure" might be two or three tokens. Each becomes a number.

Prefill. The model reads your entire prompt at once and builds an internal representation of it. This step is parallel, so it is fast relative to its size, and it scales with how long your prompt is. A short question is cheap here. A question with forty pages of retrieved documents attached is not.

Decode. Now the model generates. It produces one token, appends it to the sequence, and runs again to produce the next. This is sequential and cannot be parallelised, because token five depends on token four existing. Every token in the answer is another pass through the model.

That asymmetry is the single most useful thing to understand about inference. Reading your prompt is one parallel operation. Writing the answer is hundreds of sequential ones. It is why a long answer takes noticeably longer than a long question, and why "make it more concise" is also "make it cheaper."

The KV cache

If every generated token required re-reading the whole sequence from scratch, generation would be unusably slow. So the model keeps its intermediate work from previous tokens in memory and reuses it. This is the KV cache.

The cache is why generation is fast. It is also why inference is memory-hungry in a way that surprises people who budget only for the weights. The weights are a fixed size. The cache grows with every concurrent user and every token of context, and it lives in the same GPU memory. Long conversations and many simultaneous users compete for the same space.

This has a practical consequence. When several requests could be served by the machine that already holds the relevant cached context, sending one of them elsewhere means recomputing work that already existed. Distributing requests without regard for what is cached where costs throughput that never shows up as an error, only as a slower system.

Throughput and latency are not the same goal

Two numbers get used interchangeably and should not be.

Latency is how long one user waits. Time to first token, then how quickly tokens arrive after that.

Throughput is how many tokens the system produces per second across everyone using it.

Batching improves throughput by processing multiple requests together, which uses the GPU far more efficiently than one request at a time. It can also make any individual request slightly slower, because it waits to travel with company. A chat interface where someone is watching the cursor is a latency problem. An overnight document classification job is a throughput problem. Tuning for one degrades the other, so the first question about any inference deployment is which of the two you are actually optimising.

Where inference runs

The same weights can be executed in several places, and this is where the architectural decisions live.

A hosted API means the provider runs it. You send a prompt over the internet and receive tokens back. Simple to adopt, and your prompts are processed on infrastructure you do not control.

Your own servers means you run it. The weights sit on hardware you own or rent, the prompts do not leave your network, and you take on the operational work of keeping it healthy.

The computation is identical in both cases. What differs is who holds the data while it happens, who can see the request, and what record exists afterwards.

Why this is the part that matters

Inference is where an AI system meets real data. Not test data, not training data. The actual customer record, patient note, contract clause or case file that someone pasted into a prompt at half past four on a Tuesday.

Which means inference is where every question a regulator will ask gets answered, or does not. Which employee sent that request. Which model received it. What was in it. What came back. Whether any of that was written down.

Training is where a model's capabilities come from. Inference is where an organisation's obligations land. Those are different problems, and only one of them is yours to solve.

AI declaration: this article was drafted with AI assistance and reviewed, fact-checked and edited by the Xinity team before publication.