Model inference is the process of running a trained machine learning model on new inputs to produce outputs. For large language models, it means generating text one token at a time, and LLM serving is the engineering of doing this efficiently at scale.
What is model inference?
Model inference is the phase in which a trained model is used to make predictions on new, unseen inputs. The model's parameters are fixed: no learning happens, no gradients are computed, and no weights are updated. Inference simply runs the forward pass of the network and returns a result, whether that is a classification label, an embedding vector, or, in the case of a large language model (LLM), a sequence of generated tokens.
For LLMs, inference is the work that happens every time a user sends a prompt and receives a response. Because that response is produced one token at a time, LLM inference has performance characteristics that differ sharply from both traditional model inference and from the training that created the model. The discipline of running these models efficiently, reliably, and cost-effectively in production is commonly called LLM serving.
Inference is also where the majority of an AI system's lifetime cost typically accumulates. Training is a one-time or periodic event, but inference runs continuously for as long as the model is in production and serving users.
- Inference = using a fixed, already-trained model to produce outputs.
- No weight updates, no backpropagation, only the forward pass.
- For LLMs, output is generated autoregressively, token by token.
- LLM serving is the systems engineering of doing inference at scale.
Inference vs. training: two different workloads
Training and inference are distinct workloads with different hardware bottlenecks and cost structures. Training adjusts model weights by repeatedly running forward and backward passes over large datasets, computing gradients, and updating parameters. It is largely compute-bound and runs as a bursty, scheduled job that finishes after days or weeks, at which point its cost stops.
Inference, by contrast, runs continuously once a model is deployed and is latency-sensitive because real users wait on each response. For LLMs the decode step is frequently memory-bandwidth-bound: the bottleneck is how quickly model weights and the cache can be moved from GPU memory to the compute units, not raw arithmetic throughput. This is why GPU memory bandwidth often matters more than peak compute for serving.
These differences have real budget consequences. Industry estimates commonly attribute a majority of enterprise AI GPU spend to inference rather than training, because inference cost scales with request volume and never fully switches off while a service is live.
- Training: compute-bound, bursty, finite duration, cost ends when the run ends.
- Inference: latency-sensitive, continuous, cost scales with usage and never fully stops.
- Training updates weights; inference keeps weights frozen.
- Inference decoding is often memory-bandwidth-bound rather than compute-bound.
How LLM inference works: prefill, decode, and autoregressive generation
LLM inference proceeds in two phases. The first is prefill, in which the model processes every token of the input prompt in parallel through its self-attention layers and computes the key and value tensors for those tokens. Because all input tokens are handled together and attention cost grows quadratically with sequence length, prefill is generally compute-bound and dominates the time to the first output token.
The second phase is decode, in which the model generates output tokens one at a time. This is autoregressive generation: each new token is predicted from all preceding tokens, then appended to the sequence and fed back in to predict the next token. Decode repeats until the model emits a stop token or hits a length limit. Because each step generates only a single token while repeatedly reading a growing cache, decode is typically memory-bound rather than compute-bound.
The split matters for optimization. Prefill and decode stress different parts of the hardware, so serving systems often tune, schedule, or even physically separate them to keep accelerators busy.
- Prefill: processes the whole prompt in parallel, builds the cache, compute-bound.
- Decode: generates one token per step, autoregressive, memory-bound.
- Prefill largely sets time to first token; decode sets ongoing generation speed.
- Generation halts at a stop token or a maximum-length limit.
The KV cache and why it matters for speed and memory
Self-attention requires the key (K) and value (V) tensors of every prior token to predict the next one. Recomputing them from scratch at every decode step would make generation cost grow quadratically with output length. The key-value cache (KV cache) avoids this by storing the K and V tensors as they are computed, so each new step only computes K and V for the single new token and reuses the rest. This turns what would be a repeated quadratic operation into a linear one and is the single most important reason LLM decoding is practical.
The trade-off is memory. The KV cache grows with both the number of concurrent requests and the length of each sequence, and it can consume a large share of GPU memory. Naive memory allocation wastes much of it: the 2023 vLLM work reported that prior serving systems lost roughly 60 percent to 80 percent of KV cache memory to fragmentation and over-reservation, which limited how many requests could be batched together.
Managing this cache efficiently is therefore central to high-throughput serving and motivates techniques such as paged memory layouts discussed below.
- KV cache stores past key/value tensors so they are computed once, not every step.
- Converts quadratic per-step attention cost into linear cost.
- Cache size grows with sequence length and concurrency, pressuring GPU memory.
- Inefficient allocation historically wasted roughly 60 to 80 percent of cache memory.
Latency metrics: time to first token, tokens per second, throughput
Because generation is incremental, a single latency number is not enough to describe LLM inference performance. Practitioners track several complementary metrics. Time to first token (TTFT) measures the delay between sending a request and receiving the first output token; it is dominated by prefill and is the strongest driver of perceived responsiveness in interactive applications.
Time per output token (TPOT), sometimes called inter-token latency, measures the average time to produce each token after the first. Its inverse is the per-stream generation speed in tokens per second. End-to-end latency for a response can be approximated as TTFT plus TPOT multiplied by the number of generated tokens.
Throughput is a system-level metric: the total number of output tokens generated per second across all concurrent requests. It usually trades off against latency, since larger batches raise throughput and hardware utilization but can increase the latency seen by any individual request. Capacity planning balances these.
- TTFT: time to the first token, set mainly by prefill, drives perceived speed.
- TPOT / inter-token latency: average time per subsequent token; its inverse is tokens/sec per stream.
- Approximate latency = TTFT + TPOT x number of output tokens.
- Throughput: total tokens per second across all requests, the key capacity metric.
Cost drivers: model size, context length, batching, and hardware
Inference cost is driven by how much compute and memory each request consumes and how well the hardware stays utilized. Model size is the most direct lever: larger parameter counts require more memory and bandwidth per token and run more slowly. Context length matters because longer prompts increase prefill cost (which scales quadratically with sequence length) and enlarge the KV cache that must be held in memory throughout generation.
Batching improves cost-efficiency by amortizing the expense of streaming model weights across many simultaneous requests, raising throughput per GPU. Hardware choice rounds out the picture: since decode is often memory-bandwidth-bound, accelerators with higher memory bandwidth can outperform what raw compute figures alone would predict for serving workloads.
Because hosted APIs typically bill per input and output token, these same factors map directly onto price: longer contexts and longer responses cost more, and larger models command higher per-token rates.
- Model size: more parameters mean more memory, more bandwidth, slower tokens.
- Context length: longer inputs raise prefill cost and KV cache footprint.
- Batching: amortizes weight movement across requests, raising throughput per GPU.
- Hardware: memory bandwidth often matters more than peak compute for decode.
Optimization techniques: quantization, batching, speculative decoding, and caching
Several techniques reduce inference latency and cost, and they are often combined. Quantization lowers the numerical precision of model weights, and sometimes activations, from 16-bit or 32-bit floating point down to 8-bit or 4-bit representations. This shrinks the memory footprint and the bandwidth needed to move weights, speeding up the memory-bound decode phase, with modern low-precision formats designed to minimize quality loss.
Continuous batching keeps the GPU busy by slotting new requests into the running decode loop as soon as capacity frees up, rather than waiting for an entire batch to finish, which substantially raises throughput under concurrent load. Speculative decoding, introduced by Leviathan and colleagues in 2022, uses a small, fast draft model to propose several tokens ahead, which a larger target model then verifies in parallel; correct guesses are accepted and the rest discarded, accelerating generation without changing the output distribution.
Memory management techniques target the KV cache directly. PagedAttention, introduced in the 2023 vLLM paper, borrows the idea of paging from operating-system virtual memory: it partitions the KV cache into fixed-size blocks so memory is allocated on demand and shared across requests. The work reports near-optimal cache usage with under 4 percent waste and 2 to 4 times higher throughput than prior serving systems at comparable latency. Prompt or prefix caching reuses the prefill work for shared prompt prefixes across requests.
- Quantization: lower-precision weights cut memory and bandwidth, speeding decode.
- Continuous batching: insert new requests into the decode loop to keep GPUs busy.
- Speculative decoding: a draft model proposes tokens, a target model verifies in parallel, outputs unchanged.
- PagedAttention: paged KV cache cuts memory waste to under 4% and raises throughput 2 to 4x.
- Prefix/prompt caching: reuse prefill for shared prompt prefixes.
Self-hosting vs. hosted inference APIs (model serving)
Teams running models in production choose between hosting inference themselves and calling a managed API. Self-hosting means deploying a serving stack, often an open-source inference engine such as vLLM, on owned or rented GPUs. It offers maximum control over latency, data residency, model choice, and per-token economics at scale, but it requires provisioning hardware, managing batching and autoscaling, and absorbing idle-capacity cost.
Hosted inference APIs, offered by model providers and cloud platforms, abstract the serving stack away and bill per token or per request. They remove operational burden and scale elastically, which suits variable or early-stage workloads, but offer less control over the underlying infrastructure and can become more expensive at very high, steady volume.
The decision usually comes down to scale, control, and total cost of ownership. A common pattern is to prototype on a hosted API and migrate high-volume, latency-critical, or privacy-sensitive workloads to self-hosted serving as usage grows.
- Self-hosting: full control and better unit economics at scale, but operational overhead.
- Hosted APIs: zero serving ops and elastic scaling, billed per token or request.
- Open-source engines such as vLLM are common foundations for self-hosting.
- Choice hinges on scale, latency needs, data sensitivity, and total cost of ownership.
Inference in memory-augmented and retrieval-based systems
In retrieval-augmented generation (RAG) and memory-augmented systems, inference is invoked more than once per user interaction and on more than one type of model. An embedding model runs inference to convert documents and queries into vectors for semantic search, and a generation model then runs inference over the retrieved context to produce the final answer.
Retrieval directly shapes inference cost: injecting retrieved passages into the prompt lengthens the context, which raises prefill cost and KV cache size. Effective retrieval, prompt construction, and prefix caching therefore have a measurable impact on both latency and price in these systems.
AI memory applications and second-brain tools such as MemX rely on this pattern, running embedding inference to index stored documents, photos, and voice notes and generation inference to answer plain-English questions over what was retrieved.
- RAG runs inference twice: embedding for retrieval, then generation over the retrieved context.
- Retrieved context lengthens prompts, raising prefill cost and KV cache footprint.
- Prefix caching and tight retrieval reduce latency and cost in memory-augmented systems.
Key takeaways
- Inference runs a fixed, trained model to produce outputs with no weight updates; for LLMs this means generating text one token at a time.
- LLM inference has two phases: a parallel, compute-bound prefill that processes the prompt, and an autoregressive, memory-bound decode that emits tokens one by one.
- The KV cache makes decoding efficient by reusing past key/value tensors, but it consumes large amounts of GPU memory and is a primary serving bottleneck.
- Performance is measured with multiple metrics: time to first token, time per output token (and its inverse, tokens per second), and system throughput.
- Quantization, continuous batching, speculative decoding, and paged KV cache memory (PagedAttention) are the main levers for cutting inference latency and cost.
Frequently asked questions
Related terms
Related reading
Sources
- Efficient Memory Management for Large Language Model Serving with PagedAttention (arXiv:2309.06180)
- Fast Inference from Transformers via Speculative Decoding (arXiv:2211.17192)
- vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention (vLLM Blog)
- Mastering LLM Techniques: Inference Optimization (NVIDIA Technical Blog)
- Key metrics for LLM inference (BentoML LLM Inference Handbook)
- How PagedAttention resolves memory waste of LLM systems (Red Hat Developer)
Put the idea into practice
MemX is an AI memory agent built on these ideas: store anything, skip the folders, and find it again by asking in plain English.
Try MemX Free