Key Takeaways
- LLM inference splits into prefill (parallel, compute-bound) and decode (sequential, memory-bound) phases with very different resource profiles.
- The KV cache avoids recomputation but grows linearly with context length and concurrent users, becoming a bigger memory bottleneck than the model weights themselves.
- Continuous/in-flight batching keeps GPUs busy by swapping requests in and out at every generation step, instead of waiting for full batches to finish.
- Throughput and latency are fundamentally in tension; bigger batches raise total tokens/sec, but slow down each user's response.
- At scale, these dynamics compound into real infrastructure cost, which is why purpose-built, optimised inference hosting exists to manage batching, KV cache, and GPU allocation intelligently rather than over-provisioning hardware.
Every time you type a prompt into ChatGPT, Claude, or any AI product, a hidden process kicks in behind the scenes called inference, the stage where a trained model actually generates a response. Training happens once; inference happens millions of times a day, which is why it quietly becomes the highest ongoing cost of running an LLM product.
This post breaks down how LLM inference actually works, the two-phase generation process, the KV cache, batching strategies, and the throughput-vs-latency trade-off and why all of this makes inference expensive as usage scales.
The Two Phases of LLM Inference: Prefill vs. Decode Explained
Large language models (LLMs) do not generate complete answers in a single shot. Instead, they produce responses one token at a time. This complex generation process is split into two highly distinct phases, creating unique architectural challenges for AI infrastructure.
Here is a breakdown of how inference happens under the hood.
The Core Phases of Inference
1. The Prefill Phase
During the prefill phase, the entire input sequence is processed simultaneously. The computational costs and workload during this phase scale directly with the length of your input prompt. Because everything is processed at once, this stage is capable of parallel processing and requires massive raw compute power.
2. The Decode Phase
Once the initial context is built, the model enters the decode phase, generating one new token at a time. To avoid redundant computation, this phase relies heavily on the KV cache to store and recall previous tokens. Because it must wait for each token to be generated before predicting the next, this phase is strictly sequential and heavily constrained by memory bandwidth rather than raw compute.
Why This Matters for GPU Optimisation
According to NVIDIA's engineering documentation, understanding the split between these two phases is critical for scaling AI.
The prefill phase is highly compute-intensive, whereas the decode phase is latency-sensitive. Because these workloads are so drastically different, utilising a disaggregated approach, which separates the processing of the two phases, ensures optimal GPU utilisation and significantly increases overall throughput.
The Architectural Challenge: This phase split is the root reason why LLM inference is fundamentally hard to engineer. A single user request has two entirely different operational personalities: a short, compute-heavy burst (prefill), followed immediately by a long, thin, memory-heavy trickle (decode).
The KV Cache: The Memory That Makes LLM Generation Possible
To generate token #500 without recomputing tokens #1–499 from scratch, Large Language Models (LLMs) store intermediate attention data in a critical structure known as the KV cache (Key-Value cache). Created during the prefill phase, the KV cache sits at the core of an LLM's attention mechanism, helping the model focus on relevant parts of the input during text generation.
By maintaining a KV cache, the model only computes the Key and Value vectors for the new token at each step. It appends these to the cache and reuses the fully cached set to compute attention, completely bypassing the need to recompute the entire sequence.
While essential for efficiency, the KV cache introduces high cost and scaling challenges. The cache grows linearly with prompt length and must reside in high-speed GPU memory during the generation process to ensure fast access. Crucially, it scales not just with context length, but with concurrent users as well.
NVIDIA's own benchmarks make the magnitude of this memory footprint clear. A KV cache representing a 128k-token context window for a single user (batch size 1) consumes about 40 GB of memory with Llama 3 70B, and this requirement scales linearly with the number of users. That is, before you even load the model weights, which for a model like Llama 3 70B in half precision already require approximately 140 GB of GPU memory.
Why This Matters for AI Infrastructure Costs:
To manage these massive memory requirements, providers rely on advanced optimisation techniques. Common strategies include paged KV cache and KV cache offloading, which shift cache data to cost-efficient storage during inference to reduce overhead. Furthermore, frameworks like TensorRT-LLM include native optimisations such as support for paged KV cache, quantised KV cache, circular buffer KV cache, and KV cache reuse. Together, these techniques drastically reduce inference costs while ensuring a smooth, scalable user experience.
Batching: The Core Lever for LLM Inference Throughput
GPUs are inherently built for parallel workloads. Processing a single LLM request at a time wastes the vast majority of a GPU's computational capacity, which is why inference servers group multiple requests together into batches. However, the method used to manage these batches drastically impacts performance and cost.
The traditional approach, known as static batching, requires all requests in a batch to finish before any new request can begin. This creates significant inefficiencies; if one prompt requires a long response, the GPU sits idle while shorter requests in the same batch finish early.
Conversely, modern AI infrastructure relies on continuous batching (also referred to as iteration-level or in-flight batching). This dynamic strategy forms batches of requests at every single iteration step. By immediately swapping completed requests out and injecting new ones into the newly freed slots, the server maximises throughput without waiting for the entire batch to finish.
Static vs. Continuous Batching at a Glance:
According to NVIDIA's Triton Inference Server documentation, achieving in-flight batching requires the backend to break request processing into individual steps. At the end of each step, the model instance releases the completed requests and reschedules the ones that are still generating. Because real-world request lengths vary so drastically, continuously batching existing and new requests together drives massive improvements in both throughput and latency.
Triton also gives AI operators a direct dial for tuning this process. If a system's default configuration results in latency values that are well within budget, engineering teams can increase the maximum batch size or set a non-zero batch delay. This allows systems to intentionally trade a slight increase in latency for a significant boost in overall throughput.
Throughput vs. Latency: The Central Trade-off in LLM Inference
The crux of why AI inference systems are so difficult to optimise is that throughput and latency constantly pull in opposite directions. Balancing these two forces is the core challenge of scaling Large Language Models.
The Key Inference Metrics
Time to First Token (TTFT) directly dictates a system's responsiveness in interactive applications and primarily reflects the efficiency of the prefill phase. Conversely, Inter-Token Latency (ITL) measures decode performance. This is critical for streaming applications, where inconsistent generation speeds lead to noticeable, user-facing stuttering.
The tension between these metrics is explicit in AI research: performance metrics exhibit deep interdependence, creating a strict trade-off where optimising for throughput almost always sacrifices latency, and vice versa.
NVIDIA's own KV-cache guidance underscores this exact tension. While key-value caching prevents costly recomputation during the decode phase, it can quickly lead to memory bottlenecks, especially when utilising large batch sizes or long sequence lengths. Efficient KV cache management via techniques like PagedAttention is required to limit memory wastage, which in turn enables the larger batch sizes needed for high throughput.
Why Bigger Batches = Better Throughput but Worse Latency
When you add more requests to a batch, you alter the system dynamics in two specific ways:
- Higher Throughput: Each GPU pass now produces tokens for more users simultaneously, maximising hardware efficiency.
- Higher Latency: Every individual request must wait for the entire batch to be processed at each step, slowing down the delivery of individual tokens.
This architectural reality is why real-time chat products (which strictly require low TTFT) and bulk batch-processing jobs (which demand maximum throughput) require fundamentally different serving configurations, even when running on the exact same hardware.
Why LLM Inference Gets Expensive at Scale
When you put all these architectural pieces together, the primary cost drivers of scaling Large Language Models become starkly clear.
The Ultimate Consequence
Without careful engineering, infrastructure teams end up buying far more GPU capacity than they actually need simply to survive memory spikes and unpredictable utilisation dips. This expensive over-provisioning is precisely the gap that specialised, highly optimised inference hosting is built to close.
Conclusion: Mastering the Economics of AI Inference
Deploying Large Language Models at scale is ultimately a balancing act between computational physics and infrastructure economics. As we’ve explored, the inherent complexities of inference, from the disjointed hardware demands of the prefill and decode phases to the relentless memory consumption of the KV cache, make it impossible to rely on traditional, static computing models.
The fundamental tension between throughput and latency dictates that every AI application must be uniquely tuned. There is no one-size-fits-all approach; engineers must actively choose whether to prioritise lightning-fast Time to First Token (TTFT) for real-time chat, or maximise overall GPU utilisation for bulk processing.
As context windows expand and user adoption accelerates, brute-forcing performance by simply buying more GPUs is a financially unsustainable strategy. The future of scalable AI deployment depends entirely on purpose-built inference infrastructure. By leveraging intelligent optimisations like continuous batching, PagedAttention, and dynamic resource allocation, organisations can stop paying for idle compute, overcome memory bottlenecks, and deliver high-performance AI experiences cost-effectively.
Frequently Asked Questions (FAQ)
What are the two distinct phases of LLM inference?
LLM inference is split into the prefill phase and the decode phase. During prefill, the model processes the entire input prompt at once in a highly parallel, compute-intensive burst to build initial context. During decoding, the model generates the response sequentially, one token at a time, which relies heavily on memory bandwidth rather than raw compute.
What is the KV cache, and why is it essential?
The Key-Value (KV) cache is a memory structure created during the prefill phase that stores intermediate attention data. It allows the model to compute the vectors for only the new tokens and reuse the previously cached context, entirely bypassing the need to recompute the entire prompt sequence for every single new word.
Why does the KV cache drive up infrastructure costs?
The KV cache grows linearly with both the length of the context window and the number of concurrent users. Because this massive amount of data must reside in high-speed GPU memory for fast access, it rapidly consumes available VRAM. At scale, the KV cache often becomes a much larger memory bottleneck than the actual model weights.
How does continuous batching differ from static batching?
Static batching requires an entire batch of requests to finish before any new ones can begin, leaving GPUs idle when shorter requests finish early. Continuous (or in-flight) batching solves this by swapping completed requests out and injecting new ones in at every single generation step, ensuring the GPU maintains consistently high utilization.
What is the main trade-off in AI inference optimization?
The core architectural tension is between throughput (total tokens generated per second across the system) and latency (the speed at which an individual user receives their response). Increasing batch sizes improves overall throughput by maximizing hardware efficiency, but it degrades per-request latency because each request must wait for the entire batch to process at each step.
Why do real-time chat and bulk processing need different configurations?
Interactive chat applications are highly latency-sensitive; they require a fast Time to First Token (TTFT) so the user doesn't feel a delay. Bulk batch-processing jobs, on the other hand, care only about maximizing total throughput. Because throughput and latency pull in opposite directions, running both workloads optimally requires fundamentally different serving configurations, even on the exact same hardware.
Why is specialized inference hosting necessary at scale?
Without careful engineering, the conflicting demands of compute-heavy prefills, memory-bound decodes, unpredictable request lengths, and massive KV cache footprints lead to severe GPU underutilization and memory spikes. Specialized inference infrastructure employs dynamic resource allocation and advanced caching techniques to manage these bottlenecks intelligently, preventing teams from buying vastly more expensive GPU capacity than they actually need.
