TL;DR
- The Problem: LLM cold starts take seconds to tens of seconds because multi-gigabyte weights must move from storage into GPU VRAM before generating a single token.
- Three Bottlenecks: Cold starts stack pod scheduling, massive weight transfers, and CUDA/engine initialization into a single production-blocking delay.
- The MoE Factor: Even Mixture-of-Experts models with small active parameter counts suffer massive cold start times because the entire checkpoint must be resident in memory.
- The Fix: Warm pools eliminate cold starts by keeping containers running and weights pre-loaded in GPU VRAM, trading idle GPU costs for instant response times.
- Workload Matching: Use dedicated endpoints with a non-zero minimum replica floor for real-time applications like voice AI, and shared endpoints for asynchronous, latency-tolerant tasks.
Cold starts are the single most common reason a "serverless" LLM deployment feels lightning-fast during testing but becomes unusable in production. This post breaks down the mechanics of a cold start, what actually determines its duration, and how warm pools, when priced and sized correctly, eliminate the problem.
What Is a Cold Start in Serverless Inference?
A cold start is the latency delay that occurs between a request hitting an endpoint that has scaled to zero (or lacks a ready replica) and that endpoint returning its first token.
The severity of this delay depends entirely on the compute environment:
- CPU-Based Serverless: The cold start delay is typically sub-second because the system is only spinning up a lightweight function.
- GPU-Backed LLM Inference: The delay is drastically longer. The system must move a multi-gigabyte set of model weights from cold storage into GPU VRAM before a single token can be generated.
Because of this massive data transfer, cold starts in LLM serving are measured in seconds to tens of seconds, rather than milliseconds. This transforms what is normally a minor latency tax into a critical, production-blocking issue.
What Causes a Cold Start?
A cold start isn't just a single event; it is three sequential bottlenecks stacked on top of each other:
1. Model Weights Not Loaded in GPU Memory
This is typically the dominant cost factor. Model weights must be read from object storage or disk, staged through host RAM, transferred across the PCIe bus, and finally placed into GPU VRAM. The time required scales directly with model size and precision:
- At FP16: GPU memory required is roughly 2× the parameter count (e.g., a 70B model needs about 140 GB just to hold the weights).
- At FP8: Requirement drops to roughly half of FP16 (e.g., ~35 GB for the same 70B model).
Bigger models mean more bytes to move and longer cold starts. There is no bypassing this physical constraint; it is why a 7B model and a massive MoE model have entirely different cold start profiles.
2. Container/Pod Not Warm
Before weight loading can even begin, the orchestration layer must:
- Schedule a pod.
- Pull the container image (which can be several gigabytes for a modern inference stack complete with CUDA, drivers, and the serving framework).
- Initialize the container into a running state.
If the image is not already cached locally on the node, this pull time gets added directly on top of your delay.
3. CUDA Initialization and Engine Warm-Up
Once the container is running, the inference engine (such as vLLM, TensorRT-LLM, or a custom runtime) must still:
- Initialize the CUDA context.
- Allocate the KV cache.
- Capture CUDA graphs for the serving path.
This stage is largely CPU-bound rather than GPU-bound. Throwing a faster accelerator at it won't shorten it; it is fixed engine overhead that occurs on every cold boot regardless of model size.
The Formula: Total Cold Start = Pod Scheduling + Weight Loading + Engine Initialization
Cold Start Duration by Model Size
The scale of the problem becomes obvious when mapping real model sizes against performance impacts.
Note: The ranges below are order-of-magnitude estimates derived from weight-loading math (bytes to move ÷ realistic storage-to-GPU bandwidth). Treat them as a planning reference rather than a guarantee, as actual figures vary based on your storage backend, network path, and quantization method.
The Mixture-of-Experts (MoE) Factor
The MoE case requires a closer look. Even though only about 40B parameters are active per token during inference, the full set of expert weights must still be resident in GPU memory (or streamed in) before the router can dispatch tokens.
Consequently, cold start time tracks the total checkpoint size, not the active-parameter count. This explains why a model like GLM-5.2 behaves like a much larger model when booting cold, even though its steady-state serving cost tracks the smaller active-parameter figure.
How Warm Pools Solve the Problem
The fix for all three cold-start bottlenecks is simple in theory: do not let the cold path happen on the user's request.
A warm pool is a set of N replicas kept fully initialized in advance: containers are running, model weights are pre-loaded in GPU VRAM, and the CUDA context is live. Incoming requests route directly to a replica that has already bypassed all three cold-start stages.
What "Keeping N Instances Warm" Actually Means
Concretely, a warm replica eliminates every delay on the critical path:
- The container image is already pulled and running on the node.
- Model weights are resident in GPU VRAM, bypassing object storage reads.
- Inference engine has already finished CUDA context setup and graph capture.
Requests to a warm replica skip straight to token generation, while requests landing on a cold path pay the full sequential penalty.
The Cost of Warm Pools: Paying for Idle Compute
This is the trade-off no architecture can bypass: a warm GPU is a billed GPU, whether it is actively processing tokens or sitting idle. Keeping N instances warm continuously means paying for N×(GPU hourly rate) around the clock, even during off-peak hours.
To price and provision warm pools efficiently, follow these principles:
- Start from your traffic floor, not your average: Size your minimum warm replica count to your lowest-traffic period requiring responsiveness. Avoid your average load (which over-provisions) and your peak load (which defeats serverless economics).
- Segment your traffic by tolerance: Reserve warm floors for real-time, user-facing paths (chat, voice, autocomplete). Let batch, async, or internal jobs scale to zero and absorb the cold start.
- Treat idle-GPU cost as a fixed line item: Configure minimum replicas as your baseline infrastructure spend and maximum replicas as your burst ceiling independently.
- Re-check the floor periodically: Traffic patterns shift over time; a static warm-pool size set at launch will eventually lead to either wasted spend or unexpected latency spikes.
How Simplismart Eliminates Cold Starts
Simplismart treats cold starts as an infrastructure-layer challenge, offering fine-grained, per-endpoint warm pool configurations rather than a rigid, platform-wide default.
Configurable Minimum Instances Per Endpoint
Every deployment exposes min_pod_replicas and max_pod_replicas as first-class configurations managed directly via the SDK, CLI, or API:
Python
client.update_deployment_autoscaling(
deployment_id="deployment-uuid",
min_replicas=1,
max_replicas=3,
)
Setting min_pod_replicas above zero establishes the warm pool floor, ensuring those instances stay ready and never trigger a cold start. Conversely, max_pod_replicas caps burst expenditures.
For workloads that must never cold-start under any circumstance, scale-to-zero can be completely disabled (scale_to_zero_enabled: false).
Predictive, Traffic-Aware Sizing
Rather than relying on a static warm pool, Simplismart's autoscaling layer uses predictive, traffic-aware metrics, including request rate trends, concurrency shifts, and latency headroom, to scale proactively before load spikes hit. This matches the warm pool dynamically to usage patterns, avoiding the twin traps of 2:00 AM wastefulness and peak-hour deficits.
Sub-60-Second Provisioning From Zero
For workloads configured to scale to zero between traffic bursts, Simplismart optimizes the recovery path. Documentation indicates that pods can transition from zero to ready-to-serve in under 60 seconds, avoiding the multi-minute delays common to naive Kubernetes setups on large models.
In documented generative-media deployments, GPU readiness within a warm pool hovered around 1 second, with a full pod spinning up in roughly 7 seconds once underlying weights and engine states were cached.
Voice AI vs. Async Tasks: Choosing the Right Endpoint Type
Not every workload has the same latency requirements. Simplismart offers two distinct endpoint models, and selecting the right one is just as critical as tuning your warm pool size.
Dedicated Endpoints: The Right Answer for Real-Time Voice
A dedicated endpoint provides your deployment with exclusive infrastructure, zero compute-sharing with other workloads, absolute control over min_pod_replicas and max_pod_replicas, and complete isolation from external tenant traffic spikes.
For voice AI and conversational agents, dedicated infrastructure is mandatory:
- Zero tolerance for dead air: A 10-second cold start is a minor tax on a background batch job, but it is fatal in a voice product. When a user is live on the line, multi-second silence feels like a broken system.
- Guaranteed availability floors: Voice workloads require a non-zero min_pod_replicas configuration so the endpoint is never caught cold.
- Predictable SLAs: Isolation ensures your performance isn't impacted by shared resource pools calibrated for aggregate, unpredictable public demand.
Shared Endpoints: Cost-Effective for Async Tasks
A shared endpoint runs on Simplismart’s multi-tenant infrastructure, billing you strictly by API calls and usage rather than reserved GPU time.
This model is ideal for workloads capable of tolerating variable response times:
- Use cases: Batch text summarization, offline data enrichment, and non-interactive document processing.
- The economic trade-off: You trade away instant availability to eliminate the idle-GPU cost, letting the system absorb occasional cold starts in exchange for lower overhead.
The Takeaway
Serverless LLM cold starts are driven by three stacked bottlenecks: pod scheduling, massive weight loading, and engine initialization.
- Weight loading dominates as models scale. This explains why a massive 744B-class Mixture-of-Experts (MoE) model can trigger a 20–45 second delay on naive infrastructure, even if only a fraction of its parameters are active per token.
- Warm pools solve this by removing the load sequence from the active request path, trading off continuous payment for idle GPU time. Treating this as a conscious financial decision is vital.
- Match your endpoint to your workload: Deploy real-time applications (especially voice AI) to dedicated endpoints with a configured minimum replica floor, and route latency-tolerant background tasks to shared endpoints.
Frequently Asked Questions (FAQ)
What is a cold start in serverless LLM inference?
A cold start is the delay that occurs when a request hits an endpoint that has scaled to zero or lacks a ready replica, forcing the system to boot up from scratch before returning its first token. Unlike lightweight CPU functions that start in milliseconds, GPU-backed LLM cold starts take seconds to tens of seconds because multi-gigabyte model weights must be transferred from storage into GPU VRAM.
Why do Mixture-of-Experts (MoE) models have such long cold starts?
Even though an MoE model might only have a small number of active parameters per token during inference, the entire checkpoint (including all expert weights) must still be resident in GPU memory before the router can dispatch tokens. Consequently, cold start time tracks the total checkpoint size rather than the active-parameter count.
How do warm pools eliminate serverless cold starts?
Warm pools maintain a set of N replicas that are pre-initialized in advance. The container image is running, the model weights are already pre-loaded into GPU VRAM, and the inference engine's CUDA context is live. Incoming requests route directly to these warm replicas, bypassing all startup bottlenecks and delivering instant responses.
What is the financial trade-off of using warm pools?
A warm GPU is a billed GPU whether it is actively processing tokens or sitting idle. Keeping instances warm continuously means paying for continuous compute capacity around the clock. To manage this cost, you should size your minimum warm replica floor based on your lowest-traffic period rather than your peak or average load, and separate latency-sensitive traffic from asynchronous jobs that can tolerate scale-to-zero.
How does Simplismart handle cold start prevention?
Simplismart addresses cold starts at the infrastructure layer by providing first-class configurations for min_pod_replicas and max_pod_replicas via its SDK, CLI, and API. It pairs these controls with predictive, traffic-aware autoscaling and optimized recovery paths that can bring a pod from zero to ready-to-serve in under 60 seconds.
When should I use a dedicated endpoint vs. a shared endpoint?
Use dedicated endpoints with a non-zero minimum replica floor for real-time, user-facing applications like voice AI and conversational agents, where multi-second delays break the user experience. Use shared endpoints billed by usage for asynchronous, background workloads (such as batch summarization or offline document processing) that can absorb occasional cold starts in exchange for lower costs.
Ready to Eliminate Cold Starts in Your LLM Pipeline?
Production-grade AI demands better than multi-second delays and unpredictable latency. Whether you're building real-time voice agents that require instant responses or scaling cost-effective background processing pipelines, Simplismart gives you the infrastructure control you need, from fine-grained warm pool sizing to dedicated, isolated endpoints.
- Deploy in minutes: Configure your scaling floors, manage endpoints via SDK or API, and take total control of your serverless architecture.
- Scale intelligently: Leverage predictive, traffic-aware autoscaling that adapts to your users instead of reacting after the spike.
Stop letting cold starts stall your user experience.
Head over to Simplismart to sign up for free, explore our developer documentation, or book a demo with our engineering team today.
