Contents
A cold start is the time a new worker needs before it can answer: claim a GPU, start the container, load the weights, warm up. On a busy endpoint it happens every time traffic grows faster than the warm pool. Here is how to keep it short.
Where the seconds go
We timed a cold start for a 13B model on an H100 worker, using a full development image and weights downloaded at boot. The figure shows one run of that setup next to the same endpoint after the changes below. Your numbers will depend on image size and model.
Two phases took most of the time: pulling a large image and fetching weights over the network. Neither needs a change to the model.
1. Keep the weights next to the GPU
Downloading weights on every boot is the slowest option. Attach a volume that already holds them, or bake them into the image if they rarely change. A volume in the same region reads at local NVMe speed.
{ "name": "chat-13b", "family": "H100", "region": "eu-fra-1", "image": "registry.example.com/chat-13b:1.4", "volume": { "id": "vol_weights_13b", "mount": "/models" }, "workers": { "min": 1, "max": 24 }, "idle_timeout_s": 300}2. Slim the image
Build from a runtime base image instead of a full development image, and keep compilers in a separate build stage. Every gigabyte you cut is a gigabyte no worker has to pull.
3. Keep one worker warm
Setting min to 1 keeps a single worker ready at all times. You pay for that one worker by the second, and the first request after a quiet spell never waits. For endpoints with steady daytime traffic, a schedule that raises min during office hours works well.
4. Load once, serve many
Load the model when the module imports, outside the request handler. The warm-up cost is then paid once per worker rather than once per request.
import nexpodfrom vllm import LLM llm = LLM(model="/models/chat-13b") # runs once per worker @nexpod.handlerdef run(job): out = llm.generate(job["prompt"], max_tokens=job.get("max_tokens", 256)) return {"text": out[0].outputs[0].text}Check your own numbers
Each request in the console lists queue time, cold start and execution time as separate fields. Watch the cold start column after every change. If it does not move, the bottleneck is elsewhere. More on endpoints in the Serverless overview.