Contents
70B is the size most teams end up serving: big enough to be good, small enough for a single GPU once the weights are in FP8. B200 and B300 servers both clear that bar. So the useful question is not whether the model fits. It is how much memory is left once it does.
The short answer
Pick B300 for long contexts or many concurrent users. Pick B200 when requests are short, traffic is light, or you run several smaller models side by side. B300 costs €6.00 per GPU-hour against €5.00 for B200, and gives you about twice the cache per GPU in return.
Memory sets the batch size
While decoding, a GPU spends most of its time reading weights and KV cache out of memory. The weights are a fixed cost. The cache grows with every token of every active sequence, so the memory left after the weights decides how many requests you can batch together.
For a Llama-style 70B with grouped-query attention (80 layers, 8 KV heads, head size 128), one token of BF16 cache takes about 320 KB. An 8,192-token conversation needs roughly 2.7 GB.
What one server gives you
Every Nexpod server is a whole machine with eight GPUs on NVLink, so per-GPU numbers multiply cleanly. Run eight independent replicas with tensor parallelism of 1, or fewer, wider replicas when each request needs lower latency.
# eight replicas of a 70B FP8 model, one per GPUfor i in $(seq 0 7); do CUDA_VISIBLE_DEVICES=$i vllm serve ./llama-70b-fp8 \ --port $((8000 + i)) \ --max-model-len 32768 \ --gpu-memory-utilization 0.90 &doneTensor parallelism of 2 halves the weight cost per GPU and cuts time to first token, in exchange for some traffic between GPUs. Over NVLink that traffic is cheap, which makes TP=2 a sensible default for chat, where latency matters more than raw throughput.
Measure before you commit
- Start one B200 and one B300 server from the same image.
- Replay a day of real prompts at the concurrency you expect at peak.
- Compare tokens per second per euro, not tokens per second.
- Keep the winner and stop the other. Billing ends the second it stops.
nexctl servers create --family b200 --region eu-ams-1 --name bench-b200nexctl servers create --family b300 --region eu-ams-1 --name bench-b300nexctl servers ssh bench-b300 -- python bench.py --prompts day.jsonl --concurrency 64Conclusion
For a 70B model both families are a good fit. B300 earns its higher hourly rate when its larger batch stays busy; B200 is the better buy when it would not. Run both for an afternoon, read the bill, and decide with your own numbers. The server catalog lists every B200 and B300 model we rent.