How much GPU memory do you need to run a large language model locally? Pick a model size, a quantization format and a context length below. The calculator estimates the VRAM for the model weights, the KV cache and runtime overhead, and shows which common GPUs can hold it.
– GB of VRAM needed
- Model weights
- –
- KV cache
- –
- Runtime overhead
- –
Fits on
Estimates for Ollama and llama.cpp. vLLM reserves 90% of VRAM up front by default and uses the spare memory for more KV cache.
How the Estimate Works
The memory a model needs on the GPU has three parts:
- Model weights: number of parameters × bits per weight ÷ 8. An 8B model at 16-bit precision needs about 15 GB; at Q4_K_M (about 4.85 bits per weight) it needs about 4.5 GB.
- KV cache: the model’s working memory for the conversation. It grows with the context length and the number of parallel requests: 2 × layers × KV heads × head dimension × tokens × bytes per value. Llama 3.1 8B uses about 1 GB for an 8K context at FP16.
- Runtime overhead: CUDA context, compute buffers and fragmentation. The calculator adds 0.5 GB plus 5% of the weights.
Real usage varies by a few hundred megabytes between tools, drivers and model architectures, so leave some headroom. A result marked Tight uses more than 90% of the card and may fail with longer prompts.
Quantization Formats Explained
| Format | Bits per weight | Quality | Used by |
|---|---|---|---|
| FP16 / BF16 | 16 | Original | vLLM, Transformers |
| Q8_0 / FP8 | 8–8.5 | Practically lossless | llama.cpp, Ollama / vLLM |
| Q6_K, Q5_K_M | 5.7–6.6 | Very close to original | llama.cpp, Ollama |
| Q4_K_M | ~4.85 | Good; the usual default | llama.cpp, Ollama |
| AWQ / GPTQ 4-bit | ~4.25 | Good | vLLM |
| Q3_K_M, Q2_K | 3.4–3.9 | Noticeably worse | llama.cpp, Ollama |
Ollama’s default model tags are usually Q4_K_M. A larger model at Q4 generally beats a smaller model at Q8 of the same file size.
How to Fit a Bigger Model
- Shorten the context. Halving the context length halves the KV cache.
- Quantize the KV cache. llama.cpp and Ollama support an 8-bit KV cache, and vLLM supports FP8. It halves KV memory with little quality loss.
- Offload some layers to the CPU. llama.cpp (
-ngl) and Ollama do this automatically when VRAM runs out. It works, but every layer on the CPU slows generation down. - Use two GPUs. llama.cpp splits layers across cards, and vLLM supports tensor parallelism (
--tensor-parallel-size).
Run the Model
Once you know what fits, set up a runtime on Ubuntu 26.04 or 24.04:
| Tool | Best for | Guides |
|---|---|---|
| Ollama | Easiest setup, one user or a small team | Ubuntu 26.04 · Ubuntu 24.04 |
| llama.cpp | Full control, CPU and mixed CPU/GPU | Ubuntu 26.04 · Ubuntu 24.04 |
| vLLM | Serving many users from a GPU server | Ubuntu 26.04 · Ubuntu 24.04 |
| Open WebUI | Chat interface for any of the above | Ubuntu 26.04 · Ubuntu 24.04 |