When you run ollama run qwen3:8b, you download a 5.2 GB file, not the 16 GB the original weights take. The difference is quantization: storing each weight with fewer bits. It is the reason a 27B model fits on a 24 GB gaming card, and choosing the right level is the easiest way to trade a little quality for a lot of memory.
What Quantization Does
A model is billions of numbers (weights). Training stores them as 16-bit floating-point values (FP16 or BF16), so every billion parameters takes about 2 GB. Quantization rounds those numbers to fewer bits, typically 8 or 4. Clever schemes keep the most important weights more precise, so the model loses far less quality than the bit count suggests.
The rough memory rule is: size in GB ≈ parameters in billions × bits per weight ÷ 8, plus memory for the context (the KV cache) and the runtime.
Common Formats and Their Size
| Format | Bits per weight | 9B model | Quality |
|---|---|---|---|
| FP16 / BF16 | 16 | ~18 GB | Original weights |
| Q8_0 | ~8.5 | ~9.6 GB | Practically lossless |
| Q6_K | ~6.6 | ~7.4 GB | Very close to the original |
| Q5_K_M | ~5.7 | ~6.4 GB | Small loss |
| Q4_K_M | ~4.85 | ~5.5 GB | Good; the usual default |
| Q3_K_M | ~3.9 | ~4.4 GB | Noticeable loss on small models |
| Q2_K | ~3.4 | ~3.8 GB | Large loss; last resort |
Bits per weight for the K-quants are averages, because they mix precisions inside each block. Real downloads differ a little because some layers (such as embeddings or vision encoders) stay at higher precision. Here are the real Ollama sizes for the text-only Qwen3 family, synced from the Ollama library:
What the Names Mean
- Q4, Q5, Q8: the target number of bits per weight.
- _K: a “k-quant”, llama.cpp’s block-wise scheme that stores a scale per small block of weights. Almost every modern GGUF uses k-quants.
- _S, _M, _L: small, medium, large. Medium keeps more of the sensitive layers at higher precision than small. Q4_K_M is the usual balance.
- _0 (Q8_0, Q4_0): older, simpler block formats. Q8_0 is still the standard 8-bit choice.
- IQ (IQ3_XS, IQ2_M): “i-quants” that use an importance matrix to squeeze quality out of 2–3 bits. Good for very large models on small hardware.
Other Formats You Will See
- GGUF is the file format used by llama.cpp and Ollama. All the Q-names above are GGUF quantization types.
- AWQ and GPTQ are 4-bit formats for GPU servers such as vLLM.
- FP8 is an 8-bit float format with hardware support on NVIDIA Ada, Hopper and newer GPUs. Many vendors publish official FP8 weights.
- MXFP4 and NVFP4 are 4-bit floating-point formats. gpt-oss ships natively in MXFP4, and NVIDIA publishes Nemotron in NVFP4 for its newest GPUs.
How to Choose a Quantization in Ollama
A plain tag like qwen3:8b pulls the default build, usually Q4_K_M. To pick another level, add it to the tag. The tags page of each model on ollama.com lists what exists:
ollama pull qwen3:8b-q8_0ollama list
Rules of thumb:
- Prefer a bigger model at Q4_K_M over a smaller one at Q8_0. A 32B model at 4-bit beats an 8B model at 8-bit on almost every task.
- Use Q8_0 when the model fits easily and you want maximum accuracy, for example for code or maths with a small model.
- Avoid going below Q4 on models under 10B parameters; small models suffer most.
- For very large models that barely don’t fit, a Q3 or IQ3 build can be better than offloading layers to the CPU.
Do Not Forget the Context
Quantization shrinks the weights, but every token of context also needs memory in the KV cache. A 32K context can add several gigabytes. Ollama can quantize the cache too: set OLLAMA_FLASH_ATTENTION=1 and OLLAMA_KV_CACHE_TYPE=q8_0 for the service to roughly halve it.
To work out the total for your setup, use the LLM VRAM calculator, and see which models fit your card in best local LLMs by GPU memory.
How did it make you feel?
