Best Local LLMs for 8GB, 12GB, 16GB and 24GB GPUs (2026)

The model you can run on your own machine depends on one number: how much GPU memory (VRAM) you have. Pick a model that is too big and Ollama spills layers onto the CPU, and answers slow to a crawl. Pick one that is too small and you leave quality on the table.

This guide gives our picks for every common amount of VRAM, from CPU-only laptops to 80 GB data-centre cards. The tables are not a snapshot: they are generated from the TecAdmin AI models directory, which syncs download sizes from the Ollama library every day, so they stay current as new models come out.

How We Picked the Models

• Ollama default builds. Sizes are for the default 4-bit (Q4_K_M) download, which is what ollama run pulls when you don’t name a quantization.
• 8K-token context. The VRAM figure is the download size plus about 10% runtime overhead and 0.5 GB for an 8K context. Longer contexts need more; size them with the LLM VRAM calculator.
• Quality per GB first. Within each tier we put the strongest current model first and add alternatives with a different strength (vision, licence, reasoning, coding).
• Headroom. A pick must fit the tier with room to spare for the desktop, other apps and a longer context.

All Picks at a Glance

Click any command to copy it. Each model name links to its family page, with every size, the licence and the context window.

Best LLMs for CPU Only (4–6 GB of RAM)

No GPU? Small models still run at a usable speed on a modern CPU. Expect roughly 5 to 15 tokens per second from a 2B–4B model on a recent desktop processor, depending on memory bandwidth. Stick to 4B parameters or fewer and keep the context short.

Best LLMs for an 8 GB GPU

Eight gigabytes covers the RTX 4060, RTX 3070 and most gaming laptops. The sweet spot is a 4B–8B model at 4-bit, which leaves 2–3 GB for context. Qwen3.5 4B is the best all-rounder here and also reads images; Gemma 4 E4B adds audio input.

Best LLMs for a 12 GB GPU

With 12 GB (RTX 3060 12GB, RTX 4070) you can step up to 9B–14B models, which are noticeably better at following long instructions and writing code. Qwen3.5 9B is the default pick; Phi-4 is worth a try for maths and logic.

Best LLMs for a 16 GB GPU

Sixteen gigabytes opens the 20B class. gpt-oss-20b is a mixture-of-experts model with only 3.6B active parameters, so it is fast as well as capable. Magistral Small (24B) is the largest dense model that fits. Popular 24B models such as Mistral Small 3.2 and Devstral Small 2 need about 17 GB, so on a 16 GB card they spill slightly onto the CPU; they are listed in the 24 GB tier.

Best LLMs for a 24 GB GPU

A 24 GB card (RTX 3090, RTX 4090) or a Mac with 32 GB of unified memory runs today’s best “single-GPU” models at 4-bit. Qwen3.8 27B is our top pick for general and coding work. The MoE models in this tier (Gemma 4 26B, Qwen3-Coder 30B) activate only 3–4B parameters per token, so they generate much faster than dense models of the same size.

Best LLMs for 48 GB (Two GPUs or a Workstation Card)

At 48 GB you can run 70B dense models or keep a 30B-class model at 8-bit with a long context. Ollama splits a model across two GPUs automatically.

Best LLMs for an 80 GB GPU

One A100 or H100 80GB, or a Mac with 96–128 GB of unified memory, runs 100B-class models. For production serving to many users, use vLLM instead of Ollama.

Check How Much VRAM You Have

On a Linux server with an NVIDIA GPU, nvidia-smi shows the total and free memory per card:

nvidia-smi --query-gpu=name,memory.total,memory.used --format=csv

After loading a model, ollama ps shows how much memory it uses and whether it runs fully on the GPU. Anything other than 100% GPU in the PROCESSOR column means part of the model runs on the CPU, which is much slower:

ollama run qwen3.5:9b "Say hello"
ollama ps

How to Fit a Bigger Model

• Shorten the context. The KV cache grows with every token of context. Setting OLLAMA_CONTEXT_LENGTH=8192 for the Ollama service keeps it small.
• Quantize the KV cache. With OLLAMA_FLASH_ATTENTION=1 and OLLAMA_KV_CACHE_TYPE=q8_0, the cache takes about half the memory with little quality loss.
• Pick an MoE model. Mixture-of-experts models still need all their weights in memory, but they stay usable when part of the model runs from system RAM.
• Use a smaller quantization. A Q3 build saves about 20% over Q4_K_M, but quality drops more on small models. See LLM quantization explained.

To set these on a systemd install, run sudo systemctl edit ollama, add the variables under [Service] as Environment="OLLAMA_CONTEXT_LENGTH=8192" lines, then restart the service.

Next Steps

Install Ollama with our guide for Ubuntu 26.04 or Ubuntu 24.04, then add a web interface with Open WebUI. For coding, see the best Ollama models for coding. To browse every model with filters for memory and abilities, open the AI models directory.