vLLM is a high-throughput inference server for large language models. Where Ollama and llama.cpp are built for one user on one machine, vLLM is built to serve many requests at once: its PagedAttention memory manager and continuous batching let a single GPU answer dozens of users in parallel. It exposes an OpenAI-compatible API, so existing apps can switch to your own models by changing a URL.
This guide covers installing vLLM on Ubuntu 26.04 LTS (Resolute Raccoon) with uv, serving a model from Hugging Face, protecting the API with a key, and running it as a systemd service.
Prerequisites
- Ubuntu 26.04 on x86_64.
- An NVIDIA GPU with compute capability 7.0 or newer (Volta, Turing, Ampere, Ada, Hopper, Blackwell). vLLM is GPU-first; it is not a good fit for CPU-only machines.
- Enough VRAM for the model: roughly 2 GB per billion parameters at 16-bit precision, plus room for the KV cache.
- At least 16 GB of system RAM and 20 GB or more of free disk space.
- A user with sudo privileges.
Step 1 of 6Prepare the NVIDIA GPU Driver
Local models run on the CPU, but a GPU makes them many times faster. If your machine has an NVIDIA card, install the driver before anything else. Ubuntu 26.04 ships the ubuntu-drivers tool, which picks the recommended driver for your card:
sudo apt updateubuntu-drivers devicessudo ubuntu-drivers installsudo reboot
After the reboot, check that the driver sees your GPU:
nvidia-smi
The table lists the GPU model, the driver version and how much memory (VRAM) is free. The CUDA Version field shows the newest CUDA release the driver supports, not a CUDA toolkit installed on the system. See how to install the latest NVIDIA drivers on Ubuntu if you need a specific driver branch.
No NVIDIA GPU? You can still follow this guide on the CPU. Stick to small models (1B to 4B parameters) and expect slower answers.
Step 2 of 6Install uv
vLLM’s documentation recommends uv, a fast Python package manager. uv can also download its own Python, so you don’t depend on the system Python 3.14 that Ubuntu 26.04 uses for its own tools. Install it for your user:
sudo apt install curl -ycurl -LsSf https://astral.sh/uv/install.sh | shsource ~/.bashrcuv --version
Step 3 of 6Create a Virtual Environment and Install vLLM
Create an isolated environment with Python 3.12 and activate it:
uv venv ~/vllm-env --python 3.12 --seedsource ~/vllm-env/bin/activate
Install vLLM. The –torch-backend=auto option checks your NVIDIA driver and picks the matching PyTorch CUDA build:
uv pip install vllm --torch-backend=auto
The download is several gigabytes, mostly PyTorch and CUDA libraries, so this can take a while. Confirm the installed version:
vllm --version
Step 4 of 6Serve Your First Model
Start the server with a small model to test the setup. Qwen/Qwen3-0.6B is an openly licensed model that fits on almost any GPU:
vllm serve Qwen/Qwen3-0.6B
vLLM downloads the model to ~/.cache/huggingface, loads it onto the GPU and starts the API on port 8000. When you see Application startup complete, open a second terminal and send a request:
curl http://localhost:8000/v1/chat/completions
-H "Content-Type: application/json"
-d '{
"model": "Qwen/Qwen3-0.6B",
"messages": [{"role": "user", "content": "List three uses of the tar command."}]
}'
You can also list the loaded models with curl http://localhost:8000/v1/models.
Gated models
Some models, such as Meta’s Llama family, require you to accept a license on Hugging Face first. Accept it on the model page, create an access token under Settings → Access Tokens, and export it before starting vLLM:
export HF_TOKEN=hf_your_token_here
Step 5 of 6Tune Memory and Secure the API
By default vLLM reserves 90% of the GPU’s memory, mostly for the KV cache, so it can batch many requests. These options matter most:
| Option | What it does |
|---|---|
--gpu-memory-utilization 0.80 | Share of VRAM vLLM may use (default 0.9). Lower it if other programs need the GPU. |
--max-model-len 8192 | Maximum context length. Lowering it saves memory and fixes most startup out-of-memory errors. |
--tensor-parallel-size 2 | Split one model across 2 GPUs in the same server. |
--api-key SECRET | Require Authorization: Bearer SECRET on every request. |
--host 0.0.0.0 --port 8000 | Listen address and port. |
Never expose vLLM to the internet without –api-key. Even with a key, it’s better to keep it on a private network or behind a reverse proxy with HTTPS. Not sure which model fits your GPU? The LLM VRAM calculator estimates the memory a model needs for a given size, quantization and context length.
Step 6 of 6Run vLLM as a systemd Service
Create a unit file so vLLM starts at boot. Replace youruser with the account that owns ~/vllm-env:
sudo nano /etc/systemd/system/vllm.service
[Unit]
Description=vLLM OpenAI-compatible server
After=network-online.target
[Service]
User=youruser
Environment="HF_HOME=/home/youruser/.cache/huggingface"
ExecStart=/home/youruser/vllm-env/bin/vllm serve Qwen/Qwen3-0.6B --host 127.0.0.1 --port 8000 --api-key change-this-key --max-model-len 8192
Restart=on-failure
RestartSec=10
[Install]
WantedBy=multi-user.target
sudo systemctl daemon-reloadsudo systemctl enable --now vllmjournalctl -u vllm -f
The log shows the model loading. Press Ctrl + C to stop following it; the service keeps running.
Alternative: Run vLLM with Docker
vLLM also publishes the vllm/vllm-openai image. It needs Docker plus the NVIDIA Container Toolkit so containers can use the GPU. Once both are installed:
docker run --gpus all -p 8000:8000 --ipc=host
-v ~/.cache/huggingface:/root/.cache/huggingface
vllm/vllm-openai:latest --model Qwen/Qwen3-0.6B
Update vLLM
source ~/vllm-env/bin/activateuv pip install -U vllm --torch-backend=autosudo systemctl restart vllm
Troubleshooting
- CUDA out of memory at startup: lower –max-model-len, lower –gpu-memory-utilization, or choose a smaller or quantized (AWQ, GPTQ, FP8) model.
- No CUDA GPUs are available: check that nvidia-smi works and that you activated the virtual environment you installed vLLM in.
- 401 Unauthorized: you started the server with –api-key; send -H “Authorization: Bearer change-this-key” with each request.
Uninstall vLLM
sudo systemctl disable --now vllmsudo rm /etc/systemd/system/vllm.servicerm -rf ~/vllm-envrm -rf ~/.cache/huggingface
Conclusion
vLLM is now serving a model on Ubuntu 26.04 through an OpenAI-compatible API. Point any OpenAI SDK at http://your-server:8000/v1, or add a chat interface with Open WebUI. For a single user on a laptop or desktop, Ollama is simpler to run.
How did it make you feel?
