How to Install llama.cpp on Ubuntu 24.04 (CPU and NVIDIA GPU)

Step 1 of 6Prepare the NVIDIA GPU Driver

Local models run on the CPU, but a GPU makes them many times faster. If your machine has an NVIDIA card, install the driver before anything else. Ubuntu 24.04 ships the ubuntu-drivers tool, which picks the recommended driver for your card:

sudo apt update
ubuntu-drivers devices
sudo ubuntu-drivers install
sudo reboot

After the reboot, check that the driver sees your GPU:

nvidia-smi

The table lists the GPU model, the driver version and how much memory (VRAM) is free. The CUDA Version field shows the newest CUDA release the driver supports, not a CUDA toolkit installed on the system. See how to install the latest NVIDIA drivers on Ubuntu if you need a specific driver branch.

Install the CUDA Toolkit

Building with GPU support also needs the CUDA compiler (nvcc). Add NVIDIA’s repository for Ubuntu 24.04 with the keyring package, then install the toolkit:

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb
sudo apt install ./cuda-keyring_1.1-1_all.deb
sudo apt update
sudo apt install cuda-toolkit -y

The toolkit lives in /usr/local/cuda, which isn’t on the PATH by default. Add it to your shell profile and check the compiler:

echo 'export PATH=/usr/local/cuda/bin:$PATH' >> ~/.bashrc
source ~/.bashrc
nvcc --version

Note: On an ARM64 server, replace x86_64 with sbsa in the keyring URL. Run apt-cache policy cuda-toolkit to see which toolkit version the repository offers for Ubuntu 24.04.

No NVIDIA GPU? You can still follow this guide on the CPU. Stick to small models (1B to 4B parameters) and expect slower answers.

Step 2 of 6Install the Build Dependencies

llama.cpp is built with CMake. Install the compiler, CMake, Git and the curl development headers, which llama.cpp uses to download models:

sudo apt update
sudo apt install build-essential cmake git libcurl4-openssl-dev -y

Step 3 of 6Build llama.cpp

Clone the official repository:

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp

Now build it. Pick the command that matches your hardware.

CPU-only build

cmake -B build
cmake --build build --config Release -j$(nproc)

NVIDIA GPU build (CUDA)

cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j$(nproc)

The build takes from a few minutes (CPU) to 15 minutes or more (CUDA). The programs end up in build/bin/. Check the result:

./build/bin/llama-server --version

To make the tools available system-wide, install them to /usr/local:

sudo cmake --install build
sudo ldconfig

Note: By default, a CUDA build is optimized for the GPU in the machine that builds it. To build once and copy the binaries to servers with different GPUs, add -DGGML_NATIVE=OFF to the first cmake command.

Step 4 of 6Download a Model and Chat

llama.cpp can download GGUF models straight from Hugging Face with the -hf option. This example uses a small Gemma 3 model published by the llama.cpp team:

llama-cli -hf ggml-org/gemma-3-1b-it-GGUF

The model is cached in ~/.cache/llama.cpp, so later runs start immediately. Type a question at the prompt and press Ctrl + C to quit. On a GPU build, add -ngl 99 to offload all model layers to the GPU.

Most popular open models have GGUF versions on Hugging Face in several quantizations. Q4_K_M is a good balance of size and quality; Q8_0 is close to the original quality but twice the size. Not sure which model fits your GPU? The LLM VRAM calculator estimates the memory a model needs for a given size, quantization and context length.

Step 5 of 6Run llama-server

llama-server loads a model once and serves it over HTTP:

llama-server -hf ggml-org/gemma-3-1b-it-GGUF --host 127.0.0.1 --port 8080 -ngl 99

Open http://127.0.0.1:8080 in a browser for the built-in chat page, or call the OpenAI-compatible API from another terminal:

curl http://127.0.0.1:8080/v1/chat/completions 
  -H "Content-Type: application/json" 
  -d '{"messages": [{"role": "user", "content": "Write a one-line bash command to count files in /etc"}]}' 

Useful llama-server options:

Step 6 of 6Run llama-server as a systemd Service

Download a model to a fixed location so the service doesn’t depend on your home directory. Create a folder and save a GGUF file into it (replace the URL with the model you want):

sudo mkdir -p /opt/models
sudo wget -O /opt/models/gemma-3-1b-it-Q4_K_M.gguf https://huggingface.co/ggml-org/gemma-3-1b-it-GGUF/resolve/main/gemma-3-1b-it-Q4_K_M.gguf

Create a dedicated system user and the unit file:

sudo useradd -r -s /usr/sbin/nologin llama
sudo nano /etc/systemd/system/llama-server.service

[Unit]
Description=llama.cpp server
After=network-online.target

[Service]
User=llama
ExecStart=/usr/local/bin/llama-server -m /opt/models/gemma-3-1b-it-Q4_K_M.gguf --host 127.0.0.1 --port 8080 -ngl 99 --api-key change-this-key
Restart=on-failure

[Install]
WantedBy=multi-user.target

If the file name in the URL doesn’t exist for your model, open the model’s Files tab on Hugging Face and copy the exact .gguf name. Then start the service:

sudo systemctl daemon-reload
sudo systemctl enable --now llama-server
systemctl status llama-server