How to Install Ollama on Ubuntu 24.04 and Run LLMs Locally

Step 1 of 5Prepare the NVIDIA GPU Driver

Local models run on the CPU, but a GPU makes them many times faster. If your machine has an NVIDIA card, install the driver before anything else. Ubuntu 24.04 ships the ubuntu-drivers tool, which picks the recommended driver for your card:

sudo apt update
ubuntu-drivers devices
sudo ubuntu-drivers install
sudo reboot

After the reboot, check that the driver sees your GPU:

nvidia-smi

The table lists the GPU model, the driver version and how much memory (VRAM) is free. The CUDA Version field shows the newest CUDA release the driver supports, not a CUDA toolkit installed on the system. See how to install the latest NVIDIA drivers on Ubuntu if you need a specific driver branch.

No NVIDIA GPU? You can still follow this guide on the CPU. Stick to small models (1B to 4B parameters) and expect slower answers.

Step 2 of 5Install Ollama

Ollama publishes an official install script. It downloads the binaries to /usr/local, creates an ollama system user, sets up a systemd service and detects NVIDIA or AMD GPUs. Make sure curl is present, then run the script:

sudo apt install curl -y
curl -fsSL https://ollama.com/install.sh | sh

If you’d rather read the script before running it, download it first with curl -fsSL https://ollama.com/install.sh -o install.sh, review it, and then run sh install.sh.

When the script finishes, confirm the version and that the service is running:

ollama --version
systemctl status ollama

The service should show active (running). Ollama now listens on 127.0.0.1:11434, so only programs on this machine can reach it.

Step 3 of 5Download and Run Your First Model

Models come from the Ollama library. Start with a small one to confirm everything works. llama3.2 is Meta’s 3B-parameter model and needs about 2 GB of disk space:

ollama run llama3.2

The first run downloads the model, then opens an interactive prompt. Type a question and press Enter. Type /bye to exit. Useful commands for managing models:

Run ollama ps while a model is loaded. The PROCESSOR column should say 100% GPU on a machine with a supported GPU. A split such as 40%/60% CPU/GPU means the model doesn’t fit in VRAM; pick a smaller model or a lower-precision tag.

Not sure which model fits your GPU? The LLM VRAM calculator estimates the memory a model needs for a given size, quantization and context length.

Step 4 of 5Use the Ollama REST API

Every model you download is also available over HTTP. Send a test request with curl:

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.2",
  "prompt": "Explain what a Linux inode is in two sentences.",
  "stream": false
}' 

Ollama also offers an OpenAI-compatible endpoint at /v1/chat/completions. Most tools and SDKs that support OpenAI can use your local models by setting the base URL to http://localhost:11434/v1 and any non-empty API key:

curl http://localhost:11434/v1/chat/completions 
  -H "Content-Type: application/json" 
  -d '{"model": "llama3.2", "messages": [{"role": "user", "content": "Hello!"}]}' 

Step 5 of 5Allow Access from Other Machines (Optional)

To use Ollama from another computer or from a container, make it listen on all interfaces. Create a systemd override:

sudo systemctl edit ollama

Add these lines in the editor, then save and close it:

[Service]
Environment="OLLAMA_HOST=0.0.0.0:11434"

Reload systemd and restart the service:

sudo systemctl daemon-reload
sudo systemctl restart ollama

Important: the Ollama API has no authentication. Anyone who can reach port 11434 can run and delete your models. Allow only trusted addresses with UFW, for example a single office or VPN subnet:

sudo ufw allow from 192.168.1.0/24 to any port 11434 proto tcp

For access over the internet, put Ollama behind a reverse proxy with HTTPS and authentication, or use a chat front end such as Open WebUI, which has its own user accounts.