Ollama on AMD GPUs: every backend (CPU only, Vulkan, ROCm)

On an AMD machine, Ollama doesn't have a single "the GPU works or it doesn't" answer the way it often feels on NVIDIA. You have several distinct backends — CPU only, Vulkan, ROCm — and they differ wildly in setup effort and performance. Picking the right one matters as much as picking the right model. This guide walks through each option, how to configure it and how to choose depending on whether you're coding, running agents and what hardware you own.

Why Ollama has several AMD backends at all

Ollama runs on llama.cpp and llama.cpp itself doesn't talk to your GPU directly. It delegates the heavy lifting to an acceleration layer: CUDA on NVIDIA and on AMD either ROCm (AMD's full compute stack, HIP-based) or Vulkan (a cross-vendor graphics/compute API). Each backend ships as a different set of libraries, which is why an install can be GPU-accelerated on one machine and silently CPU-bound on the next.

The good news: you don't reinstall Ollama to switch. You control the behaviour with environment variables and, in some cases, separate builds. Here's the comparison table of all three modes up front — the rest of the article covers each in depth.

At a glance: CPU vs Vulkan vs ROCm

Before anything, here's how to know what your current Ollama is doing.

Check what you're currently running

The check takes two steps: load a model, then look at where it runs.

# Load a model (any will do):
ollama run mistral

Wait for the model to finish loading and generate a reply. In a second terminal, check where the model runs:

# Is the model on GPU or CPU?
ollama ps

The PROCESSOR column is the honest answer: 100% GPU means full offload, while something like 48%/52% CPU/GPU means the model is split — usually a sign your model doesn't fit in VRAM. In that case, no backend switch will save you: pick a smaller model or a lower quantization first (see the complete Ollama guide).

Option 1: CPU only

Sometimes you want to force the CPU — to test, to compare or because the GPU is busy with something else (a game, a render). Ollama has no one-line "CPU only" flag, but you can force it reliably by hiding the GPU from the process.

How to force CPU only

The mechanism is the same one used to exclude a GPU in multi-GPU setups: mask the device files so Ollama can't see them.

If you run Ollama as a systemd service (the common case on GNU/Linux), you can add the env var to the service — see option 2 for the exact snippet. Simpler and more portable: run a one-off server with the GPU masked. Stop the service, then run it in a terminal:

sudo systemctl stop ollama
# Run a CPU-only server in a terminal:
HIP_VISIBLE_DEVICES=-1 ollama serve

HIP_VISIBLE_DEVICES=-1 tells the ROCm layer that zero GPUs are visible, so every layer runs on CPU. The Vulkan build has its equivalent — see option 2.

CPU only: good or bad fit?

Option 2: GPU with Vulkan

Vulkan is the "it just works" path: one universal backend that runs on AMD, NVIDIA and Intel GPUs alike, with no vendor compute stack to install.

How to configure it

Vulkan support is built into recent Ollama releases and is enabled by default on GNU/Linux when Ollama can't (or isn't configured to) use ROCm. Two things to know:

1. Make sure ROCm isn't hijacking the run. If ROCm libraries are installed but something's off, you can explicitly prefer Vulkan:

# Prefer the Vulkan backend over ROCm
OLLAMA_VULKAN=1 ollama serve

2. Or go Vulkan-only by hiding the GPU from HIP. If you want to force the Vulkan path (for instance to A/B it against ROCm):

HIP_VISIBLE_DEVICES=-1 OLLAMA_VULKAN=1 ollama serve

The differences with ROCm are summarized in the at-a-glance table early in this article. The practical summary: Vulkan is the default choice — zero install, broad support and within 10–20% of ROCm on most models. ROCm wins when you chase every last token/second or run big models where its better memory efficiency buys you an extra layer in VRAM.

[Service]
Environment="OLLAMA_VULKAN=1"

Option 3: GPU with ROCm

ROCm is AMD's answer to CUDA: the official, best-optimized compute stack. When it works, it's the fastest way to run models on an AMD GPU. When it doesn't, it's the most demanding to set up — which is why we have a whole separate article on installing ROCm on GNU/Linux. The short version:

amdgpu-install -y --usecase=rocm --no-dkms

(The --no-dkms part matters: it avoids the unstable DKMS kernel module and keeps the in-kernel AMD driver.)

Which GPUs are actually supported

This is the part that decides everything. Officially supported cards are mostly RDNA2 and RDNA3 discrete GPUs — the RX 6000 and RX 9000/7000 families — plus datacenter Instinct cards. Many other cards work anyway via the HSA_OVERRIDE_GFX_VERSION trick, which makes ROCm pretend your GPU is a supported one:

Set the override the same way as any other env var:

HSA_OVERRIDE_GFX_VERSION=11.0.0 ollama serve

[Service]
Environment="HSA_OVERRIDE_GFX_VERSION=11.0.0"

Verify ROCm sees your card

rocminfo | grep -i gfx
# or, for a quick health check:
rocm-smi

If rocminfo lists your GPU and ollama ps shows 100% GPU (or a partial offload), you're on ROCm. If Ollama starts but ollama ps still shows CPU, check the two usual suspects: the user running the service must have access to the GPU (add it to the render and video groups) and the ROCm version must match what your Ollama build expects.

ROCm-specific tuning knobs

• OLLAMA_NUM_GPU — number of GPU layers to offload (0 = CPU only, 999 = try everything). Rarely needed; Ollama's scheduler is good at this.
• OLLAMA_SPLIT_MODE=layer — how multi-GPU setups split the model (default is fine for most).
• HIP_VISIBLE_DEVICES=0,1 — restrict which GPUs ROCm uses when you have several.

Which option for which job

Hardware-first decision table

By use case

Coding and agents (the demanding cases). Agentic coding — the kind our local coding models guide covers and tools like OpenCode do with a local backend — is the most demanding workload you can throw at a local machine: long prompts, huge contexts, thousands of tokens per minute of sustained generation. Every % of speed compounds. On supported hardware, use ROCm; on anything else, use Vulkan. And in both cases make sure the whole model fits in VRAM (100% GPU in ollama ps) — a partially-offloaded model will make an agent feel broken, because tool-call round-trips multiply your latency penalty.

Inline autocomplete. A tiny FIM model (1B–3B) wants minimal latency, not throughput. These are so small they run fine on either backend — even CPU — but if the GPU is free, offload them; the difference shows up in perceived editor snappiness.

Chat, RAG, summarization. Human reading speed is the bottleneck. Even 10–15 tokens/s feels fine. Here Vulkan vs ROCm matters little; pick whichever is already working. Prioritize fitting the model + a comfortable num_ctx over raw speed.

Experimenting / unstable drivers. If your ROCm install is flaky, don't fight it for hours: switch to Vulkan and move on. Vulkan's failure mode is graceful (CPU fallback), ROCm's is not.

Troubleshooting quick reference