Running Ollama

Ollama is a popular runtime for serving large language models locally, with a simple CLI and an HTTP API. It runs models on the CPU out of the box and uses NVIDIA GPUs automatically when they are present, which makes it a natural fit for RLC Pro AI systems with or without accelerators.

You install Ollama from the upstream project the same way you would on any Rocky Linux system. This guide walks through installation, serving, and API usage on RLC Pro AI 9.7.

Install Ollama

The upstream installer detects the platform, installs to /usr/local, creates a dedicated service account, and registers a systemd service:

curl -fsSL https://ollama.com/install.sh | sh

>>> Installing ollama to /usr/local
>>> Downloading ollama-linux-amd64.tar.zst
>>> Creating ollama user...
>>> Adding ollama user to render group...
>>> Adding ollama user to video group...
>>> Adding current user to ollama group...
>>> Creating ollama systemd service...
>>> Enabling and starting ollama service...

Verify the service

The installer enables and starts ollama.service immediately:

systemctl status ollama --no-pager

● ollama.service - Ollama Service
     Loaded: loaded (/etc/systemd/system/ollama.service; enabled; preset: disabled)
     Active: active (running) since Fri 2026-07-17 03:10:57 UTC; 1min 23s ago

The service runs as the unprivileged ollama user with automatic restarts, and listens on localhost only by default:

ss -tln | grep 11434

LISTEN 0      4096       127.0.0.1:11434      0.0.0.0:*

ollama --version

ollama version is 0.32.1

Pull and run a model

Pull a model from the Ollama library. This example uses a small model that runs comfortably on CPU-only systems:

ollama pull qwen2.5:0.5b
ollama list

NAME            ID              SIZE      MODIFIED
qwen2.5:0.5b    a8b0c5157701    397 MB    Less than a second ago

Chat with it interactively, or pass a one-shot prompt; the reply streams token by token, at interactive speed for a small model even on CPU-only systems:

ollama run qwen2.5:0.5b "In one short sentence, what is Rocky Linux?"

Model choice is workload- and hardware-dependent: larger models need proportionally more memory (the model file is loaded into RAM on CPU-only hosts, or VRAM on GPU hosts). Browse the Ollama library for available models and sizes.

Models are stored under the service account's home at /usr/share/ollama/.ollama/; plan disk capacity there for the models you intend to keep.

Use the HTTP API

Everything the CLI does is available over the REST API on port 11434:

curl -s http://localhost:11434/api/generate -d '{
  "model": "qwen2.5:0.5b",
  "prompt": "Say hello in five words or fewer.",
  "stream": false
}'

{"model":"qwen2.5:0.5b","created_at":"2026-07-17T03:13:55.014592437Z","response":"Hello! How can I help you today?","done":true,...}

List installed models via the API:

curl -s http://localhost:11434/api/tags

Ollama also exposes an OpenAI-compatible endpoint at /v1, so existing OpenAI SDK clients can point at http://localhost:11434/v1. See the Ollama API documentation for the full surface.

Serving on the network

By default the API is reachable only from the local host. To serve other machines, set OLLAMA_HOST through a systemd drop-in:

sudo systemctl edit ollama

Add:

[Service]
Environment="OLLAMA_HOST=0.0.0.0"

Then restart and confirm the listener moved to all interfaces:

sudo systemctl restart ollama
ss -tln | grep 11434

LISTEN 0      4096               *:11434            *:*

Open the port in the firewall:

sudo firewall-cmd --add-port=11434/tcp --permanent
sudo firewall-cmd --reload

GPU acceleration

On RLC Pro AI hosts with a supported NVIDIA data center GPU, Ollama detects and uses the GPU automatically. The NVIDIA driver stack and the CUDA runtime (cuda-cudart) ship pre-installed (see AI Frameworks), so no additional driver or CUDA runtime setup is required. On hosts without a GPU, everything above works on the CPU.