Running Ollama

Ollama is a popular runtime for serving large language models locally, with a simple CLI and an HTTP API. It runs models on the CPU out of the box and uses NVIDIA GPUs automatically when they are present, which makes it a natural fit for RLC Pro AI systems with or without accelerators.

You install Ollama from the upstream project the same way you would on any Rocky Linux system. This guide walks through installation, serving, and API usage on RLC Pro AI 9.7.

Install Ollama

The upstream installer detects the platform, installs to /usr/local, creates a dedicated service account, and registers a systemd service:


curl -fsSL https://ollama.com/install.sh | sh


>>> Installing ollama to /usr/local
>>> Downloading ollama-linux-amd64.tar.zst
>>> Creating ollama user...
>>> Adding ollama user to render group...
>>> Adding ollama user to video group...
>>> Adding current user to ollama group...
>>> Creating ollama systemd service...
>>> Enabling and starting ollama service...

Note

Piping a remote script into a shell warrants a look at what you are running; the script is readable at https://ollama.com/install.sh. On RLC Pro AI it completes without any prerequisite packages, and the service runs under SELinux enforcing with no policy changes.

Verify the service

The installer enables and starts ollama.service immediately:


systemctl status ollama --no-pager


● ollama.service - Ollama Service
     Loaded: loaded (/etc/systemd/system/ollama.service; enabled; preset: disabled)
     Active: active (running) since Fri 2026-07-17 03:10:57 UTC; 1min 23s ago

The service runs as the unprivileged ollama user with automatic restarts, and listens on localhost only by default:


ss -tln | grep 11434


LISTEN 0      4096       127.0.0.1:11434      0.0.0.0:*


ollama --version


ollama version is 0.32.1

Pull and run a model

Pull a model from the Ollama library. This example uses a small model that runs comfortably on CPU-only systems:


ollama pull qwen2.5:0.5b
ollama list


NAME            ID              SIZE      MODIFIED
qwen2.5:0.5b    a8b0c5157701    397 MB    Less than a second ago

Chat with it interactively, or pass a one-shot prompt; the reply streams token by token, at interactive speed for a small model even on CPU-only systems:


ollama run qwen2.5:0.5b "In one short sentence, what is Rocky Linux?"

Model choice is workload- and hardware-dependent: larger models need proportionally more memory (the model file is loaded into RAM on CPU-only hosts, or VRAM on GPU hosts). Browse the Ollama library for available models and sizes.

Models are stored under the service account's home at /usr/share/ollama/.ollama/; plan disk capacity there for the models you intend to keep.

Use the HTTP API

Everything the CLI does is available over the REST API on port 11434:


curl -s http://localhost:11434/api/generate -d '{
  "model": "qwen2.5:0.5b",
  "prompt": "Say hello in five words or fewer.",
  "stream": false
}'


{"model":"qwen2.5:0.5b","created_at":"2026-07-17T03:13:55.014592437Z","response":"Hello! How can I help you today?","done":true,...}

List installed models via the API:


curl -s http://localhost:11434/api/tags

Ollama also exposes an OpenAI-compatible endpoint at /v1, so existing OpenAI SDK clients can point at http://localhost:11434/v1. See the Ollama API documentation for the full surface.

Serving on the network

By default the API is reachable only from the local host. To serve other machines, set OLLAMA_HOST through a systemd drop-in:


sudo systemctl edit ollama

Add:


[Service]
Environment="OLLAMA_HOST=0.0.0.0"

Then restart and confirm the listener moved to all interfaces:


sudo systemctl restart ollama
ss -tln | grep 11434


LISTEN 0      4096               *:11434            *:*

Open the port in the firewall:


sudo firewall-cmd --add-port=11434/tcp --permanent
sudo firewall-cmd --reload

Warning

The Ollama API has no built-in authentication. Only expose it on trusted networks, or front it with a reverse proxy that enforces access control.

GPU acceleration

On RLC Pro AI hosts with a supported NVIDIA data center GPU, Ollama detects and uses the GPU automatically. The NVIDIA driver stack and the CUDA runtime (cuda-cudart) ship pre-installed (see AI Frameworks), so no additional driver or CUDA runtime setup is required. On hosts without a GPU, everything above works on the CPU.

AI Frameworks Overview