Troubleshooting

Common issues on RLC Pro AI systems and how to diagnose them.

Entitled repositories are missing from dnf repolist

Symptom: after depot enable rlc-ai-9, running dnf repolist shows only ciq-depot-client.

Cause: the Depot-generated repository configuration embeds credentials and is readable only by root.

Fix: run repository operations with sudo:

sudo dnf repolist

Cannot reach CIQ Depot

Symptom: dnf fails with Could not resolve host or timeouts against depot.ciq.com.

Diagnosis:

getent hosts depot.ciq.com
curl -sI https://depot.ciq.com | head -1

Ensure DNS resolution works and HTTPS (port 443) egress to depot.ciq.com is allowed. On cloud-init provisioned VMs, note that cloud-init writes /etc/resolv.conf once at provision time and drops dns=none into NetworkManager's configuration; if your DNS servers changed after provisioning, nmcli con mod ... ipv4.dns alone will not update /etc/resolv.conf until NetworkManager is allowed to manage it (a drop-in with dns=default under /etc/NetworkManager/conf.d/ that sorts after 99-cloud-init.conf restores normal behavior).

GPG key prompt or signature errors on first update

Symptom: the first transaction after enablement prompts to import a key, or fails signature verification.

Explanation: CIQ packages are signed; importing the CIQ Signing Key V2 once resolves it:

Importing GPG key 0xC7780912:
 Userid     : "CIQ Signing Key V2 <engineering@ciq.co>"
 From       : https://ciq.com/keys/rpm-gpg-key-ciq
Is this ok [y/N]: y

nvidia-smi fails or no GPU is detected

Symptom:

nvidia-smi

NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

Diagnosis: first confirm the host actually has an NVIDIA device visible:

lspci | grep -i nvidia
lsmod | grep nvidia

On hosts without a GPU (or VMs without GPU passthrough) both return nothing, and the nvidia-smi failure is expected: the driver stack is installed but has no device to bind. This is a supported configuration; PyTorch runs on the CPU. On hosts that DO have an NVIDIA device but show no loaded modules, check kernel messages for driver binding errors:

sudo journalctl -k | grep -i nvidia | tail -20

torch.cuda.is_available() returns False

The pre-installed PyTorch is a CUDA 12.8 build on every system; False means no usable GPU is visible, not a broken installation. Work through the GPU detection steps above. On CPU-only systems, False is the expected value and workloads run on the CPU.

Container pull fails with authentication required

The Depot registry rejects unauthenticated pulls. Log in with your Portal credentials:

podman login depot.ciq.com -u <DEPOT_USERNAME> -p <TOKEN>

See the container deployment guides and the Depot container guide.

Permission denied on container volume mounts

SELinux is enforcing by default. Label volumes with :Z or :z instead of disabling SELinux. See SELinux on RLC Pro AI.

Reading system logs

RLC Pro AI systems log through the journal. Prefer journalctl over legacy log files:

# Kernel/driver messages
sudo journalctl -k

# A specific unit, e.g. the Depot refresh timer
journalctl -u depot-refresh.timer

# Everything since the last boot
journalctl -b

Getting help

Contact CIQ Support with the output of sudo dnf repolist, rpm -qa | grep -E 'ciq|clk|ciqai', and the relevant journalctl excerpts.