Distributed GPU Training with NCCL

Overview

The NVIDIA Collective Communications Library (NCCL) is the communication layer that lets multiple GPUs exchange data during distributed jobs -- the GPU counterpart to MPI. It is the backbone of multi-GPU AI training.

NCCL is a library, not a process launcher: you do not wire it up directly. Your framework -- almost always PyTorch -- uses NCCL internally, and a launcher such as torchrun starts the processes and assigns ranks. Because NCCL is not a wire-up mechanism, Fuzzball has no dedicated NCCL multinode.implementation. Instead you run these workloads with the generic implementation, which hands your launcher the per-node rank and head-node address it needs.

Single node, multiple GPUs

A single node with several GPUs needs no multinode block at all. torchrun --standalone launches one process per GPU on the node, and PyTorch uses NCCL to communicate between them:

Loading fuzzball/4.4/fuzzfiles/nccl-torchrun-1n.fz...

Set --nproc-per-node to the GPU count. Fuzzball automatically raises the job's locked-memory limit (RLIMIT_MEMLOCK) to its memory allocation, which covers NCCL's pinned communication buffers.

Multiple nodes

To span nodes, use the generic implementation. It exposes the per-node rank as $RANK and the head node's address as $MULTINODE_NODE_IP -- exactly torchrun's --node-rank and --master-addr. The exec_all helper runs torchrun on every node:

Loading fuzzball/4.4/fuzzfiles/nccl-torchrun-generic.fz...

Each node launches --nproc-per-node processes (one per GPU) and rendezvouses at the head node; PyTorch coordinates the rest through NCCL. See Distributed Workflows with Generic Multinode for the full generic environment and helper-function reference.

What your container must provide

Because Fuzzball runs jobs as an unprivileged user, everything must be baked into the image -- you cannot install packages at runtime:

CUDA, NCCL, and your framework (e.g. PyTorch). The NGC PyTorch images bundle all three, plus torchrun.
For InfiniBand/RoCE, the NCCL network plugin plus libibverbs/libfabric matching your cluster's fabric.

The GPU code in the image must be built for your GPU's architecture. A CUDA toolkit older than the GPU fails at kernel launch with Cuda failure 'invalid argument'.

InfiniBand / RDMA

On a fabric-equipped cluster, pin the device so NCCL uses RDMA instead of falling back to TCP:


    env:
      - NCCL_IB_HCA=mlx5

If NCCL cannot find a high-speed device it falls back to sockets, visible in the job log as NET/IB : No device found followed by Using network Socket. Fuzzball exposes /dev/infiniband to multinode containers automatically and raises the locked-memory limit RDMA pinning needs; the fabric libraries must be present in your container, and the interconnect provisioned on the nodes.

Multi-Node NVLink

On systems with Multi-Node NVLink (MNNVL), the GPUs of several nodes share one NVLink domain, and NCCL can use NVLink between those nodes instead of the network. For that, each node's container needs an IMEX channel. Ask for one by setting NVIDIA_IMEX_CHANNELS in the job's env, and pin the job to the MNNVL nodes:


    env:
      - NVIDIA_IMEX_CHANNELS=0
    resource:
      devices:
        nvidia.com/gpu: 4    # per node; --nproc-per-node must match
      annotations:
        nvidia.com/gpu.product: GB200    # a GPU model only the MNNVL nodes have
    multinode:
      nodes: 2
      implementation: generic

The rest of the job is the same as the multinode example above. Fuzzball adds /dev/nvidia-caps-imex-channels/channel0 to the container on every node, and places the job's nodes inside one NVLink domain whenever one has enough free nodes. When none does, the job spans domains and NCCL uses the network between them. The nvidia.com/gpu.product annotation matches any product name that contains its value; without a pin like it, a job that lands on nodes without IMEX fails when its containers are created.

This needs:

IMEX set up on the nodes by your administrator; see Multi-Node NVLink.
An NCCL release in the image that supports Multi-Node NVLink.

Warning

On an MNNVL system without IMEX, NCCL 2.27.3 and later stop at startup instead of falling back to the network, because they cannot set up NVLink between nodes. Set NCCL_MNNVL_ENABLE=0 in the job's env to run over the network instead.

If the job fails at startup:

Cuda failure 800 'operation not permitted' means the job cannot write to the IMEX channel. Ask your administrator to check its permissions.
An error that mentions MNNVL means IMEX is missing or not running on the nodes.

Note

Every job that uses channel 0 in an NVLink domain shares it, so a job can read GPU memory that another job in the same domain shares through IMEX. Treat each NVLink domain as one trust domain.

NCCL without PyTorch

For NCCL programs that use MPI to bootstrap (for example NVIDIA's nccl-tests), launch them with mpirun on the generic implementation rather than torchrun. The generic command example shows the pattern -- your command invokes mpirun -H $MULTINODE_HOSTLIST --mca plm_rsh_agent $MULTINODE_SSH_WRAPPER -np $MULTINODE_TOTAL_SLOTS <program>, and the program uses NCCL for the actual GPU communication.

Distributed Workflows with Generic Multinode Overview