Substrate Installation

Please select your environment.

Compute node: install Substrate

Now that you have a running instance of Fuzzball Orchestrate it is time to connect Compute nodes to it. Compute nodes run the Fuzzball Substrate container platform to create containerized environments for Fuzzball jobs. To do this, Compute nodes require some additional configuration (firewall, NFS, etc.).

You can start by enrolling the Compute node in CIQ Depot and installing required packages.

# DEPOT_USER="" # populate with your username for CIQ Depot

# ACCESS_KEY="" # populate with the Depot key obtained from the CIQ sales/support team

# VERSION="{{< version_no_v >}}" # matches orchestrate version without the leading v

# dnf install -y https://depot.ciq.com/dlv2/depot-el9.$(uname -m)/depot.rpm

# depot register --user "${DEPOT_USER}" --token "${ACCESS_KEY}"

# depot enable fuzzball

# dnf install -y fuzzball-substrate fuzzball-substrate-orchestrate-${VERSION}-1 fuzzball-cli-${VERSION}-1

If your compute node has an NVIDIA GPU, be sure to install the following package as well.

# dnf install -y fuzzball-substrate-nvidia-gpu-plugin # optional

Compute node: configure the node

You should configure firewalld to prevent interference with intra-cluster communication. Either disable it or configure the internal interface to use the trusted zone.

# INTERNAL_INTERFACE="" # populate this with the value for your COMPUTE NODE environment (e.g. enp8s0)

# firewall-cmd --permanent --zone=trusted --add-interface=${INTERNAL_INTERFACE}

# firewall-cmd --reload

Now you can install and configure NFS on the Compute node so that Fuzzball can share configuration and logging files with the Server node.

# dnf install -y nfs-utils

You can place the following contents in the file /etc/systemd/system/fuzzball-shared.mount to create a mount unit for use with systemd. This will ensure that NFS starts properly again after a reboot.

# NFS_SERVER_IP="" # populate with the internal IP address of the Server node (e.g. 10.0.0.3)

# cat >/etc/systemd/system/fuzzball-shared.mount<<EOF
[Unit]
Description=Mount Fuzzball NFS Share
After=network-online.target remote-fs-pre.target
Wants=network-online.target
Before=remote-fs.target

[Mount]
What=${NFS_SERVER_IP}:/srv/fuzzball/shared
Where=/fuzzball/shared
Type=nfs
Options=_netdev,hard,retry=5,timeo=600,retrans=3
TimeoutSec=300

[Install]
WantedBy=remote-fs.target
EOF

systemd attempts a mount unit only once per start. If the NFS server isn't reachable or its name doesn't resolve yet, for example because the Compute node boots before the Server node, the mount fails and stays failed. The following service retries the mount until it succeeds, waiting 10 seconds after each failed attempt.

# cat >/etc/systemd/system/fuzzball-shared-wait.service<<EOF
[Unit]
Description=Mount the Fuzzball NFS share, retrying until it is available
Wants=network-online.target
After=network-online.target

[Service]
Type=oneshot
ExecStart=/bin/sh -c 'until systemctl start fuzzball-shared.mount; do sleep 10; done'
TimeoutStartSec=infinity
EOF

The following drop-in makes Substrate wait for that service, so Substrate starts only once the share is mounted and doesn't fail when the share is late.

# mkdir -p /etc/systemd/system/fuzzball-substrate.service.d

# cat >/etc/systemd/system/fuzzball-substrate.service.d/10-require-shared.conf<<EOF
[Unit]
Requires=fuzzball-shared-wait.service
After=fuzzball-shared-wait.service fuzzball-shared.mount
PartOf=fuzzball-shared.mount
EOF

Now create the shared mount point on the Compute node and load/enable/start the newly created mount unit.

# mkdir -p /fuzzball/shared

# systemctl daemon-reload

# systemctl enable --now fuzzball-shared.mount

# ls /fuzzball/shared/
substrate

The final command demonstrates that a configuration directory has already been placed on NFS by Fuzzball Orchestrate.

Compute node: configure and start Substrate

Now that your compute node is fully configured, you are ready to configure Fuzzball Substrate to act as a Compute node agent and enable/start it.

You begin by adding the following variables to /etc/default/fuzzball-substrate.env. This lets Substrate know where it should look to find its configuration files. /fuzzball/shared in our example is the mount point of the file system shared by Orchestrate and the Substrate nodes.

# cat >/etc/default/fuzzball-substrate.env<<EOF
FB_POLL_CONFIG=/fuzzball/shared/substrate/substrate-config.yaml

# aids crash recovery
FB_MOUNTNS=true
EOF

Now you can enable/start the service like so.

# systemctl enable --now fuzzball-substrate.service

If the NFS share isn't mounted, this command waits until it is. At boot, Substrate likewise waits for the share, and boot doesn't finish until the share is mounted: systemctl is-system-running reports starting, and units ordered after multi-user.target wait too. Run journalctl -u fuzzball-shared-wait.service -u fuzzball-shared.mount to see the mount attempts and why they fail. To stop retrying and cancel the Substrate start, run systemctl stop fuzzball-shared-wait.service. Once the share is mounted, the service exits, so inactive is its normal state.

Next steps

You now have Fuzzball Orchestrate running and Fuzzball Substrate installed on a Compute node. Follow the Initial Configuration guide to add your Compute node to the cluster, configure Fuzzball Orchestrate to use storage, and start adding users.

A Docker Compose stack already runs one or more Substrate containers on the Orchestrate host. This tab adds Compute nodes on separate hosts, for compute capacity beyond what the Orchestrate host provides.

Each step is labeled with the host it runs on:

• Orchestrate host -- the machine running the Docker Compose stack, where the fuzzball CLI is installed and logged in.
• Compute node -- a separate machine you are adding. Repeat the Compute node steps on each one.

The commands for both hosts are written for Rocky Linux. On an Ubuntu Orchestrate host, such as DGX OS, open the ports with ufw instead of firewall-cmd, and install the nfs-kernel-server package instead of nfs-utils; Ubuntu starts the NFS server when the package is installed.

The Docker Compose stack must be running for the whole procedure. See Starting and stopping a stack.

The stack's api.<domain> and nats.<domain> names must also resolve on the Orchestrate host, because several steps run fuzzball commands there. If fuzzball cluster docker-compose info works but other fuzzball commands can't reach the API, see DNS resolution. For a domain that only resolves through /etc/hosts, deploy with --update-etc-hosts or add the entry yourself.

Environment variables used in this guide

Several steps use the following variables. Set the ones for each host in that host's shell.

Orchestrate host: open the required ports

The Compute node connects to the following TCP ports on the Orchestrate host. Make sure no firewall between the two hosts blocks them.

On a Rocky Linux Orchestrate host running firewalld, open the ports in the default zone. The nfs service opens port 2049; leave it out if the Orchestrate host isn't your NFS server.

# firewall-cmd --permanent --add-port=443/tcp --add-port=4222/tcp --add-port=5353/tcp

# firewall-cmd --permanent --add-service=nfs

# firewall-cmd --reload

Confirm that the ports and services lines include them:

# firewall-cmd --list-all

If another firewall sits between the two hosts, such as a network firewall or a cloud security group, open the same ports there as well.

Orchestrate host: generate the Substrate configuration

Generate the files the Compute node needs to connect to Orchestrate. Pass the Orchestrate host's IP address with --orchestrate-ip.

Orchestrate also connects back to each Compute node. By default it uses the address the node sends its own traffic from, which is the node's address on the route to --orchestrate-ip. If Orchestrate must reach the node at a different address, for example because of NAT or a separate network, also pass --substrate-ip with that address. It becomes the node's ID and the address Orchestrate connects to, so generate a separate configuration for each Compute node that needs it.

$ ORCHESTRATE_IP="" # populate with the IP address of the Orchestrate host (e.g. 10.0.0.5)

$ fuzzball cluster docker-compose generate-substrate-config ./substrate-config --orchestrate-ip ${ORCHESTRATE_IP}

The command writes five files into ./substrate-config: fuzzball-substrate.yml, orchestrate.yaml, ca.crt, nginx-ca.crt, and hosts. The directory must be empty; pass --force to overwrite existing files.

Copy the directory to the Compute node, for example with scp -r ./substrate-config root@<compute-node>:. Run the Compute node commands that follow from the directory that contains substrate-config; with the scp example, that's root's home directory.

Compute node: install Substrate

Enroll the Compute node in CIQ Depot and install the Substrate packages. The version of fuzzball-substrate-orchestrate must match your Orchestrate version.

# DEPOT_USER="" # populate with your username for CIQ Depot

# ACCESS_KEY="" # populate with the Depot key obtained from the CIQ sales/support team

# VERSION="{{< version_no_v >}}" # matches orchestrate version without the leading v

# dnf install -y https://depot.ciq.com/dlv2/depot-el9.$(uname -m)/depot.rpm

# depot register --user "${DEPOT_USER}" --token "${ACCESS_KEY}"

# depot enable fuzzball

# dnf install -y fuzzball-substrate fuzzball-substrate-orchestrate-${VERSION}-1

If your Compute node has an NVIDIA GPU, install the following package as well.

# dnf install -y fuzzball-substrate-nvidia-gpu-plugin # optional

Don't start Substrate yet.

Compute node: configure the node

Configure firewalld so it doesn't interfere with traffic between nodes. Orchestrate connects back to each Compute node on TCP ports 7331 to 7333 to assign and start jobs, and the nodes of a multinode job talk to each other directly. Either disable firewalld or add the internal interface to the trusted zone.

# INTERNAL_INTERFACE="" # populate with the internal interface of the Compute node (e.g. enp8s0)

# firewall-cmd --permanent --zone=trusted --add-interface=${INTERNAL_INTERFACE}

# firewall-cmd --reload

If the Orchestrate host reaches the Compute node through a different interface, or a network firewall sits between them, open TCP ports 7331 to 7333 from the Orchestrate host there as well. If the address Orchestrate uses differs from the node's own outbound address, see --substrate-ip in Orchestrate host: generate the Substrate configuration.

The Compute node reaches Orchestrate by name, so api.<domain> and nats.<domain> must resolve to the Orchestrate host on the Compute node too. Substrate can't check in with Orchestrate until they do. Append the generated hosts entry, which maps both names to --orchestrate-ip, to /etc/hosts on every Compute node:

# cat substrate-config/hosts >>/etc/hosts

Add the entry even with the default <host-ip>.nip.io domain. That domain encodes the Orchestrate host's primary outbound address, which on a host with more than one network may not be the address the Compute node can reach. Skip this step only if your site's DNS already resolves both names to an address the Compute node can reach.

Install the NFS client. Substrate uses it to mount volumes from the NFS server, which you set up later, when a job needs them.

# dnf install -y nfs-utils

The generated configuration keeps the node's container image cache in /mnt/fuzzball-sharedfs/image. Create that directory. With more than one Compute node, you mount the shared image cache there later.

# mkdir -p /mnt/fuzzball-sharedfs/image

Compute node: install the configuration and start Substrate

Install the generated configuration files. Install them after the packages, because installing or updating the Substrate package can replace them with its defaults. orchestrate.yaml contains the key the node uses to authenticate to Orchestrate, so only root can read it.

# install -m 0644 substrate-config/fuzzball-substrate.yml /etc/fuzzball-substrate/fuzzball-substrate.yml

# install -d -m 0755 /etc/fuzzball-substrate/extension.conf.d

# install -m 0600 substrate-config/orchestrate.yaml /etc/fuzzball-substrate/extension.conf.d/orchestrate.yaml

# sed -i '/host-rootfs-path:/d' /etc/fuzzball-substrate/fuzzball-substrate.yml

Add the deployment's two certificate authorities to the system trust store:

# install -m 0644 substrate-config/ca.crt /etc/pki/ca-trust/source/anchors/fuzzball-internal-ca.crt

# install -m 0644 substrate-config/nginx-ca.crt /etc/pki/ca-trust/source/anchors/fuzzball-nginx-ca.crt

# update-ca-trust

Enable and start Substrate:

# systemctl enable --now fuzzball-substrate.service

Orchestrate host: set up a shared file system

External Compute nodes use shared storage for two separate things:

• The image cache. Before a job runs, one node in its segment pulls and converts the job's container image into the image cache, and the node that runs the job then uses it from there. The two aren't always the same node, so with more than one Compute node, every node must see the same cache directory. A single Compute node can use its local directory.
• Volumes. Workflows that mount volumes need a storage provisioner the node can serve. The stack's default provisioner uses a Docker volume that only its own Substrate containers can reach. Workflows without volumes don't need one.

This guide shows one way to provide both: the Orchestrate host exports two NFS directories. Every Compute node mounts /srv/fuzzball/images as its image cache, and Fuzzball's NFS storage driver serves volumes from /srv/fuzzball/storage, mounting each volume only when a job needs it. A shared external segment limits these volumes to the external Compute nodes; the stack's own Substrate containers keep the default provisioner.

Other storage works too, including an existing NFS server or a parallel file system such as Lustre, GPFS, or BeeGFS, but needs a different configuration. See Driver Types and Volume Configuration and Management. To use an existing NFS server, skip the rest of this section and use its address and export paths in the next section and when you add the provisioner.

Install and start the NFS server:

# dnf install -y nfs-utils

# systemctl enable --now nfs-server

Create the directories and export them. Substrate runs as root when it writes the image cache, creates volume directories, and sets their ownership, so the exports need no_root_squash.

# COMPUTE_SUBNET="" # populate with the network range of your Compute nodes (e.g. 10.0.0.0/24)

# mkdir -p /srv/fuzzball/images /srv/fuzzball/storage

# echo "/srv/fuzzball/images ${COMPUTE_SUBNET}(rw,sync,no_subtree_check,no_root_squash)" >>/etc/exports

# echo "/srv/fuzzball/storage ${COMPUTE_SUBNET}(rw,sync,no_subtree_check,no_root_squash)" >>/etc/exports

# exportfs -ra

Compute node: mount the shared image cache

With more than one Compute node, mount the image cache export on every Compute node at the path the generated configuration uses. With a single Compute node, you can skip this section.

Add the mount to /etc/fstab so it returns after a reboot, then mount it and restart Substrate so it uses the shared cache:

# ORCHESTRATE_IP="" # populate with the IP address of the Orchestrate host (e.g. 10.0.0.5)

# echo "${ORCHESTRATE_IP}:/srv/fuzzball/images /mnt/fuzzball-sharedfs/image nfs4 _netdev,hard 0 0" >>/etc/fstab

# systemctl daemon-reload

# mount /mnt/fuzzball-sharedfs/image

# systemctl restart fuzzball-substrate.service

Orchestrate host: add the node to the cluster

Orchestrate assigns work only to nodes that match a provision definition. The stack's built-in docker-nodes definition matches only its own Substrate containers, so add a definition for the Compute node.

fuzzball cluster config set replaces the whole configuration, so start from the current one to keep the docker-nodes definition:

$ fuzzball cluster config get >provisioner.yaml

Add a definition for the Compute node to the definitions list in provisioner.yaml, replacing <compute-hostname> with the node's hostname, as printed by hostname on that node:

  - id: compute1
    provisioner: static
    segment: external
    provisionerSpec:
      condition: |-
        hostname() matches "^<compute-hostname>$"

The segment must match the segment of the storage provisioner you add in the next section.

hostname() matches takes a regular expression, so one definition can cover several Compute nodes, for example "^compute[0-9]{2}$" for compute01 to compute99. Anchor the pattern with ^ and $: without them, compute1 also matches compute10. In a regular expression, [01-58] is a set of characters, not a numeric range. See the Provisioner Configuration Guide for all options.

Apply the configuration:

$ fuzzball cluster config set provisioner.yaml

Then restart Substrate on the Compute node so it picks up the new definition:

# systemctl restart fuzzball-substrate.service

Orchestrate host: add a provisioner for the shared file system

Create a storage provisioner that serves volumes from the NFS export. Its segment matches the Compute node's provision definition, so only the external Compute nodes serve it.

$ cat >shared-storage.yaml <<EOF
shared-storage:
  description: "NFS storage for external Compute nodes"
  driver:
    type: nfs
    target: "${ORCHESTRATE_IP}:/srv/fuzzball/storage"
    version: 4
  segment: external
  access: all
  ephemeral: all
EOF

$ fuzzball volume provisioner add shared-storage -f shared-storage.yaml

version: 4 pins NFSv4, which needs only port 2049. A provisioner's segment can't be changed after it is created. See NFS driver for the other driver options and Segments for how segments work.

Troubleshooting

If the Compute node doesn't appear in fuzzball node list, or jobs don't run on it, start with these checks on the Compute node:

# getent hosts api.<domain> nats.<domain>

# curl -sS -o /dev/null -w '%{http_code}\n' https://api.<domain>/

# journalctl -u fuzzball-substrate -n 100

The first command should print the Orchestrate host's address for both names. The second prints an HTTP status code if the node trusts the deployment's certificate, and a certificate error if it doesn't.

Next steps

The Compute node is now part of the cluster and has storage. Confirm that it is listed and matched to the compute1 definition:

$ fuzzball node list -o yaml

Find the node by its host_name and check that its definition_id is compute1. A node with definition_id: N/A is connected but matches no provision definition, so it never gets work.

If other people will use the cluster, create users for them and organize them into organizations and groups, which control the resources each user can see and share. Follow Initial Entity Management to create an organization, a group, and a user, then see Fuzzball Entity Management for the full reference.

Storage provisioners belong to the organization that created them, so users in a new organization can't use shared-storage. As an owner of each organization whose users will run jobs on the Compute node, repeat Orchestrate host: add a provisioner for the shared file system. Point each organization's provisioner at its own directory on the NFS server, such as /srv/fuzzball/storage/<org-name>, and create that directory first; the NFS driver can't keep organizations apart within one directory. See Tenant Isolation.