Skip to Content
Local Deployment GuidesDeployment

Deployment

After fulfilling the requirements, you are ready to deploy Fuzzball. Use the Kubernetes Operator tab for a multi-node, production-style deployment across server and compute nodes, or the Docker Compose tab for a single-host deployment suited to local development and testing.

selectKubernetes OperatorDocker Compose

Please select your environment.

The deployment takes place across two groups of nodes: an odd number of Server node(s) that run Fuzzball Orchestrate, and one or more Compute node(s) that run Fuzzball Substrate. This guide assumes one Server node and one Compute node; where a production-ready, high availability (HA) cluster differs, special instructions are called out inline.

Environment variables used in deployment

This guide is designed to be completed primarily through copy-and-paste instructions. Some values cannot be anticipated and are included as environment variables that you are expected to set. Be aware that you will switch from one node to another at various points, so variables need to be assigned again in the new environment.

The following variables are determined by your specific network configuration:

  • IP1: The first IP address in your metallb pool (e.g. 10.0.0.99). See the appendix on installing a minimal K8s environment for details.
  • NFS_SERVER_IP: The internal IP address of your NFS server (e.g. 10.0.0.3). If you serve NFS from your Server node, this is simply the internal IP address of the Server node. See the appendix on installing a minimal K8s for details.
  • REALM_ID: A UUID used to configure the Keycloak instance. Create it with the uuidgen command.
  • PUBLIC_INTERFACE: The literal name of the public interface — for instance, enp1s0.
  • INTERNAL_INTERFACE: The literal name of the internal interface — for instance, enp8s0.

The remaining variables relate to accessing CIQ Depot and downloading artifacts from the container registry:

  • DEPOT_USER: A username to access CIQ Depot, issued by your CIQ sales/support team once you subscribe to Fuzzball.
  • ACCESS_KEY: A token you receive from the CIQ sales/support team after subscribing to Fuzzball.
  • VERSION: The version of the Fuzzball Operator used for the deployment. You can find version information in Depot.
  • CHART: The URL of the Helm Chart used to deploy the Fuzzball Operator.
  • IMAGE: The URL of the image for the Fuzzball Operator.
  • OWNER_EMAIL: The email address of the Fuzzball organization owner used for login.
  • STORAGE_CLASS: The StorageClass used internally by K8s to support the Fuzzball cluster.

Server node: install the Fuzzball Operator

The Server node will run Fuzzball Orchestrate, which is deployed as a K8s operator. We must therefore begin by obtaining and installing the operator.

First, we need an access key to obtain artifacts from Depot. You will receive this key from the CIQ sales/support team upon subscribing to Fuzzball. Save it somewhere safe (with appropriate permission so that only you can access it).

Now you can use the Depot access key to login to the registry using Helm.

# DEPOT_USER="" # populate with your username for CIQ Depot

# ACCESS_KEY="" # populate with the Depot key obtained from the CIQ sales/support team

# helm registry login depot.ciq.com --username "${DEPOT_USER}" --password "${ACCESS_KEY}"

Once you have logged into the depot registry, you can install the Fuzzball operator. You can replace the VERSION below if you need a different version of Fuzzball to be installed. (You can find different Fuzzball versions by signing into Portal and looking at Fuzzball under "my products".)

# VERSION="<Version />"

# CHART="oci://depot.ciq.com/fuzzball/fuzzball-images/helm/fuzzball-operator"

# STORAGE_CLASS="longhorn" # adjust this if you are using a different StorageClass

# helm upgrade --install fuzzball-operator "${CHART}" \
  --namespace fuzzball-system --create-namespace \
  --version "${VERSION}" \
  --set "image.tag=${VERSION}" \
  --set "imagePullSecrets.name=repository-ciq-com" \
  --set "imagePullSecrets.inline.registry=depot.ciq.com" \
  --set "imagePullSecrets.inline.username=${DEPOT_USER}" \
  --set "imagePullSecrets.inline.password=${ACCESS_KEY}" \
  --set "storageClassName=${STORAGE_CLASS}"

After executing this command, you can check that the operator is installed properly with the following commands:

# kubectl get pvc -n fuzzball-system
NAME                                   STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
fuzzball-operator-controller-manager   Bound    pvc-dfc2f2c3-8df0-4ec8-8503-2d927b13402e   20Gi       RWO            longhorn       <unset>                 11s

This will show that the fuzzball-operator-controller-manager is Bound to a volume.

# kubectl get crd fuzzballorchestrate.deployment.ciq.com
NAME                                     CREATED AT
fuzzballorchestrate.deployment.ciq.com   2025-12-11T21:42:25Z

This command shows that a new custom resource has been defined.

# kubectl get pod -n fuzzball-system
NAME                                                    READY   STATUS    RESTARTS   AGE
fuzzball-operator-controller-manager-6dd9bd7f56-hnb9b   2/2     Running   0          81s

This will show that 2/2 pods instantiating the fuzzball-operator-controller-manager are running. (It might take a few minutes to complete, so don't worry if there are initially 0/3 ready.)

You can also inspect the configuration of the operator with the following command:

# helm get values -n fuzzball-system fuzzball-operator
USER-SUPPLIED VALUES:
image:
  repository: depot.ciq.com/fuzzball/fuzzball-images/fuzzball-operator
  tag: <Version />
imagePullSecrets:
  inline:
    password: <YOUR_PASSWORD_HERE>
    registry: depot.ciq.com
    username: dgodlove@ciq.com
  name: repository-ciq-com
storageClassName: longhorn
Note

You can rerun the helm upgrade command if you want to update your version of Fuzzball or to change selected settings.

Customizing resource names

Most Kubernetes resources created by the chart derive their name from a computed prefix. The prefix is determined as follows:

  1. If fullnameOverride is set, the prefix is exactly that value.
  2. Otherwise, the effective name is nameOverride if set, or the chart name (fuzzball-operator).
  3. If the release name already contains the effective name, the prefix is the release name alone. Otherwise, the prefix is <release-name>-<effective-name>.

Two optional Helm values control this behavior:

  • nameOverride: Substitutes an alternative chart name in step 2. For example, --set nameOverride=fbop with release name fuzzball-operator produces the prefix fuzzball-operator-fbop (release does not contain fbop), so the Deployment becomes fuzzball-operator-fbop-controller-manager.

  • fullnameOverride: Skips the release name entirely and sets the prefix directly. For example, --set fullnameOverride=fbop always produces fbop-controller-manager regardless of the release name.

The table below summarizes how these values combine with the release name to form resource prefixes:

--set flagsRelease nameResult prefix
(none)fuzzball-operatorfuzzball-operator
(none)fbfb-fuzzball-operator
nameOverride=fbopfuzzball-operatorfuzzball-operator-fbop
nameOverride=fbopfbopfbop
fullnameOverride=fbop(any)fbop

Each affected resource appends a stable suffix to this prefix:

SuffixResource(s)
-controller-managerDeployment, ServiceAccount, PersistentVolumeClaim
-controller-manager-metrics-serviceService
-controller-manager-registryService (only when airgap.enabled=true)
-leader-election-roleRole
-leader-election-rolebindingRoleBinding
-manager-rolebindingClusterRoleBinding
-proxy-rolebindingClusterRoleBinding
-update-crd-saServiceAccount (post-upgrade hook)
-update-crd-roleClusterRole (post-upgrade hook)
-update-resourcesClusterRoleBinding (post-upgrade hook)
-update-crds-jobJob (post-upgrade hook)
Note

The following resources are not affected by nameOverride or fullnameOverride:

  • CRDs (fuzzballorchestrate.deployment.ciq.com, fuzzballfederate.deployment.ciq.com) — Kubernetes enforces <plural>.<group> naming and these cannot be renamed.
  • ClusterRoles fuzzball-metrics-reader and fuzzball-proxy-role — fixed names generated by the kubebuilder/kube-rbac-proxy scaffolding.
  • imagePullSecrets Secret — controlled by imagePullSecrets.name, not the fullname helper.

Server node: configure Fuzzball Deployment

Now that the K8s Fuzzball Operator is installed on the Server node, we are ready to use it to deploy Fuzzball! The first step is to generate a suitable fuzzball.yaml file to configure the new deployment. Here is an example.

Note

At this time, there are variables that appear in the fuzzball.yaml file that can only be set to the given value. Do not assume that any variable in the YAML file can be used to configure the Fuzzball Orchestrate CRD.

Note

The following code block describes several commands to set the values of environment variables and one (extremely long) cat command that will use these values to create the appropriate Custom Resource Definition (CRD) file.

# DEPOT_USER="" # populate with your username for CIQ Depot

# ACCESS_KEY="" # populate with the Depot key obtained from the CIQ sales/support team

# IP1="" # populate with the first IP address in your metallb pool (e.g. 10.0.0.99)

# NFS_SERVER_IP="" # populate with the IP address of your NFS server

# REALM_ID=$(uuidgen --random | tr 'A-Z' 'a-z')

# OWNER_EMAIL="admin@ciq.com" # populate with the email of the owner of the Fuzzball organization

# STORAGE_CLASS="longhorn" # change this to local-path if you installed Local Storage Path

# cat > fuzzball.yaml <<EOF
apiVersion: deployment.ciq.com/v1alpha1
kind: FuzzballOrchestrate
metadata:
  labels:
    app.kubernetes.io/name: fuzzball-orchestrate
    app.kubernetes.io/part-of: fuzzball
  name: fuzzball-orchestrate
spec:
  image:
    username: ${DEPOT_USER}
    password: ${ACCESS_KEY}
    exclusive: false
  ingress:
    create:
      domain: "${IP1}.nip.io"
      proxy:
        type: LoadBalancer
        annotations:
          metallb.io/allow-shared-ip: ingress-and-fuzzball
          metallb.io/loadBalancerIPs: ${IP1}
  database:
    create:
      enableDebugPod: true
      storage:
        class: ${STORAGE_CLASS}
  keycloak:
    create:
      ingress:
        hostname: auth.${IP1}.nip.io
      realmName: Fuzzball
      realmId: ${REALM_ID}
      username: keycloak
      ownerEmail: "${OWNER_EMAIL}"
      createDatabase: true
  tls:
    certManager:
      create: {} # Empty object = deploy cert-manager with defaults
    trustManager:
      create: {} # Empty object = deploy trust-manager with defaults
  fuzzball:
    substrate:
      nfs:
        destination: "/fuzzball/shared"
        server: ${NFS_SERVER_IP}
        path: "/srv/fuzzball/shared"
    audit:
      storage:
        class: ${STORAGE_CLASS}
    objectCache:
      storage:
        class: ${STORAGE_CLASS}
    storage:  # example of how to specify resources for fuzzball services
      resources:
        requests:
          cpu: 250m
          memory: 256Mi
        limits:
          cpu: 1000m
          memory: 1Gi
    workflow:
      callbackService:
        type: LoadBalancer
        annotations:
          metallb.io/allow-shared-ip: ingress-and-fuzzball
          metallb.io/loadBalancerIPs: ${IP1}
      autoscaling: # example for auto-scaling workflow service
        enabled: true
        minReplicas: 1
        maxReplicas: 3
        targetCPUUtilization: 80
        targetMemoryUtilization: 80
    substrateBridge:
      log:
        storage:
          class: ${STORAGE_CLASS}
      dns:
        externalService:
          type: NodePort
    orchestrator:
      provisioner:
        enabled: false
      autoscaling:
        enabled: false
        minReplicas: 1
        maxReplicas: 3
        targetCPUUtilization: 80
        targetMemoryUtilization: 80
      resources:
        requests:
          cpu: "500m"
          memory: "512Mi"
    jetstream:
      replicas: 3
      externalService:
        type: NodePort
      storage:
        class: ${STORAGE_CLASS}
        size: 10Gi
EOF
Note

If your cluster already has cert-manager installed, you can configure Fuzzball to use it instead of deploying a new instance. See Deploying with External cert-manager for details.

Because of the environment variables you've set, this command will automatically populate the field spec.image.password with a CIQ Depot access key and spec.keycloak.create.realmId with a v4 UUID. It will also set the domain and hostname with the appropriate IP addresses for your setup. There are a few more customizations you may want to make before proceeding.

  • It is recommended that you update spec.ingress.create.domain with a site-specific domain.
  • Optionally update spec.keycloak.create.password with a unique password for the Keycloak admin user.
  • Optionally define spec.keycloak.create.ingress.hostname with a site-specific domain name.

Once you have the fuzzball.yaml file created and customized to your liking, you are ready to use it to deploy Fuzzball Orchestrate.

Server node: deploy Fuzzball Orchestrate

With the Fuzzball K8s operator installed and the fuzzball.yaml configuration file in place, you are ready to deploy Fuzzball Orchestrate to the Server node.

You can do so with the following command:

# kubectl apply -f fuzzball.yaml

This command proceeds asynchronously. So it will look like it is completed even though it will still require some time for the K8s Fuzzball deployment to finish. This usually takes several minutes.

You can monitor the deployment as it occurs with the following command:

# kubectl logs -l app.kubernetes.io/name=fuzzball-operator -n fuzzball-system -f --tail=-1
[...snip]
Diagnostics:
  ciq:core/v1:Identity (identity):
    Waiting for keycloak to be ready: SUCCESS

  pulumi:pulumi:Stack (fuzzball-orchestrate-fuzzball-orchestrate):
    Using fuzzballCRD image username and password

Resources:
    + 9 created
    ~ 7 updated
    16 changes. 235 unchanged

Duration: 16s

2025-02-04T22:49:08Z	DEBUG	events	Resources have been deployed successfully	{"type": "Normal", "object": {"kind":"FuzzballOrchestrate","name":"fuzzball-orchestrate","uid":"786ef498-cc8b-4d31-9729-8e0c2a6d9dcb","apiVersion":"deployment.ciq.com/v1alpha1","resourceVersion":"44728"}, "reason": "DeploymentSucceeded"}
2025-02-04T22:49:08Z	INFO	Updated Fuzzball status to ReconciliationComplete - Reconciliation completed successfully	{"controller": "fuzzballorchestrate", "controllerGroup": "deployment.ciq.com", "controllerKind": "FuzzballOrchestrate", "FuzzballOrchestrate": {"name":"fuzzball-orchestrate"}, "namespace": "", "name": "fuzzball-orchestrate", "reconcileID": "ebe8f5ac-8ef7-449a-81bc-080c455aae86"}

After deployment, you can inspect the configuration that was used with the following command:

# kubectl get fuzzballorchestrate fuzzball-orchestrate -o yaml

You can edit the current configuration in-place. These changes will be immediately applied and updated by the Fuzzball operator.

# kubectl edit fuzzballorchestrate fuzzball-orchestrate

Server node: verify the deployment

Once the deployment is complete, you can confirm that Fuzzball Orchestrate is installed correctly by inspecting that expected objects have been created. (Note that you may have more or fewer replicas of some pods.)

Look at the newly started pods:

# kubectl get pods -n fuzzball
NAME                                         READY   STATUS      RESTARTS   AGE
fuzzball-admin-0                             1/1     Running     0          8m49s
fuzzball-admin-ui-548f599b68-9j9vc           1/1     Running     0          8m29s
fuzzball-agent-848db9f89c-lmdmq              1/1     Running     0          7m19s
fuzzball-audit-8dd47456-wf2qq                1/1     Running     0          8m39s
fuzzball-auth-spicedb-77d7dd9957-jz2lr       1/1     Running     0          8m47s
fuzzball-cluster-admin-599c87b794-4txmh      1/1     Running     0          7m54s
fuzzball-jetstream-0                         1/1     Running     0          7m20s
fuzzball-jetstream-1                         1/1     Running     0          7m20s
fuzzball-jetstream-2                         1/1     Running     0          7m20s
fuzzball-load-default-catalog-sv2jg          0/1     Completed   0          3m35s
fuzzball-openapi-65c78cc7cb-wdmmw            1/1     Running     0          8m41s
fuzzball-orchestrator-779cd894d6-h9x9b       1/1     Running     0          5m17s
fuzzball-storage-6597d744b4-zz5n5            1/1     Running     0          5m15s
fuzzball-substrate-bridge-6cc6f74859-gtc2h   1/1     Running     0          6m9s
fuzzball-sync-0                              1/1     Running     0          4m20s
fuzzball-ui-7d664769bb-tkqhl                 1/1     Running     0          8m28s
fuzzball-workflow-7d8cd4cf6-8rt5t            1/1     Running     0          4m48s

Check for any pods that are not in the running state:

# kubectl get pods -A --field-selector status.phase!=Running,status.phase!=Succeeded
No resources found

Check that ingress is configured and ready.

# kubectl get ingress -A
NAMESPACE           NAME                        CLASS    HOSTS                                ADDRESS      PORTS     AGE
fuzzball            fuzzball-admin-ui           kong     ui-admin.10.1.96.99.nip.io           10.1.96.99   80, 443   8m16s
fuzzball            fuzzball-config             kong     api.10.1.96.99.nip.io                10.1.96.99   80, 443   9m35s
fuzzball            fuzzball-openapi            kong     api.10.1.96.99.nip.io                10.1.96.99   80, 443   9m11s
fuzzball            fuzzball-substrate-bridge   kong     substrate-bridge.10.1.96.99.nip.io   10.1.96.99   80, 443   6m14s
fuzzball            fuzzball-ui                 kong     ui.10.1.96.99.nip.io                 10.1.96.99   80, 443   8m10s
fuzzball-identity   identity-keycloakx          <none>   auth.10.1.96.99.nip.io               10.1.96.99   80, 443   10m

Confirm that all persistent volume claims (PVCs) are bound.

# kubectl get pvc -A
NAMESPACE         NAME                                      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
fuzzball-database data-database-postgresql-0                Bound    pvc-52d62829-32b5-41e7-90c0-11d9ddfcfccb   8Gi        RWO            longhorn       <unset>                 12m
fuzzball-system   fuzzball-operator-controller-manager      Bound    pvc-dfc2f2c3-8df0-4ec8-8503-2d927b13402e   20Gi       RWO            longhorn       <unset>                 22m
fuzzball          audit-cold-storage                        Bound    pvc-7bcd7abc-0d11-419b-b453-3a6bddccbb67   1Gi        RWO            longhorn       <unset>                 10m
fuzzball          fuzzball-substrate-logs                   Bound    pvc-e282b15b-6ab2-4b64-8a39-d6c58a382835   5Gi        RWO            longhorn       <unset>                 7m39s
fuzzball          jetstream-data-fuzzball-jetstream-0       Bound    pvc-ad6481d4-d469-4343-a8a3-9b2117e53ab4   10Gi       RWO            longhorn       <unset>                 8m47s
fuzzball          jetstream-data-fuzzball-jetstream-1       Bound    pvc-4cb91882-d556-4ffc-a931-76ed45f8761a   10Gi       RWO            longhorn       <unset>                 8m47s
fuzzball          jetstream-data-fuzzball-jetstream-2       Bound    pvc-a161913d-6629-4b36-b109-a83943041fba   10Gi       RWO            longhorn       <unset>                 8m47s
fuzzball          substrate-shared-config-fuzzball-sync-0   Bound    pvc-2953abd9-5e29-45d5-b731-8dcf935a76d4   5Gi        RWX            nfs-client     <unset>                 5m47s

Compute node: install Substrate

Now that you have a running instance of Fuzzball Orchestrate it is time to connect Compute nodes to it. Compute nodes run the Fuzzball Substrate container platform to create containerized environments for Fuzzball jobs. To do this, Compute nodes require some additional configuration (firewall, NFS, etc.).

You can start by enrolling the Compute node in CIQ Depot and installing required packages.

# DEPOT_USER="" # populate with your username for CIQ Depot

# ACCESS_KEY="" # populate with the Depot key obtained from the CIQ sales/support team

# VERSION="{{< version_no_v >}}" # matches orchestrate version without the leading v

# dnf install -y https://depot.ciq.com/public/files/depot-client/depot/depot.x86_64.rpm

# depot register --user "${DEPOT_USER}" --token "${ACCESS_KEY}"

# depot enable fuzzball

# dnf install -y fuzzball-substrate fuzzball-substrate-orchestrate-${VERSION}-1 fuzzball-cli-${VERSION}-1

If your compute node has a GPU, be sure to install the following package as well.

# dnf install -y fuzzball-substrate-nvidia-gpu-plugin # optional

Compute node: configure the node

You should configure firewalld to prevent interference with intra-cluster communication. Either disable it or configure the internal interface to use the trusted zone.

# INTERNAL_INTERFACE="" # populate this with the value for your COMPUTE NODE environment (e.g. enp8s0)

# firewall-cmd --permanent --zone=trusted --add-interface=${INTERNAL_INTERFACE}

# firewall-cmd --reload

Now you can install and configure NFS on the Compute node so that Fuzzball can share configuration and logging files with the Server node.

# dnf install -y nfs-utils

You can place the following contents in the file /etc/systemd/system/fuzzball-shared.mount to create a mount unit for use with systemd. This will ensure that NFS starts properly again after a reboot.

# NFS_SERVER_IP="" # populate with the internal IP address of the Server node (e.g. 10.0.0.3)

# cat >/etc/systemd/system/fuzzball-shared.mount<<EOF
[Unit]
Description=Mount Fuzzball NFS Share
After=network-online.target remote-fs-pre.target
Wants=network-online.target
Before=remote-fs.target

[Mount]
What=${NFS_SERVER_IP}:/srv/fuzzball/shared
Where=/fuzzball/shared
Type=nfs
Options=_netdev,hard,retry=5,timeo=600,retrans=3
TimeoutSec=300

[Install]
WantedBy=remote-fs.target
EOF

# mkdir -p /etc/systemd/system/fuzzball-substrate.service.d

# cat >/etc/systemd/system/fuzzball-substrate.service.d/10-require-shared.conf<<EOF
[Unit]
Requires=fuzzball-shared.mount
After=fuzzball-shared.mount
EOF

Now create the shared mount point on the Compute node and load/enable/start the newly created mount unit.

# mkdir -p /fuzzball/shared

# systemctl daemon-reload

# systemctl enable --now fuzzball-shared.mount

# ls /fuzzball/shared/
substrate

The final command demonstrates that a configuration directory has already been placed on NFS by Fuzzball Orchestrate.

Warning

If the NFS mount fails to load or the /fuzzball/shared directory doesn't allow you to share between the nodes, stop and troubleshoot your NFS configuration before continuing with the instructions. If the Compute node NFS configuration looks good, check the NFS configuration on the Server node.

Compute node: configure and start Substrate

Now that your compute node is fully configured, you are ready to configure Fuzzball Substrate to act as a Compute node agent and enable/start it.

You begin by adding the following variables to /etc/default/fuzzball-substrate.env. This lets Substrate know where it should look to find its configuration files. /fuzzball/shared in our example is the mount point of the file system shared by Orchestrate and the Substrate nodes.

# cat >/etc/default/fuzzball-substrate.env<<EOF
FB_POLL_CONFIG=/fuzzball/shared/substrate/substrate-config.yaml

# aids crash recovery
FB_MOUNTNS=true
EOF

Now you can enable/start the service like so.

# systemctl enable --now fuzzball-substrate.service

At this point you have installed Fuzzball Orchestrate on a server node and Fuzzball Substrate on a compute node! Next, follow the initial configuration guide to add your compute node to the cluster, configure Fuzzball Orchestrate to use storage, and start adding users.

For a single-host deployment suited to local development, evaluation, and testing, use the fuzzball cluster docker-compose deploy flow. The complete walkthrough -- install, deploy, DNS resolution, TLS trust, lifecycle, upgrading, and troubleshooting -- lives on its own page:

Docker Compose Stack

Hardware-specific guides build on it: NVIDIA DGX Spark and AMD Ryzen AI Max (Strix Halo).

RequirementsDocker Compose Stack