Deployment

Please select a supported cloud provider.

Deployment Modes: Marketplace vs. Non-Marketplace

Fuzzball on AWS supports two deployment modes:

• Marketplace mode (default): Uses AWS Marketplace Elastic Container Registry (ECR) for images. This mode enables usage-based billing through AWS Marketplace metering, and it is recommended for most installations.
• Non-Marketplace mode (opt-in): Populates a customer-owned ECR with images from CIQ Depot or offline tarballs, and disables Marketplace metering. This method is required for restricted AWS partitions, air-gapped environments, or locked-down accounts.

Quick Comparison

Once deployed, a stack's mode is immutable; updates preserve the existing mode and reject any attempt to switch between Marketplace and non-Marketplace modes. See the updating documentation for mode preservation details.

This guide covers both deployment methods. Choose the correct method for your environment.

Instructions

$ fuzzball cluster aws cleanup

$ fuzzball cluster aws cleanup --all

$ fuzzball cluster aws preflight

$ fuzzball cluster aws preflight --provision-roles

$ fuzzball cluster aws deploy

$ fuzzball cluster aws deploy \
  --domain "$DOMAIN" \
  --organization-admin "admin@example.com" \
  --cluster-admin-arns "arn:aws:iam::123456789012:user/fuzzballAdmin" \
  --instance-types "t3a.2xlarge,p3.2xlarge" \
  --non-interactive

$ fuzzball cluster aws deploy \
  --domain "$DOMAIN" \
  --organization-admin "admin@example.com" \
  --cluster-admin-arns "arn:aws:iam::123456789012:user/fuzzballAdmin" \
  --permissions-boundary-arn "arn:aws:iam::123456789012:policy/my-boundary-policy" \
  --non-interactive

$ fuzzball cluster aws cleanup

$ fuzzball cluster aws cleanup --all

$ fuzzball cluster aws deploy \
  --depot-user <your-depot-username> \
  --depot-access-token <your-depot-access-token> \
  --domain "$DOMAIN" \
  --organization-admin "admin@example.com" \
  --cluster-admin-arns "arn:aws:iam::123456789012:user/fuzzballAdmin" \
  --instance-types "t3a.2xlarge,p3.2xlarge" \
  --non-interactive

$ export DEPOT_USER=<your-depot-username>

$ export DEPOT_ACCESS_TOKEN=<your-depot-access-token>

$ fuzzball cluster aws deploy \
  --domain "$DOMAIN" \
  --organization-admin "admin@example.com" \
  --cluster-admin-arns "arn:aws:iam::123456789012:user/fuzzballAdmin" \
  --instance-types "t3a.2xlarge,p3.2xlarge" \
  --non-interactive

$ ls /path/to/offline-images/
aws-pulumi-runner.tar       fuzzball-openapi.tar
fuzzball-admin.tar          fuzzball-orchestrate.tar
fuzzball-audit-archive.tar  fuzzball-substrate-bridge.tar
fuzzball-jetstream.tar      fuzzball-substrate-orchestrate.tar
fuzzball-operator.tar       fuzzball-sync.tar

$ fuzzball cluster aws deploy \
  --container-images-dir /path/to/offline-images \
  --domain "$DOMAIN" \
  --organization-admin "admin@example.com" \
  --cluster-admin-arns "arn:aws:iam::123456789012:user/fuzzballAdmin" \
  --instance-types "t3a.2xlarge,p3.2xlarge" \
  --non-interactive

$ fuzzball cluster aws deploy \
  --depot-user <user> \
  --depot-access-token <token> \
  --depot-host depot.internal.example.com \
  ...

$ fuzzball cluster aws update \
  --depot-user <user> \
  --depot-access-token <token>

Error: stack "fuzzball-20260101-120000" was deployed in non-Marketplace mode; updating it requires --depot-user and --depot-access-token (or DEPOT_USER / DEPOT_ACCESS_TOKEN env vars) so the new image versions can be hydrated into the customer ECR. Alternatively, pass --container-images-dir to load container images from local tar files (Helm charts still come from depot).

Error: stack "fuzzball-20260101-120000" was deployed against AWS account 123456789012 but the current session is account 987654321098; re-run with the AWS profile used to deploy this stack.

Error: stack "fuzzball-20260101-120000" was deployed against AWS region us-west-2 but the current session is region us-east-1; re-run with --region us-west-2.

$ fuzzball cluster aws delete --stack-name fuzzball-20260604-150405

$ fuzzball cluster aws cleanup --all

Lifecycle management

Once deployed, the following subcommands let you manage the cluster:

Use fuzzball cluster aws <subcommand> --help for the full list of options for each command.

$ fuzzball cluster gcp cleanup

Log in to the OCI registry in your GCP project

To deploy Fuzzball in GCP, you must log in via Helm to the OCI registry that you set up during the Requirements section.

To create an access token and use it to log into the OCI registry via Helm, execute the following command:

$ gcloud auth configure-docker ${REGION}-docker.pkg.dev

$ gcloud auth print-access-token | helm registry login ${REGION}-docker.pkg.dev \
  --username oauth2accesstoken \
  --password-stdin

Once you are logged in, you are ready to deploy Fuzzball!

Deployment

The simplest way to deploy is using interactive mode.

$ fuzzball cluster gcp deploy

The CLI will prompt you for all required parameters including the following:

• GCP project
• Region and zone
• Fuzzball version (defaults to the CLI version if omitted)
• Domain name
• Organization owner email

For non-interactive deployments, you can use the following. (See the Requirements section for information on setting these environment variables.)

$ fuzzball cluster gcp deploy \
  --project "$PROJECT_ID" \
  --region "$REGION" \
  --zone "$ZONE" \
  --version "$VERSION" \
  --domain "$SELECTED_DOMAIN" \
  --dns-zone-name "$MANAGED_ZONE" \
  --dns-zone-project "$PROJECT_ID" \
  --keycloak-owner-email "admin@example.com" \
  --deployment-name "unique-name" \
  --instance-types=n1-standard-4,n2-standard-8,g2-standard-8 \
  --non-interactive

For a full list of the options and arguments that can be specified during deployment, use the fuzzball cluster gcp deploy --help command.

A successful deployment will create a fuzzball cli context for the cluster in ${XDG_CONFIG_HOME:-$HOME/.config}/fuzzball/fuzzball.yaml and print out a basic summary of the cluster properties including commands to set up kubectl access to the underlying Kubernetes cluster (see below). You can also obtain similar information later by running

$ fuzzball cluster gcp info --project $PROJECT_ID --region $REGION

 SUCCESS  GCP authentication verified for project 'PROJECT'
 INFO  Searching for Fuzzball deployments to view info for...
 SUCCESS  Found 1 Fuzzball deployment(s):

  1. DEPLOYMENT_NAME (Version: v3-3-0, Region: us-central1, Status: ACTIVE)

 INFO  Using deployment: DEPLOYMENT_NAME

 INFO  Deployment Information

  Deployment:     DEPLOYMENT_NAME
  Project:        PROJECT
  Region:         REGION
  Status:         ACTIVE
  Version:        v3.3.0
  Domain:         DOMAIN
  Created:        2026-04-08t09-20-19z

 INFO  Cluster URLs

  API:            https://api.DOMAIN
  UI:             https://ui.DOMAIN
  Keycloak:       https://auth.DOMAIN

 INFO  Context Configuration

  To connect the Fuzzball CLI to this deployment, run:

    fuzzball context create DEPLOYMENT_NAME \
      --api-url https://api.DOMAIN \
      --auth-url https://auth.DOMAIN

    fuzzball context use DEPLOYMENT_NAME

DNS Zone Project Configuration

By default, the Pulumi runner looks for Cloud DNS zones in the deployment project. If your Cloud DNS zone lives in a different GCP project from the deployment project, you must pass the --dns-zone-project flag during deployment:

$ fuzzball cluster gcp deploy \
  --project "$PROJECT_ID" \
  --region "$REGION" \
  --zone "$ZONE" \
  --version "$VERSION" \
  --domain "$SELECTED_DOMAIN" \
  --dns-zone-name "$MANAGED_ZONE" \
  --dns-zone-project "$DNS_PROJECT_ID" \
  --keycloak-owner-email "admin@example.com" \
  --deployment-name "unique-name" \
  --instance-types=n1-standard-4,n2-standard-8,g2-standard-8 \
  --non-interactive

Replace $DNS_PROJECT_ID with the project ID hosting the Cloud DNS zone. This parameter is required when the DNS zone is managed in a separate project from the Fuzzball deployment.

The --dns-zone-project flag ensures that cert-manager's DNS01 ACME challenge solver can list Cloud DNS zones in the correct project during Let's Encrypt certificate issuance.

Substrate Image Project Configuration

After completing your deployment, if substrate images have been mirrored to a customer-owned GCP project, you must configure the substrateImageProject field in the FuzzballOrchestrate CRD.

This is required for:

• Air-gapped environments that cannot access the CIQ public image project
• Deployments with organizational policies that prohibit external image references
• Environments requiring custom image approval workflows

To configure a custom substrate image project, edit your FuzzballOrchestrate resource:

spec:
  fuzzball:
    orchestrator:
      provisioner:
        enabled: true
        gcp:
          enabled: true
          project: my-gcp-project
          region: us-central1
          zone: us-central1-a
          substrateImageProject: my-mirrored-images-project

Apply the updated configuration:

$ kubectl apply -f fuzzball-orchestrate.yaml

When substrateImageProject is empty or omitted, the provisioner uses the default CIQ public image project.

Managing and Monitoring your cluster

Kubernetes (kubectl)

You can use Kubernetes through the kubectl command to monitor your cluster as it deploys and to manage the underlying pods and resources. First you need to configure your local kubectl installation to use your GCP deployment. Issue the following commands:

$ gcloud container clusters list --project=$PROJECT_ID --region=$REGION

This will give you your cluster name. Now you can use it to execute the following:

$ gcloud container clusters get-credentials <cluster-name> --project $PROJECT_ID --region $REGION

You now have enough information to proceed to the Initial Configuration section and log in and finish setting your cluster up.

At this point you can also run commands like the following to monitor your deployment and check the health of your cluster:

$ kubectl logs -l app.kubernetes.io/name=fuzzball-operator -n fuzzball-system -f --tail=-1

$ kubectl get pods -n fuzzball

Fuzzball CLI

The fuzzball cluster gcp info or the deployment-info-wait.sh script referenced in the initial login section display a command to create the appropriate context to add and log into your cluster along with the credentials needed for the automatically provisioned users including the cluster admin user. Once you have done so, you can use the fuzzball command directly to monitor and manage many aspects of your deployment.

For instance, the list, status, info, and logs commands allow you to view information about your running deployment(s).

And the update, destroy, and cleanup commands allow you to manage your cluster directly.

The generate-substrate-config command generates the configuration an external substrate node (for example, one attached over a VPN) needs to join the cluster, and prints the node-side installation steps. It requires the deployment's dynamic provisioner to be enabled.

Use the --help flag in the CLI to list these commands and see information about running each of them.

$ fuzzball cluster azure cleanup

Authenticate to Azure and Depot

The Azure deployment uses the Azure CLI session for control-plane operations, plus your CIQ Depot credentials for pulling Fuzzball container images and Helm charts. Set up both before deploying.

$ az login

$ az account set --subscription "$SUBSCRIPTION_ID"

Capture your Depot credentials in environment variables (these are picked up automatically by the CLI):

$ export DEPOT_USER=<your depot username>

$ export DEPOT_ACCESS_TOKEN=<your depot access token>

Deployment

The simplest way to deploy is using interactive mode.

$ fuzzball cluster azure deploy

The CLI will prompt you for all required parameters including the following:

• Azure subscription (if you have multiple)
• Azure location (region)
• Resource group (defaults to fuzzball)
• Fuzzball version (defaults to the CLI version if omitted)
• Domain name
• Organization owner email
• DNS provider configuration

For non-interactive deployments, you can use the following. (See the Requirements section for information on setting these environment variables.)

$ fuzzball cluster azure deploy \
  --subscription "$SUBSCRIPTION_ID" \
  --location "$LOCATION" \
  --resource-group "$RESOURCE_GROUP" \
  --version "$VERSION" \
  --domain "$DOMAIN" \
  --keycloak-owner-email "admin@example.com" \
  --instance-types "Standard_D4s_v5,Standard_NC6s_v3" \
  --dns-provider azuredns \
  --dns-azuredns-subscription "$SUBSCRIPTION_ID" \
  --dns-azuredns-resource-group "$DNS_ZONE_RG" \
  --dns-azuredns-zone-name "$DOMAIN" \
  --non-interactive

For a full list of the options and arguments that can be specified during deployment, use the fuzzball cluster azure deploy --help command.

A successful deployment will create a fuzzball CLI context for the cluster in ${XDG_CONFIG_HOME:-$HOME/.config}/fuzzball/fuzzball.yaml and print out a basic summary of the cluster properties including commands to set up kubectl access to the underlying AKS cluster (see below). You can also obtain similar information later by running:

$ fuzzball cluster azure info --subscription $SUBSCRIPTION_ID --resource-group $RESOURCE_GROUP

Managing and Monitoring your cluster

Kubernetes (kubectl)

You can use Kubernetes through the kubectl command to monitor your cluster as it deploys and to manage the underlying pods and resources. First you need to configure your local kubectl installation to use your AKS deployment. Issue the following commands:

$ az aks list --resource-group $RESOURCE_GROUP --query "[].name" -o tsv

This will give you your AKS cluster name. Now you can use it to fetch credentials:

$ az aks get-credentials --resource-group $RESOURCE_GROUP --name <cluster-name>

You now have enough information to proceed to the Initial Configuration section and log in and finish setting your cluster up.

At this point you can also run commands like the following to monitor your deployment and check the health of your cluster:

$ kubectl logs -l app.kubernetes.io/name=fuzzball-operator -n fuzzball-system -f --tail=-1

$ kubectl get pods -n fuzzball

Fuzzball CLI

Once you have created and switched to the Fuzzball context for your cluster, you can use the fuzzball command directly to monitor and manage many aspects of your deployment. The Azure cloud subcommand provides the following operations:

• fuzzball cluster azure list — list every Fuzzball deployment in the subscription.
• fuzzball cluster azure status — show health and resource state for a specific deployment.
• fuzzball cluster azure info — print deployment details (domain, version, endpoints).
• fuzzball cluster azure logs — fetch operator and orchestrate logs from the AKS cluster.
• fuzzball cluster azure update — upgrade Fuzzball version or change configuration in place.
• fuzzball cluster azure destroy — tear down a deployment and delete its resource group.
• fuzzball cluster azure cleanup — sweep up orphaned resources from a failed or partial destroy.
• fuzzball cluster azure generate-substrate-config — generate the configuration an external substrate node (for example, one attached over a VPN) needs to join the cluster; the command prints the node-side installation steps. Requires the dynamic provisioner.

Use the --help flag on any of these to see the full flag list.

After fulfilling the prerequisites listed in the requirements and discovering your cluster's domain using the domain discovery procedure, you are ready to deploy Fuzzball on CoreWeave.

Step 1: Install the Fuzzball Operator

The Fuzzball operator manages the deployment and lifecycle of Fuzzball on your Kubernetes cluster.

Authenticate with the CIQ Depot

First, use your Depot credentials to authenticate with the Helm registry:

$ DEPOT_USER="your-depot-username"

$ ACCESS_KEY="your-depot-access-key"

Replace your-depot-username and your-depot-access-key with the credentials provided by CIQ.

$ helm registry login depot.ciq.com --username "${DEPOT_USER}" --password "${ACCESS_KEY}"

Install the Operator Helm Chart

Install the Fuzzball operator using the Helm chart from the depot:

$ VERSION="<Version />"

$ CHART="oci://depot.ciq.com/fuzzball/fuzzball-images/helm/fuzzball-operator"

$ STORAGE_CLASS="shared-vast"

$ helm upgrade --install fuzzball-operator "${CHART}" \
  --namespace fuzzball-system \
  --create-namespace \
  --version "${VERSION}" \
  --set "image.tag=${VERSION}" \
  --set "imagePullSecrets.name=repository-ciq-com" \
  --set "imagePullSecrets.inline.registry=depot.ciq.com" \
  --set "imagePullSecrets.inline.username=${DEPOT_USER}" \
  --set "imagePullSecrets.inline.password=${ACCESS_KEY}" \
  --set "storageClassName=${STORAGE_CLASS}"

Verify the Operator Installation

Check that the operator pod is running:

$ kubectl get pods -n fuzzball-system
NAME                                              READY   STATUS    RESTARTS   AGE
fuzzball-operator-controller-manager-xxxxx-xxxxx  2/2     Running   0          2m

Expected output should show the operator pod in Running state with 2/2 containers ready.

Step 2: Deploy Fuzzball with FuzzballOrchestrate

Create a FuzzballOrchestrate custom resource to deploy Fuzzball on CoreWeave. Here's a complete example configuration:

apiVersion: deployment.ciq.com/v1alpha1
kind: FuzzballOrchestrate
metadata:
  name: fuzzball-coreweave
  namespace: fuzzball-system
spec:
  # Image registry configuration
  image:
    username: <depot-username>
    password: <depot-password>
    exclusive: false

  # Database configuration
  database:
    create:
      enableDebugPod: false
      storage:
        class: shared-vast

  # Fuzzball version and cluster configuration
  fuzzball:
    version: <Version />
    cluster:
      name: fuzzball-coreweave

    # CoreWeave provisioner configuration
    orchestrator:
      provisioner:
        enabled: true
        coreweave:
          enabled: true
          storage:
            accessMode: ReadWriteMany
            class: shared-vast
            size: 100Gi  # Adjust based on workflow needs

    # Shared container image cache
    config:
      sharedPVC:
        accessMode: ReadWriteMany
        class: shared-vast
        size: 10Gi  # Adjust based on caching needs

  # Ingress and networking configuration
  ingress:
    create:
      # Use your discovered CoreWeave domain
      domain: <YOUR_COREWEAVE_DOMAIN>
      proxy:
        type: LoadBalancer
        annotations:
          # Public LoadBalancer for internet access
          service.beta.kubernetes.io/coreweave-load-balancer-type: public
          # Wildcard DNS for all services
          service.beta.kubernetes.io/external-hostname: '*.<YOUR_COREWEAVE_DOMAIN>'

  # Keycloak identity management
  keycloak:
    create:
      createDatabase: true
      # Generate with: uuidgen
      realmId: <uuid-v4-realm-id>
      ownerEmail: <owner-email>
      # Change this password after first login!
      defaultUserPassword: <initial-user-password>

  # TLS certificate configuration
  tls:
    # cert-manager for certificate management
    certManager:
      create: {}
    # trust-manager for CA certificate distribution
    trustManager:
      create: {}
    # Let's Encrypt certificate issuer
    ingressIssuer:
      create:
        letsEncrypt:
          email: <letsencrypt-email>
          issuer: letsencrypt-prod

Replace the placeholder values with your configuration. Use the domain discovered in the domain discovery procedure.

Apply the Configuration

Save the configuration to a file and apply it with kubectl:

$ kubectl apply -f fuzzball-orchestrate-coreweave.yaml

Monitor Deployment Progress

Watch the deployment status:

$ kubectl get fuzzballorchestrate -A -w

Wait until the status shows Ready. This typically takes several minutes for a full deployment.

Verify Deployment Components

Check the deployed resources:

$ kubectl get fuzzballorchestrate -A

$ kubectl get pvc -n fuzzball

$ kubectl get pods -n fuzzball

For detailed deployment status and troubleshooting, check the operator logs:

$ kubectl logs -l control-plane=fuzzball-operator-controller-manager -n fuzzball-system -f

Alternative Deployment Pattern: Static Provisioning

The deployment steps above configure dynamic provisioning, where Fuzzball creates and destroys CoreWeave nodes on-demand. If you prefer to manage node pools yourself for predictable capacity or faster startup times, see Static Node Pool Provisioning for an alternative deployment approach.