FuzzballOrchestrate CRD Reference

The FuzzballOrchestrate Custom Resource Definition (CRD) is the primary way to deploy and configure a Fuzzball cluster on Kubernetes. This CRD provides comprehensive control over all aspects of a Fuzzball deployment, from basic infrastructure (database, ingress, authentication) to advanced features (multi-cloud provisioning, billing integration, autoscaling).

Basic Example

The minimal configuration requires image credentials, ingress configuration, database setup, and authentication. This example is suitable for local or on-premises deployments:


apiVersion: deployment.ciq.com/v1alpha1
kind: FuzzballOrchestrate
metadata:
  name: fuzzball-orchestrate
spec:
  image:
    username: ${DEPOT_USER}
    password: ${ACCESS_KEY}
    exclusive: false
  ingress:
    create:
      domain: "10.0.0.99.nip.io"
      proxy:
        type: LoadBalancer
        annotations:
          metallb.io/loadBalancerIPs: 10.0.0.99
  database:
    create:
      storage:
        class: longhorn
  keycloak:
    create:
      ingress:
        hostname: auth.10.0.0.99.nip.io
      realmName: Fuzzball
      realmId: 550e8400-e29b-41d4-a716-446655440000
      username: keycloak
      ownerEmail: "admin@example.com"
      createDatabase: true
  tls:
    certManager:
      create: {}
    trustManager:
      create: {}
  fuzzball:
    substrate:
      nfs:
        destination: "/fuzzball/shared"
        server: 10.0.0.10
        path: "/srv/fuzzball/shared"
    jetstream:
      replicas: 3
      storage:
        class: longhorn
        size: 10Gi

Configuration Sections

Note

Many credential fields (passwords, API keys, TLS certificates, image pull secrets) support references to Kubernetes Secrets as an alternative to inline values. Secret references are recommended for production deployments. See Kubernetes Secret References for the full list of fields and usage examples.

Image Configuration

Controls where container images are pulled from and authentication:

Parameter	Type	Required	Default	Description
`repository`	string	No	`depot.ciq.com/fuzzball/fuzzball-images`	Container image registry
`credentialsSecretName`	string	No	-	Name of a `kubernetes.io/dockerconfigjson` Secret with registry credentials (recommended)
`username`	string	No	-	Registry authentication username (deprecated — use `credentialsSecretName`)
`password`	string	No	-	Registry authentication password (deprecated — use `credentialsSecretName`)
`exclusive`	boolean	No	`true`	If true, all images must come from specified repository

Example with inline credentials:


spec:
  image:
    repository: depot.ciq.com/fuzzball/fuzzball-images
    username: my-depot-user
    password: my-depot-token
    exclusive: false

Example with Secret reference (recommended for production):


spec:
  image:
    credentialsSecretName: depot-pull-secret
    exclusive: false

Ingress Configuration

Defines how the cluster is exposed to the network. Choose either create (new ingress) or external (existing ingress controller).

Option A: Create New Ingress

Parameter	Type	Required	Default	Description
`domain`	string	Yes	-	Base domain for all Fuzzball services
`proxy.type`	string	Yes	`LoadBalancer`	Service type: `LoadBalancer` or `NodePort`
`proxy.annotations`	map	No	`{}`	Annotations for the proxy service (e.g., MetalLB config)
`proxy.http.nodePort`	integer	No	-	NodePort for HTTP (if type is NodePort)
`proxy.tls.nodePort`	integer	No	-	NodePort for HTTPS (if type is NodePort)

Example with LoadBalancer:


spec:
  ingress:
    create:
      domain: fuzzball.example.com
      proxy:
        type: LoadBalancer
        annotations:
          metallb.io/loadBalancerIPs: 10.0.0.99
          metallb.io/allow-shared-ip: ingress-and-fuzzball

CoreWeave LoadBalancer Annotations

CoreWeave requires specific annotations on the LoadBalancer service for automatic DNS configuration:

Annotation	Value	Description
`service.beta.kubernetes.io/coreweave-load-balancer-type`	`public`	Creates an internet-accessible load balancer
`service.beta.kubernetes.io/external-hostname`	`*.<domain>`	Wildcard domain for automatic DNS resolution

CoreWeave example:


spec:
  ingress:
    create:
      domain: a1b2c3-my-cluster.coreweave.app
      proxy:
        type: LoadBalancer
        annotations:
          service.beta.kubernetes.io/coreweave-load-balancer-type: public
          service.beta.kubernetes.io/external-hostname: '*.a1b2c3-my-cluster.coreweave.app'

Example with NodePort:


spec:
  ingress:
    create:
      domain: fuzzball.example.com
      proxy:
        type: NodePort
        http:
          nodePort: 30080
        tls:
          nodePort: 30443

Option B: Use External Ingress

Parameter	Type	Required	Default	Description
`domain`	string	Yes	-	Base domain for all Fuzzball services
`className`	string	Yes	-	Ingress class name (e.g., `nginx`, `traefik`)
`annotations`	map	No	`{}`	Annotations for ingress resources

Example:


spec:
  ingress:
    external:
      domain: fuzzball.example.com
      className: nginx
      annotations:
        cert-manager.io/cluster-issuer: letsencrypt-prod

Database Configuration

Fuzzball requires a PostgreSQL database. Choose either create (deploys a dedicated PostgreSQL alongside Orchestrate) or external.

Option A: Create Dedicated Database

Parameter	Type	Required	Default	Description
`enableDebugPod`	boolean	No	`false`	Deploy a debug pod with database tools
`annotations`	map	No	`{}`	Annotations for database resources
`storage.class`	string	No	-	StorageClass for database PVC
`storage.size`	string	No	-	Size of database storage
`storage.accessMode`	string	No	`ReadWriteOnce`	Access mode for PVC
`storage.annotations`	map	No	`{}`	Annotations for PVC

Example:


spec:
  database:
    create:
      enableDebugPod: true
      storage:
        class: longhorn
        size: 200Gi
        accessMode: ReadWriteOnce

Option B: Use External Database

Parameter	Type	Required	Default	Description
`host`	string	Yes	-	Database hostname or IP
`port`	string	No	`5432`	Database port
`driver`	string	No	`postgres`	Database driver
`credentials.user`	string	Yes	-	Database username
`credentials.password`	string	No	-	Database password (inline)
`credentials.passwordRef`	object	No	-	Reference to a K8s Secret containing the database password (preferred over inline)
`sslMode`	string	No	`verify-full`	SSL mode: `disable`, `require`, `verify-ca`, `verify-full`
`rdsSecretId`	string	No	-	AWS Secrets Manager ARN for RDS credentials
`certificate.caCert`	string	No	-	CA certificate for SSL verification (inline)
`certificate.caCertURL`	string	No	-	URL to download CA certificate
`certificate.caCertRef`	object	No	-	Reference to a K8s Secret containing the CA certificate (preferred over inline)
`certificate.clientCert`	string	No	-	Client certificate for mTLS (inline)
`certificate.clientCertRef`	object	No	-	Reference to a K8s Secret containing the client certificate (preferred over inline)
`certificate.clientKey`	string	No	-	Client key for mTLS (inline)
`certificate.clientKeyRef`	object	No	-	Reference to a K8s Secret containing the client key (preferred over inline)

Example with external PostgreSQL:


spec:
  database:
    external:
      host: postgres.example.com
      port: "5432"
      driver: postgres
      credentials:
        user: fuzzball
        passwordRef:
          name: db-credentials
          key: password
      sslMode: verify-full

Example with AWS RDS:


spec:
  database:
    external:
      host: mydb.abc123.us-east-1.rds.amazonaws.com
      rdsSecretId: arn:aws:secretsmanager:us-east-1:123456789012:secret:rds-db-credentials-abc123
      sslMode: verify-full
      certificate:
        caCertURL: https://truststore.pki.rds.amazonaws.com/global/global-bundle.pem

Keycloak Authentication

For Keycloak-backed deployments, Fuzzball delegates authentication to Keycloak. Choose either create (deploy Keycloak alongside Orchestrate) or external. As of v3.3.0, Keycloak is optional — deployments that use the local-organization flow authenticate directly with Fuzzball without a Keycloak instance.

Option A: Deploy Keycloak

Note

As of v3.3.0, plaintext password and defaultUserPassword fields are deprecated. Use the passwordRef and defaultUserPasswordRef fields to reference passwords stored in Kubernetes Secrets. The plaintext fields continue to work but will be removed in a future release.

Parameter	Type	Required	Default	Description
`ownerEmail`	string	Yes	-	Email address for the Fuzzball organization owner
`realmId`	string	No	(generated)	Keycloak realm ID (must be UUID v4, lower case)
`realmName`	string	No	`Fuzzball`	Keycloak realm display name
`username`	string	No	`keycloak`	Keycloak admin username
`password`	string	No	(generated)	Deprecated. Keycloak admin password (plaintext). Use `passwordRef` instead.
`passwordRef.name`	string	No	-	Name of Kubernetes Secret containing the Keycloak admin password
`passwordRef.key`	string	No	-	Key within the Secret for the Keycloak admin password
`passwordRef.namespace`	string	No	`fuzzball-system`	Namespace of the Secret; defaults to the Fuzzball system namespace
`defaultUserPassword`	string	No	-	Deprecated. Default password for new users (plaintext). Use `defaultUserPasswordRef` instead.
`defaultUserPasswordRef.name`	string	No	-	Name of Kubernetes Secret containing the default user password
`defaultUserPasswordRef.key`	string	No	-	Key within the Secret for the default user password
`defaultUserPasswordRef.namespace`	string	No	`fuzzball-system`	Namespace of the Secret; defaults to the Fuzzball system namespace
`createDatabase`	boolean	No	`true`	Create dedicated database for Keycloak
`replicas`	integer	No	`1`	Number of Keycloak replicas
`ingress.hostname`	string	No	`auth.{domain}`	Hostname for Keycloak UI
`ingress.className`	string	No	-	Ingress class name
`ingress.tls.cert`	string	No	-	TLS certificate (inline)
`ingress.tls.key`	string	No	-	TLS private key (inline)
`ingress.tls.secretName`	string	No	-	Name of a `kubernetes.io/tls` Secret containing the TLS cert and key (preferred over inline)
`ingress.annotations`	map	No	`{}`	Ingress annotations

Preferred example using SecretKeyReference:


spec:
  keycloak:
    create:
      ownerEmail: admin@example.com
      realmName: MyOrganization
      realmId: 550e8400-e29b-41d4-a716-446655440000
      username: keycloak
      passwordRef:
        name: keycloak-credentials
        key: admin-password
      defaultUserPasswordRef:
        name: keycloak-credentials
        key: default-user-password
      createDatabase: true
      ingress:
        hostname: auth.fuzzball.example.com

Initial Setup

Generating a Realm ID

Generate a UUID v4 for the realm ID:


$ uuidgen

Use the output in your configuration:


realmId: 550e8400-e29b-41d4-a716-446655440000

Setting the Owner Account

The ownerEmail and defaultUserPassword fields create the initial administrator account with full cluster administration privileges. Change the default password after first login.


ownerEmail: admin@example.com
defaultUserPassword: initial-secure-password

LDAP Integration

Keycloak can be configured to federate users from an LDAP/Active Directory server:

Parameter	Type	Required	Description
`ldap.url`	string	Yes	LDAP server URL (ldap:// or ldaps://)
`ldap.startTLS`	boolean	No	Use StartTLS for ldap:// connections
`ldap.insecure`	boolean	No	Skip TLS certificate verification
`ldap.vendor`	string	No	LDAP vendor: `ad`, `rhds`, `tivoli`, `edirectory`, `other`
`ldap.bindDN`	string	No	Bind DN for LDAP authentication
`ldap.bindPassword`	string	No	Password for bind DN (inline)
`ldap.bindPasswordRef`	object	No	Reference to a K8s Secret containing the bind password (preferred over inline)
`ldap.searchScope`	string	No	Search scope: `single` or `subtree`
`ldap.users.dn`	string	Yes	Base DN for user search
`ldap.users.attributes.*`	string	Yes	User attribute mappings
`ldap.users.objectClasses`	[]string	Yes	User object classes
`ldap.users.filter`	string	No	LDAP filter for users
`ldap.groups.dn`	string	Yes	Base DN for group search
`ldap.groups.membershipAttributeType`	string	Yes	Membership type: `dn` or `uid`
`ldap.groups.userGroupsStrategy`	string	Yes	Strategy for loading groups
`ldap.groups.attributes.*`	string	Yes	Group attribute mappings
`ldap.groups.objectClasses`	[]string	Yes	Group object classes
`ldap.groups.filter`	string	No	LDAP filter for groups

LDAP example:


spec:
  keycloak:
    create:
      ownerEmail: admin@example.com
      ldap:
        url: ldaps://ldap.example.com
        vendor: ad
        bindDN: cn=fuzzball,ou=service,dc=example,dc=com
        bindPassword: ldap-bind-password
        searchScope: subtree
        users:
          dn: ou=users,dc=example,dc=com
          attributes:
            username: sAMAccountName
            rdn: sAMAccountName
            uuid: objectGUID
            uidNumber: uidNumber
            gidNumber: gidNumber
          objectClasses:
            - person
            - organizationalPerson
          filter: "(memberOf=cn=fuzzball-users,ou=groups,dc=example,dc=com)"
        groups:
          dn: ou=groups,dc=example,dc=com
          membershipAttributeType: dn
          userGroupsStrategy: get_groups_from_user_memberof_attribute
          attributes:
            groupName: cn
            groupMembership: member
            userMembership: sAMAccountName
            memberOf: memberOf
            gidNumber: gidNumber
          objectClasses:
            - group
          filter: "(cn=fuzzball-*)"

Option B: Use External Keycloak

Parameter	Type	Required	Description
`url`	string	Yes	Keycloak server URL
`realmId`	string	Yes	Existing realm ID (UUID v4)
`realmName`	string	Yes	Existing realm name
`username`	string	Yes	Admin username
`password`	string	No	Admin password (inline)
`passwordRef`	object	No	Reference to a K8s Secret containing the admin password (preferred over inline)
`ownerEmail`	string	Yes	Organization owner email
`useLDAPFederation`	boolean	No	Whether LDAP is configured

Example:


spec:
  keycloak:
    external:
      url: https://keycloak.example.com
      realmId: 550e8400-e29b-41d4-a716-446655440000
      realmName: ExistingRealm
      username: admin
      passwordRef:
        name: keycloak-credentials
        key: admin-password
      ownerEmail: admin@example.com

TLS Configuration

Optional configuration for certificate management and Let's Encrypt:

cert-manager Deployment

Parameter	Type	Required	Description
`certManager.create`	object	No	Deploy cert-manager (empty object for defaults)
`certManager.serviceAccount.annotations`	map	No	Service account annotations (for IRSA)
`trustManager.create`	object	No	Deploy trust-manager (empty object for defaults)
Default behavior (empty objects for certManager and trustManager) is to issue self-signed certificates

Certificate Issuers

Parameter	Type	Required	Description
`internalIssuer`	object	No	Internal certificate issuer configuration
`ingressIssuer.create.letsEncrypt.email`	string	No	Email for Let's Encrypt notifications
`ingressIssuer.create.letsEncrypt.issuer`	string	No	Issuer name (e.g., `letsencrypt-prod`, `letsencrypt-staging`)
`ingressIssuer.create.letsEncrypt.solvers`	[]object	No	ACME challenge solvers (dns01 or http01)
`ingressIssuer.external.issuerName`	string	No	Name of existing ClusterIssuer for ingress TLS (preferred)
`ingressIssuer.external.internalCAIssuerName`	string	No	Name of existing ClusterIssuer for ingress TLS (deprecated, use `issuerName`)
`internalIssuer.external.issuerName`	string	No	Name of existing ClusterIssuer for internal TLS (preferred)
`internalIssuer.external.internalCAIssuerName`	string	No	Name of existing ClusterIssuer for internal TLS (deprecated, use `issuerName`)
`internalIssuer.create.customCA.inline.secretName`	string	No	Reference to an existing `kubernetes.io/tls` Secret providing the CA cert/key for the internal issuer (otherwise the operator generates a self-signed CA)
`ingressIssuer.create.customCA.inline.secretName`	string	No	Reference to an existing `kubernetes.io/tls` Secret providing the CA cert/key for the ingress issuer

Note

When using external issuers, the operator can automatically extract the CA certificate from the ClusterIssuer and distribute it via trust-manager, assuming trust-manager is running. Enable trustManager.create so the operator deploys it, or install trust-manager separately. When trust-manager is available, manual creation of the trusted-ca-certs-bundle ConfigMap is no longer required. trustManager.create can be set independently of certManager.create when using external issuers.

Example with Let's Encrypt (DNS-01):


spec:
  tls:
    certManager:
      create: {}
    trustManager:
      create: {}
    ingressIssuer:
      create:
        letsEncrypt:
          email: admin@example.com
          issuer: letsencrypt-prod
          solvers:
            - dns01:
                route53:
                  region: us-east-1
                  hostedZoneID: Z1234567890ABC

Example with Let's Encrypt (HTTP-01):


spec:
  tls:
    certManager:
      create: {}
    ingressIssuer:
      create:
        letsEncrypt:
          email: admin@example.com
          issuer: letsencrypt-prod
          solvers:
            - http01:
                ingress:
                  class: nginx

Fuzzball Services Configuration

The fuzzball section controls all Fuzzball services and their configurations.

Global Settings

Parameter	Type	Description
`version`	string	Fuzzball version (defaults to operator version)
`cluster.name`	string	Cluster name (default: `unset-cluster`)
`cluster.kind`	string	Cluster kind/type

Substrate Configuration

Controls how substrate nodes connect and operate:

Parameter	Type	Description
`substrate.nfs.destination`	string	Mount point on substrate nodes
`substrate.nfs.server`	string	NFS server IP or hostname
`substrate.nfs.path`	string	NFS export path
`substrate.secureRegistries`	[]string	Private registries requiring authentication
`substrate.imageProxy`	string	HTTP(S) proxy for image pulling
`substrate.imageNoProxy`	[]string	Hosts to exclude from proxy
`substrate.mtls.*`	object	mTLS configuration for substrate

Example:


spec:
  fuzzball:
    cluster:
      name: my-fuzzball-cluster
      kind: on-premises
    substrate:
      nfs:
        destination: /fuzzball/shared
        server: 10.0.0.10
        path: /srv/fuzzball/shared
      secureRegistries:
        - depot.ciq.com
      imageProxy: http://proxy.example.com:3128
      imageNoProxy:
        - localhost
        - 127.0.0.1
        - .example.com

Kubernetes Scheduling Configuration

Three scheduling fields at the top level of the CRD spec (spec.nodeSelector, spec.tolerations, spec.affinity) apply to every Fuzzball service pod as global defaults:

Parameter	Type	Description
`nodeSelector`	map	Label selector: only schedule pods onto nodes with matching labels
`tolerations`	[]object	Allow pods to schedule onto nodes with matching taints
`affinity`	object	Node and pod affinity/anti-affinity rules (standard Kubernetes `affinity` object)

Each individual service spec supports the same three fields as per-component overrides. When set on a service, the override replaces (does not merge with) the global value for that service.

Example — place all services on a dedicated node pool, but keep Jetstream on storage-optimized nodes:


spec:
  nodeSelector:
    node-pool: fuzzball-system        # global default
  tolerations:
    - key: fuzzball
      operator: Equal
      value: system
      effect: NoSchedule
  fuzzball:
    jetstream:
      nodeSelector:
        node-pool: storage-optimized  # overrides global for Jetstream only
      tolerations:
        - key: storage
          operator: Exists
          effect: NoSchedule

Example — node affinity for a specific availability zone:


spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: topology.kubernetes.io/zone
                operator: In
                values:
                  - us-east-1a

See the Kubernetes documentation for full syntax details on tolerations and affinity objects.

Service-Specific Configuration

Most Fuzzball services support the following common parameters:

Parameter	Type	Description
`replicas`	integer	Number of pod replicas
`autoscaling.enabled`	boolean	Enable horizontal pod autoscaling
`autoscaling.minReplicas`	integer	Minimum replicas when autoscaling
`autoscaling.maxReplicas`	integer	Maximum replicas when autoscaling
`autoscaling.targetCPUUtilization`	integer	Target CPU percentage (default: 80)
`autoscaling.targetMemoryUtilization`	integer	Target memory percentage (default: 80)
`resources.requests`	map	Resource requests (cpu, memory) as key-value pairs
`resources.limits`	map	Resource limits (cpu, memory) as key-value pairs
`serviceAccount.annotations`	map	Service account annotations (for IRSA)
`nodeSelector`	map	Override global `nodeSelector` for this service only
`tolerations`	[]object	Override global `tolerations` for this service only
`affinity`	object	Override global `affinity` for this service only

Example for orchestrator service:


spec:
  fuzzball:
    orchestrator:
      replicas: 2
      autoscaling:
        enabled: true
        minReplicas: 1
        maxReplicas: 5
        targetCPUUtilization: 70
        targetMemoryUtilization: 75
      resources:
        requests:
          cpu: 500m
          memory: 512Mi
        limits:
          cpu: 2000m
          memory: 2Gi
      serviceAccount:
        annotations:
          eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/fuzzball-orchestrator

Available Configuration Sections

The following configuration sections are available under spec.fuzzball:

Top-Level Configuration:

version - Fuzzball version (defaults to operator version)
config - Global configuration (shared storage, etc.)
cluster - Cluster name and metadata
substrate - Substrate node configuration (NFS, registries, proxies, mTLS)

Active Services:

agent - Agent service (workflow execution)
audit - Audit logging service
auth - Authentication service (SpiceDB)
billing - Billing and marketplace integration
clusterAdmin - Admin UI and cluster setup
jetstream - NATS Jetstream message broker
openapi - OpenAPI documentation service
orchestrator - Orchestrator service
storage - Storage service
substrateBridge - Substrate bridge service (DNS, logging, Kubernetes integration)
ui - Web UI service
workflow - Workflow service
workflowCatalog - Workflow catalog configuration

The following services have been deprecated as of version v3.0 and will be removed in a future version (use recommended replacements):

account - Deprecated (use agent)
dns - Deprecated (use substrateBridge)
kube - Deprecated (use substrateBridge)
log - Deprecated (use substrateBridge)
organization - Deprecated (use agent)
provision - Deprecated (use orchestrator)
schedule - Deprecated (use orchestrator)
secret - Deprecated (use agent)
user - Deprecated (use agent)
workflowEngine - Deprecated (use jetstream)

Jetstream Configuration

NATS Jetstream has specific configuration requirements:

Parameter	Type	Default	Description
`replicas`	integer	`3`	Number of Jetstream replicas (recommended: 3)
`storage.class`	string	-	StorageClass for Jetstream PVCs
`storage.size`	string	`10Gi`	Storage size per replica
`externalService.type`	string	-	External service type (NodePort, LoadBalancer)

Example:


spec:
  fuzzball:
    jetstream:
      replicas: 3
      storage:
        class: longhorn
        size: 20Gi
      externalService:
        type: NodePort

Storage Service Configuration

Note

This storage is necessary for internal services. Admins can configure storage for computational jobs later.

Parameter	Type	Description
`storage.storage.class`	string	StorageClass for storage service

Example:


spec:
  fuzzball:
    storage:
      storage:
        class: longhorn

Audit Service Configuration

Parameter	Type	Description
`audit.storage.class`	string	StorageClass for audit logs

Example:


spec:
  fuzzball:
    audit:
      storage:
        class: longhorn

Substrate Bridge Configuration

Parameter	Type	Description
`substrateBridge.log.storage.class`	string	StorageClass for substrate logs
`substrateBridge.dns.externalService.type`	string	DNS service type

Note

NodePort is recommended for local deployments.

Example:


spec:
  fuzzball:
    substrateBridge:
      log:
        storage:
          class: longhorn
      dns:
        externalService:
          type: NodePort

Orchestrator Provisioner Configuration

The orchestrator provisioner enables multi-cloud and HPC integration:

Parameter	Type	Description
`enabled`	boolean	Enable provisioner
`substrateComputeDirectory`	string	Working directory for substrate operations
`aws`	object	AWS provisioner configuration
`coreweave`	object	CoreWeave provisioner configuration
`slurm`	object	Slurm provisioner configuration
`pbs`	object	PBS provisioner configuration

AWS Provisioner

Parameter	Type	Description
`aws.enabled`	boolean	Enable AWS provisioner
`aws.region`	string	AWS region
`aws.subnetIDs`	[]string	VPC subnet IDs
`aws.securityGroupIDs`	[]string	Security group IDs
`aws.instanceProfileARN`	string	IAM instance profile ARN
`aws.usePublicIP`	boolean	Assign public IPs
`aws.sshEnabled`	boolean	Enable SSH access
`aws.sshKeyPairName`	string	EC2 key pair name
`aws.sshPrivateKeyPem`	string	SSH private key
`aws.depotUser`	string	CIQ Depot username for substrate nodes
`aws.depotAccessToken`	string	CIQ Depot access token for substrate nodes
`aws.cloudInitScripts`	[]string	Custom cloud-init scripts to run on instances
`aws.sharedFs`	map	Shared filesystem configuration

Example:


spec:
  fuzzball:
    orchestrator:
      provisioner:
        enabled: true
        aws:
          enabled: true
          region: us-east-1
          subnetIDs:
            - subnet-0123456789abcdef0
          securityGroupIDs:
            - sg-0123456789abcdef0
          instanceProfileARN: arn:aws:iam::123456789012:instance-profile/FuzzballSubstrate
          usePublicIP: false

CoreWeave Provisioner

The CoreWeave provisioner enables dynamic provisioning of compute nodes on CoreWeave infrastructure. When enabled, Fuzzball will automatically create and manage CoreWeave NodePool resources and deploy substrate pods as DaemonSets on provisioned nodes.

CoreWeave deployments use two types of shared storage PVCs:

Workflow Data PVC (fuzzball-shared-storage): Mounted at /mnt/shared-storage on substrate nodes and exposed to workflows via the HostPath CSI driver
Image Cache PVC (fuzzball-sharedfs): Mounted at /mnt/fuzzball-sharedfs on substrate nodes for internal container image caching (not exposed to workflows)

Warning

CoreWeave's shared-vast storage requires Native Protocol Limit view policy. Ensure this is configured before deployment.

Workflow Data Storage Parameters

Parameter	Type	Required	Default	Description
`coreweave.enabled`	boolean	Yes	-	Enable CoreWeave provisioner
`coreweave.storage.class`	string	No	-	StorageClass for CoreWeave shared storage PVCs (typically `shared-vast`)
`coreweave.storage.size`	string	No	-	Size of shared storage per substrate node
`coreweave.storage.accessMode`	string	No	`ReadWriteOnce`	Access mode for shared storage PVC (use `ReadWriteMany` for multi-node access)
`coreweave.storage.annotations`	map	No	`{}`	Additional annotations for storage resources

Example:


spec:
  fuzzball:
    orchestrator:
      provisioner:
        enabled: true
        coreweave:
          enabled: true
          storage:
            class: shared-vast
            size: 100Gi
            accessMode: ReadWriteMany

Workflow Data Storage Sizing

Workload Type	Recommended Size	Rationale
Light workflows	50Gi	Minimal data processing
Standard workflows	100Gi	Typical data processing needs
Heavy workflows	250Gi+	Large datasets, intermediate files
Data-intensive	500Gi+	Big data processing, ML training

Shared PVC Configuration (`fuzzball.config.sharedPVC`)

The shared PVC (fuzzball-sharedfs) is mounted at /mnt/fuzzball-sharedfs on substrate nodes for internal container image caching across compute nodes. This improves workflow startup times by avoiding repeated image downloads. This storage is not exposed to workflows.

Parameter	Type	Required	Default	Description
`fuzzball.config.sharedPVC.accessMode`	string	Yes	-	Volume access mode (must be `ReadWriteMany`)
`fuzzball.config.sharedPVC.class`	string	Yes	-	Storage class name (must be `shared-vast`)
`fuzzball.config.sharedPVC.size`	string	Yes	-	Total shared cache size

Example:


spec:
  fuzzball:
    config:
      sharedPVC:
        accessMode: ReadWriteMany
        class: shared-vast
        size: 10Gi

Shared PVC Sizing

Environment Type	Recommended Size	Rationale
Testing/Development	10Gi	Few container images
Small Production	25Gi	Limited image variety
Large Production	50Gi+	Many different images

Slurm Provisioner

See Slurm Integration Documentation for detailed configuration.

Parameter	Type	Required	Default	Description
`slurm.enabled`	boolean	Yes	-	Enable Slurm provisioner
`slurm.sshHost`	string	Yes	-	SSH host for remote Slurm instance
`slurm.sshPort`	integer	No	`22`	SSH host port for remote Slurm instance
`slurm.username`	string	Yes	-	SSH login username for remote Slurm instance
`slurm.password`	string	No	-	SSH login password for remote Slurm instance
`slurm.sshHostPublicKey`	string	No	-	SSH host public key for remote Slurm instance
`slurm.sshPrivateKeyPem`	string	No	-	SSH private key PEM for remote Slurm instance
`slurm.sshPrivateKeyPassPhrase`	string	No	-	Passphrase for encrypted SSH private key
`slurm.binaryPath`	string	No	-	Custom path to Slurm binaries (if not in $PATH)
`slurm.connectionTimeout`	integer	No	`30`	SSH connection timeout in seconds
`slurm.sudoPath`	string	No	-	Path to sudo binary on compute nodes
`slurm.options`	map	No	`{}`	Additional Slurm sbatch options
`slurm.skipHostKeyVerification`	boolean	No	`false`	Skip SSH host key verification (not recommended)

Basic example:


spec:
  fuzzball:
    orchestrator:
      provisioner:
        enabled: true
        slurm:
          enabled: true
          sshHost: slurm-head.example.com
          sshPort: 22
          username: fuzzball-service
          sshPrivateKeyPem: |
            <full private key in PEM format>
          sshHostPublicKey: "slurm-head.example.com ecdsa-sha2-nistp256 AAAAE2..."

PBS Provisioner

See PBS Integration Documentation for detailed configuration.

Parameter	Type	Required	Default	Description
`pbs.enabled`	boolean	Yes	-	Enable PBS provisioner
`pbs.sshHost`	string	Yes	-	SSH host for remote PBS instance
`pbs.sshPort`	integer	No	`22`	SSH host port for remote PBS instance
`pbs.username`	string	Yes	-	SSH login username for remote PBS instance
`pbs.password`	string	No	-	SSH login password for remote PBS instance
`pbs.sshHostPublicKey`	string	No	-	SSH host public key for remote PBS instance
`pbs.sshPrivateKeyPem`	string	No	-	SSH private key PEM for remote PBS instance
`pbs.sshPrivateKeyPassPhrase`	string	No	-	Passphrase for encrypted SSH private key
`pbs.binaryPath`	string	No	-	Custom path to PBS binaries (if not in $PATH)
`pbs.validateSubstrate`	boolean	No	`false`	Validate substrate before use
`pbs.defaultQueue`	string	No	-	Default PBS queue name
`pbs.pbsServer`	string	No	-	PBS server hostname
`pbs.connectionTimeout`	integer	No	`30`	SSH connection timeout in seconds
`pbs.options`	map	No	`{}`	Additional PBS qsub options
`pbs.sudoPath`	string	No	-	Path to sudo binary on compute nodes
`pbs.skipHostKeyVerification`	boolean	No	`false`	Skip SSH host key verification (not recommended)

Basic example:


spec:
  fuzzball:
    orchestrator:
      provisioner:
        enabled: true
        pbs:
          enabled: true
          sshHost: pbs-head.example.com
          sshPort: 22
          username: fuzzball-service
          password: secure-password

LOTA Object Storage

Preview Status

Support for CoreWeave LOTA object storage is in preview status and is currently subject to more rapid change to address customer requirements than other features of Fuzzball. If you are interested in using LOTA with Fuzzball on CoreWeave, we recommend contacting CIQ as part of your deployment planning process.

Overview

CoreWeave's LOTA provides S3-compatible object storage for workflow data ingress and egress. Configure LOTA credentials to enable workflows to read from and write to LOTA buckets.

Configuration Parameters

Parameter	Type	Required	Description
`type`	string	Yes	Must be `s3` for LOTA
`secret.access-key-id`	string	Yes	LOTA access key ID
`secret.secret-access-key`	string	Yes	LOTA secret access key
`secret.endpoint`	string	Yes	LOTA endpoint URL (`https://cwlota.com`)
`secret.region`	string	Yes	LOTA bucket region (e.g., `ord1`, `lga1`)

Create LOTA Credentials

Step 1: Create Credentials File

Create a YAML file with your LOTA credentials:


# lota-credentials.yaml
type: s3
secret:
  access-key-id: AKIAIOSFODNN7EXAMPLE
  secret-access-key: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
  endpoint: https://cwlota.com
  region: us-east-02a

Replace placeholder values with your actual LOTA credentials from CoreWeave.

Step 2: Create Secret in Fuzzball


$ fuzzball secret create -s user --from-file \
  lota-credentials lota-credentials.yaml

The -s user flag makes the secret available to all users in your organization.

Use LOTA in Workflows

Reference the LOTA credentials in workflow volume definitions:


volumes:
  shared-storage:
    reference: volume://user/fuzzball-shared-vast/shared-storage
    ingress:
      - source:
          uri: "s3://my-bucket/input/data.txt"
          secret: secret://user/lota-credentials
        destination:
          uri: "file://data.txt"
    egress:
      - source:
          uri: "file://results.tar.gz"
        destination:
          uri: "s3://my-bucket/output/results.tar.gz"
          secret: secret://user/lota-credentials

Replace my-bucket with your LOTA bucket name.

Common Configuration Patterns

Minimal Local Deployment

For testing and development on a local Kubernetes cluster. This deploys the Fuzzball control plane. To run workflows, you'll need to add shared storage (fuzzball.substrate.nfs or fuzzball.config.sharedPVC) and configure compute nodes.


apiVersion: deployment.ciq.com/v1alpha1
kind: FuzzballOrchestrate
metadata:
  name: fuzzball-orchestrate
spec:
  image:
    username: depot-user
    password: depot-token
    exclusive: false

  ingress:
    create:
      domain: localhost.nip.io
      proxy:
        type: NodePort

  database:
    create:
      storage:
        class: local-path

  keycloak:
    create:
      ownerEmail: admin@localhost
      createDatabase: true

  tls:
    certManager:
      create: {}
    trustManager:
      create: {}

  fuzzball:
    jetstream:
      replicas: 1
      externalService:
        type: NodePort
      storage:
          class: local-path
        substrateBridge:
        log:
          storage:
            class: "local-path"
        dns:
          externalService:
            type: NodePort

Production On-Premises Deployment

For production deployments with high availability:


apiVersion: deployment.ciq.com/v1alpha1
kind: FuzzballOrchestrate
metadata:
  name: fuzzball-orchestrate
spec:
  image:
    username: depot-user
    password: depot-token
    exclusive: true

  ingress:
    create:
      domain: fuzzball.company.com
      proxy:
        type: LoadBalancer
        annotations:
          metallb.io/loadBalancerIPs: 10.0.100.50

  database:
    create:
      storage:
        class: longhorn
        size: 500Gi

  keycloak:
    create:
      ownerEmail: admin@company.com
      replicas: 2
      createDatabase: true

  tls:
    certManager:
      create: {}
    trustManager:
      create: {}
    ingressIssuer:
      create:
        letsEncrypt:
          email: admin@company.com
          issuer: letsencrypt-prod
          solvers:
            - dns01:
                route53:
                  region: us-east-1
                  hostedZoneID: Z1234567890ABC

  fuzzball:
    cluster:
      name: production-cluster

    substrate:
      nfs:
        server: nfs.company.com
        path: /fuzzball/shared
        destination: /fuzzball/shared

    orchestrator:
      replicas: 3
      autoscaling:
        enabled: true
        minReplicas: 2
        maxReplicas: 10

    agent:
      replicas: 5
      autoscaling:
        enabled: true
        minReplicas: 3
        maxReplicas: 20

    jetstream:
      replicas: 3
      storage:
        class: longhorn
        size: 50Gi

Cloud Deployment with External Services

Using external database and authentication:


apiVersion: deployment.ciq.com/v1alpha1
kind: FuzzballOrchestrate
metadata:
  name: fuzzball-orchestrate
spec:
  image:
    username: depot-user
    password: depot-token

  ingress:
    external:
      domain: fuzzball.cloud.com
      className: nginx
      annotations:
        cert-manager.io/cluster-issuer: letsencrypt-prod

  database:
    external:
      host: postgres.abc123.us-east-1.rds.amazonaws.com
      rdsSecretId: arn:aws:secretsmanager:us-east-1:123456789012:secret:fuzzball-db
      credentials:
        user: ""      # Leave empty when using rdsSecretId
        password: ""  # Leave empty when using rdsSecretId
      sslMode: verify-full
      certificate:
        caCertURL: https://truststore.pki.rds.amazonaws.com/global/global-bundle.pem

  keycloak:
    external:
      url: https://keycloak.cloud.com
      realmId: 550e8400-e29b-41d4-a716-446655440000
      realmName: FuzzballProd
      username: admin
      password: keycloak-password
      ownerEmail: admin@cloud.com

  fuzzball:
    config:
      sharedPVC:
        accessMode: ReadWriteMany
        class: efs-sc
        size: 100Gi

    orchestrator:
      provisioner:
        enabled: true
        aws:
          enabled: true
          region: us-east-1
          subnetIDs:
            - subnet-0123456789abcdef0
          securityGroupIDs:
            - sg-0123456789abcdef0
          instanceProfileARN: arn:aws:iam::123456789012:instance-profile/FuzzballSubstrate

      serviceAccount:
        annotations:
          eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/fuzzball-orchestrator

Managing FuzzballOrchestrate Resources

Viewing Configuration

Check deployment status:


$ kubectl get fuzzballorchestrate
NAME                   STATUS    AGE
fuzzball-orchestrate   Running   10m

View the current configuration:


$ kubectl get fuzzballorchestrate fuzzball-orchestrate -o yaml

View status and events:


$ kubectl describe fuzzballorchestrate fuzzball-orchestrate

Updating Configuration

Edit the configuration in-place (changes are applied immediately):


$ kubectl edit fuzzballorchestrate fuzzball-orchestrate

Apply configuration from a file:


$ kubectl apply -f fuzzball.yaml

Monitoring Deployment

The operator reports deploy and destroy progress directly on the custom resource. Use kubectl describe to see real-time status during any operation:


$ kubectl describe fuzzballorchestrate fuzzball-orchestrate

Note

The operator uses Pulumi internally to manage the underlying Kubernetes and cloud resources. As a result, events and .status.failedResources entries identify resources by Pulumi URN — a stable identifier of the form urn:pulumi:<stack>::<provider>::<type>::<name>. You can treat a URN as an opaque label unique to the specific resource Pulumi is tracking; you only need to recognize that the trailing <name> segment is human-readable (e.g., fuzzball-coldboot, fuzzball-load-default-catalog).

Status Conditions

Two conditions are maintained on the resource:

Condition	Status	Reason	Meaning
`Progressing`	`True`	`Deploying`	Deploy in progress
`Ready`	`False`	`Deploying`	Deploy in progress
`Progressing`	`False`	`DeployComplete`	Deploy finished successfully
`Ready`	`True`	`Succeeded`	Deploy finished successfully
`Progressing`	`False`	`DeployFailed`	Deploy failed (see `Message` for error)
`Ready`	`False`	`DeployFailed`	Deploy failed
`Progressing`	`True`	`Destroying`	Destroy in progress
`Ready`	`False`	`Terminating`	Destroy in progress
`Progressing`	`False`	`DestroyComplete`	Destroy finished — resource pending garbage collection
`Progressing`	`False`	`DestroyFailed`	Destroy failed (see `Message` for error)
`Ready`	`False`	`DestroyFailed`	Destroy failed

Each condition also carries a Message field summarizing the state in human-readable form: "Deploy in progress" or "Destroy in progress" while the operation is running, "Deploy completed" or "Destroy completed" on success, or the underlying error string when it failed.

Note

After a successful destroy, Ready is not updated — the operator removes the finalizer and Kubernetes garbage-collects the resource. Only Progressing transitions to False / DestroyComplete before the resource disappears.

Progress Field

While a deploy or destroy is running, .status.progress is populated with real-time counters and the currently active resources:


Status:
  Progress:
    Phase:            Deploying
    Completed:        153
    In Flight:        2
    Failed:           0
    Elapsed Seconds:  310
    Start Time:       2026-04-23T01:00:22Z
    Current Resources:
      Kind:   Command    Op: create    Elapsed Seconds: 0
      Urn:    urn:pulumi:…::fuzzball-coldboot
      Kind:   Job        Op: create    Elapsed Seconds: 0
      Urn:    urn:pulumi:…::fuzzball-load-default-catalog

Phase is either Deploying or Terminating. completed, inFlight, and failed count individual infrastructure resource operations. Once the operation finishes, .status.progress is cleared.

Kubernetes Events

The operator emits events on the resource throughout the operation lifecycle. View them at the bottom of kubectl describe output or with:


$ kubectl get events --field-selector involvedObject.name=fuzzball-orchestrate --sort-by='.lastTimestamp'

Deploy events:

Event	Type	Description
`DeployStarted`	Normal	Deploy has begun
`DeployCompleted`	Normal	Deploy succeeded — includes elapsed time and resource count
`ResourceOpFailed`	Warning	A single tracked resource's create/update/replace/delete failed
`DeployFailed`	Warning	Deploy failed — see `.status.failedResources` and operator logs

Destroy events:

Event	Type	Description
`DestroyStarted`	Normal	Destroy has begun
`DestroyingDependency`	Normal	A named dependency stack (e.g., `Fuzzball`, `Database`) is being torn down
`DestroyedDependency`	Normal	A dependency stack completed teardown
`DestroyCompleted`	Normal	All resources destroyed — includes elapsed time and count
`ResourceOpFailed`	Warning	A single tracked resource's create/update/replace/delete failed
`DestroyFailed`	Warning	Destroy failed

ResourceOpFailed is emitted per affected resource (not once per operation), so it can fire multiple times during a single Deploy or Destroy. Its message has the form <Kind> <op> failed: <URN> (for example, Job create failed: urn:pulumi:fuzzball::…::fuzzball-load-default-catalog). It is the most precise signal for pinpointing which resource broke before the rolled-up DeployFailed / DestroyFailed event fires.

The same event patterns apply to both FuzzballOrchestrate and FuzzballFederate resources.

Example — successful Orchestrate deploy:


Events:
  Normal  DeployCompleted  0s   Deploy completed in 2m11s; 93 resources applied

Example — 164-resource teardown:


Events:
  Normal  DestroyStarted        1m41s  Starting Destroy for fuzzball-local-dev
  Normal  DestroyingDependency  1m41s  Destroying Identity
  Normal  DestroyingDependency  1m37s  Destroying Fuzzball
  Normal  DestroyingDependency  53s    Destroying Namespace
  Normal  DestroyingDependency  48s    Destroying Database
  Normal  DestroyingDependency  10s    Destroying TLS
  Normal  DestroyedDependency   0s     Destroyed Database
  Normal  DestroyedDependency   0s     Destroyed Fuzzball
  Normal  DestroyedDependency   0s     Destroyed Identity
  Normal  DestroyedDependency   0s     Destroyed Namespace
  Normal  DestroyedDependency   0s     Destroyed TLS
  Normal  DestroyCompleted      0s     Destroy completed in 1m44s; 164 resources destroyed

Watch operator logs during deployment:


$ kubectl logs -l app.kubernetes.io/name=fuzzball-operator -n fuzzball-system -f

Important Notes

Resource Scope: FuzzballOrchestrate is a cluster-scoped resource (not namespace-scoped)
Short Names: Can use fb or fuzz as shortcuts in kubectl commands
Mutual Exclusivity:
- Database: Use either create or external, not both
- Ingress: Use either create or external, not both
- Keycloak: Use either create or external, not both
Autoscaling: When enabled, replicas is ignored in favor of minReplicas/maxReplicas
Storage Classes: Ensure specified storage classes exist in your cluster
UUIDs: realmId must be a valid UUID v4 format
NFS Requirements: NFS server must be accessible from all nodes

Troubleshooting

Deployment Fails

When a deploy fails, the Ready and Progressing conditions are both set to False with reason DeployFailed. The Message field on the condition contains the Pulumi diagnostic error:


Status:
  Conditions:
    Type: Progressing    Status: False    Reason: DeployFailed
    Message: an unhandled error occurred: waiting for RPCs: marshaling properties:
             awaiting input property "clientId": while getting fuzzball realm: ...
    Type: Ready          Status: False    Reason: DeployFailed
    Message: (same as above)
Events:
  Warning  DeployFailed  0s   Deploy failed after 5m15s. See .status.failedResources and operator logs.

The Message field is capped at 256 characters; a trailing ... in the output indicates the text was truncated. Consult the operator logs (shown further down) for the full error.

Check .status.failedResources for a structured list of every resource that failed, including the Pulumi URN, the operation type (create, update, replace, or delete), the error message, and how many times it was retried:


$ kubectl get fuzzballorchestrate fuzzball-orchestrate -o jsonpath='{.status.failedResources}' | jq .

Each entry contains:

urn — Pulumi URN identifying the specific resource that failed
op — operation that was attempted
error — the diagnostic error message
attemptCount — number of retries before giving up

.status.failedResources is retained on the resource until the next stack operation begins — the operator overwrites the slice with a fresh (empty) one as soon as the next deploy or destroy starts, so capture this output before retrying if you want a post-mortem record.

Check operator logs for additional context:


$ kubectl logs -l app.kubernetes.io/name=fuzzball-operator -n fuzzball-system --tail=100

Check FuzzballOrchestrate status and events:


$ kubectl describe fuzzballorchestrate fuzzball-orchestrate

Pods Not Starting

Check pod events:


$ kubectl get pods -n fuzzball

$ kubectl describe pod <pod-name> -n fuzzball

Check PVC status:


$ kubectl get pvc -A

Ingress Not Working

Check ingress configuration:


$ kubectl get ingress -A

$ kubectl describe ingress <ingress-name> -n fuzzball

Verify domain DNS resolution and load balancer IP assignment.

FuzzballOrchestrate CRD Reference

Basic Example

Configuration Sections

Image Configuration

Ingress Configuration

Option A: Create New Ingress

CoreWeave LoadBalancer Annotations

Option B: Use External Ingress

Database Configuration

Option A: Create Dedicated Database

Option B: Use External Database

Keycloak Authentication

Option A: Deploy Keycloak

Initial Setup

Generating a Realm ID

Setting the Owner Account

LDAP Integration

Option B: Use External Keycloak

TLS Configuration

cert-manager Deployment

Certificate Issuers

Fuzzball Services Configuration

Global Settings

Substrate Configuration

Kubernetes Scheduling Configuration

Service-Specific Configuration

Available Configuration Sections

Jetstream Configuration

Storage Service Configuration

Audit Service Configuration

Substrate Bridge Configuration

Orchestrator Provisioner Configuration

AWS Provisioner

CoreWeave Provisioner

Workflow Data Storage Parameters

Workflow Data Storage Sizing

Shared PVC Configuration (fuzzball.config.sharedPVC)

Shared PVC Sizing

Slurm Provisioner

PBS Provisioner

LOTA Object Storage

Overview

Configuration Parameters

Create LOTA Credentials

Step 1: Create Credentials File

Step 2: Create Secret in Fuzzball

Use LOTA in Workflows

Common Configuration Patterns

Minimal Local Deployment

Production On-Premises Deployment

Cloud Deployment with External Services

Managing FuzzballOrchestrate Resources

Viewing Configuration

Updating Configuration

Monitoring Deployment

Status Conditions

Progress Field

Kubernetes Events

Important Notes

Troubleshooting

Deployment Fails

Pods Not Starting

Ingress Not Working

See Also

Shared PVC Configuration (`fuzzball.config.sharedPVC`)