Deployment

After fulfilling the prerequisites from the requirements doc, you are ready to deploy Federate in an existing K8s cluster that already supports Fuzzball.

Ensure that your K8s context is Controlling the Appropriate Fuzzball Cluster

If you are running kubectl on the same node that is hosting the K8s installation and Fuzzball Orchestrate deployment (common in on-prem deployments) then you don't need to worry about setting up a kubeconfig file. If you are using the kubectl command on a local host to control a K8s deployment on a remote server (common in a cloud deployment) you will need to double check that your kubeconfig file is pointing at the right cluster.

In AWS, the command to update your kubeconfig file will look something like the following:

$ aws eks update-kubeconfig --region us-east-1 --name <cluster name and ID>

Check the Cloud Admin Guides for information on retrieving the appropriate command to update your kubeconfig file.

Create a fuzzball-federate.yaml file for the Federate Deployment

Fuzzball Federate uses the Fuzzball operator that was already installed to deploy your Fuzzball Orchestrate cluster with a new Custom Resource Definition (CRD) file. We'll call this file fuzzball-federate.yaml. Many of the values are predetermined, but there are some values you need to supply for your particular installation.

You can create an appropriate fuzzball-federate.yaml file by populating the following environment variables with their correct values and running the cat command to create the file.

Fill in your CIQ depot credentials here:

$ DEPOT_USER="" # populate with your username for CIQ Depot

$ ACCESS_KEY="" # populate with the Depot key obtained from the CIQ sales/support team

Decide on the version of Federate that you want to deploy (for example, <Version />):

$ FED_VERSION=<Version /> # change as necessary

You need to determine the certificate issuer for your externally facing URLs. This is set at Fuzzball deployment. In the AWS marketplace, it will be letsencrypt-prod. On-prem, your deployment may have (insecure) self-signed certificates in which case the appropriate value will be internal-ca-issuer, it might be letsencrypt-prod if you used letsencrypt, or it could be another issuer. You can find this information with kubectl get clusterissuer.

$ CERT_ISSUER=letsencrypt-prod # change as necessary

You will need to supply the domain where the cluster is hosted, the realm ID of the keycloak instance, and the email address that was set as the owner of the keycloak instance. You can obtain this information as described here.

$ DOMAIN=myfuzzballdomain.com # change as necessary

$ KC_REALMID=$(uuidgen --random | tr 'A-Z' 'a-z')

$ KC_EMAIL="" # populate with the keycloak owner email address

Federate's audit service, database, and cluster configuration service (JetStream) all need a storage class that actually exists in your cluster. The name varies by platform: local on-prem clusters typically provide local-path, while cloud clusters expose their own classes. On AWS EKS the EBS-backed class is usually named gp2 or gp3.

List the storage classes available in your cluster and identify the one you want to use (the default class is marked (default)):

$ kubectl get storageclass
NAME            PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
gp2 (default)   kubernetes.io/aws-ebs   Delete          WaitForFirstConsumer   false                  9d

Set the STORAGE_CLASS environment variable to the class you want Federate to use. On AWS EKS this will commonly be gp2 or gp3; on a local on-prem cluster it is often local-path:

$ STORAGE_CLASS="" # choose a class from 'kubectl get storageclass'

Choose the initial password for the Federate Keycloak admin user. Set it to a strong value of your own rather than leaving the placeholder:

$ KC_PASSWORD="" # populate with a strong initial keycloak admin password

Once you have all of these environment variables set in your terminal, you can copy and paste the following code block to create the fuzzball-federate.yaml file in your current working directory with the values you chose.

$ cat >fuzzball-federate.yaml<<EOF
apiVersion: deployment.ciq.com/v1alpha1
kind: FuzzballFederate
metadata:
  labels:
    app.kubernetes.io/name: fuzzball-federate
    app.kubernetes.io/part-of: fuzzball
  name: fuzzball-federate
spec:
  image:
    username: ${DEPOT_USER}
    password: ${ACCESS_KEY}
    exclusive: false
  fuzzball:
    version: ${FED_VERSION}
    cluster:
      kind: FEDERATE
    audit:
      storage:
        class: ${STORAGE_CLASS}
      localStorage: true
    jetstream:
      storage:
        class: ${STORAGE_CLASS}
  database:
    create:
      enableDebugPod: true
      storage:
        class: ${STORAGE_CLASS}
  tls:
    # Externally facing issuer, likely letsencrypt but could be something else.
    # Issuers in the cluster can be found with 'kubectl get clusterissuer'.
    ingressIssuer:
      external:
        issuerName: ${CERT_ISSUER}
    internalIssuer:
      external:
        issuerName: internal-ca-issuer
  # Assumes existing Kong deployment from FuzzballOrchestrate
  ingress:
    external:
      domain: federate.${DOMAIN}
      className: kong
  keycloak:
    create:
      createDatabase: true
      realmId: ${KC_REALMID}
      username: keycloak
      password: ${KC_PASSWORD}  # or use passwordRef to reference a K8s Secret
      ownerEmail: ${KC_EMAIL}
EOF

Using an External Keycloak Instance

The example above deploys a new Keycloak instance alongside Federate (keycloak.create). If your organization already operates a Keycloak server, you can connect Federate to it instead by using keycloak.external in place of keycloak.create.

Replace the keycloak section of fuzzball-federate.yaml with the following:

  keycloak:
    external:
      url: https://keycloak.example.com   # URL of your existing Keycloak instance
      realmId: <uuid-v4-realm-id>         # Existing realm ID (UUID v4; mixed case accepted)
      realmName: ExistingRealm            # Realm display name (used as the Fuzzball org name)
      username: admin                     # Keycloak admin from the master realm (not the target realm)
      ownerEmail: admin@example.com       # Must match an existing user in the target realm
      passwordRef:
        name: keycloak-credentials        # Name of a Kubernetes Secret
        key: admin-password               # Key within the Secret

Trusting Tokens from Another Keycloak

By default each cluster only trusts JWTs issued by its own Keycloak. In a federated topology the Orchestrate and Federate clusters typically run separate Keycloak instances, so a token minted by one cluster's Keycloak is rejected by the other. To let a cluster accept tokens issued by a second Keycloak — for example, so the Federate cluster honors tokens from the Orchestrate cluster's Keycloak — add that Keycloak's issuer URL to fuzzball.config.additionalTrustedIssuers.

Add the field to the fuzzball section of fuzzball-federate.yaml:

  fuzzball:
    config:
      additionalTrustedIssuers:
        - https://auth.${DOMAIN}/   # The other cluster's Keycloak issuer URL

Each entry must be an absolute http(s) URL. You can list more than one issuer. The same field is available on FuzzballOrchestrate, so the trust can be configured in either direction.

Workflow Catalog

Federate clusters support the same workflow catalog as Orchestrate clusters. The official CIQ catalog is enabled by default and syncs on a schedule via a CronJob. You can control this behavior with the fuzzball.workflowCatalog field:

  fuzzball:
    workflowCatalog:
      catalogs:
        official:
          enabled: true                                         # Set to false to disable
          uri: https://github.com/ctrliq/ciq-fuzzball-catalog  # Default catalog URI

The operator always deploys a fuzzball-catalog-sync CronJob, regardless of the enabled setting. This CronJob periodically runs fuzzball workflow catalog source reload --all, which refreshes every catalog Fuzzball currently knows about.

When enabled is true (the default), the operator additionally deploys a one-time fuzzball-load-default-catalog Job during deployment that seeds the official CIQ catalog so the recurring sync has something official to refresh. Setting enabled: false skips this Job, so the official catalog is never registered with Fuzzball — the fuzzball-catalog-sync CronJob still runs, but it will only refresh other catalogs (if any) that have been registered manually. No additional configuration is required to start using the catalog.

Application Replication

In addition to the official CIQ catalog, any non-CIQ applications created on an Orchestrate cluster are automatically replicated to the connected Federate cluster. This happens in two ways:

• On registration: When a Federate cluster is registered with an Orchestrate cluster, all existing non-CIQ applications are bulk-synced to Federate automatically.
• On change: Any subsequent create, update, or delete of a non-CIQ application on Orchestrate is pushed to the connected Federate cluster in real time.

No configuration is required for application replication. Replicated applications are stamped with the source cluster ID so they can be distinguished from applications created natively on the Federate cluster.

Secret Materialization in Federate

When Federate forwards workflow score requests to downstream clusters, it automatically materializes all secret references embedded in the workflow specification. This includes:

• Job-scoped secrets: Image pull secrets (image.secret), image decryption secrets (image.decryption-secret), and environment variable references using secret:// URIs
• Service-scoped secrets: The same secret reference patterns are supported in the services section of a workflow and are handled identically to job-scoped secrets
• Volume secrets: S3 and other storage credentials referenced in ingress/egress configurations

Secret materialization ensures that workflows referencing secrets at any scope (user, group, organization, or cluster) will have those secrets resolved and forwarded to the target cluster. Administrators do not need to manually replicate secrets to downstream clusters; Federate handles this automatically as part of the score request.

Cluster Configuration

Federate deployments include a dedicated cluster configuration service — a 3-replica fuzzball-jetstream StatefulSet in the Federate namespace — that stores the cluster configuration applied with fuzzball cluster config set. This means the same configuration management commands used against Orchestrate clusters also work against a Federate cluster.

On a Federate cluster, only the global settings (for example, scheduler options) in the configuration apply. Node provisioner definitions are not used on Federate clusters — Federate receives its node provisioner information from its registered Orchestrate clusters — so any definitions entries in a configuration applied to Federate are stored but ignored.

The example fuzzball-federate.yaml above sets the service's storage class via the fuzzball.jetstream field of the FuzzballFederate resource. The same field also lets you tune the replica count (replicas, default 3) and the volume size (storage.size, default 10Gi).

Apply the fuzzball-federate.yaml to the K8s Cluster to Deploy Federate

Now you can run the following command to deploy Federate into the K8s cluster that supports your Fuzzball Orchestrate installation.

$ kubectl apply -f fuzzball-federate.yaml

And you can watch as the Fuzzball Operator carries out the deployment of Federate in your K8s cluster like so. The process usually takes 10 or 15 minutes to complete.

$ kubectl logs -l app.kubernetes.io/name=fuzzball-operator -n fuzzball-system -f --tail=-1

[snip...]

Resources:
    ~ 3 updated
    136 unchanged

Duration: 4s

2025-03-13T17:13:00Z	DEBUG	events	Resources have been deployed successfully	{"type": "Normal", "object": {"kind":"FuzzballFederate","name":"fuzzball-federate","uid":"6bdda038-b41c-489d-9ec9-f2fb5311388a","apiVersion":"deployment.ciq.com/v1alpha1","resourceVersion":"412932"}, "reason": "DeploymentSucceeded"}
2025-03-13T17:13:00Z	INFO	Updated Fuzzball status to ReconciliationComplete - Reconciliation completed successfully	{"controller": "fuzzballfederate", "controllerGroup": "deployment.ciq.com", "controllerKind": "FuzzballFederate", "FuzzballFederate": {"name":"fuzzball-federate"}, "namespace": "", "name": "fuzzball-federate", "reconcileID": "a63ae801-a7fb-40d1-a733-bf3582bfd463"}

At this point, you can access your new Federate cluster and configure it to point to your existing Fuzzball Orchestrate cluster(s).