Fuzzball v4.4.0 release notes

Fuzzball v4.4.0 is a feature release. The in-product node health model is replaced by a bundled Prometheus monitoring stack. This new stack can place holds on nodes to cordon, evict or deprovision based on configured policy. Workflow stages can now be retried when they lose their node or exit with a designated code, and later workflow jobs can run based on a dependency fail state. Workflows can build their own container images, and service endpoints can outlive the workflow that serves them.

Substrate now converts images to EROFS on compute nodes at runtime, adding support for layered images and UID/GID.

v4.4.0 rolls up v4.3.1, so these notes are complete for anyone upgrading directly from v4.3.0; items marked (also in v4.3.1) already shipped there.

New Features

Node Holds

Cordoning a node used to replace its status with CORDONED, so the status could no longer show whether the node itself was healthy. A cordon, evict or deprovision is now a hold kept alongside the status, recording who placed it and why, and a node can carry any number of them.

• Three effects. cordon places no new work on the node. evict also takes off what is running: a step declaring policy.retry.on: node is requeued elsewhere, and anything else fails with the hold's reason. deprovision places no new work and tears the node down once it is empty.
• Multiple sources for holds. A hold is attributed to admin, monitoring or provisioner from the caller's identity. fuzzball node uncordon releases admin cordon holds only, so it never clears evidence monitoring gathered or a hold the provisioner placed.
• CLI. fuzzball node hold list|add|release provides a general-purpose interface to holds, in addition to fuzzball node cordon|evict|deprovision. cordon and uncordon now accept several nodes simultaneously, cordon accepts a --reason parameter, and deprovision gains --evict, --force and --all. fuzzball node remove now clears an offline node's record and releases its holds, so retired hardware can be removed from the node list.
• Seeing holds. fuzzball node list gains a HOLDS column and fuzzball node show lists each hold with its effect, source, reason and when it was placed. The Web UI shows holds as sortable, filterable node list columns, adds cordon, uncordon and evict actions and the node event log to node details, and replaces the dashboard health rollup with per-cluster counts by hold state. Node events now record provisioning and each rank's allocations.
• A site-owned node that goes offline keeps its admin holds and unreleased monitoring holds, so a node held for repair returns still held.

Node Health Monitoring

The bundled monitoring stack now watches compute nodes and acts on what it finds, through Prometheus rules rather than logic internal to Fuzzball.

• Alerts that place holds. Seven node-health rules ship with the stack. FuzzballNodeAttributedJobFailures and FuzzballNodeMemoryErrors cordon a node and FuzzballNodeGPUUncorrectableECC evicts one, while still notifying the site's own receiver. Holds arrive through an Alertmanager webhook receiver that authenticates with mTLS on its own port, and the operator issues client certificates to bundled and external Alertmanagers. The fb_action and fb_release alert labels are documented for sites writing their own rules.
• A ceiling on automatic holds. Monitoring holds may cover at most nodeHealth.maxMonitoringHeldPercent (default 20%) of a provisioner definition's nodes; past that, holds are refused and FuzzballNodeHoldCeilingReached fires.
• Node metrics. Compute nodes serve fuzzball_node_* metrics, including disk I/O errors, read-only disks, memory errors and failed device plugins, and run a bundled node_exporter. A failed device plugin is also reported as a device_plugin_failed scheduling reason and in fuzzball node show.
• Reaching them. A Prometheus service discovery endpoint at /v4/nodes/-/sd_config tracks dynamically provisioned nodes, and a /v4/metrics/node proxy relays scrapes over each node's existing connection for clusters that cannot reach node IPs; it requires the new scrape_node_metrics permission. The nodeMetrics central config block and monitoring.nodeMetricsMode choose between them. fuzzball context token prints the active access token for a scrape config's bearer header.
• Dashboards. Fleet Health, Node Health and Node Resources Grafana dashboards ship with the stack, the last with LSF lsload-style load indices. The rules, routes and dashboards are also published for import by sites running their own monitoring.
• Prometheus, Alertmanager and Grafana values can be overridden per component with valuesOverride. fuzzball_scheduler_node_attributed_job_failures is deprecated in favor of fuzzball_node_attributed_job_failures; both are published.

Retrying Workflow Steps

policy.retry was accepted and ignored in previous releases. It now decides whether a step may run again, and from the beginning -- Fuzzball does not checkpoint, so declare it only for work that is safe to repeat.

    policy:
      retry:
        on: [node, 75]
        attempts: 3

• on is required. Without it nothing is retried. node covers an evict hold and a node going offline alike. Exit codes 1-255 retry a job that exits with a listed code; 130 and 143 are not accepted.
• attempts is one budget shared by every cause. Node retries are also capped by the cluster-wide nodeHealth.maxEvictionRestarts (default 3), and the lower of the two applies.
• policy.restart still works: on-node-fault is treated as retry: {on: node}. Declaring nothing leaves a step running through a routine drain and fails it if the node is lost.
• Workflow events gain retried and retries_exhausted, and failure messages name the field to set. The Web UI workflow editor can set a job's retry policy.

On-Failure Dependencies

A job or service can now run only when another fails, for cleanup or notification steps. Declare it with status: FAILED in depends-on. When the dependency succeeds instead, the dependent is skipped and shown as Canceled. Single-node jobs and services are supported, and the dependent does not run when the dependency is canceled or preempted.

Persistent Service Endpoints

An endpoint can now outlive the workflow serving it, so a URL handed out once keeps working across redeployments. fuzzball endpoint create NAME creates one, and a workflow attaches by declaring an endpoint of that name with persistent: true. A new workflow claiming an endpoint in use takes it over, canceling the previous one either at submission or once the new service is ready (--strategy start|ready). Endpoint commands move to fuzzball endpoint, and fuzzball workflow endpoints is deprecated. Endpoint names use lowercase letters, digits, - and _. Workflows can read their persistent endpoints' URLs from new FB_SERVICE_<SERVICE>_ENDPOINT_* environment variables, and the Web UI Connect button becomes a menu when a workflow exposes more than one endpoint.

Building Images in a Workflow

A workflow can now build the container images its jobs need, instead of relying on an image built and pushed to a registry beforehand. Images are declared in a top-level images: section and built on a compute node as stages of the workflow.

images:
  tools.sif:
    from:
      uri: docker://rockylinux/rockylinux:9
    post: |
      dnf install -y jq ripgrep
jobs:
  analyze:
    image:
      uri: fuzzball://user/tools.sif

• Definitions. A build starts from a registry or object cache image, writes any files into it, then runs the post script as root inside a confined container. The result is stored in the object cache with scope: user or group, with an optional ttl.
• Ordering is inferred. A job waits for the images it uses, an image built from another in the same workflow is built after it, and circular definitions are rejected at submission. An unchanged definition is served from the cache rather than rebuilt.
• Watching a build. Each step of a build is reported as a workflow event, and the post script's output is the log of the image's stage in the CLI and the Web UI.

Distributed GPU Jobs

Fuzzball v4.4 improves support for multi-GPU jobs, particularly in support of AI training workloads.

• Multi-node NVLink. With fuzzball-substrate-nvidia-gpu-plugin 0.0.20 or later, multinode jobs are placed inside one NVLink domain when one has enough free nodes, and can request an IMEX channel with NVIDIA_IMEX_CHANNELS.
• Pinned memory. Job containers set RLIMIT_MEMLOCK to the job's memory allocation, so NCCL, RDMA and DPDK can lock the memory they need. A job that requests no memory keeps the default.
• AWS. The provisioner selects an NVIDIA Fabric Manager AMI for EFA-equipped NVLink instance types (p4d, p4de, p5, p5e, p5en).
• A new guide covers distributed training with NCCL and torchrun.

Enhancements

Scheduling Diagnostics

• fuzzball workflow why takes a workflow id on its own and explains every stage still waiting, in fuzzball workflow describe order; --all includes finished stages.
• Placement attempts are reported. INSUFFICIENT_RESOURCES and BACKFILL_NO_FIT now say how many nodes were looked at, why each group was passed over, and what fell short on the roomiest node. A new NODE_ANNOTATIONS_MISMATCH reason covers nodes with capacity but without the requested annotations.
• Submission explains rejections. A job no node of a definition could hold even when idle is rejected, naming the request and the largest node, rather than queued forever, and the error lists why each definition was rejected.
• scheduling_blocked events report, once per cause, a stage that too few nodes could hold, and fuzzball workflow events shows their detail. stage_failed and container_exited events carry the container exit code.

Images

• EROFS. Container images are converted to EROFS on compute nodes.
• Object cache uploads are visible. Uploading a pulled image to the object cache is reported as stage events with progress, retried, and abandoned when it stalls, without failing the stage.

Authentication and Web Security

• Password policy and lockout. New passwords must be at least 12 characters with mixed case, a digit and a special character, and must not contain fuzzball, the username or email. These rules apply to Fuzzball Local and to Fuzzball-managed Keycloak realms that have no password policy of their own, and AddOrganizationMember requires a password. Local password login, and sign-in to newly created organization Keycloak realms, locks an account for 15 minutes after 5 failed attempts.
• Security headers. HTTPS responses from the Web UI and REST API carry Strict-Transport-Security, and the Web UI sends Content-Security-Policy and X-Content-Type-Options.
• Public endpoints identify organization members who present a Fuzzball token, let them wake an idle pool, and no longer forward that token to the service.

Web UI

• Stage details show the node a job or service ran on, workflows show the set of nodes used, and pending jobs show an estimated backfill start time.
• The workflows list gains a sortable Started column, and job logs gain inline text search.
• Node-local persistent volumes show their holding node, and storage provisioner group fields offer the organization's groups.
• Object preview distinguishes too-large, unsupported and error states, with a "Preview anyway" action for large files.
• The workflows list defaults to Submitted, newest first; storage lists page through results; catalog search matches descriptions; and read-only templates offer Copy and Edit (also in v4.3.1).

Deployment and Infrastructure

• New resource fields. spec.image.pullPolicy lets pods restart from cached images while the registry is unreachable. spec.clusterDomain supports clusters whose DNS domain is not cluster.local, and the operator detects it otherwise. spec.fuzzball.jetstream.externalService.type accepts LoadBalancer for Substrate hosts that cannot reach node IPs. spec.keycloak.create.resources sizes Keycloak.
• Let's Encrypt staging. fuzzball cluster <cloud> deploy and update accept --lets-encrypt-issuer; update keeps the deployed issuer.
• Docker Compose accepts a PULL_POLICY override for offline restarts.
• The macOS CLI is signed and notarized, so it runs without removing the quarantine attribute.
• Workflow tokens can read secret data, so controller-submitted child workflows resolve secret:// references; creating or modifying secrets is refused.
• Provisioner definitions using the deprecated unsharedPath are rewritten to local on upgrade, with no change in behavior.
• fuzzball-admin audit list pages, orders and filters events (also in v4.3.1).

Fixes

Security

• Fixed an issue where workflow catalog applications and sources from one group or organization could be listed, rendered or read by users in another. A catalog source's access token is now returned only by the admin API.
• Fixed an issue where volume get, update, status and delete by ID could reach volumes in other organizations.
• Fixed Filestore and hostpath isolation between organizations.
• Fixed the operator's status messages publishing credentials in many error shapes, including nested keys, one-line YAML, request dumps and unencoded Basic passwords.
• Fixed workflow validation errors echoing secret-bearing field values.
• Fixed an issue where the Web UI access-token cookie was readable from JavaScript.
• Fixed provisioned nodes exposing credentials in cloud-init and console logs, fuzzball debug dump bundles containing ConfigMap credentials, and the instance metadata guard staying off after a failed startup check (also in v4.3.1).

Workflows

• Fixed task-array workflows hanging, and ignoring cancel, when one task exited non-zero while siblings were still running.
• Fixed jobs staying STARTED and holding capacity after their node lost their lease, such as after a substrate restart. They now fail as stopped responding.
• Fixed single-architecture images failing at pull on mixed-architecture clusters. Jobs and services are now pinned to the architecture an image tag names.
• Fixed fuzzball workflow start, validate, score and upgrade processing {{ ... }} in workflow files as a template; files are sent unchanged.
• Fixed invalid definitions and mounts of undeclared volumes failing with an internal error instead of InvalidArgument.
• Fixed upgrades of v1 workflows silently resolving nameless persistent references such as volume://user/persistent; start and upgrade now warn.
• Fixed fuzzball workflow validate rejecting v1 Fuzzfiles that start accepts, and workflow upgrade leaving simple v1 files unconverted. Validate now upgrades v1 the way a submission does.
• Fixed workflows that set policy on a service being rejected. The field is accepted again and ignored with a warning, since services do not support execution policies.
• Fixed the endpoint proxy answering 404 "Endpoint not found" whenever the endpoint lookup failed, including during an Orchestrate upgrade before the workflow database was migrated. Other lookup failures now return 503, and the database is migrated before the services start.
• Fixed ephemeral volumes binding to a provisioner whose segment had no ready nodes; auto-selected volumes now pick the first eligible provisioner by name.
• Fixed volume egress racing services, Open MPI multinode jobs hanging on the node-local bridge, and services that exited nonzero ending FINISHED (also in v4.3.1).

Scheduler and Provisioner

• Fixed dynamic nodes running forever after their workflow ended, and AWS, GCP and OCI nodes losing their provisioner annotations on substrate restart.
• Fixed a torn-down node being torn down again on every idle sweep, which could terminate a new instance that reused its address.
• Fixed provisioned instances that never joined the cluster being left running. They are now deleted 15 minutes after the registration deadline.
• Fixed workflows failing with a provision_instances_instance_id_key duplicate key error when a new instance reused an earlier one's name.
• Fixed autoscaled services being rejected, or scale-ups reported as capacity-exhausted, while a pool's capacity was unknown.
• Fixed services declaring the same host port being placed on one node.
• Fixed a newly registered node not waking the scheduler, leaving work waiting up to 60 seconds.
• Fixed node listing failing on nodes running a peer rank of a multinode job.
• Fixed a job that stopped responding counting against its node after it was stopped, finished or being moved.
• Fixed multi-cloud compute grants being ignored, and new organizations not getting a default storage provisioner.
• Fixed appending provisioner definitions through the admin API failing because Orchestrate had not opened the central configuration store.
• Fixed bandwidth metering being disabled on every compute node because its eBPF programs failed to load.
• Fixed the orchestrate extension crashing on hosts whose /etc/os-release contains a blank line, such as Oracle Linux 9.
• Fixed maxNodes being exceeded by bursts, provisioned nodes ignoring the configured shared filesystem mount point and writing unvalidated fstab entries, AWS nodes not trusting the cluster CA, and heartbeating nodes being deregistered (also in v4.3.1).

Deployment

• Fixed FuzzballOrchestrate reporting Ready while a Deployment was still rolling out; a stalled rollout now sets Degraded.
• Fixed Orchestrate and Federate reporting Updating indefinitely after an early reconcile failure, and Degraded carrying only a generic message.
• Fixed operator upgrades stalling when the audit or substrate-bridge pod moved to a node that could not attach its ReadWriteOnce volume.
• Fixed the operator downloading Pulumi providers after every restart, which blocked reconciles without internet access, and being OOMKilled; its default memory limit is now 4Gi.
• Fixed the operator hanging when an external CA ClusterIssuer's secret was outside the cert-manager namespace; set tls.certManager.clusterResourceNamespace.
• Fixed custom-named ingress-nginx classes missing gRPC and TLS annotations; set ingress.external.controller.
• Fixed a 502 when signing in through an external nginx ingress.
• Fixed CLI and UI login through a bare NodePort ingress with the new spec.ingress.publicPort; setting it changes the token issuer, so users must log in again.
• Fixed Fuzzball pods failing admission on clusters that enforce Pod Security restricted by default.
• Fixed air-gapped installs failing to pull the database reloader image.
• Fixed a federate in-place PostgreSQL upgrade deadlocking when Keycloak was unavailable, and FuzzballFederate ignoring nodeSelector, tolerations and affinity for the fuzzball-admin pod.
• Fixed in-cluster Substrate failing to reach JetStream and DNS over NodePort on a node running no JetStream or substrate-bridge pod.
• Fixed Docker Compose sizing Keycloak's heap from host RAM; it is now capped at 1GiB, in a 1.75GiB container.
• Fixed JetStream connections and buckets not surviving outages, the pre-upgrade database restore reporting failures as success, internal tokens expiring after a host resumed from suspend, and Compose TLS, DNS and sign-in failures (also in v4.3.1).

Web UI, Catalog and CLI

• Fixed the Workflow Editor dropping a workflow's preemptible, retry and restart settings when loaded, edited or started. It also keeps the images: section intact.
• Fixed the Workflow Editor leaving uploaded, pasted and v1-syntax workflows unconverted. It now upgrades them to v4 through the server and flags mounts of undefined volumes.
• Fixed an issue where a workflow opened in the Workflow Editor could not be started when its definition was not version: v4, such as one submitted from the CLI as a v1 file. Start failed with cannot unmarshal !!str ... into int64 because the YAML carried a quoted cache-ttl. The editor now writes cache-ttl as an integer and leaves an unset one out.
• Fixed unchecking Follow in a stage's log pane not stopping auto-scrolling while logs streamed.
• Fixed live log streams failing permanently after idle disconnects, expired sessions leaving the page broken, and docs preview links opening the landing page.
• Fixed rerunning a workflow appending a (#N) suffix to its name.
• Fixed existing organization members getting an invite email on password reset.
• Fixed organization owners not seeing the account-owner workflow summary.
• Fixed the storage provisioner form showing "undefined" and losing fixed-choice settings on edit.
• Fixed monospace rendering in workflow stage logs.
• Fixed catalog sync ignoring the documented key_art field.
• Fixed editing a catalog entry clearing its tags, key art, featured flag and provider, saving a template in the editor removing its enum options, and workflow catalog update overwriting the name and category when -n or -g was omitted.
• Fixed federate clusters labeling official catalog apps as External and rejecting the official catalog when allowedHosts is set.
• Fixed the password change link from fuzzball context update-user --new-password failing with "Invalid parameter: redirect_uri" on deployments such as Docker Compose.
• Fixed the MCP catalog tools reporting the whole catalog unreachable when a listing failed, and group_list misdescribing its default scope.
• Fixed the MCP workflow_validate tool reporting a workflow valid when the server rejected it. It now returns valid: false with the server's reason, warns when the server could not be asked, and accepts cluster_id.
• Fixed fuzzball context validate's volume ingress and inter-job DNS checks failing on cloud deployments: the ingress check asked for a provisioner named default, and the DNS check tried only once before the job it called was ready.
• Fixed the dashboard showing 0 running workflows, dates ignoring the browser locale, missing image edges in the workflow graph, and fuzzball cluster config get -o json printing YAML (also in v4.3.1).