Building Images in a Workflow

Sometimes no public container has quite what your job needs. Rather than building an image on your workstation and pushing it to a registry first, you can describe the image in the Fuzzfile itself and let Fuzzball build it as part of the workflow.

Image building is off unless your site turns it on. A build runs the definition's post script as root on a compute node, so an administrator has to set features.enableWorkflowImageBuilds on the deployment first. Until they do, starting a workflow that contains an images: section fails with building images in a workflow is not enabled on this deployment.

Images are described in a top-level images: section, which is a map of image names to image definitions:

version: v4
images:
  lolcow.sif:
    from:
      uri: docker://ubuntu
    files:
      /README.md: |
        A Fuzzball-built image for use in a lolcow demo.
    post: |
      apt update && apt install -y fortune cowsay lolcat
jobs:
  lolcow:
    image:
      uri: fuzzball://user/lolcow.sif
    script: "fortune | cowsay | lolcat"
    resource:
      cpu:
        cores: 1
      memory:
        size: 1GB

Fuzzball builds lolcow.sif before the lolcow job starts, stores it in the object cache, and the job then uses it like any other image.

Image definitions

The map key is the name the built image is stored under in the object cache. Names follow the same rules as other object names:

• lower case, and made up only of the characters a-z, 0-9, _, -, . and /;
• starting with a letter or a digit, so not with ., - or /;
• with no // in them, and not ending in /.

A name may include / to organise images into folders, for example team/analysis/base.sif.

Each definition takes the following fields:

Fuzzball builds an image in this order: it pulls from, writes any files into it, then runs post inside it. Because files are written first, the post script can act on them.

from takes the same fields as a job's image, so a base image in a private registry or an encrypted SIF works the same way here as it does for a job:

images:
  internal.sif:
    from:
      uri: docker://registry.example.com/team/base:2026.1
      secret: secret://account/registry-credentials

Referring to a built image

A built image is stored in the object cache under its scope, so a job or service refers to it with an fb:// or fuzzball:// URI made of the scope and the image name:

images:
  private.sif:
    scope: user      # the default
    from:
      uri: docker://ubuntu
  shared.sif:
    scope: group
    from:
      uri: docker://ubuntu
jobs:
  uses-private:
    image:
      uri: fuzzball://user/private.sif
  uses-shared:
    image:
      uri: fuzzball://group/shared.sif

Use scope: group when you want everyone in your group to be able to use the image; leave it at user to keep it to yourself. See Object URIs and namespaces.

Adding files

files maps an absolute path in the image to the contents written there:

images:
  configured.sif:
    from:
      uri: docker://rockylinux/rockylinux:9
    files:
      /etc/myapp/config.yaml: |
        threads: 4
        verbose: true
      /opt/entrypoint.sh: |
        #!/bin/sh
        exec /usr/bin/myapp --config /etc/myapp/config.yaml
    post: |
      chmod +x /opt/entrypoint.sh

Paths must be absolute. This field is for small text files such as configuration; use a volume with ingress for large data.

The post script

post runs inside the image with root privileges within the container, which is what lets it install packages. If the script does not begin with an interpreter line, #!/bin/sh is assumed.

images:
  tools.sif:
    from:
      uri: docker://rockylinux/rockylinux:9
    post: |
      #!/bin/bash
      dnf install -y epel-release
      dnf install -y jq ripgrep
      dnf clean all

The script runs as root inside the build container, which is what lets a package manager install into the image. That container is confined: it holds only the capabilities needed to unpack files and set their ownership, it gets its own network namespace, nothing it runs can acquire further privileges, and a seccomp filter refuses it a new user namespace. Installing packages, writing files and running programs from the image all work; most things beyond that do not.

If the post script exits non-zero the build fails, the image is not saved, and any job that referred to it does not run.

Output from post is the log of the image's own stage, so that is where to look when a build fails. The stage is listed by fuzzball workflow status <workflow>, and is named for the image's URI and the architecture it was built for rather than for the key in images::

fuzzball workflow log <workflow> "fuzzball://user/lolcow.sif (amd64)"

It works while the build is running as well as after it has finished.

Watching a build

A build reports its progress as the workflow's events, so fuzzball workflow events shows where it has got to rather than a stage that sits there until it finishes: the base image being pulled, the build starting, each file written into the image, the post script starting and finishing with how long it took, the result being compressed, and the image being saved to the object cache.

fuzzball workflow events <workflow> --follow

--follow keeps printing them as they arrive, which is the useful form while a build is still running. Without it you get the events so far and the command returns.

Build order and dependencies

Fuzzball works out the build order from the definitions, so you do not declare it yourself.

A job that refers to a built image waits for that image to be built. An image whose from refers to another image in the same workflow is built after it:

version: v4
images:
  base.sif:
    from:
      uri: docker://rockylinux/rockylinux:9
    post: |
      dnf install -y gcc make
  app.sif:                              # built after base.sif
    from:
      uri: fuzzball://user/base.sif
    post: |
      dnf install -y openmpi-devel

This is useful when several images share an expensive common layer: build it once as its own image and start the others from it.

Definitions that depend on each other in a circle cannot be built in any order, so Fuzzball rejects the workflow when you submit it rather than starting a run that cannot finish:

version: v4
images:
  image1.sif:
    from:
      uri: fuzzball://user/image1.sif    # refers to itself
  image2a.sif:
    from:
      uri: fuzzball://user/image2b.sif   # these two refer
  image2b.sif:
    from:
      uri: fuzzball://user/image2a.sif   # to each other

Notes and limits

• Building takes time. An image is built on a compute node as a stage of your workflow, so a workflow that builds an image takes longer to reach its first job than one that pulls a ready-made image. Where an image changes rarely, consider building it once into scope: group and referring to it from later workflows by URI, rather than rebuilding it every run.
• A built image is a cached object, not a permanent artifact. It is subject to the object cache's TTL and cleanup, so it can be removed once it stops being used. Set ttl if you need a specific lifetime, and do not treat the object cache as a registry of record.
• images: needs a current CLI. An older Fuzzball CLI rejects a Fuzzfile containing an images: section with field images not found. Upgrade the CLI if you see that.
• Some things a post script might try are refused. It cannot mount a filesystem, create a new namespace, create device nodes, or load kernel modules, and it reaches the network through its own namespace rather than the compute node's. A script that needs one of those fails with Operation not permitted, and that is deliberate rather than a bug — build the image outside Fuzzball and refer to it by URI instead. Installing packages, writing files, changing ownership and running programs from the image are all unaffected.
• This is not a full container build system. It is meant for straightforward customisation — a base image plus some packages and configuration. For complicated builds, build the image with Apptainer or Docker, push it to a registry, and refer to it by URI.