Building Images in a Workflow

Sometimes no public container has quite what your job needs. Rather than building an image on your workstation and pushing it to a registry first, you can describe the image in the Fuzzfile itself and let Fuzzball build it as part of the workflow.

Images are described in a top-level images: section, which is a map of image names to image definitions:

version: v4
images:
  lolcow.sif:
    from:
      uri: docker://ubuntu
    files:
      /README.md: |
        A Fuzzball-built image for use in a lolcow demo.
    post: |
      apt update && apt install -y fortune cowsay lolcat
jobs:
  lolcow:
    image:
      uri: fuzzball://user/lolcow.sif
    script: |
      export PATH=/usr/games:$PATH
      fortune | cowsay | lolcat
    resource:
      cpu:
        cores: 1
      memory:
        size: 1GB

Fuzzball builds lolcow.sif before the lolcow job starts, stores it in the object cache, and the job then uses it like any other image. A built image keeps the environment of its base image, so a job must extend PATH itself when packages install somewhere the base does not search: Ubuntu puts fortune, cowsay and lolcat under /usr/games, which is why the job above adds it.

Image definitions

The map key is the name the built image is stored under in the object cache. Names follow the same rules as other object names:

• lower case, and made up only of the characters a-z, 0-9, _, -, . and /;
• starting with a letter or a digit, so not with ., - or /;
• with no // in them, and not ending in /.

A name may include / to organize images into folders, for example team/analysis/base.sif.

Each definition takes the following fields:

Fuzzball builds an image in this order: it pulls from, writes any files into it, then runs post inside it. Because files are written first, the post script can act on them.

from takes the same fields as a job's image, so a base image in a private registry works the same way here as it does for a job. The one exception is an encrypted SIF: a build converts its base to EROFS and cannot open an encrypted one, so a from with a decryption-secret is rejected when the workflow is submitted.

images:
  internal.sif:
    from:
      uri: docker://registry.example.com/team/base:2026.1
      secret: secret://account/registry-credentials

Adding files

files maps an absolute path in the image to the contents written there:

images:
  configured.sif:
    from:
      uri: docker://rockylinux/rockylinux:9
    files:
      /etc/myapp/config.yaml: |
        threads: 4
        verbose: true
      /opt/entrypoint.sh: |
        #!/bin/sh
        exec /usr/bin/myapp --config /etc/myapp/config.yaml
    post: |
      chmod +x /opt/entrypoint.sh

Files are written owned by root with mode 0660, and the post script may change them: the image keeps what the script leaves behind, so making an entrypoint executable with chmod +x, as in the example, or editing a configuration file works.

Paths must be absolute, and no two files may share a file name: the build stages every file in one directory before placing each at its path, so /etc/app/config.yaml and /opt/app/config.yaml cannot both be in the same image definition. Give them distinct names, or write one of them from the post script instead. The post script itself is staged the same way under the name .erofsbuild.sh, so that name is reserved at any path. This field is for small text files such as configuration; use a volume with ingress for large data.

The post script

post runs inside the image with root privileges within the container, which is what lets it install packages. If the script does not begin with an interpreter line, #!/bin/sh is assumed. The script starts in the base image's working directory, or / if the image sets none, so use absolute paths or cd into a directory first.

images:
  tools.sif:
    from:
      uri: docker://rockylinux/rockylinux:9
    post: |
      #!/bin/bash
      dnf install -y epel-release
      dnf install -y jq ripgrep
      dnf clean all

The script runs as root inside the build container, which is what lets a package manager install into the image. That container is confined: it holds only the capabilities needed to unpack files and set their ownership, it gets its own network namespace, nothing it runs can acquire further privileges, and a seccomp filter refuses it a new user namespace. Installing packages, writing files and running programs from the image all work; most things beyond that do not.

If the post script exits non-zero the build fails, the image is not saved, and any job that referred to it does not run.

Referring to a built image

A built image is stored in the object cache under its scope, so a job or service refers to it with an fb:// or fuzzball:// URI made of the scope and the image name:

images:
  private.sif:
    scope: user      # the default
    from:
      uri: docker://ubuntu
  shared.sif:
    scope: group
    from:
      uri: docker://ubuntu
jobs:
  uses-private:
    image:
      uri: fuzzball://user/private.sif
  uses-shared:
    image:
      uri: fuzzball://group/shared.sif

Use scope: group when you want everyone in your group to be able to use the image; leave it at user to keep it to yourself. See Object URIs and namespaces.

Watching a build

Output from post is the log of the image's own stage, so that is where to look when a build fails. The stage is listed by fuzzball workflow describe <workflow>, and is named for the image's URI and the architecture it was built for rather than for the key in images::

$ fuzzball workflow log <workflow> "fuzzball://user/lolcow.sif (amd64)"

The log command works while the build is running as well as after it has finished.

A build reports its progress as workflow events rather than as a stage that sits idle until it finishes, so fuzzball workflow events shows how far it has progressed. Fuzzball emits an event for each of the following:

• the base image being pulled
• the build starting
• each file written into the image
• the post script starting, and finishing with how long it took
• the result being converted to an EROFS layer on top of the base image
• the image being saved to the object cache

A build whose base image, files and post script have not changed since a previous build is served from the cache instead. The events then report the cache hit, and the stage has no log.

$ fuzzball workflow events <workflow> --follow

--follow keeps printing them as they arrive, which is the useful form while a build is still running. Without it you get the events so far and the command returns.

Build order and dependencies

Fuzzball works out the build order from the definitions, so you do not declare it yourself.

A job that refers to a built image waits for that image to be built. An image whose from refers to another image in the same workflow is built after it:

version: v4
images:
  base.sif:
    from:
      uri: docker://rockylinux/rockylinux:9
    post: |
      dnf install -y gcc make
  app.sif:                              # built after base.sif
    from:
      uri: fuzzball://user/base.sif
    post: |
      dnf install -y openmpi-devel

This is useful when several images share an expensive common layer: build it once as its own image and start the others from it.

Definitions that depend on each other in a circle cannot be built in any order, so Fuzzball rejects the workflow when you submit it rather than starting a run that cannot finish:

version: v4
images:
  image1.sif:
    from:
      uri: fuzzball://user/image1.sif    # refers to itself
  image2a.sif:
    from:
      uri: fuzzball://user/image2b.sif   # these two refer
  image2b.sif:
    from:
      uri: fuzzball://user/image2a.sif   # to each other

Notes and limits

• Building takes time. An image is built on a compute node as a stage of your workflow, so a workflow that builds an image takes longer to reach its first job than one that pulls a ready-made image. Where an image changes rarely, consider building it once into scope: group and referring to it from later workflows by URI, rather than rebuilding it every run.
• A built image is a cached object, not a permanent artifact. It is subject to the object cache's TTL and cleanup, so it can be removed once it stops being used. Set ttl if you need a specific lifetime, and do not treat the object cache as a registry of record.
• images: needs a current CLI. An older Fuzzball CLI rejects a Fuzzfile containing an images: section with field images not found. Upgrade the CLI if you see that.
• Some things a post script might try are refused. It cannot mount a filesystem, create a new namespace, create device nodes, or load kernel modules, and it reaches the network through its own namespace rather than the compute node's. A script that needs one of those fails with Operation not permitted, and that is deliberate rather than a bug -- build the image outside Fuzzball and refer to it by URI instead. Installing packages, writing files, changing ownership and running programs from the image are all unaffected.
• This is not a full container build system. It is meant for straightforward customization -- a base image plus some packages and configuration. For complicated builds, build the image with Apptainer or Docker, push it to a registry, and refer to it by URI.