V3 to V4 Storage Migration

When upgrading from Fuzzball V3 to V4, the storage system is automatically migrated from the V3 three-tier model (Storage Driver, Storage Class, Storage Volume) to the V4 two-tier model (Storage Provisioner, Storage Volume). The upgrade creates V4 provisioners from your existing storage classes, links your volumes to them, and backfills volume ownership.

However, the automated migration handles database records only. Depending on how your V3 storage classes were configured, the actual directories on your storage backend may need adjustment so that V4 can locate them. This guide walks through the post-upgrade steps to get your storage fully functional.

Before you upgrade: Generate the migration report

Before upgrading to V4, generate a migration report on your V3 cluster. The report previews exactly what the automated migration will do: which storage classes will be converted to provisioners, what driver type each will use, and which volumes will be linked. This report is your reference for verifying the upgrade succeeded and for planning any post-upgrade path reconciliation.

Prerequisites for the preflight report

• fuzzball CLI configured and authenticated against the V3 cluster
• kubectl access to the fuzzball-system namespace
• jq installed
• The storage-migrate-preflight bundle from CIQ Support

Running the preflight tool

Obtain the storage-migrate-preflight.zip bundle from CIQ Support. Then:

# unzip storage-migrate-preflight.zip
# cd storage-migrate-preflight
# ./storage-migrate-preflight.sh

The tool discovers your cluster, connects to the operator pod, and runs a dry-run of the migration. Example output:

storage-migrate: previewing migration for cluster 491506d4-33fd-6849-5a22-e832418a3615
storage-migrate: found 2 storage class(es)
storage-migrate: processing class "persistent" (id=e8778dde-..., persistent=true)
  org e700253c-...: 2 volume(s) in class "persistent"
    -> will create provisioner "persistent" (driver=HOSTPATH)
    -> volume "migration-test-vol-1" will be linked to provisioner "persistent"
    -> volume "migration-test-vol-2" will be linked to provisioner "persistent"
storage-migrate: processing class "ephemeral" (id=7f33ca42-..., persistent=false)
  no unmigrated volumes for class "ephemeral"
storage-migrate: completed successfully — 2 class(es) processed

What each line means:

• "will create provisioner" — a V4 provisioner will be created for this storage class in this organization, with the specified driver type
• "will be linked to provisioner" — this existing volume will be associated with the new provisioner
• "no unmigrated volumes" — the storage class exists but has no volumes that need migration

Save the report

Save the JSON report for reference during the post-upgrade steps:

# ./storage-migrate-preflight.sh --json | jq . > preflight-report.json

Review the report and confirm:

1. The driver type mapping is correct for each storage class (HOSTPATH, NFS, or EFS)
2. The expected volumes will be linked to the right provisioners
3. No unexpected warnings about corrupted class definitions or orphaned volumes

Once the report looks correct, proceed with the V4 upgrade. See Updating Fuzzball for the upgrade procedure.

Post-upgrade steps

The remainder of this guide covers steps to complete after the V4 upgrade has finished. The operator runs storage-migrate automatically during reconciliation, which converts your V3 storage classes to V4 provisioners as previewed in the preflight report.

Prerequisites

Before starting the post-upgrade steps:

• Fuzzball V4 upgrade is complete and the operator has reconciled successfully
• fuzzball CLI is installed and authenticated against the upgraded cluster
• kubectl access to the fuzzball-system namespace
• Filesystem access to the storage backend (NFS mount point, host node, or AWS console)
• Pre-upgrade migration report has been generated and reviewed (see above)

# kubectl logs -l app.kubernetes.io/name=fuzzball-operator -n fuzzball-system | grep storage-migrate

Step 1: Verify the automated migration

After the V4 upgrade completes, confirm that provisioners were created from your V3 storage classes.

List all provisioners:

# fuzzball volume provisioner list

You should see one provisioner per V3 storage class per organization. For example, if you had a persistent storage class and two organizations, you will see two provisioners both named persistent, one for each organization.

Compare against the preflight-report.json you saved before the upgrade — the provisioner count and names should match the dry-run output.

Inspect a specific provisioner:

# fuzzball volume provisioner info persistent

List volumes linked to a provisioner:

# fuzzball volume list --provisioner persistent

# kubectl logs -l app.kubernetes.io/name=fuzzball-operator -n fuzzball-system | grep storage-migrate

Step 2: Determine whether path reconciliation is needed

V3 storage classes used template-based directory naming that placed volumes inside subdirectories. For example, a V3 persistent class on a typical on-premises deployment stored volumes at:

/mnt/fuzzball-sharedfs/persistent/<group-UUID>/

V4 provisioners expect volumes as flat directories directly under the driver's base path:

/mnt/fuzzball-sharedfs/<volume-name>/

If your V3 storage classes used a basePath, subDir, or share parameter (most default configurations do), then the directories on your storage backend are nested one level deeper than V4 expects. You need to create symbolic links to bridge this gap.

If your V3 volumes are already at the top level of the storage backend with no nesting, skip to Step 4.

How to check

SSH into a host that can access the storage backend (the NFS server, or any node that mounts the shared filesystem) and list the top-level directories:

# ls -la /mnt/fuzzball-sharedfs/

If you see directories like persistent/ or ephemeral/ that contain your actual volume data as subdirectories, path reconciliation is needed.

Step 3: Create symlinks for path reconciliation

On-premises NFS or hostpath

This is the most common scenario. V3 default classes create a persistent/ and/or ephemeral/ subdirectory under the shared filesystem, with per-group or per-workflow UUID directories inside.

On the machine that hosts or mounts your shared filesystem, run the discover-and-link script for each V3 sub-path that contains volumes:

# ./discover-and-link.sh /mnt/fuzzball-sharedfs persistent

Scanning V3 volumes in: /mnt/fuzzball-sharedfs/persistent
  LINK: /mnt/fuzzball-sharedfs/e700253c-... -> /mnt/fuzzball-sharedfs/persistent/e700253c-.../
  LINK: /mnt/fuzzball-sharedfs/a1b2c3d4-... -> /mnt/fuzzball-sharedfs/persistent/a1b2c3d4-.../

Done. Created 2 symlink(s), skipped 0.

If you also have an ephemeral/ sub-path with data you want to preserve:

# ./discover-and-link.sh /mnt/fuzzball-sharedfs ephemeral

#!/bin/bash
# discover-and-link.sh — Creates symlinks to reconcile V3 storage paths for V4.
#
# Usage:
#   ./discover-and-link.sh <storage-root> <v3-sub-path>
#
# Examples:
#   ./discover-and-link.sh /mnt/fuzzball-sharedfs persistent
#   ./discover-and-link.sh /srv/nfs/fuzzball persistent

set -euo pipefail

usage() {
    echo "Usage: $0 <storage-root> <v3-sub-path>"
    echo ""
    echo "Arguments:"
    echo "  storage-root   Absolute path to the provisioner base directory"
    echo "                 (e.g., /mnt/fuzzball-sharedfs)"
    echo "  v3-sub-path    V3 subdirectory containing volume directories"
    echo "                 (e.g., persistent or ephemeral)"
    exit 1
}

if [ $# -ne 2 ]; then
    usage
fi

STORAGE_ROOT="$1"
V3_SUBPATH="$2"
V3_DIR="${STORAGE_ROOT}/${V3_SUBPATH}"

if [ ! -d "$STORAGE_ROOT" ]; then
    echo "ERROR: Storage root does not exist: $STORAGE_ROOT" >&2
    exit 1
fi

if [ ! -d "$V3_DIR" ]; then
    echo "ERROR: V3 directory not found: $V3_DIR" >&2
    echo "This may mean the storage class did not use a '$V3_SUBPATH' sub-path," >&2
    echo "or the storage is mounted at a different location." >&2
    exit 1
fi

created=0
skipped=0

echo "Scanning V3 volumes in: $V3_DIR"

for vol_dir in "$V3_DIR"/*/; do
    [ -d "$vol_dir" ] || continue

    vol_name=$(basename "$vol_dir")
    link_path="${STORAGE_ROOT}/${vol_name}"

    if [ -e "$link_path" ] || [ -L "$link_path" ]; then
        echo "  SKIP: $link_path already exists"
        skipped=$((skipped + 1))
        continue
    fi

    echo "  LINK: $link_path -> $vol_dir"
    ln -s "$vol_dir" "$link_path"
    created=$((created + 1))
done

echo ""
echo "Done. Created $created symlink(s), skipped $skipped."
if [ "$created" -gt 0 ]; then
    echo "Verify with: ls -la $STORAGE_ROOT"
fi

Verifying symlinks

After creating symlinks, confirm they point to the correct locations:

# ls -la /mnt/fuzzball-sharedfs/

You should see symlinks alongside the original V3 directories:

drwxr-xr-x  4 root root 4096 May 26 10:00 persistent/
drwxr-xr-x  2 root root 4096 May 26 10:00 ephemeral/
lrwxrwxrwx  1 root root   52 May 26 10:05 e700253c-... -> /mnt/fuzzball-sharedfs/persistent/e700253c-.../
lrwxrwxrwx  1 root root   52 May 26 10:05 a1b2c3d4-... -> /mnt/fuzzball-sharedfs/persistent/a1b2c3d4-.../

If your storage root is different

The default storage path is /mnt/fuzzball-sharedfs. If your cluster uses a different path, adjust accordingly. You can check the provisioner's driver configuration to find the correct base path:

# fuzzball volume provisioner info persistent -o yaml

Look for the path (hostpath driver) or target (NFS driver) field in the driver configuration.

GCP and Azure deployments

GCP and Azure V3 configurations typically used a subDir parameter (e.g., subDir=persistent) under a shared NFS export. The same symlink approach applies — mount the NFS export and run the script against the appropriate sub-path.

OCI deployments

OCI provisions separate NFS exports for /persistent and /ephemeral. Mount each export independently and verify whether the volume directories are nested. If they are, run the symlink script against each mount point.

Step 4: Scan for unlinked volumes

The automatic storage-migrate process links all V3 volumes that existed in the database. However, if you have volume directories on the storage backend that were not tracked in V3 (for example, directories created manually or by external tools), you can use the scan command to discover and import them.

# fuzzball volume provisioner edit <provisioner-name>

Preview what the scan will find:

# fuzzball volume provisioner scan persistent --dry-run

If the discovered volumes look correct, run the scan without --dry-run to import new volumes and remove stale records:

# fuzzball volume provisioner scan persistent

Verify the imported volumes:

# fuzzball volume list --provisioner persistent

Step 5: Verify with a test workflow

Submit a simple workflow that mounts one of your migrated volumes to confirm data is accessible:

version: v4
volumes:
  test-vol:
    use: persistent
    name: <your-volume-name>
jobs:
  verify:
    image: ubuntu:22.04
    mounts:
      /data: test-vol
    script: |
      echo "Listing migrated volume contents:"
      ls -la /data/

# fuzzball workflow start verify-migration.yaml

Monitor the workflow and confirm the volume contents are visible:

# fuzzball workflow events <workflow-id>

If the workflow runs successfully and your data is listed, the migration is complete.

Troubleshooting

Provisioner list is empty

The automatic storage-migrate did not run or encountered an error. Check the operator logs:

# kubectl logs -l app.kubernetes.io/name=fuzzball-operator -n fuzzball-system | grep storage-migrate

If the migration failed, it can be re-run safely (it is idempotent). Contact CIQ support for assistance with manual re-runs.

Volume shows in the database but workflows cannot mount it

The V4 driver cannot find the volume directory at the expected path. Verify:

1. The provisioner's driver base path is correct:

   # fuzzball volume provisioner info <provisioner-name> -o yaml
   

2. The volume directory (or symlink) exists at <base-path>/<volume-name>/:

   # ls -la /mnt/fuzzball-sharedfs/<volume-name>
   

3. If the path is missing, create a symlink to the actual data location (see Step 3).

Scan discovers unexpected volume names

If you ran the scan before creating symlinks, it may have discovered the V3 directory structure names (like persistent or ephemeral) as volume names. To fix this:

1. Remove the incorrectly imported volumes (this only removes the database record, not the data):

   # fuzzball volume delete <provisioner> <wrong-name> --preserve-data
   

2. Create the correct symlinks (Step 3).

3. Re-run the scan:

   # fuzzball volume provisioner scan <provisioner>
   

Provisioner has the wrong driver type

The automatic migration auto-detects the driver type from the V3 class configuration. If it chose incorrectly (for example, hostpath instead of NFS), edit the provisioner to correct it:

# fuzzball volume provisioner edit <provisioner-name>

This opens the provisioner definition in your editor. Update the driver section with the correct type and configuration, then save.

Symlinks are not followed by the storage driver

This is rare but can occur with highly restricted NFS configurations. As an alternative, you can edit the provisioner's driver configuration to point directly at the V3 sub-path:

# fuzzball volume provisioner edit <provisioner-name>

Change the path (hostpath) or target (NFS) to include the V3 sub-path. For example, change path: /mnt/fuzzball-sharedfs to path: /mnt/fuzzball-sharedfs/persistent.