Container Runtime

containerd and runc, built from source

Stratum ships containerd and runc, compiled here rather than repackaged, and runs containerd as a supervised component like any other.

Why the paths are unusual

The socket and state directories are deliberately not the containerd defaults.

A node may already have a distribution containerd installed and running. Using the default paths would mean two containerd instances competing for the same socket and state directory. Stratum's live under /run/stratum and /var/lib/stratum, so the two coexist and stratum uninstall removes one directory without touching the other.

State living under the Stratum data directory is also a hard requirement rather than tidiness: the overlayfs snapshotter cannot mount overlay-on-overlay, which is exactly what happens when containerd's state sits on a container's own overlay root.

The sandbox image comes from the payload

Every pod runs a pause container. By default containerd pulls that from registry.k8s.io on the first pod — which fails outright on an airgapped cluster and reintroduces an upstream registry dependency on every other one.

Stratum points containerd at the sandbox image carried in the payload, by the name the payload registers. No pod start requires a registry.

Payload images and kubelet's pull-credential check

Before a pod uses an image that is already on the node, the kubelet can check that the pod is allowed to pull it. Stratum exempts the payload registry's namespace from that check: imagePullCredentialsVerificationPolicy: NeverVerifyAllowlistedImages, with ghcr.io/ctrliq/stratum/* in preloadedImagesVerificationAllowlist in release builds. A payload image is therefore always used from the node, even after something pulls the same image under another name, such as docker.io/library/busybox.

The exemption covers every image under that namespace, not only the ones in the payload. An image pulled under it with a pull secret can be used from the node by any pod, without that secret.

Every other image is checked. An image the kubelet pulls, directly or through a registry mirror, is checked against the credentials it was pulled with, as upstream Kubernetes does by default.

An image you load onto a node by hand, for example with ctr images import, is also checked, and here Stratum departs from upstream: upstream's default policy never checks an image the kubelet did not pull. With imagePullPolicy: IfNotPresent the check is a pull from the image's registry, which fails on an airgapped node. With imagePullPolicy: Never the pod fails with ErrImageNeverPull even though the image is on the node. Serve your own images from a registry mirror instead. This applies to clusters you upgrade as well as new ones: every stratum up rewrites the kubelet's configuration. See Images loaded onto a node by hand.

Configuration

containerd's config is generated, not hand-maintained: Stratum takes containerd's own default and applies the settings above. Regenerating is a restart, and the resulting config is a normal containerd config file you can read.

The cgroup driver is matched to the kubelet's automatically. A mismatch between the two is a classic source of pods that start and then die in ways that do not point at their cause.

A containerd Stratum did not start

A containerd process left over from an earlier run can keep serving /run/stratum/containerd/containerd.sock after the supervisor has lost track of it. For example, a second stratum up killed part-way can leave one behind. The supervisor's own containerd then cannot bind the socket, exits, and restarts every 30 seconds, while the leftover containerd keeps serving pods.

When the supervisor's containerd exits and the socket is still live, the supervisor prints this line once:

supervisor: containerd exited but /run/stratum/containerd/containerd.sock is still served by another process — an orphan from an earlier run. This node is NOT healthy: ...

From then on, the supervisor reports containerd as Foreign, not as running. A Foreign component fails the upgrade and rollback health gate. The supervisor keeps restarting containerd. It clears Foreign once one of its own containerd processes has stayed up for 5 seconds. That is a heuristic: it shows the process did not exit the way it does when another process owns the socket, not that it holds the socket.

stratum version --components checks the same thing. It lists the versions of the bundled components. When the CRI socket is live and the containerd serving it reports a different version, it adds a line like this:

WARN: containerd <installed version> is installed but the runtime serving /run/stratum/containerd/containerd.sock reports <running version> — another containerd, likely an orphan from an earlier run, holds the socket

It prints no warning when nothing serves the socket or when it cannot read either version.

To recover, find the process that holds the socket, check it, and stop only that process. Do not run stratum uninstall for this: it deletes the node's state, including etcd data and certificates.

1. Find the process that owns the socket. Match on the socket, not on the name containerd: a host that also runs Docker or the NVIDIA container toolkit has its own containerd, and a name match will list it too.

   sudo ss -xlp 'src = /run/stratum/containerd/containerd.sock'
   # or
   sudo fuser -v /run/stratum/containerd/containerd.sock
   

2. Check what that process is. For an orphan from an earlier Stratum run, the path is a Stratum containerd, normally /var/lib/stratum/bin/containerd:

   sudo readlink /proc/<pid>/exe
   

   After an upgrade the orphan's binary may have been replaced, so the path can end in (deleted) or point at a backup copy such as /var/lib/stratum/bin/containerd.bak-20260928-100000. Both are still a Stratum containerd. A path anywhere else, such as /usr/bin/containerd, is not the orphan: leave it alone.

3. Stop only that process:

   sudo kill <pid>
   

Never kill a containerd that does not hold /run/stratum/containerd/containerd.sock. The host's own containerd, or the supervisor's short-lived child that is failing to start, is not the orphan.

The supervisor's next restart then takes over the socket. Run stratum version --components again to confirm the warning is gone.

Registry mirrors

Images in the payload never need a registry. To pull your own workload images through a mirror, describe the mirror in /etc/stratum/registries.yaml on each node. The format is the one RKE2 and k3s use, so an existing file works without changes for the fields Stratum supports: mirrors endpoints, and auth and tls under configs. A rewrite block is ignored (see below). For example:

# /etc/stratum/registries.yaml
mirrors:
  docker.io:
    endpoint:
      - "https://mirror.example.com:5000"
  "*":                                      # every other registry
    endpoint:
      - "https://mirror.example.com:5000"
configs:
  "mirror.example.com:5000":                # keyed by the endpoint's host:port
    auth:
      username: <user>
      password: <password>
    tls:
      ca_file: /etc/pki/mirror-ca.crt

The file can hold registry passwords, and Stratum does not change its mode. Keep it owned by root with mode 0600:

sudo chown root:root /etc/stratum/registries.yaml
sudo chmod 0600 /etc/stratum/registries.yaml

Stratum translates the file into containerd's own registry configuration: one /etc/containerd/certs.d/<registry>/hosts.toml per mirrored registry (_default for "*"), and a drop-in, /etc/containerd/conf.d/stratum-registry.toml, that points containerd at that directory. A hosts.toml that holds credentials is written with mode 0600.

The file is applied each time the node starts. To apply a change without restarting the node:

sudo stratum registry check    # show what would be configured; secrets are not printed
sudo stratum registry apply    # write it

containerd reads hosts.toml on every pull, so changes to mirror endpoints, credentials or TLS take effect at once. apply restarts containerd only when the drop-in is created, changed or removed: on the first apply, and when the last mirror is removed. Deleting registries.yaml and running apply removes the mirrors and the drop-in, and restarts containerd.

What is not here

There is no alternate runtime handler and no RuntimeClass in this distribution. Every pod runs under runc. Support for an additional runtime is a seam that has not been built, rather than a feature that is configured somewhere.