Installation

The artifact

Stratum ships as one executable with an appended payload:

stratum  ≈ 610 MB
  ELF                supervisor + operator
  payload/bin        12 component binaries
  payload/images     OCI archive, 7 images

There is no separate airgap edition, no image list to pre-pull, and no registry credentials to configure. The artifact is the airgap package. On an SELinux host, such as Rocky Linux, RHEL or RLC, the node also needs the distribution's SELinux policy packages from its own mirror; see Required packages.

stratum which reports what a staged tree contains; stratum stage --root <dir> unpacks the payload without starting anything.

Prerequisites

firewalld

RLC and RHEL ship firewalld turned on, and so do some providers' Rocky Linux images. When it runs, it rejects pod traffic to the node itself: Cilium's per-pod interfaces belong to no zone, so they fall into the default zone. Every Service call a pod makes to the API server arrives that way, so without a change CoreDNS and local-path-provisioner never become Ready (dial tcp 10.96.0.1:443: connect: no route to host).

When firewalld is running, stratum up first pins the standard LXC bridge pattern lxcbr+ to the host's non-trusted default zone, unless you already assigned it another non-trusted zone. Firewalld evaluates this more-specific prefix before lxc+. It then binds Cilium's host-side pod interface pattern lxc+ to the trusted zone in both runtime and permanent configurations. If the bridge exclusion cannot be established, Stratum does not add pod trust and warns; an older binding it cannot remove is reported with a cleanup command. It repeats the checks on every start and never runs firewall-cmd --reload. Pods entering through the trusted interfaces can reach services on the node. Traffic arriving on an external NIC does not become trusted merely by claiming a pod-CIDR source address.

The lxc+ rule is a name-prefix match. The standard lxcbr0 bridge is excluded, and Stratum checks for other existing lxc-named bridges and physical interfaces on each start. If it finds one, it removes any wildcard binding it owns and warns. You can bind each Cilium veth by its exact name, or rename the other interface. An unrelated veth deliberately named with the lxc prefix can still match the wildcard.

Earlier builds of this change bound the entire pod CIDR as a trusted source. On the next start, Stratum removes source bindings it recorded as its own and replaces them with the interface binding. A failed removal is reported and its ownership record is kept for retry.

Worker join commands may include --pod-cidr for a non-default range and persist it in node state. The interface binding does not rely on this value, so a worker joined before it was saved needs no manual CIDR migration for firewalld.

The following cases need operator attention and produce warnings:

• lxc+ is already bound to a different zone, which Stratum does not move;
• the LXC bridge cannot be pinned outside the trusted zone;
• an lxc-named interface that is not a Cilium pod interface exists;
• an earlier source binding could not be removed;
• firewall-cmd fails.

stratum uninstall removes the pod interface binding, the LXC bridge exclusion, and any remaining legacy source bindings Stratum added, from the configurations it added them to. A binding you made yourself, even one Stratum completed in the other configuration, keeps your half. If firewalld refuses a removal, uninstall reports the exact command and keeps /var/lib/stratum with its ownership record so a retry can finish cleanup; other teardown targets are still removed.

Ports between nodes

On a multi-node cluster, nodes have to reach each other on their node IPs. stratum up opens none of these ports, because which interfaces and zones face the other nodes depends on your network. Allow them in the host firewall and in any network firewall between nodes:

The table covers IPv4 and the Cilium configuration Stratum ships (VXLAN tunnel, kube-proxy replacement).

With a host firewall on, pods also have to reach the node. A pod's call to the kubernetes Service (10.96.0.1:443) reaches the API server as <node IP>:6443, still from the pod's address. For a pod on the same node, the lxc+ binding described above admits it. A pod on another node is not covered by that binding; the tested configuration below admits it by trusting the pod range (10.245.0.0/16 by default).

A tested firewalld configuration

This configuration was tested with Stratum v1.36.4-stratum.0-rc.6 on Rocky Linux 9.8 (firewalld 1.3.4, nftables backend, SELinux enforcing) with one control plane and two workers. That build predates the lxc+ binding described above: the zone and that binding together have not been tested on a multi-node cluster. Workers joined; pods, Services and DNS worked across nodes; kubectl logs and exec worked; and Cilium's health checks passed from every node. It kept working through firewall-cmd --reload and systemctl restart firewalld with the cluster running.

The recipe and its reload behavior apply to firewalld's nftables backend, the one tested. Check which backend a node uses:

sudo grep -E '^[[:space:]]*FirewallBackend[[:space:]]*=' /etc/firewalld/firewalld.conf

FirewallBackend=nftables is the tested backend. If the file is readable and no line matches (grep prints nothing and exits 1), firewalld also uses nftables. If grep reports an error, such as a missing or unreadable file, the check has failed: it says nothing about the backend.

Run it on every node. Replace 10.50.0.0/24 with your node subnet and 10.245.0.0/16 with your pod range if you changed it:

sudo firewall-cmd --permanent --new-zone=stratum
sudo firewall-cmd --permanent --zone=stratum --add-source=10.50.0.0/24
sudo firewall-cmd --permanent --zone=stratum \
  --add-port=6443/tcp --add-port=10250/tcp --add-port=2379-2380/tcp \
  --add-port=8472/udp --add-port=4240/tcp --add-port=9448/tcp \
  --add-service=ssh
sudo firewall-cmd --permanent --zone=trusted --add-source=10.245.0.0/16
sudo firewall-cmd --reload

• The stratum zone is bound to the node subnet as a source, not to an interface. firewalld matches source-bound zones first, and the binding does not compete with the zone NetworkManager assigns to each connection. Traffic from outside the node subnet, including the public interface, still falls into that interface's own zone.
• 9448/tcp was not part of the tested configuration, which had one control plane. With more than one, an API server can send admission requests to the webhook on another control plane, and the webhook fails closed, so the port has to be open between them.
• --add-service=ssh was not part of the tested configuration either. It keeps SSH from the node subnet working once the zone takes effect.
• firewalld zones accept ICMP unless you block it, so the health checks' ICMP needs no rule.
• The trusted binding admits every packet with a pod-range source address, whichever interface it arrives on. That is why stratum up no longer adds such a binding itself. Make sure nothing outside the cluster can send your nodes traffic from the pod range, for example by dropping it at your network firewall.
• Stratum never removes bindings you made, so these survive stratum up and stratum uninstall.

On the nftables backend, reloading, restarting or stopping firewalld leaves Cilium's rules in place. Cilium keeps them in the ip filter, nat, mangle and raw tables, and firewalld 1.3.4 manages only its own inet firewalld table.

NetworkManager

On Rocky Linux 9.8, NetworkManager needs no configuration. It leaves Cilium's interfaces (cilium_host, cilium_net, cilium_vxlan, lxc_health and each lxc*) unmanaged by default. To check a host:

nmcli device status

Each of those interfaces shows unmanaged. The tested image did not include nm-cloud-setup.

SELinux

Stratum installs and runs on an SELinux-enforcing host, which is the default on Rocky Linux, RHEL and RLC.

Required packages

On a host where SELinux is enabled, Stratum needs these packages from the distribution's AppStream and BaseOS repositories, so any mirror of those repositories has them:

• container-selinux, which defines the SELinux types for a container runtime and its files.
• policycoreutils-python-utils, which provides semanage.
• libselinux-utils, which provides matchpathcon. It is installed wherever SELinux is, as policycoreutils requires it.

sudo dnf install container-selinux policycoreutils-python-utils libselinux-utils

A new node without them is refused with an error that names what is missing when SELinux is enforcing, and under the CIS profile in permissive mode too. Otherwise, on a permissive host and on any node that was brought up before (a restart, a reboot or an upgrade), stratum up prints a warning and continues. Such a node also only warns when the contexts cannot be registered or its files relabelled. A node that was already labelled stays labelled, and its next start runs the relabel again. The labels are applied on the first start after the packages are installed; to apply them now, run sudo systemctl restart stratum.

Stratum was tested with container-selinux 2.245 on Rocky Linux 9.8.

What Stratum labels

Every stratum up registers these file contexts with semanage fcontext, reusing the types container-selinux defines, and relabels the files to match:

A context is registered on the path the distribution's policy labels it by: on Rocky Linux 9, which labels /run as /var/run, the /run/stratum context is listed as /var/run/stratum, and the unit's as /usr/lib/systemd/system/stratum.*.

Started by systemd, stratum, containerd, the shims and the kubelet then run as container_runtime_t. Stratum also creates /opt/local-path-provisioner and labels it, so the local-path provisioner can create volumes there. On the start that first registers the contexts, the stratum up process itself keeps the domain it started in; it moves to container_runtime_t when it next restarts, for example at the next reboot.

The first start that registers the contexts relabels everything under /var/lib/stratum and /run/stratum, which takes longer on a node with many images; it prints a line when it does. It stays on those directories' filesystems, so it does not enter the mounts of running containers or pod volumes, whose labels belong to containerd and the kubelet. It writes /var/lib/stratum/selinux-relabel-pending when it starts and removes it when it finishes, and the first one to finish writes /var/lib/stratum/selinux-labels, which marks the node as labelled. A relabel that was interrupted or failed leaves the pending file, and the next start relabels again. On a node that was labelled before, selinux-labels stays in place, so it does not mean the latest relabel finished: the pending file does that. A start that cannot register the contexts leaves the pending file too, and changes no labels. A start with SELinux disabled removes selinux-labels, so the next start with it enabled relabels everything. A reinstall relabels too, as stratum uninstall removes both. Later starts relabel only the binaries, the unit file, the local-path directory and CoreDNS's upstream directory. Once a node is labelled, payload extraction, stratum upgrade and stratum rollback label each file they write before it replaces the old one.

On Rocky Linux 9 an equivalence for /run/stratum/containerd would not apply, because the policy labels /run as /var/run first, so Stratum registers copies of the policy's /var/run/containerd contexts under /var/run/stratum/containerd in its place. A sandbox's shm is then container_runtime_tmpfs_t, as it is under /run/containerd.

Stratum records the contexts it registered in /var/lib/stratum/selinux-contexts and deletes any it no longer registers, such as a copy of a policy rule that a container-selinux update removed. Contexts you added for other paths are never touched.

To list the contexts Stratum added:

sudo semanage fcontext -l -C

To run without these labels, set spec.selinux: false in the node's /etc/stratum/config.yaml and restart Stratum. A worker reads it on every start too. Stratum then registers no contexts, relabels nothing and labels no file it writes. The CIS profile refuses spec.selinux: false.

Opting out does not remove the contexts an earlier start registered. /var/lib/stratum/selinux-contexts lists them, one per line: a file context as SPEC FILE-TYPE TYPE, an equivalence as TARGET = SOURCE. Delete each file context, then each equivalence, then relabel. For example, on Rocky Linux 9:

sudo semanage fcontext -d -f a '/var/run/stratum(/.*)?'
sudo semanage fcontext -d -e /var/lib/containerd /var/lib/stratum/containerd
sudo restorecon -R -x /var/lib/stratum /run/stratum /usr/local/bin/stratum /etc/systemd/system/stratum.service

Lines starting with policy record the policy's own contexts and are not Stratum's to delete. stratum uninstall removes Stratum's contexts for you.

Containers

On a labelled node, Stratum's containerd runs with enable_selinux = true in /var/lib/stratum/config/containerd.toml, and each container gets its own SELinux label:

A pod's seLinuxOptions, such as level, are applied.

A node is labelled once /var/lib/stratum/selinux-labels exists (see above). On a node that opted out with spec.selinux: false, or that lacks the policy packages, enable_selinux stays off. Every container then keeps the label it inherits, all containers on the node share it, and SELinux separates nothing. Stratum rewrites the containerd config and restarts containerd when this changes. Running containers keep their labels until they are recreated.

hostPath volumes. A container_t pod cannot read a host directory with a host type, such as usr_t or var_lib_t. Label a directory that pods should use container_file_t, or run the pod privileged, which the baseline and restricted Pod Security levels refuse:

sudo semanage fcontext -a -t container_file_t '/srv/data(/.*)?'
sudo restorecon -R /srv/data

local-path volumes are container_file_t:s0, with no categories, so they are not separated by category from one another, as on k3s.

Troubleshooting. No AVC denials does not mean containers are confined: an unconfined domain has nothing to deny. To see the label a container runs with, replace PROCESS_NAME with the name of a process running in it, and to list recent denials:

ps -eZ | grep PROCESS_NAME
sudo ausearch -m AVC -ts recent

sudo stratum hardening check reports the selinux controls, which fail when the node is not labelled, containerd's enable_selinux is off, containerd is not container_runtime_t, a container runs in a domain other than container_t or spc_t, or a pod was started before containerd began confining containers (see SELinux confinement).

What a node exposes

Nothing in the Stratum stack protects the node itself. The bundled Cilium runs with its host firewall off (enable-host-firewall: "false"). With no host firewall, these listeners answer on every interface, the public one included:

etcd listens on 127.0.0.1 and the node IP only, so it is reachable wherever the node IP is.

Protect nodes from outside the Stratum stack: with a network firewall or security group, a firewalld zone like the one above, or both. Turning firewalld off removes the only filter on the host.

Troubleshooting: no route to host to port 6443

firewalld rejects a packet with an ICMP "administratively prohibited" reply, which Go reports as connect: no route to host. So when a worker join fails with dial tcp <control-plane-ip>:6443: connect: no route to host, or a pod logs dial tcp 10.96.0.1:443: connect: no route to host, check the host firewall before the routes. A worker join retries an unreachable control plane for up to two minutes, then fails.

To see what firewalld rejects, log denied packets on the receiving node and read the kernel log. Changing the setting reloads firewalld, so use these commands only on the nftables backend while Stratum runs (see the backend check):

sudo firewall-cmd --set-log-denied=all
sudo journalctl -k | grep _REJECT
sudo firewall-cmd --set-log-denied=off

On the iptables backend, don't change the setting on a running node. Read what is already logged instead:

sudo firewall-cmd --get-log-denied
sudo journalctl -k | grep _REJECT

If the first command prints off, firewalld logs no rejects, and the kernel log shows none.

An entry such as filter_IN_public_REJECT ... DPT=6443 names the zone that rejected the packet and the port it was for.

Host settings

stratum up prepares the host on every start: it loads the overlay and br_netfilter modules and lists them in /etc/modules-load.d/k8s.conf so they load again after a reboot, turns swap off, and writes /etc/sysctl.d/99-stratum.conf with the settings pod networking needs (ip_forward and bridge-nf-call-iptables). Under the CIS profile it also writes /etc/sysctl.d/99-stratum-cis.conf; see Hardening. It also raises two inotify limits, which the kubelet, the control plane and every root-running container share: fs.inotify.max_user_instances to at least 8192 and fs.inotify.max_user_watches to at least 524288. The distribution default of 128 instances runs out on a single node with a few dozen pods, and a workload then fails with too many open files. These are minimums: a host already set higher keeps its value.

stratum stage and stratum up set their own umask to 022, so the shell or systemd unit that starts them does not decide the node's file modes. Other stratum commands keep the caller's umask. CIS and STIG profiles often set root's umask to 027 or 077. Inherited, that umask leaves /var/lib/stratum and the containerd content store at 0750 or 0700. Either way, other users lose access, so containers that run as a non-root user cannot read their own image or mounted ConfigMaps, and CoreDNS crash-loops. Files Stratum creates with an explicit 0600 or 0700 mode, such as keys, keep that mode.

Topologies

Node (default).

sudo stratum up

One node that is both control plane and workload host: the control plane runs as kubelet-managed static pods and the node is registered and schedulable. Suitable for single-node clusters and hyperconverged deployments. For a dedicated control-plane machine, taint the node after it registers:

kubectl taint nodes <node> node-role.kubernetes.io/control-plane:NoSchedule

Worker (workload-only).

sudo stratum up --worker --server https://<endpoint>:6443 \
  --token <token> --discovery-token-ca-cert-hash sha256:<pin>

On a cluster with a non-default pod CIDR, service CIDR or cluster domain, stratum token create adds --pod-cidr, --service-cidr or --cluster-domain to this command. Run the command it prints as it is.

The worker's kubelet asks for its client certificate with the join token. The operator approves it only when the node name is not already registered and no other pending bootstrap CSR or previously approved bootstrap certificate claims that name. An approved name claim persists even after Kubernetes removes its CSR or Node: removing either object does not revoke the client certificate. A token joins new node names; it cannot obtain a second certificate for a name that has already been claimed.

A request the operator will not approve stays pending, and the operator records why on it. List the pending bootstrap requests, then read one's reason, the node name it asks for and the join token that sent it (system:bootstrap:<token-id>):

sudo stratum kubectl get csr --field-selector spec.signerName=kubernetes.io/kube-apiserver-client-kubelet
sudo stratum kubectl get csr <csr> -o jsonpath='{.metadata.annotations.stratum\.ciq\.com/node-client-pending}{"\n"}{.metadata.annotations.stratum\.ciq\.com/requested-node}{"\n"}{.spec.username}{"\n"}'

kubectl describe csr <csr> shows the same annotations. To check the subject the request actually carries, decode it:

sudo stratum kubectl get csr <csr> -o jsonpath='{.spec.request}' | base64 -d | openssl req -noout -subject

Deny a request you do not expect, and delete the token that sent it:

sudo stratum kubectl certificate deny <csr>
sudo stratum kubectl -n kube-system delete secret bootstrap-token-<token-id>

A machine rebuilt under a previously used node name waits for you, including after stratum uninstall removes its old Node. Give the rebuilt machine a new name when possible. If the name must be reused, first verify that the old node credential can no longer authenticate. Then delete the name's claim (the listing below shows which claim holds which name), so the operator records the new request as its owner, and approve the request by hand:

sudo stratum kubectl -n kube-system delete configmap <claim>
sudo stratum kubectl certificate approve <csr>

A request whose join token has since expired or been deleted is denied instead; rejoin that machine with a new token.

Deleting the Node alone does not revoke its old certificate or release the operator's persistent name claim. To list those claims, run:

sudo stratum kubectl -n kube-system get configmaps -l stratum.ciq.com/node-client-claim=true -o custom-columns=CLAIM:.metadata.name,NODE:.data.nodeName,OWNER:.data.csrName

Every controller start and every stratum token create reserve the upstream auto-approval binding without granting it to join tokens. This prevents token creation on another control plane from allowing a token holder to claim an existing node's identity. If every control plane runs with spec.operator: false, approve new node CSRs manually after checking the request and node name. The same applies if all operators are later disabled.

Configuration

Everything declarative lives in one strictly-parsed file:

# /etc/stratum/config.yaml
spec:
  cis: false
  operator: true
  selinux: true               # the default; false skips SELinux labelling, see SELinux
  network:
    nodeIP: 192.168.1.21        # this node's cluster address; autodetected when unset, see Node IP
    podCIDR: 10.245.0.0/16      # the default
    serviceCIDR: 10.96.0.0/12   # the default
    clusterDomain: cluster.local  # the default; fixed once the cluster is initialized
    cilium:                     # optional; the same on every control plane, see Cilium settings
      mtu: 9000
  vip:
    address: 192.168.1.10

Pass it explicitly with --config, or leave it at the default path.

A node's role is not configuration: it is chosen when the node is brought up, with stratum up for a control plane and stratum up --worker for a worker.

A worker join (stratum up --worker --server ...) reads only spec.network.nodeIP and spec.selinux from the config file. A worker keeps its node IP from the join. A restart reads only spec.selinux from the file, and keeps the saved setting when the file does not set it or cannot be read. If the file sets any other field, the join refuses to start and names the field, even when --node-ip is given. This includes cis, operator, network.podCIDR, network.serviceCIDR, network.clusterDomain and network.cilium. On a worker, cis, podCIDR, serviceCIDR and clusterDomain come from the --cis, --pod-cidr, --service-cidr and --cluster-domain flags instead; operator has no flag equivalent and is refused. A key that is present counts as set even when its value is false, 0 or "", so operator: false is refused. A key with no value (operator:) and an empty mapping (vip: {}) set nothing, at any depth, and are accepted. Do not copy a control plane's file to a worker. Give the worker a file that sets only its own address:

# /etc/stratum/config.yaml on a worker
spec:
  network:
    nodeIP: 192.168.1.31

A worker join takes its IP from the first of these that is set:

1. --node-ip
2. spec.network.nodeIP
3. $NODE_IP
4. the address of the default-route interface

On a host with more than one NIC, set the address explicitly, because the default route may be on a network the control plane cannot reach.

A node's IP is fixed at its first start. After that, the node keeps the address saved in its NodeState, and editing spec.network.nodeIP or $NODE_IP has no effect on it. That is what lets a rebooted node come back on the address its certificates and etcd membership name.

The pod and service CIDRs are fixed when the cluster is created. spec.network.podCIDR and serviceCIDR are read only on the cluster's first controller boot, which records them in /var/lib/stratum/pki/netconfig. A joining control plane does not read them: it inherits the cluster's recorded ranges. On every later start a node uses its recorded ranges. If the config names different ranges, the node ignores them and logs a warning that changing them needs a cluster rebuild. See Network configuration.

The cluster domain is recorded the same way, but a config that names a different domain is refused rather than ignored: stratum up does not start the control plane until you remove the field or set it to the recorded domain. See Cluster domain.

On a control plane, --node-ip on the command line outranks the saved address, and the startup log says which source the IP came from; stratum up still refuses to actually start on a different one, rather than starting etcd on an address its cluster does not know. A worker restart always reuses its saved address instead — --node-ip has no effect there, whatever it says.

Changing a control plane's IP in place is not supported: for an HA control plane, first remove this node's etcd member from a remaining control plane (stratum etcdctl member remove <member-id>, see stratum etcdctl member list), then remove this node (stratum uninstall) and rejoin it with the new IP. To recover a failed or interrupted join on the same node and address, re-run stratum up --join. The join reconciles its recorded intent against the cluster's member list before adding anything (see High availability).

Node IP

When nothing sets the node IP, stratum up uses the source address of the host's default route. No packet leaves the host, so the route needs no internet behind it. With no default route, it uses the host's global unicast IPv4 address if there is exactly one; loopback, link-local and IPv6 addresses don't count. With none or several, stratum up stops before doing anything and asks you to set spec.network.nodeIP or $NODE_IP:

sudo NODE_IP=192.168.1.21 stratum up

On a multi-NIC host, the default route often isn't the cluster network, so set the address explicitly there too.

Reboot persistence

On a systemd host, stratum up installs and enables /etc/systemd/system/stratum.service every time it starts. There is no separate step. After a reboot, systemd runs stratum up again and the node rejoins the cluster without anyone logging in.

The unit runs the node's steady-state command, not the command you typed:

Join flags are never written to the unit: --join, --server, --token, --token-file, --discovery-token-ca-cert-hash, --certificate-key and --certificate-key-file. They are single-use credentials, and a reboot never tries to join again.

Environment for the unit

A process started by systemd doesn't inherit your shell's environment. To give the node environment variables across a systemd-managed restart, the unit reads an optional file:

EnvironmentFile=-/etc/stratum/stratum.env

The leading - tells systemd to ignore the file if it doesn't exist, so the unit starts normally without it. Stratum never creates this file; you do. Its main use is S3 credentials for scheduled snapshot uploads. stratum snapshot schedule never writes environment credentials to disk, so unless snapshot.conf already holds them, the node needs them in its environment. See Snapshot & Restore.

The file holds secrets, so make it owned by root with mode 0600:

sudo install -m 0600 -o root -g root /dev/null /etc/stratum/stratum.env
sudoedit /etc/stratum/stratum.env

Write one KEY=value per line. systemd reads the file only when it starts the unit, so changes take effect on the next systemd-managed start, for example after a reboot. A stratum up you started by hand uses your shell's environment instead. stratum uninstall deletes the file along with the rest of /etc/stratum.

The unit is reinstalled and re-enabled whenever the command changes, or whenever it is not already enabled — so a systemctl enable that failed on one stratum up is retried on the next one, rather than accepted forever. If the unit cannot be installed, the node still starts and prints a warning. On a host without systemd, Stratum skips the unit, and restarting the node after a reboot is up to you.

If the unit is masked, stratum up does not touch it. It warns and starts the node without installing or enabling the unit, rather than overwriting the mask. To restore reboot persistence, unmask the unit and run stratum up again. Use the command that matches how the unit was masked:

sudo systemctl unmask stratum             # masked with systemctl mask
sudo systemctl unmask --runtime stratum   # masked with systemctl mask --runtime

To keep a node from starting at boot without uninstalling it — this also stops the currently running node, not just future boots, but only when it is running as stratum.service (for example, after a reboot). A first stratum up launched by hand keeps running; only the unit is disabled and stopped:

sudo systemctl disable --now stratum

Uninstall

sudo stratum uninstall          # shows what it would stop and delete
sudo stratum uninstall --yes    # does it

Uninstall also removes the firewalld pod-interface binding, the LXC bridge exclusion, and any older source bindings stratum up added, and lists them in the preview. Bindings you made yourself are not touched.

On an SELinux host it removes the file contexts and equivalences stratum up registered (see SELinux), each only while the policy still holds it as Stratum registered it, so a context you set for the same path is kept. It then relabels /usr/local/bin/stratum, which it leaves in place, to the policy's default.

Uninstall first disables stratum.service, so a reboot partway through does not bring the node back. Disabling only affects the next boot, so uninstall also writes a runtime drop-in, /run/systemd/system/stratum.service.d/uninstall-no-restart.conf, that sets Restart=no. If stratum up exits during teardown, systemd does not start it again. The drop-in lives under /run, so it never outlives a reboot, and the unit file in /etc is not modified.

Uninstall does not stop the unit yet, because kubelet has to keep running to remove the control-plane static pods. It stops the control plane, kubelet, every container running on the node (SIGTERM, then SIGKILL after 30 seconds), and containerd, and unmounts everything below the state directories.

Before it stops stratum.service and the top-level stratum up process, uninstall checks that it can query the node-start lock. If that query fails, uninstall aborts and tries to restore the node, as for any other failure at this stage.

Uninstall then stops stratum.service, and separately stops the top-level stratum up process it found. It stops the process directly because systemctl stop only affects an active unit, and a node started by hand was never started through systemd. Once that process has exited, uninstall checks the start lock again. A stratum up running from a different copy of the binary isn't recognized as the same process, so this second check is what catches it.

If the lock is still held, or can't be queried, uninstall refuses before removing node state. It doesn't try to restore the node, because the processes it stopped are gone. It can't restore an unidentified stratum up that may still hold the lock and may still be running. The node is left stopped, with the unit disabled, the Restart=no drop-in in place and node state kept. Stop any remaining stratum up on the node, then run stratum uninstall --yes again.

Otherwise, uninstall removes the Restart=no drop-in, then tears down the pod network and cgroup state the stopped containers leave behind, in this order:

1. the network namespace mounts under /run/netns that this node's containers ran in, and the mounts under /run/cilium;
2. Cilium's network links (cilium_*, lxc_health and the lxc* pod interfaces; lxcbr0 is left alone);
3. Cilium's tc filters on the remaining interfaces, the ones whose name starts with cil_ or cilium (the qdisc stays);
4. Cilium's CILIUM_* iptables and ip6tables chains, one rule at a time: it deletes each rule whose target (-j or -g) is a CILIUM_* chain, then flushes and deletes the Cilium chains. A CILIUM_* name inside a comment doesn't count. It never restores a saved ruleset, so rules another program adds while uninstall runs are left alone;
5. Cilium's BPF pins: the /sys/fs/bpf/cilium directory and the cilium_* maps in /sys/fs/bpf/tc/globals. Other programs' pins, including anything else under /sys/fs/bpf/tc and /sys/fs/bpf/xdp, are left, and the bpffs mount itself stays. /run/cilium goes too;
6. Cilium's cgroup programs: socket programs named cil_* still attached to the root cgroup, which kernels before 5.7 (and nodes upgraded from them) attach without a bpf link, so removing the pins doesn't detach them;
7. the empty kubepods cgroups (see below);
8. /var/lib/kubelet, if nothing but Stratum's stale kubelet sockets is in it.

Cilium's links, tc filters, chains, pins and cgroup programs are found by name, and another Cilium on the same host uses the same names. So before the teardown starts, uninstall checks for another kubelet or container runtime by process name: kubelet, containerd, crio, dockerd, cri-dockerd, podman, conmon, crun, kubelite, k0s, and anything starting with k3s or rke2. The match is deliberately broad, so an unrelated process with one of these names also counts. If it finds one, or can't read /proc, it skips every Cilium step: the /run/cilium mounts, links, tc filters, netfilter rules, BPF pins and /run/cilium, and cgroup programs. It names the process it found and exits nonzero, so the cleanup is never reported as clean. What is provably Stratum's still goes: the network namespaces in the record below, the kubelet sockets, and the rest of the uninstall. The kubepods cgroups have their own check, described below.

Before it stops any container, uninstall asks Stratum's containerd which network namespaces this node's containers run in. A namespace is detached only when a Stratum container's spec names its path in /run/netns (or the same directory through /var/run/netns). A namespace with the same name elsewhere doesn't count, and a host containerd's cni-* namespaces are left alone.

Uninstall saves that record to /run/stratum-uninstall-netns.json before it stops the workload containers. By then the control-plane static pods and kubelet are already stopped. The record is JSON with three fields: complete, paths (the namespaces to detach) and unread (the containers it couldn't read). If the record can't be saved, uninstall stops before touching any workload container and deletes nothing. As for any failure before stratum.service is stopped, it then tries to undo the earlier shutdown steps and start the node's components again, and says so if it can't.

If a record from an earlier run is already there but can't be read, uninstall treats it the same way: it stops before touching any workload container, deletes nothing and leaves the file exactly as it was, because it may name namespaces the current containers no longer show. Check what is mounted under /run/netns against the record, detach what it names, delete the record and re-run stratum uninstall --yes.

Uninstall deletes the record only when it is complete and every namespace in it has been detached and removed. If a detach fails, re-run stratum uninstall --yes to finish, even after the node's state has been deleted: the re-run reads the saved record. A reboot clears the record along with the mounts it names.

If a container or its spec can't be read, the record is saved as incomplete, with the unread containers listed. Uninstall still detaches the namespaces it did identify, then reports an error that names what it couldn't read and exits nonzero. The record stays incomplete: a re-run, even after the state is deleted, reports the same error. To finish, reboot (/run is a tmpfs), or detach the remaining mounts under /run/netns yourself and delete /run/stratum-uninstall-netns.json. The /run/cilium mounts are detached either way.

The record can also become unreadable after the containers were read, or a re-run may find no containers to read and rely on the record alone. If the record can't be read when uninstall comes to detach the namespaces, that is a failure: uninstall keeps /var/lib/stratum and carries on with the rest of the cleanup.

If containerd was already stopped with no shim left, uninstall can't read the containers, and there may be no saved record either. If Stratum's containerd metadata under /var/lib/stratum/containerd is still there and a cni-* namespace is mounted under /run/netns, uninstall can't tell whose namespace it is. It leaves the namespace mounted, reports the cleanup as incomplete, keeps /var/lib/stratum and exits nonzero; the rest of the cleanup still runs. Detach any of those namespaces that are Stratum's, or reboot, then re-run stratum uninstall --yes.

Uninstall never stops a process in the kubepods cgroups. The pod cgroup names are shared with any other kubelet on the host, so uninstall removes the empty kubepods cgroups, deepest first, only when no other kubelet or container runtime from the list above is running. If one is, or if /proc can't be read, the whole tree is kept, and uninstall names the process it found and exits nonzero. A cgroup that still has a process in it is kept and reported, and its empty siblings are removed.

With the systemd cgroup driver, those cgroups belong to transient kubepods*.slice units. Once the tree has been removed, which means it was empty, uninstall stops kubepods.slice, which stops the units beneath it. Otherwise systemd can create kubepods.slice again. It does the same for an active kubepods.slice unit whose directory is already gone, and removes a directory systemd creates again during the stop. Nothing is stopped while another kubelet or container runtime is running or while a cgroup is still in use. The cgroupfs driver has no slice unit, so there is nothing to stop.

Stratum's kubelet keeps its data in /var/lib/stratum/kubelet, but it still leaves two sockets at fixed paths under /var/lib/kubelet: device-plugins/kubelet.sock and pod-resources/kubelet.sock. Uninstall removes a socket only when all of these hold:

• nothing is mounted at or below /var/lib/kubelet;
• neither /var/lib/kubelet nor the socket's directory is a symlink;
• a connection to the socket is refused.

It then removes the emptied socket directories, and /var/lib/kubelet if that empties it. If a socket is already gone, its directory is still removed when it is empty. Anything else there is kept: other files, another plugin's socket, and any socket uninstall can't show to be stale. That includes a live socket, a datagram socket, and one where the connection fails with a permission error or times out. When something is kept, uninstall prints left /var/lib/kubelet in place: … with the reason and moves on, so another kubelet's data there is safe. If the mount table can't be read, everything is kept and the failure is reported.

Uninstall never deletes through a mount. If a mount under /run/cilium or /sys/fs/bpf/cilium can't be detached, or the mount table can't be read, that directory is kept and reported as a failure. The same applies to the cilium_* maps in /sys/fs/bpf/tc/globals: if anything is mounted at or below /sys/fs/bpf/tc, they are kept. The bpffs mount at /sys/fs/bpf itself doesn't count.

It then removes the unit file, the state directory and the other paths the preview lists, and reloads systemd.

The commands the teardown steps run (ip, tc, iptables, bpftool and systemctl) are each limited to 30 seconds, and iptables waits at most 5 seconds for the xtables lock. A command that times out is reported as a failure, and uninstall skips the rest of that step and carries on with the next. If the iptables lock wait runs out, for example because another process holds the lock, the iptables step ends the same way and the ip6tables pass, which shares the lock, is skipped. That applies while reading the rules with iptables-save too, not only while deleting them. The containerd (ctr) calls, including the namespace-record reads, keep their own 2-minute limit, and the commands uninstall runs outside the teardown steps have no limit.

If a teardown step fails, uninstall carries on with the remaining steps and paths, then reports the failure at the end and exits non-zero. Run it again to finish: each step skips state that is already gone, so running uninstall on a node it has already cleaned changes nothing.

Removing Cilium's tc filters needs the tc command. Kernels before 6.6 attach Cilium's programs as tc filters, and uninstall can't tell a node with no filters from one that still has them, so it checks on every kernel. If tc is missing, uninstall continues the remaining cleanup, reports incomplete TC teardown and exits nonzero. Install iproute-tc on Rocky and re-run uninstall.

Removing Cilium's netfilter rules needs iptables-save and ip6tables-save to find them. If either is missing, uninstall continues the remaining cleanup, reports that Cilium's rules could not be checked or removed, and exits nonzero. Install iptables and re-run uninstall.

Detaching Cilium's cgroup programs needs bpftool to list them. If it is missing, uninstall continues the remaining cleanup, reports that the programs could not be checked or detached, and exits nonzero. Install bpftool and re-run uninstall.

Uninstall looks for these tools, and for ip, systemctl and umount, only in /usr/sbin, /usr/bin, /sbin and /bin, never on your PATH. A tool installed anywhere else, such as a bpftool built into /usr/local/sbin, counts as missing. Install the distribution package, or link the tool into one of those directories.

On a worker, whether the Node is marked for removal depends only on whether the node's state directory, /var/lib/stratum, was removed. That directory holds the credential the mark needs, so the mark has to happen in the same run that deletes it.

• State removed: uninstall marks the Node and prints what's left to do in the cluster, even if something else failed: a teardown step, a later path, or the systemd reload. If anything failed, it doesn't print stratum uninstalled.; it exits nonzero, reports cleanup failures and says the node's state was removed. Fix the reported problem and re-run stratum uninstall --yes to finish the local cleanup.
• State kept or not removed: if removing /var/lib/stratum fails, or uninstall keeps it because a firewalld removal failed, the saved network-namespace record couldn't be read, or a mounted cni-* namespace couldn't be attributed, the Node is not marked. Uninstall exits nonzero and reports cleanup failures. A re-run can still mark the Node, because the credential is still there.

Cilium state is found by name and pin path, not through the Cilium agent, so a few things are not cleaned up:

• XDP programs attached without a bpf link.
• Cilium's nftables mode. Only the iptables and ip6tables chains are removed.

Uninstall works on a node that is running pods.

If systemctl fails to disable the unit, uninstall stops before touching anything. If a step fails before stratum.service is stopped, uninstall aborts before deleting any state: it removes the Restart=no drop-in, restores the unit's boot enablement as it found it, and tries to start the node's components again. A unit you had disabled yourself stays disabled.

After stratum.service and the top-level process are stopped and the start lock is confirmed free, uninstall no longer rolls back. If it then fails to remove the Restart=no drop-in, it prints a warning and carries on deleting state. The next stratum up on that node removes any leftover drop-in before installing the unit, so a reinstall doesn't normally start with Restart=no. If that cleanup fails too, stratum up still starts, with a warning, the same as when installing or enabling the unit fails.

Before uninstalling a node that is part of a larger cluster:

• Drain it from a control plane first, so its pods move gracefully and their disruption budgets are respected:

  kubectl drain <node> --ignore-daemonsets --delete-emptydir-data
  

  Uninstall stops whatever is still running without regard to disruption budgets.

• A worker's Node object is removed for you. The node's own credential is not allowed to delete its Node, so uninstall marks it instead, and the operator on the control plane deletes it once the worker's kubelet has been gone for about 2 minutes. Only the node itself can write that mark (the stratum-node-uninstall-mark admission policy refuses it from anyone else), and a worker reinstalled under the same name loses it again. On a cluster that runs no operator (spec.operator: false), or if the mark could not be written, it stays in the cluster as NotReady until removed from a control plane; uninstall prints the command:

  kubectl delete node <node>
  

  A control plane's Node is never removed automatically: remove it the same way, together with its etcd member, below.

• For a control plane in an HA cluster, remove its etcd member too, from a remaining control plane, so the cluster does not keep counting it towards quorum: find it with stratum etcdctl member list, then stratum etcdctl member remove <member-id>. A node that stays installed and rejoins after a failed join does not need this: re-running stratum up --join reconciles the join's intent journal against the cluster's member list and attempts to resume the join (see High availability).