Upgrades & Rollback

One node

sudo stratum upgrade --target v1.36.4 --from /var/lib/stratum/upgrade/stratum-v1.36.4

--from names the upgrade binary: the new stratum binary itself, staged onto the node as a single file. Its embedded payload carries everything the upgrade installs — control-plane images, host binaries, addon images. Running the new binary directly, without --from, upgrades from its own payload.

The upgrade always runs as the version it installs, because the control-plane manifests it renders come from that binary, not from its payload. So the installed binary, given a different --from, hands off to it: it prints handing off to the upgrade binary … and the upgrade binary carries on with the same arguments. Leaving out --target targets the upgrade binary's version.

An installed binary from a release before this handoff behavior cannot hand off to --from. For the first upgrade from such a release, run the staged new binary directly instead:

sudo /var/lib/stratum/upgrade/stratum-v1.36.4 upgrade

--target is the bare Kubernetes version, vMAJOR.MINOR.PATCH. The build suffix a release carries (-stratum.0, -rc.2) is not part of it, and a suffixed target is refused with the version to use instead.

To upgrade to a prerelease build, pass the Kubernetes version as --target and point --from at the prerelease binary. For v1.37.1-stratum.0-rc.3:

sudo stratum upgrade --target v1.37.1 \
  --from /var/lib/stratum/upgrade/stratum-v1.37.1-stratum.0-rc.3

The per-node primitive imports the payload images and re-renders the control-plane static pod manifests one component at a time (controllers), swaps the host binaries (every node), gating on health as it goes and rolling the whole node back automatically if a gate fails.

One upgrade or rollback runs on a node at a time. A second one started while the first is running, by hand or by a cluster upgrade's Job, refuses at once with another upgrade or rollback is already running on this node and changes nothing. --dry-run changes nothing, so it is never refused.

A whole cluster

sudo stratum upgrade --cluster --yes --target v1.36.4 \
  --from /var/lib/stratum/upgrade/stratum-v1.36.4

--cluster drains and upgrades every node, so it refuses to run without --yes. To preview the rollout without changing anything, pass --dry-run in place of --yes.

With --cluster, omitting --target uses the version of the binary that starts the rollout, even when --from names a newer binary for each node. Specify --target when starting a rollout from an older installed binary.

--from is the same path on every node, and each node runs the file found there. In a cluster that mixes architectures, stage each node's own build at that path: the amd64 binary on amd64 nodes, the arm64 one on arm64 nodes.

Controllers first, always one at a time — two controllers down at once would cost etcd its quorum. Workers follow, up to --workers-max-unavailable of them concurrently (default 1, which is serial).

Each node is drained before it is touched and uncordoned after, unless it was already cordoned (see Nodes you cordoned). Draining evicts through the eviction API, so PodDisruptionBudgets are enforced by the API server however many nodes drain at once — raising --workers-max-unavailable shortens the window without letting an upgrade evict past a PDB. The first node failure stops new upgrades from starting; nodes already in flight finish rather than being abandoned part-way.

A node whose upgrade was interrupted after its kubelet was replaced reports the new version before its upgrade has finished. The plan shows it as finish, and the run completes it. An interrupted upgrade to a different version is refused before anything is drained: finish it with that version's --target, or undo it with stratum rollback on that node.

A single-node cluster

Upgrading a controller restarts its API server. On a single-node cluster, that is the API server the run talks to, so the run waits it out. It keeps polling the node's upgrade Job while the API server is unreachable, for as long as that Job is allowed to run. It then retries recording the new version and uncordoning the node for up to 5 minutes. Expect connection errors in between; they are not a failure.

Each node is still drained first, so on one node every workload is down from the drain until the node is uncordoned.

upgrade --cluster runs as the binary you start it with. Unlike a one-node upgrade, it doesn't hand off to --from. Only a release with this fix waits out the restart, so start the run from the new binary itself, the file you pass to --from:

sudo /var/lib/stratum/upgrade/stratum-v1.37.1 upgrade --cluster --target v1.37.1 \
  --from /var/lib/stratum/upgrade/stratum-v1.37.1 --yes

Started from an older installed stratum, the run still halts when the API server restarts. The new node Job then finishes the upgrade and uncordons the node, as described in the next section. Re-run the cluster upgrade afterwards to finish it.

Nodes you cordoned

The run uncordons only the nodes it cordoned. A node that is already cordoned when the run reaches it is still drained and upgraded, but it stays cordoned afterwards. This holds for a halted run, a re-run, and the Job's own uncordon described below.

The run records who owns each cordon in the node's stratum.ciq.com/upgrade-cordoned annotation. Its value is <kind>@<holder>, where <holder> is the run's identity on the cluster-upgrade lease, in the form <host>/<pid>/<start time>. For example, v1.37.1@ctl1/48211/1791249600000000000. The kind is one of these:

Only a claim whose kind is a valid version counts as the upgrade's own. The run treats any other value as an admin's cordon and leaves the node cordoned. That includes a misspelled kind, one with extra whitespace, or a kind added by a newer release.

A value with no @<holder> was written by an older stratum. The run treats it by its kind alone.

The run removes the annotation when the node finishes. Don't set or edit it yourself.

The holder ties each claim to one run. Only one run holds the lease at a time:

• A run that takes over the lease from a halted run takes over that run's claims. A cordon the halted run made stays the upgrade's to undo, and a node that was already cordoned stays preexisting. This holds even if the halted run was taking the node to a different version than yours.
• A run never finishes a node whose claim names another run. If a second run takes the lease while the first is still retrying its final write, the first run stops and leaves the second run's cordon and annotations in place.

If you cordon a node while the run has it cordoned, kubectl cordon changes nothing, so the run still uncordons the node when it finishes. Cordon it again after the node finishes.

Each cordon and uncordon write is checked against the version of the node the run read. If something else changes the node in between, the run reads the node again and decides again rather than overwriting the change.

Uncordoning after a halted run

If the run that started a cluster upgrade stops, the controller's own upgrade can still finish and uncordon the node. When a controller's upgrade Job passes its last health gate, it uncordons its own node if both of these are true:

• the node is cordoned
• the cluster upgrade owns that cordon: the kind in its stratum.ciq.com/upgrade-cordoned annotation is this upgrade's target

An older stratum that started the run records no upgrade-cordoned annotation. In that case, the Job treats the cordon as the upgrade's own when the node's stratum.ciq.com/upgrade-in-progress annotation names the target. An older run doesn't record whether the node was already cordoned, so the Job also uncordons a node you cordoned before that run. To keep your cordon, start the run from the new binary, as shown in A single-node cluster.

A node you cordoned before the run is marked preexisting, so the Job leaves it cordoned. When the Job uncordons the node, it sets upgrade-cordoned to released@<holder>, keeping the holder of the run that cordoned it. Its log says:

stratum upgrade: uncordoned <node> (the cluster upgrade that cordoned it may have halted)

The upgrade-in-progress annotation stays on the node. Re-run the cluster upgrade to record the new version and clear both annotations. Workers don't uncordon themselves, because upgrading a worker does not restart the API server.

If the Job cannot reach the API server within 4 minutes, the upgrade still succeeds. The Job logs a WARN that names the command to run:

  WARN: could not check whether <node> is still cordoned: <error>; if the cluster upgrade cordoned it (you did not cordon it yourself before the upgrade), run: /usr/local/bin/stratum kubectl uncordon <node>

Run it as sudo /usr/local/bin/stratum kubectl uncordon <node>, and only if the cluster upgrade cordoned the node, not you. The command uses the installed binary's full path because sudo may not search /usr/local/bin. On Rocky Linux 9, for example, sudo stratum fails with command not found.

Recovering from a halted cluster upgrade

If a run stops after it drained a node, it prints the recovery steps for that node:

  <node> is left cordoned, and its upgrade Job may still be running. To recover:
    1. <path> kubectl -n kube-system get job/<job>       (wait for it to succeed or fail; "not found" means it
       already finished and was cleaned up)
    2. <path> upgrade --cluster --target <target> --from <path> --yes    (finishes the node)
       or, if the node is at the target and healthy, and this run cordoned it (you did not cordon it yourself
       before the upgrade): <path> kubectl uncordon <node> (halting; later nodes untouched)

<path> is the halted run's --from path. Every command in the message runs that binary, including the kubectl commands, for two reasons:

• upgrade --cluster doesn't hand off to --from, and the stratum on your PATH may be an older release that halts again.
• The node has no kubectl on its PATH.

A path that contains spaces or other shell characters is printed in single quotes, so you can paste each command as printed. Run the commands with sudo.

The per-node Job runs on the node, not in the stratum process, so it keeps running after the run stops. Follow the steps in order:

1. Wait for the Job named in the message to succeed or fail. If the Job is not found, it already finished and was cleaned up. Re-running before the Job finishes is refused on that node, because only one upgrade or rollback runs on a node at a time.
2. Run the command from step 2 of the message. It finishes the halted node and goes on to the nodes the halted run had not reached yet. Run it with --dry-run in place of --yes first. The plan lists the halted node as finish.

If you don't want to re-run yet, and the node is already at the target and healthy, you can run sudo <path> kubectl uncordon <node> to restore its workloads. Skip this if you cordoned the node yourself before the run. Run the step 2 command later to upgrade the remaining nodes and clear the annotations.

A new build of the same Kubernetes release

A stratum release can change what it installs — containerd, a fix — without changing the Kubernetes version, so a node's kubelet version alone cannot say whether it runs the build you are rolling out. A node already at --target is therefore planned as check build. Before the run cordons it, a short Job on that node compares its installed stratum with the staged one, byte for byte. A node that already runs this build is left alone, not drained. One that runs another build — any difference at all, including a rebuild of the same source — is upgraded like any other node. --target is the same Kubernetes version in both cases:

sudo stratum upgrade --cluster --yes --target v1.36.4 \
  --from /var/lib/stratum/upgrade/stratum-v1.36.4-<build>

A node left alone this way can still carry a cordon from an earlier run that halted without finishing the node. If that cordon is the upgrade's own, the run uncordons the node and removes its stratum.ciq.com/upgrade-cordoned annotation, so you don't need to run kubectl uncordon. The cordon is the upgrade's own when the claim's kind is a valid version, whatever the version, or when there is no claim and the node has an upgrade-in-progress annotation. A preexisting or released claim, or a cordon with neither annotation, is yours, and the node stays cordoned. As in Uncordoning after a halted run, an older stratum didn't record whether the node was already cordoned, so a node you cordoned before that older run is uncordoned too. A claim left on a node that is no longer cordoned is marked released, so a cordon you set later is not taken for the upgrade's.

Every node runs its own copy of the file at --from, so the run pins them all to the copy on the node it runs on, and refuses to start if that binary cannot report its build or its version. Every node's copy must be the same build — the source it was built from and how (build tags, dashboard assets), the same on every architecture — and a node of the same architecture as the one running the rollout must hold exactly the same bytes. That is checked by a short Job on each Ready node before it is cordoned — behind --target, mid-upgrade or at it — so a node with anything else staged refuses there, and the rollout stops without draining it: a stale or different binary left on one node cannot answer for, or be installed as, the one being rolled out. A NotReady node behind --target cannot run that Job; its binary is checked by the upgrade itself, after the drain, and a mismatch then leaves it cordoned. A node at --target that is NotReady cannot be checked; it is left alone, the rest of the cluster is upgraded, and the run ends with an error naming it. (A NotReady control plane is allowed only where preflight finds room for it; see below.) The run ends by saying how many nodes it upgraded and how many already ran the build.

The build check is made by the installed stratum that starts the rollout, not by the --from binary. If the installed stratum comes from a release made before the build check existed, it drains every node, and installs the new build only if the --from binary has the check. For that first rollout, upgrade each node with sudo stratum upgrade --from <binary> instead.

On one node, sudo stratum upgrade --from <binary> --check asks the same question and changes nothing. It exits 0 when the node already runs that version and build, 10 when it needs the upgrade, and 1 when it cannot tell — for example while another upgrade or rollback is running on the node, which it refuses to answer during. Treat 1 as neither: a cluster upgrade stops at that node without draining it.

--dry-run prints the plan without changing anything. Run it first.

What a cluster upgrade creates

A cluster upgrade drives each node through Kubernetes objects you can inspect while it runs:

To follow one node's upgrade:

sudo stratum kubectl -n kube-system get jobs -l stratum.ciq.com/upgrade=<node>
sudo stratum kubectl -n kube-system logs job/stratum-upgrade-<node>

Long node names change both names, at different lengths:

• Job name. When stratum-upgrade-<node> would exceed 63 characters (a node name over 47), the Job name is truncated and given a hash suffix.
• Label. When the node name is not a valid label value (over 63 characters, for example), the label value is node-<hash> instead.

Every Job carries the node's full name in its stratum.ciq.com/node annotation, whatever its length.

The lease. The run holds the lease for 60 seconds at a time and renews it every 20 seconds. A second upgrade --cluster refuses while the lease is held, and the error names its holder. If the first run's process is gone, the lease expires on its own within 60 seconds. You do not need to delete it.

The annotations. A node that still has upgrade-in-progress was interrupted during its upgrade. The next run finishes it (the plan shows finish) or refuses if it was going to a different version. Every node, worker or control plane, sets k8s-version and version on its own Node each time its kubelet starts, and a cluster upgrade sets them when it finishes a node.

The control-plane label. Every control plane labels its own Node when it starts. If no Node has the label, the upgrade falls back to the etcd member list to tell control planes from workers, and refuses if it cannot read it.

HA clusters with custom CIDRs

Do this check before you upgrade an HA cluster that uses a custom spec.network.podCIDR or serviceCIDR, if its control planes run a build from before the recorded CIDRs took precedence. On those builds, a control plane that joined with --join could take the default CIDRs on a restart. The new build keeps whatever a node has recorded, so an upgrade would keep the defaults on that node.

1. Before you change any node, work out the cluster's pod and service CIDRs. No single file is the authority, because on the affected builds any controller could have recorded the defaults. Cross-check these sources:

   • spec.network in the config.yaml the cluster was first installed with, if you still have it.

   • The kubernetes Service's ClusterIP, the first address of the service CIDR:

     kubectl get svc -n default kubernetes -o jsonpath='{.spec.clusterIPs}{"\n"}'
     

   • Each node's .spec.podCIDRs, which are inside the pod CIDR:

     kubectl get nodes -o custom-columns=NAME:.metadata.name,PODCIDRS:.spec.podCIDRs
     

   • The CIDR flags on every control plane, from step 2, and each control plane's /var/lib/stratum/pki/netconfig. The file's first line is the pod CIDR and its second line is the service CIDR. A control plane that shows different ranges from the sources above has drifted, and step 3 fixes it.

   If the sources above the control planes do not agree on one pod CIDR and one service CIDR, stop: see the rule at the end of this section.

2. On every control plane, check that both CIDR flags match the CIDRs from step 1:

   sudo grep -h -e --service-cluster-ip-range -e --cluster-cidr \
     /var/lib/stratum/config/manifests/kube-apiserver.yaml \
     /var/lib/stratum/config/manifests/kube-controller-manager.yaml
   

3. On each control plane whose flags differ, set both CIDRs from step 1 in its /etc/stratum/config.yaml, or the file its unit passes with --config. The older build applies spec.network on every start, so copying netconfig alone does not survive a restart:

   spec:
     network:
       podCIDR: <cluster pod CIDR>
       serviceCIDR: <cluster service CIDR>
   

   Then restart Stratum with sudo systemctl restart stratum and repeat step 2 on that node. After the upgrade these values match the recorded CIDRs, so you can leave them in place. They produce no warning.

4. Check that every address in each Service's .spec.clusterIPs is inside the service CIDR of its IP family. Skip ExternalName Services and headless Services, whose cluster IP is None:

   kubectl get svc -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,TYPE:.spec.type,IPS:.spec.clusterIPs
   

5. On every node, check that the kubelet's clusterDNS is the kube-dns Service's ClusterIP:

   sudo grep clusterDNS /var/lib/stratum/config/kubelet.conf
   

6. On every node, pick an existing pod whose image has nslookup. It must not use the host network, must have dnsPolicy: ClusterFirst, and must have no custom dnsConfig. Other pods can legitimately use a different resolver. For a suitable pod this prints ClusterFirst (or false ClusterFirst) and nothing after it:

   kubectl get pod -n <namespace> <pod> -o jsonpath='{.spec.hostNetwork} {.spec.dnsPolicy} {.spec.dnsConfig}{"\n"}'
   

   Check that its resolver is the kube-dns ClusterIP and that a lookup succeeds:

   kubectl exec -n <namespace> <pod> -- cat /etc/resolv.conf
   kubectl exec -n <namespace> <pod> -- nslookup kubernetes.default
   

If the sources in step 1 disagree, or any check still fails after step 3, on a control plane or a worker, stop. The cluster needs a separate recovery before you upgrade it, and this page does not cover that recovery.

See Network configuration for how the CIDRs are recorded.

Clusters whose config sets clusterDomain

Do this check before you upgrade any cluster from a build that predates custom-domain support. Those builds accepted spec.network.clusterDomain and ignored it, so every cluster they initialized runs on cluster.local, whatever its config file says. Unlike a CIDR that differs from the recorded one, which the new build ignores with a warning, a different domain is refused: stratum up does not start a control plane whose config file sets a domain other than the one the cluster runs on:

spec.network.clusterDomain "corp.internal" does not match this cluster's domain "cluster.local" recorded in /var/lib/stratum/pki/netconfig: the cluster domain cannot be changed after init; remove the field or set it to "cluster.local"

Before you upgrade, check the config file on every control plane (by default /etc/stratum/config.yaml). If it sets clusterDomain to anything other than cluster.local, remove the line or set it to cluster.local. The cluster keeps running on cluster.local either way: the domain can't be changed on an existing cluster. Workers need no change: a worker takes the domain from the --cluster-domain join flag, never from its config file.

Images loaded onto a node by hand

Do this check before you upgrade a cluster from a build that predates the kubelet pull-credential policy. Those builds left the kubelet on upstream's default, which never checks an image the kubelet did not pull. The new build checks every image outside the payload registry's namespace, as described in Payload images and kubelet's pull-credential check.

After the upgrade, a pod whose image you loaded onto the node by hand, for example with ctr images import, fails the next time its container starts on an airgapped node: with ErrImagePull if its imagePullPolicy is IfNotPresent, or ErrImageNeverPull if it is Never. Before you upgrade, push those images to a registry the nodes can reach, through a registry mirror if they are airgapped, so the kubelet pulls them itself.

SELinux confinement

Do this check before you upgrade a cluster on Rocky Linux, RHEL or RLC from a build that predates SELinux labelling. On a node whose host runs SELinux and has the policy packages (see Required packages), the first start of the new build:

1. registers Stratum's SELinux file contexts and relabels /var/lib/stratum, /run/stratum and /opt/local-path-provisioner once. On a node with many images this takes a while; it runs before containerd and the kubelet start again, within the three-minute window the supervisor restart allows;
2. turns on containerd's SELinux labelling and restarts containerd. Running containers keep running, with the labels they started with.

From then on every new pod runs as container_t with its own categories, unless it is privileged (see Containers). Pods that were already running stay unconfined until they are recreated: once every node runs the new build, restart them, for example with kubectl rollout restart. A container process the node started before then shows a label without categories, such as system_u:system_r:spc_t:s0, in ps -eZ, where one started since shows container_t with its own categories. sudo stratum hardening check counts them under selinux.4.

A non-privileged pod that mounts a hostPath it did not get from Stratum is denied it the next time it is recreated, unless the directory is labelled container_file_t. Find such pods before you upgrade:

kubectl get pods -A -o json | jq -r '.items[] | select([.spec.containers[].securityContext.privileged] | any | not) | select([.spec.volumes[]?.hostPath] | any) | .metadata.namespace + "/" + .metadata.name'

Label their directories, or set spec.selinux: false on the nodes that run them before you upgrade.

Rolling back. stratum rollback to a build whose control-plane manifests predate running as spc_t turns containerd's SELinux labelling off before it restores them, so the restored control plane is not confined out of its own data; containers it starts are then unconfined. Upgrading to such a build with stratum upgrade --from fails at the etcd gate and rolls back; set spec.selinux: false and restart Stratum first if you need to go back that far.

Nodes that finalized a CA rotation on an earlier build

Do this check before you upgrade a cluster that ran stratum certs rotate-ca on a build that verified kubelet serving certificates only under --cis. That rotation did not renew bootstrap-issued kubelet serving certificates, so a worker, or a control plane of a CIS cluster, can still serve one signed by the CA the rotation dropped. The new build's API servers verify kubelet serving certificates in every profile, so kubectl logs, exec and port-forward to such a node fail from the first control-plane upgrade until that node is upgraded.

upgrade --cluster probes every kubelet before it starts, --dry-run included, and names the nodes whose serving certificate the cluster CA does not sign, and any node it could not check. It warns rather than refuses, because the upgrade is what repairs them: when an upgraded node's kubelet starts, it moves the untrusted certificate aside and requests a new one, which the operator approves. A node whose supervisor is not restarted by the upgrade keeps the old certificate until Stratum next restarts there (sudo systemctl restart stratum).

Clusters whose certificates carry system:masters

Releases before the admin.conf and super-admin.conf split issued admin, kube-vip and apiserver-kubelet-client in system:masters. After the upgrade, each control plane's first start of the new release creates the bindings their new certificates need, then reissues every leaf and restarts its control-plane components, one control plane at a time once the cluster upgrade has released the maintenance lease. Expect one more restart of each control plane's components shortly after the upgrade finishes.

Copies of admin.conf taken off a control plane before the upgrade keep system:masters until they expire. Replace them with copies of the new admin.conf, and rotate the cluster CA if the old ones must stop working sooner. See Upgraded clusters and system:masters.

HA join tokens and the shared certs

This applies when you upgrade an HA cluster from a build that gave every join token read access to stratum-certs, the encrypted shared CA material. The new build lets only control-plane tokens read it.

During the upgrade, mint control-plane tokens only on an upgraded controller. One minted by the earlier release is not in the control-plane bootstrap group, and its join fails with Forbidden once an upgraded controller has narrowed the grant.

After the last controller is upgraded:

1. Narrow the grant by minting any token on an upgraded controller. Until then, worker tokens can still read an existing stratum-certs. The next real join token does this, and so does a short-lived worker token you never hand out: sudo stratum token create --ttl 1h.

2. Check that a worker token is refused (no) and a control-plane token is not (yes). The second check prints no on a cluster that has never uploaded the shared certs, since no grant exists yet.

   sudo stratum kubectl auth can-i get secret/stratum-certs -n kube-system --as=system:bootstrap:probe --as-group=system:bootstrappers --as-group=system:bootstrappers:stratum:nodes
   sudo stratum kubectl auth can-i get secret/stratum-certs -n kube-system --as=system:bootstrap:probe --as-group=system:bootstrappers --as-group=system:bootstrappers:stratum:nodes --as-group=system:bootstrappers:stratum:control-planes
   

3. If sudo stratum kubectl -n kube-system get secret stratum-certs finds the Secret, re-key it with sudo stratum certs rotate --rekey-shared --yes so a copy read from now on is useless with the old key. Do this before you hand out a new certificate-key file: re-keying invalidates any key file already copied, so a join in progress needs a fresh --control-plane token. Skip it when there is no Secret, since re-keying would create one with no owner that is never garbage-collected.

None of these steps protects ciphertext a worker token already read together with the certificate-key that decrypts it. If you do not trust the worker tokens and nodes from before the upgrade, rotate the CA with sudo stratum certs rotate-ca --cluster --yes. That replaces the cluster CA, the front-proxy CA and the ServiceAccount signing key. It does not replace the etcd CA key, or on a --cis cluster the at-rest encryption key, which the same Secret carries; Stratum has no rotation for either today, so a cluster whose shared certs and certificate-key may both have leaked should treat those two keys as compromised.

Rolling back. stratum rollback does not restore the old grant, and the earlier release does not widen it again. Its control-plane tokens are not in the control-plane bootstrap group, so their joins fail with Forbidden. Once every controller runs the earlier release, delete the binding, and the next stratum token create --control-plane recreates it the way that release expects. This gives worker tokens read access to stratum-certs again, as before the upgrade.

sudo stratum kubectl -n kube-system delete rolebinding stratum:certs-reader

Node client certificate approval

Earlier releases bound stratum-csr-autoapprove to every worker join token, so the controller-manager approved any node client certificate a token asked for, including one naming a node that already existed. This release keeps that binding with no subjects and has the operator approve a worker's first client certificate only for a node name no other node holds. Every upgraded controller empties the binding when it starts, whether or not it runs an operator.

During the upgrade, mint join tokens on an upgraded controller and join new workers once the operator's leader is upgraded. Until then a worker's certificate request stays pending; approve it by hand as Installation describes.

Rolling back. stratum rollback leaves the binding empty, and the earlier release does not grant it again, because it treats an existing binding as already set up. New workers then wait for you to approve their certificate requests by hand. To return to the earlier behaviour instead, restore the subject knowingly. This gives every worker join token the power to claim any node name again, including an existing node's:

sudo stratum kubectl patch clusterrolebinding stratum-csr-autoapprove --type=json -p '[{"op":"add","path":"/subjects","value":[{"kind":"Group","apiGroup":"rbac.authorization.k8s.io","name":"system:bootstrappers:stratum:nodes"}]}]'

The rules it enforces before touching anything

Preflight refuses rather than proceeding when:

• the target is not valid vMAJOR.MINOR.PATCH
• the change crosses a major version
• the target is a downgrade — compared as a whole version, so a patch downgrade is caught too, not just a minor one
• the target skips a minor; Kubernetes supports one minor at a time
• the control plane is not quorum-healthy — a majority of controllers Ready, and, while any controller is NotReady, still a majority with one more drained. Three controllers with one NotReady are refused: draining a second would leave etcd without quorum. Bring it back, or upgrade it on its own with stratum upgrade --from, then run the cluster upgrade.

The skew checks look at the lowest node version and the downgrade check at the highest, because those gates point in opposite directions: every node converges on the target, so the laggard is what would jump several minors, while going below the leader is what constitutes a downgrade.

Rollback

sudo stratum rollback

Restores the previous binaries and restarts every host process on them. When the backup set restores host binaries, it needs only the stratum supervisor to be running: a node whose kubelet or containerd is crash-looping on an upgrade's new binaries is exactly what it recovers. A rollback that restores no host binary — an --etcd-snapshot-only one, or a set from an upgrade interrupted before its host binaries were swapped — has nothing to recover them with, and still needs every host process running. stratum upgrade refuses on a node with one down, and suggests stratum rollback when an interrupted upgrade left a set that can recover it: roll back, then re-run the upgrade. The per-node upgrade path also rolls back on its own when the post-upgrade health gate fails, so an unattended upgrade that goes wrong does not leave a half-upgraded node.

A cluster initialized with a non-default spec.network.clusterDomain can't be rolled back to a release that predates custom-domain support: that release can't read or keep the domain this cluster records. A cluster on the default cluster.local rolls back normally.

stratum rollback --dry-run prints what it would restore and changes nothing.

What an upgrade leaves behind

Before an upgrade replaces a file, it keeps the old one beside it as <file>.bak-<stamp>, with a UTC stamp in the form YYYYMMDD-HHMMSS. The stratum binary's backup is in /usr/local/bin, host binaries' backups are in /var/lib/stratum/bin, and the control-plane manifests are kept in /var/lib/stratum/upgrade/manifests-bak-<stamp>. All files with the same stamp make one backup set, and stratum rollback restores the newest one. After each successful upgrade, only the two newest sets are kept.

From its first change until the node passes the health gate, an upgrade records itself in /var/lib/stratum/upgrade/in-progress.json, with the stamp of its backup set and its target version. This covers the binary swap and the restarts, the part a rollback can undo. A node that has this file had an upgrade interrupted. stratum upgrade to the same target finishes it, and stratum rollback undoes it. If the file cannot be read, both commands refuse and name it. Check the *.bak-<stamp> backups, then delete the file by hand.

The upgrade removes the file once the health gate passes, before it rewrites the host-process configs and arguments. A failure in that last step leaves no file: a rerun treats the node as already at the target, and stratum rollback does not restore those configs. So a missing file does not prove that no upgrade was interrupted. Check the stratum upgrade output for an error after the health gate.

Rolling back across an etcd minor version

An upgrade can move etcd to a newer minor version. The newer etcd moves its data to its own storage version as soon as it runs, and the older etcd refuses to start on that data. So before it restores the older etcd, a rollback compares etcd's storage version with the version it restores. The rollback an upgrade runs on its own when it fails does the same.

• If the storage version is not above the restored etcd's, the rollback goes ahead as usual. This is the case when the upgrade did not move etcd, or when the newer etcd never ran on this node's data.

• On a single control plane, the rollback first runs etcd's own downgrade and waits up to two minutes for the storage version to come down, before it restores anything. Every write made since the upgrade is kept. --dry-run shows the downgrade it would run.

• On a cluster with more than one control plane, the rollback refuses and changes nothing. An etcd downgrade is cluster-wide, and rolling back one node can't do it on its own. Its error names the version it restores, as <major>.<minor>. Downgrade etcd to that version first, from any one control plane, with VERSION set to it:

  sudo stratum etcdctl downgrade enable "$VERSION"
  

  Wait until every member shows that version under STORAGE VERSION:

  sudo stratum etcdctl endpoint status --cluster -w table
  

  Then run sudo stratum rollback on each control plane, one at a time. etcd ends the downgrade on its own once every member runs the older version. To stay on the newer version instead, run sudo stratum etcdctl downgrade cancel.

• If etcd does not answer, the rollback stops it and reads the storage version from the data on disk, keeping etcd's manifest at /var/lib/stratum/upgrade/etcd-manifest-stopped.yaml while it is stopped. If the data is not newer than the etcd it restores, etcd stays stopped until the restore puts the older manifest in place. If the data is newer, or cannot be read, the rollback puts etcd's manifest back and refuses: only a running etcd can downgrade its data. Get etcd answering on its current version, then run the rollback again. If the rollback fails while etcd is stopped, its error names the kept manifest; running stratum rollback again uses it.

Restoring etcd with a rollback

sudo stratum rollback --etcd-snapshot /var/backups/etcd/pre-upgrade-<stamp>.db --yes

--etcd-snapshot <file> also restores etcd from that snapshot after the binaries are rolled back. Use it when an upgrade changed data in etcd that you want to undo, with the pre-upgrade snapshot taken before that upgrade. The flag stops the API server and etcd, so it needs --yes.

It works only on a single control plane. A restore resets etcd to a new one-member cluster, which would break a multi-member cluster's quorum, so rollback refuses when etcd has more than one member, or when it cannot confirm the member count. For an HA cluster, use snapshot restore and rejoin instead.

The node health gate

After the host binaries are swapped, the upgrade waits for the node to pass a health gate. stratum rollback uses the same gate. If the gate fails during an upgrade, the node rolls back on its own. The gate passes only when all of these hold:

• every host process is running under the supervisor, and none is Foreign
• the containerd answering the CRI socket reports the version of the containerd installed on disk
• on a controller, the API server is healthy
• on every node, the kubelet version in the node's nodeInfo matches the kubelet installed on disk

In an upgrade, each version check waits up to 30 seconds in total, including reading the installed version. In stratum rollback, the whole gate shares one 60-second window, and the wait for host processes counts against it: each check gets what is left, and the gate fails when the window is used up. The gate reads nodeInfo with the node's own kubelet credential, which the Kubernetes Node authorizer allows to read that node's Node object, so workers are checked the same way controllers are.

A gate that fails on containerd's version names both versions:

containerd <installed version> is installed but the runtime serving /run/stratum/containerd/containerd.sock reports <running version> — another containerd, an orphan from an earlier run, holds the socket; stop it so the supervisor can start the installed one

That message means a different containerd, one the supervisor did not start, still holds the socket. Remove it as described in A containerd Stratum did not start, then run the upgrade again.

A gate that fails on kubelet's version reports kubelet <installed> is installed but node <name> reports kubelet <running>.

The supervisor restart

Replacing /usr/local/bin/stratum does not replace the running stratum up, the supervisor that starts containerd, kubelet and the operator. It keeps running the previous build, and gives everything it starts that build's arguments and paths, until it restarts. So the last step of a successful upgrade, and of stratum rollback, restarts the supervisor on the installed stratum and then runs the health gate again. The supervisor stops containerd, kubelet and the operator and starts them again; running containers keep running.

• A supervisor recent enough to support it restarts itself in place, keeping its process ID, whether systemd or a shell started it.
• An older supervisor, the first time a node is upgraded from a release without this step, is restarted with sudo systemctl restart stratum when systemd runs it. When it was started by hand, the upgrade warns and leaves it running: stop it (its containers keep running) and start it again without the join flags, with sudo stratum up --worker on a worker or sudo stratum up --config <path> on a controller (adding --cis if it was started with it). If systemd cannot say whether it runs the supervisor, or the unit's ExecStart runs a binary other than /usr/local/bin/stratum, the upgrade reports that as an error and restarts nothing.
• A rerun of stratum upgrade that finds the node already at its target still restarts a supervisor that is not running the installed stratum, and stratum upgrade --check reports such a node as needing the upgrade, so a cluster upgrade rerun picks it up.

If the supervisor does not come back on the installed stratum with every process running within three minutes, or the gate after it fails, the upgrade reports it and leaves the installed build in place: it does not roll back.

sudo stratum version --components and the MCP cluster_status tool report a supervisor that is not running the installed stratum. Both need root to ask the supervisor, so without it stratum version says nothing about it.

Staging the upgrade binary

For a cluster upgrade, the upgrade binary must be staged at the same path on every node the rollout will touch (there is no in-cluster transport for it). Each per-node Job execs that staged binary directly, so the path is the one place guaranteed to hold the new version and its payload on every node — which is why the flag is required rather than inferred.