Snapshot & Restore

Taking snapshots

sudo stratum snapshot save                 # now
sudo stratum snapshot schedule             # the recurring timer
sudo stratum snapshot list

Snapshots are etcd snapshots taken through the managed datastore, so they are consistent rather than a copy of files on disk.

The scheduled path runs from the supervisor on each control plane. If object storage is configured, snapshots are shipped there as well as kept locally.

Where snapshots go

Snapshots are written to the local snapshot directory, /var/backups/etcd unless spec.snapshots.dir sets another, with UTC timestamps in the form YYYYMMDD-HHMMSS:

<host> is the short hostname, so control planes sharing one S3 bucket never overwrite or prune each other's snapshots. snapshot save --name <file> sets the file name. It is joined onto the snapshot directory as given, so use a plain file name: a path such as ../<file> writes outside the directory.

Retention goes by file name. After every snapshot save, scheduled snapshot and snapshot prune, Stratum keeps the newest retain snapshots for this host and deletes older ones that match:

It does not matter whether --name was used. A --name that matches the pattern is pruned like any other snapshot. Anything that does not match, including pre-upgrade-*.db and other custom names, stays until you delete it.

snapshot list shows local files only when their names end in .db. It lists S3 objects whatever their names.

The scheduled snapshot, snapshot save, snapshot list and snapshot prune all use the snapshot directory, and so do the MCP snapshot tools; see Where retention and the upload target come from. Pre-upgrade snapshots stay in /var/backups/etcd even when spec.snapshots.dir is set, so snapshot list does not show them then.

Schedule settings

How often the scheduled snapshot runs, and where snapshots are kept, is set under spec.snapshots in /etc/stratum/config.yaml:

spec:
  snapshots:
    intervalHours: 12       # the default
    disabled: false         # true stops the scheduled snapshot
    dir: /var/backups/etcd  # the default; the local snapshot directory, an absolute path

Restart the node for changes to affect the scheduled loop. snapshot save, list and prune read the directory on each invocation. The schedule runs on control planes only.

Retention (default 5 per control plane) and the S3 target come from snapshot.conf or spec.snapshots; see Where retention and the upload target come from.

Where retention and the upload target come from

The scheduled snapshot loop, stratum snapshot save, list, prune, restore s3://… and schedule, and the MCP snapshot tools all read the retention count and S3 target from one of two sources:

• If /etc/stratum/snapshot.conf exists, its RETAIN and S3_* values apply. The spec.snapshots.retain and spec.snapshots.s3 values in config.yaml are ignored, even if the file leaves a value unset.
• If snapshot.conf does not exist, spec.snapshots.retain and spec.snapshots.s3 from config.yaml apply. If config.yaml cannot be read or parsed, or you passed stratum up --config a file that has since been deleted, the built-in defaults apply: retain 5 and no S3 target. The command prints a warning to stderr that names the file and the error.

Here config.yaml means the file the controller was started with: /etc/stratum/config.yaml, or the path you passed to stratum up --config. The controller records that path each time it starts, and the CLI and MCP tools read the recorded path. Until the controller has restarted on this release, they read /etc/stratum/config.yaml even if the controller uses another file. They also read /etc/stratum/config.yaml if the node state that holds the recorded path cannot be read or parsed, and then print a warning to stderr, because the retention they use may not be the one the controller uses. The scheduled loop always uses the controller's own file.

To configure snapshots from config.yaml:

# /etc/stratum/config.yaml
spec:
  snapshots:
    intervalHours: 12             # the default
    retain: 5                     # the default
    dir: /var/backups/etcd        # the default
    s3:
      bucket: etcd-backups
      prefix: prod/               # optional
      region: us-east-1           # optional
      endpoint: https://minio.example.com:9000   # optional; omit for AWS
      pathStyle: true             # optional; MinIO, Ceph and most non-AWS endpoints need it

Only stratum snapshot schedule and the MCP snapshot_schedule tool create snapshot.conf. Both start from the settings already in effect, so on a node without the file they keep the S3 target from config.yaml. The MCP tool changes only retention. With the CLI, --s3-bucket replaces the whole S3 target. Once the file exists, change retention or the target by running stratum snapshot schedule again or by editing the file.

spec.snapshots.dir sets the local snapshot directory. The default is /var/backups/etcd. The scheduled loop, stratum snapshot save, list and prune all use it, read from the same config.yaml as above. If that file cannot be read, they use /var/backups/etcd.

dir must be an absolute path. A relative value, such as backups or ./backups, is ignored and every command, the scheduled loop included, uses /var/backups/etcd. When the stratum snapshot commands or the MCP snapshot and rollback tools read the directory, they print warning: spec.snapshots.dir "<dir>" is not absolute; using /var/backups/etcd to stderr. Earlier releases resolved a relative value against the working directory of whichever process read it. If you set one, replace it with the absolute path of the directory your snapshots are in, or snapshot list and retention no longer see them.

A dir with a .. element, such as /srv/snaps/../x, is ignored the same way, with warning: spec.snapshots.dir "<dir>" contains a ".." element; using /var/backups/etcd. Write the path without ... A . element, repeated slashes and a trailing slash are cleaned without a warning, so /srv/./snaps, //srv//snaps and /srv/snaps/ all mean /srv/snaps.

snapshot restore resolves a NAME in the snapshot directory and downloads an s3:// snapshot into it, and the MCP snapshot_restore and rollback tools resolve a bare name in it. The MCP snapshot_list, snapshot_save and snapshot_prune tools use the same directory, so they see the same snapshots as the stratum snapshot commands.

config.yaml has no credential fields. Wherever the target comes from, the node reads S3 credentials as described below.

Credentials for remote storage

stratum snapshot schedule --s3-bucket <bucket> configures the upload target. It never writes credentials from your environment to disk, but it keeps any S3_ACCESS_KEY or S3_SECRET_KEY already in /etc/stratum/snapshot.conf. On every scheduled snapshot, the running node resolves each key separately, in this order:

The first one that is set wins. If snapshot.conf already holds a key, the matching AWS_* variable is ignored, and uploads keep working after a reboot without stratum.env. stratum snapshot schedule prints a note when it finds credentials in snapshot.conf. To move them off disk, set the STRATUM_S3_* variables in stratum.env (below), remove S3_ACCESS_KEY and S3_SECRET_KEY from snapshot.conf, and check that uploads still work.

When systemd restarts the node, for example after a reboot, the new process has none of your shell's environment. If your credentials come only from the environment, local snapshots keep working but uploads to S3 stop. To keep uploads working, put the variables in /etc/stratum/stratum.env, owned by root with mode 0600:

# /etc/stratum/stratum.env
STRATUM_S3_ACCESS_KEY=<access-key>
STRATUM_S3_SECRET_KEY=<secret-key>

The unit loads this file each time systemd starts it. See Environment for the unit for how to create it.

Restoring

sudo stratum snapshot list
sudo stratum snapshot restore etcd-snapshot-<host>-<stamp>.db --yes

snapshot restore takes a NAME as snapshot list prints it and finds it in the local snapshot directory (spec.snapshots.dir, by default /var/backups/etcd). It also takes a path to a file, or s3://<bucket>/<key>. An argument with no / in it is always read as a name in the snapshot directory, so to restore a file in the current directory, pass ./NAME. An absolute path is used as given. A relative path is prefixed with the current directory and is not otherwise changed, so .. after a symlink still follows the symlink.

An s3:// snapshot always comes from the configured S3 target (see Where retention and the upload target come from), because that target's credentials are the only ones the node has. The URL must name the target's bucket. A key with a directory part must be exactly the key the target fetches, <prefix>/<name>: with prefix snaps, s3://<bucket>/snaps/<name> works. The key is compared as written, so snaps/./<name>, snaps//<name> and other/../<name> are refused. A bare key, s3://<bucket>/<name>, is read as <name> under the target's prefix. If the configured prefix starts with / or ends in more than one /, the key is that prefix as written: with prefix /snaps only s3://<bucket>//snaps/<name> matches, and with snaps// only s3://<bucket>/snaps//<name>. s3://<bucket>/snaps/<name> is refused for both. Pass the bare key, or set the prefix as snaps.

Restore refuses a URL that names another bucket, another prefix, or no object, and says which bucket or key the configured target would use. It refuses before it creates the snapshot directory or downloads anything, so nothing on the node changes.

An s3:// snapshot is downloaded into the snapshot directory under its key's base name, replacing any local file of that name, and restored from there. Before the download, restore opens the snapshot directory one path component at a time, from /, without following symlinks. It creates each missing component with mode 0700. It refuses the directory, and nothing is downloaded, if:

• any component of the path is a symlink, such as link in /var/backups/link/etcd. The error names the component. This also refuses a directory reached through a symlinked parent, such as /var/backups linked to another disk: set spec.snapshots.dir to the real path instead.
• any component is not a directory.
• the directory is writable by group or others, or is owned by another user.

Otherwise it leaves the directory's mode as it is. Restore writes through the directory it opened and checked, so renaming the directory or replacing it with a symlink during the download does not move where the file lands. These rules apply to snapshot restore s3://… only. snapshot save, list, prune and the scheduled loop still follow symlinks in spec.snapshots.dir.

The file is written with mode 0600 to a temporary name, .<name>.<digits>, and renamed into place only when the download completes, so a failed download leaves an existing local file of that name untouched. A download killed partway, by SIGKILL or a power loss, leaves that temporary file behind, and it holds a partial copy of the snapshot. The next snapshot restore s3://… of the same name removes such files that are regular files and were last written more than 15 minutes ago. It leaves younger ones, which may belong to a download still running, and every other file. If it cannot remove one, it prints warning: removing stale download temp files in <dir>: <error> and carries on with the restore. Temporary files for other names stay until those names are restored; you can delete them by hand.

The MCP snapshot_restore tool resolves a bare name in the same directory. It refuses any other relative path, so pass an absolute path for a file elsewhere.

Restore rebuilds this node's datastore from the snapshot and restarts the control plane. It is a destructive, deliberate operation, so it needs --yes.

• The current datastore is not deleted. It is moved to /var/lib/stratum/etcd.pre-restore-<UTC stamp> first.
• If the restore fails before etcd is running on the restored data, it rolls back: the original datastore goes back in place, and etcd and the API server are restarted on it. Whatever the failed attempt wrote is kept at /var/lib/stratum/etcd.failed-restore-<UTC stamp>. The error says what was rolled back.
• If the original datastore cannot be put back, the control plane is left stopped rather than started on the wrong data. The error says where the original datastore and the failed restore are.
• A node that has lost its datastore entirely can be restored. If that restore fails, there is nothing to roll back to, so the control plane stays stopped.
• A snapshot path that is a directory, such as . or .., is refused before anything is stopped.
• A node staged by an older release has no etcdutl, which restore needs. Restore refuses before stopping anything and asks you to run stratum stage.

What the restore restarts

A restore takes etcd's resourceVersions back to the snapshot's. A component that was watching the cluster before the restore has already seen later resourceVersions, so it is never told that its view is stale. Without a restart, kubelet keeps running the pods it started after the snapshot and never starts the restored ones, Cilium can drop the backend of the kubernetes Service, so pods that reach the API through 10.96.0.1 fail with no route to host, CoreDNS can answer headless-Service names from post-snapshot endpoints, and the local-path provisioner can leave a restored PersistentVolumeClaim unprovisioned.

The restore checks the health of this node's own API server (https://127.0.0.1:6443), not the VIP in admin.conf, which can be down before the other control planes have rejoined. A failed restore's rollback is checked the same way. The health check keeps probing for up to two minutes, and every kubectl call the restore makes runs under a deadline: a call that gets no answer is killed after 30 s and named as the one that failed, so a stalled API request cannot hold the recovery, or the steps printed after it, open. Once the API server is healthy on the restored data, the restore restarts these, in order, and prints one line for each:

1. kubelet on this node, through the supervisor. It goes first because the others are pods it has to recreate.

2. kube-controller-manager, kube-scheduler and, on a cluster with a VIP, kube-vip, on this node. Each is skipped on a node without its manifest. They are restarted from the manifests on disk, not re-rendered, so an upgrade rollback keeps the older images it restored.

3. The stratum operator, through the supervisor. It reconciles on an interval and refreshes the shared requester-signing key periodically. Restarting also discards its pre-restore reconcile state, re-registers its webhook endpoint, and reacquires its leader lease against the restored cluster. It runs on control planes only, and is skipped elsewhere.

4. The in-cluster workloads that watch the cluster, through this node's own API server rather than the VIP, which may still lead to a control plane that has the old data (rollout restart). Each is found by its namespace and skipped if the cluster does not run it:

   • the Cilium agents and operator, ds/cilium and deploy/cilium-operator in kube-system. Cilium builds its service table and endpoints from its watches.
   • CoreDNS, deploy/coredns in kube-system. It watches Services and EndpointSlices, so left on its pre-restore watch it answers headless-Service names from post-snapshot endpoints.
   • the local-path provisioner, deploy/local-path-provisioner in local-path-storage, which a cluster whose default storage is local-path runs. It watches PersistentVolumeClaims, so a restored claim stays unprovisioned until it restarts.
   • the dashboard, deploy/stratum-dashboard in stratum-dashboard, when the cluster runs it. It reaches the API with the cluster CA it read at start.

   Each target is always restarted, even if another restart fails. The restore then waits up to 60 s in total for all the rollouts that started, waiting on them at once; the 60 s also ends any of those calls that has not answered. They come last because a rollout needs the controller manager and scheduler to be running on current watches.

The restore does not restart cilium-envoy, which takes its configuration from a mounted ConfigMap and the Cilium agent and opens no API watch, or containerd. It never restarts workloads you run, or controllers and operators you installed: that would widen a destructive command to what it does not own. Third-party controllers are in step 5 below.

A rollout that does not finish in 60 s is a WARN line, not a failure: the restart was done and the rollout keeps going. Expect it on a cluster with workers. A rollout may not finish while a worker's kubelet still holds a pre-restore watch, and that is fixed by the steps the restore prints below, after it returns. The restore still succeeds and prints what is left to do, since none of it depends on those rollouts. Check one with sudo stratum kubectl --server https://127.0.0.1:6443 -n kube-system rollout status ds/cilium, or the other workload and namespace in its place.

If a component cannot be restarted at all, the restore still tries the rest, then fails with etcd restored, but not every component holding a pre-restore watch or start-up state was restarted, followed by the names of the ones that failed. The restored data stays in place: do not run the restore again. It prints the same list of what is left to do as a successful restore, preceded by how to restart the named components by hand: kubelet, the operator and the static pods by killing the process, which is then restarted, and the in-cluster workloads, one command per namespace, with

sudo stratum kubectl --server https://127.0.0.1:6443 -n kube-system rollout restart ds/cilium deploy/cilium-operator deploy/coredns
sudo stratum kubectl --server https://127.0.0.1:6443 -n local-path-storage rollout restart deploy/local-path-provisioner
sudo stratum kubectl --server https://127.0.0.1:6443 -n stratum-dashboard rollout restart deploy/stratum-dashboard

Run these on the seed node, with --server as shown: until the other control planes have rejoined, the VIP in admin.conf can be down or lead to a control plane that still holds the old data.

An upgrade rollback restores etcd the same way. There, a component that could not be restarted is a WARN line rather than an error, so that retrying the rollback does not restore etcd a second time. The rollback prints the list in After a restore after a clean restore and after that warning, and the MCP rollback result carries it too, without the items about rejoining and repeating the restarts: a rollback restores a single-member cluster, which has no other control planes.

After a restore

The restore restarts components on this node, and the Cilium, CoreDNS and local-path workloads in the cluster. It does not touch the other nodes. It prints what is left to do when it succeeds, and when it fails only because some components could not be restarted:

1. Rejoin every other control plane, one at a time, with stratum snapshot rejoin --force pointed at this node. Their etcd still holds the pre-restore cluster, which a rejoin refuses to discard without --force until that cluster has lost quorum.

2. Restart kubelet on every other node: workers, and each control plane once it has rejoined. Their kubelets also hold a pre-restore watch. Stratum has no command to restart kubelet yet; stopping the process makes the supervisor restart it:

   sudo pkill -x kubelet
   

3. Check that every Pod is Running with a current IP:

   sudo stratum kubectl get pods -A -o wide
   

4. If the cluster has other control planes, repeat the restarts once every one of them has rejoined. Cilium, CoreDNS, the operator, kubelet, kube-controller-manager and kube-scheduler may reach the API server through the VIP or a Service backed by every control plane. Until the other control planes have rejoined, the VIP can lead to one that still holds the old data, so what those components reloaded in the meantime can be stale. On each control plane, run these. Kubelet recreates the controller containers once they exit, and the supervisor restarts kubelet and the operator:

   sudo pkill -x kubelet
   sudo pkill -TERM -f '^([^[:space:]]*/)?[k]ube-controller-manager([[:space:]]|$)'
   sudo pkill -TERM -f '^([^[:space:]]*/)?[k]ube-scheduler([[:space:]]|$)'
   sudo pkill -TERM -f '^[^[:space:]]+ operator( [^[:space:]]+)* --kubeconfig (/var/lib/stratum/kubeconfig/super-admin\.conf|/var/lib/stratum/kubeconfig/admin\.conf)( [^[:space:]]+)* --manifests-dir /var/lib/stratum/manifests( [^[:space:]]+)* --helm-cache /var/lib/stratum/helm( [^[:space:]]+)* --leader-elect([[:space:]]|$)'
   

   The operator's pattern names its subcommand and Stratum's own flags, not its executable, because that can be named after a release artifact such as stratum-linux-amd64. It matches no other process, and not the pkill command itself.

   Then restart the in-cluster workloads once:

   sudo stratum kubectl -n kube-system rollout restart ds/cilium deploy/cilium-operator deploy/coredns
   sudo stratum kubectl -n local-path-storage rollout restart deploy/local-path-provisioner
   sudo stratum kubectl -n stratum-dashboard rollout restart deploy/stratum-dashboard
   

5. Restart the controllers and operators you installed yourself. cert-manager, an ingress or GitOps controller and a database operator hold watches too, and the restore does not restart them. Anything that watches the API, and keeps running across the restore, may be holding the pre-restore view. For each one, restart it and check its pods:

   sudo stratum kubectl -n <namespace> rollout restart deploy/<name>
   sudo stratum kubectl -n <namespace> rollout status deploy/<name>
   

   Use ds/<name> or sts/<name> for a DaemonSet or StatefulSet. Find candidates with sudo stratum kubectl get deploy,ds,sts -A. Until the other control planes have rejoined, add --server https://127.0.0.1:6443 so the command reaches the restored API server and not the VIP.

Cleaning up after a restore

sudo stratum snapshot prune-restores          # list the copies and their sizes
sudo stratum snapshot prune-restores --yes    # remove them

Losing the controller machine

Use this when a controller is gone with its disk and no other control plane is left: it was the only control plane, or every other control plane is also destroyed, or confirmed powered off and kept off. The replacement becomes the cluster's only control plane, and you add more control planes afterwards with a normal join.

If any other control plane is still running, or you cannot confirm its state, do not use this procedure. The cluster may still have quorum and data newer than any snapshot, and a restore would throw that data away. Treat the lost machine as a failed member of a running cluster, or contact support. Step 1 below checks this before anything is started.

stratum up also refuses to start a controller whose saved node state records that its etcd cluster has had more than one member when its etcd data is gone, because such a node could overwrite a cluster that still has quorum. You lift that refusal with --confirm-etcd-quorum-lost only after step 1. The check reads the restored node.yaml, so it is only as current as your backup of that file. See Back up again after adding control planes.

stratum snapshot save writes etcd's data and nothing else. snapshot restore expects the node's PKI to be in place already, and it reads the node's address from its saved node state. A replacement controller that starts without the old PKI generates a new cluster CA. The workers' kubelets and every kubeconfig you have handed out trust the old CA, so they cannot reach the new one. The node's own kubeconfigs are not a problem: stratum up rebuilds them from the PKI on every start.

What to back up

Copy these off the controller together with the snapshots, and again after anything that changes them, such as a certificate or CA rotation, a change to the cluster config, a change under /var/lib/stratum/manifests/, or enabling CIS.

You do not need /var/lib/stratum/config/manifests/. Those are the control-plane static pods, which stratum up generates.

Back up again after adding control planes

The first controller records that the cluster has grown within about a minute of seeing a second etcd member, and only while it runs a release that includes this check. It sets etcdHasPeers: true in its node.yaml. Its etcdState stays new and its etcdInitialCluster keeps one member, because those fields are etcd's own startup arguments. Until the record is set, a backup of that file looks like a cluster that had only one controller, so stratum up cannot protect a replacement restored from it.

After you add control planes, or after you upgrade a cluster that already has them, check the first controller:

sudo grep etcdHasPeers /var/lib/stratum/config/node.yaml

Once it shows etcdHasPeers: true, take a new backup of that controller and discard the older ones. No output means the record is not set yet.

For example, as root, create the archive under umask 077 so it is readable by root only from the moment it exists, encrypt it, then remove the unencrypted copy. Keep it outside /var/lib/stratum and /etc/stratum. Add your --config file to the list if it lives elsewhere.

The steps are chained with &&, so each one runs only if the one before it succeeded. The unencrypted archive is removed only after gpg has written the encrypted one. If any step fails, the chain stops there: fix the cause and run it again, and do not copy off a backup from a run that did not reach backup complete.

sudo -i
umask 077
backup=/root/stratum-node-$(hostname)-$(date -u +%Y%m%dT%H%M%SZ).tgz
tar -czpf "$backup" \
  /var/lib/stratum/pki \
  /var/lib/stratum/config/node.yaml \
  /var/lib/stratum/manifests \
  /var/lib/stratum/kubeconfig \
  /etc/stratum \
  /etc/systemd/system/stratum.service \
  && gpg --symmetric --cipher-algo AES256 --output "$backup.gpg" "$backup" \
  && shred -u "$backup" \
  && echo "backup complete: $backup.gpg"
exit

Copy only the .gpg file off the host. Keep the passphrase where you keep other recovery secrets, not next to the archive.

Replacing the controller

Do these steps in order. The files must be in place before the replacement's first start, because that start is when it would otherwise create a new CA.

The commands in this procedure call Stratum as /usr/local/bin/stratum. On a stock Rocky host, sudo and sudo -i leave /usr/local/bin off the PATH, so sudo stratum fails with command not found.

1. If the cluster had other control planes, confirm that every one of them is gone before you start anything on the replacement. Each one must be destroyed, or confirmed powered off through its console or cloud provider, and kept off. Not answering on the network does not count as gone.

   Do not start or restore while another control plane may be running

   The restored node state still names the old etcd members, so the replacement's first start can contact any of them that is still running. A restore then resets the cluster to the snapshot. If another control plane is still running, the cluster may have quorum and writes newer than the snapshot, and nothing you can run from the replacement proves it does not: a timeout from etcdctl looks the same whether quorum is lost or the request never reached etcd. stratum up refuses some of these starts on its own, but not one restored from a backup that predates the cluster's growth.

   If any other control plane is still running, or you cannot confirm its state, stop, and do not continue. Treat the lost machine as a failed member of a running cluster, or contact support.

   Take the list of control planes from your own records: your inventory, your cloud provider's console, or a node list saved while the cluster was running. Do not go by the backed-up node.yaml. It can name only the first controller, or only the members that existed when this one joined, so it can make a cluster with several control planes look like one with a single controller. If you cannot say for certain whether the cluster had other control planes, stop.

   Skip this step only if your records show that the cluster had one controller.

2. Bring up the replacement machine with the same hostname and the same IP address as the old controller. The certificates and the saved node state name that address, and the workers connect to it.

3. Install the same Stratum release the old controller ran, at /usr/local/bin/stratum. Do not start it yet.

4. Copy the encrypted backup to /root on the replacement. Then decrypt it and unpack it at /, keeping ownership and modes. Give the full path of the .gpg file, because sudo -i starts the shell in /root:

   sudo -i
   umask 077
   enc=/root/stratum-node-<host>-<timestamp>.tgz.gpg
   plain=/root/stratum-node-restore.tgz
   rm -f "$plain" \
     && gpg --output "$plain" --decrypt "$enc" \
     && tar -xzpf "$plain" -C / \
     && shred -u "$plain" \
     && echo "files restored"
   exit
   

   The rm -f removes any archive left by an earlier attempt, so a failed decryption cannot fall through to stale files. If the chain does not reach files restored, stop. Do not start Stratum until it does: a start without the restored files creates a new CA.

   If the old controller ran with a --config file outside /etc/stratum, check that it is back at the same path now. A missing config file is not an error: stratum up starts with the defaults instead, which can replace the cluster's CIDRs and VIP.

5. Start the node with the restored unit:

   sudo systemctl enable --now stratum
   sudo journalctl -u stratum -n 20 --no-pager
   

   The output names the address it uses, for example node IP 10.60.1.10 (saved node state). What happens next depends on what the restored node.yaml records:

   • A single-member cluster (the cluster had one controller, or the backup predates its growth): the control plane comes up on an empty datastore. Go on to step 6.
   • A cluster that had more than one member (a control plane that joined later, or a first controller whose backup shows etcdHasPeers: true): startup refuses with refusing to start an empty member, which could overwrite a cluster whose other members still hold quorum.

   The refusal is expected here. Step 1 confirmed that no other control plane is running, so stop the unit and start the node once by hand with --confirm-etcd-quorum-lost. Use the flags from the unit's ExecStart, shown by systemctl cat stratum, for example:

   sudo systemctl stop stratum
   sudo /usr/local/bin/stratum up --config /etc/stratum/config.yaml --confirm-etcd-quorum-lost
   

   This runs in the foreground, so do step 6 in a second terminal. What the node does until the restore depends on its saved state:

   • A control plane that joined later (etcdState: existing): etcd waits for its old peers, which are gone, so the API stays down and kubectl fails with errors such as no route to host.
   • The first controller (etcdState: new with etcdHasPeers: true): etcd starts a new, empty datastore and the API comes up on it.

   Either way, go on to step 6. The flag is never written to the unit. Once step 7 passes, stop the foreground process with Ctrl-C and run sudo systemctl start stratum: the restored datastore now has a WAL, so the unit starts without the flag.

   On a CIS cluster, if /var/lib/stratum/pki/encryption-config.yaml is missing, startup refuses with this node has saved node state ... but no etcd data. Restore the original key from the backup. Do not work around the check: the restored Secrets are encrypted with that key, and a new one cannot read them.

6. Put the snapshot on the node and restore it. /var/backups/etcd/ does not exist on a fresh node, so the commands create it first, readable by root only, because the snapshot holds the cluster's Secrets. They are chained with &&, so the restore runs only if the copy succeeded and never picks up an older file of the same name:

   sudo install -d -m 0700 /var/backups/etcd \
     && sudo install -m 0600 <snapshot>.db /var/backups/etcd/ \
     && sudo /usr/local/bin/stratum snapshot restore /var/backups/etcd/<snapshot>.db --yes
   

   If /var/backups/etcd already exists with looser permissions, run sudo chmod 700 /var/backups/etcd first.

   If /etc/stratum/snapshot.conf came back in step 4 and its S3 target can be reached, you can pass s3://<bucket>/<key> instead, with that target's bucket and prefix.

   The restore restarts kubelet, kube-controller-manager, kube-scheduler, kube-vip, the stratum operator, Cilium, CoreDNS and the local-path provisioner, as described in What the restore restarts. It then lists what is left to do. Skip its first and fourth items, rejoining every other control plane (with stratum snapshot rejoin --force) and repeating the restarts after they have rejoined: none is left to rejoin, and step 8 adds new ones. Its other items, restarting kubelet, checking the Pods and restarting the controllers you installed yourself, are step 7.

7. Restart kubelet on every worker. Workers need no rejoin, but each kubelet still holds a watch from before the restore:

   sudo pkill -x kubelet
   

   Restart the controllers and operators you installed yourself, as After a restore describes in its step 5, since the restore does not.

   Then check the cluster from the replacement. The replacement and every worker should be listed and Ready, and every Pod Running with a current IP:

   sudo /usr/local/bin/stratum kubectl get nodes
   sudo /usr/local/bin/stratum kubectl get pods -A -o wide
   

   On a cluster that had other control planes, they stay listed as NotReady, because their Node objects are in the snapshot. They are no longer etcd members: the restore made the replacement the only member. Delete each one:

   sudo /usr/local/bin/stratum kubectl delete node <node-name>
   

8. To run more than one control plane again, add them with a normal join from freshly installed machines, as in Growing to three controllers. Mint the token on the replacement and use the control-plane join command it prints, not one written by hand:

   sudo /usr/local/bin/stratum token create --control-plane
   

   The printed command already carries --cis on a CIS cluster. Before you run it on a new machine, give that machine the same spec.vip in its /etc/stratum/config.yaml as the replacement, so its certificate covers the VIP. Run it as sudo /usr/local/bin/stratum up --join .... Do not start Stratum again on a machine from the old cluster with its old disk.

9. Once the cluster is verified, remove the copies the restore kept, as in Cleaning up after a restore.

If the replacement was started before the files were restored, it has already created a new CA, and restoring the files over a running node is not supported. Start the replacement over:

sudo /usr/local/bin/stratum uninstall --yes

Then start again from step 3.

Rejoining a member

sudo /usr/local/bin/stratum snapshot rejoin --server https://<seed-node-ip>:6443 --yes

That form goes ahead only when rejoin can confirm that this node's own etcd cluster has lost quorum, which is the exception. In the usual cases it refuses, and the same command with --force is what you run once you have checked that this node's data can be discarded (see below):

sudo /usr/local/bin/stratum snapshot rejoin --server https://<seed-node-ip>:6443 --yes --force

--server is the seed control plane's own address: the node the cluster is being rebuilt from, such as the one that ran snapshot restore. It is not the VIP. Rejoin talks to that node's etcd directly, and etcd does not listen on the VIP, so a VIP is refused.

Run it on each of the other control planes, never on the seed. Rejoin refuses a --server that is this node's own address, with or without --force, and says that this node is the seed. It checks this first, before it looks at anything else, because the seed's address would otherwise reach the seed's own etcd and, with --force, wipe the restored data the cluster is running on. On a dual-stack node, both halves of the node's address count as its own.

With --force, rejoin also refuses a --server that answers as this node's own etcd member, which catches this node reached by another name: a hostname, loopback or another interface. It asks the --server etcd for its member list, takes the member that answered, and refuses if one of that member's peer URLs is on one of this node's own addresses. It never reads this node's own etcd, so it does not matter whether that etcd is running, stopped, or restarted without quorum and never became ready. A --server that is a different member of the same cluster is not refused. That is the repair below of a diverged member against its healthy majority, where this node's data is what --force is there to discard. A re-run after a rejoin that joined this node but failed to start the apiserver is not refused either, and discards nothing: this node's etcd already serves as its member, so the re-run resumes without wiping. The cost is that a mistyped address of another member of the old cluster is not caught either. It wipes this node's data and adds it back to that cluster, and it does not touch the data on the seed you restored. Check --server against the node that ran snapshot restore. If the answering member cannot be found in the list, or the list cannot be read, or if its peer URLs are missing or not IP addresses, rejoin refuses. Nothing is changed when either check refuses.

This is the one to reach for when a controller's etcd has diverged or its data directory is gone, but the cluster is healthy. It re-adds this node as a member and lets it resynchronise from the surviving majority, rather than restoring an older point-in-time over a cluster that has moved on. In that case it needs --force, as described below.

etcd's argv is re-rendered from the persisted node state on every start, which is how membership changes take effect with a component restart rather than a full node restart.

After a restore, rejoin the other control planes one at a time. Each one is added as an etcd learner, a member with no vote, and promoted to a voter only once it has caught up with the seed. Until then the seed's quorum does not depend on it, so the seed keeps serving however long the new member takes to start. Rejoin waits for as long as the member keeps making progress. etcd answers nothing until it has applied the snapshot the seed sends it, so until then progress is its data directory changing on disk. Once it answers it can still be behind, and progress is its applied index advancing until the seed accepts its promotion. Rejoin gives up when there has been no progress for five minutes, and in any case after thirty minutes, so a member that never converges still ends the command. The two endings leave different states:

• No progress for five minutes: a learner has its etcd stopped, so the re-run replaces it. Its etcd is left running instead if the seed does not confirm it is still a learner, because a promote whose reply was lost may have made it a voter.
• Thirty minutes with progress: its etcd is left running, so the re-run resumes the wait.

Ctrl-C stops the wait; the member keeps catching up on its own, and re-running rejoin while its etcd is still running picks the wait back up.

Re-running rejoin is safe, including after a run that timed out or was interrupted. It does not start over. It looks at the seed's membership and at this node's etcd first:

Rejoin reads the seed's membership before it touches anything. If that read cannot be served because the cluster has lost quorum — which is the state the last row describes — rejoin falls back to a read that does not need one, and then allows itself only the "keeps everything and resumes" action. It never removes a member or wipes a data dir on the strength of a membership it could not confirm.

Before a first rejoin wipes anything, it checks whether this node's own etcd cluster has lost quorum. It reads the member list from this node's etcd and sorts the voting members into three groups:

• Answers in this cluster: this node, plus every peer that reports the same cluster ID.
• Answers from another cluster: for example a restored seed on an old member's address. These are not counted.
• Cannot be reached: a timeout, a TLS failure or no connection. Such a peer may be down, or may be on the other side of a network partition with a majority of its own, so it counts as possibly in the cluster.

Learners are not counted. Rejoin goes ahead without --force only when the members that answer in this cluster plus the ones that cannot be reached still fall short of a majority. With three voters, both other voters must answer from another cluster. With five, one of the other four can be unreachable as long as the remaining three answer from another cluster.

In every other case rejoin refuses with not wiping this node's etcd: <reason> ... Nothing was changed, because this node's data may be newer than the seed's. The reason is one of these:

• The cluster still has quorum, for example 2 of its 3 voting members answer, 2 are needed.
• Rejoin cannot rule out quorum, for example 1 of its 3 voting members answer in this cluster and 1 could not be reached (they may hold a majority), 2 are needed.
• Rejoin cannot tell. This node's etcd is not running, its membership cannot be read (a TLS error or a 5-second timeout), or it reports no voting members.

The last case also covers a survivor whose etcd restarted after quorum was lost, for example after a reboot. That etcd never becomes ready, and even a member list that needs no quorum times out. Rejoin refuses with cannot read this node's etcd membership (context deadline exceeded) and needs --force, even though quorum really is lost.

So most rejoins need --force, including the ones this command is usually run for. A diverged member of a healthy cluster still sees quorum. A member whose data directory is gone has no etcd running, so rejoin cannot tell. After a seed restore, the other control planes still form a cluster among themselves: with two of three left, that cluster has quorum.

Pass --force only when you know that this node's data can be discarded. That is the case in these situations:

• You restored the seed on purpose with snapshot restore, and mean to throw away what the other control planes hold.

• This node's old cluster is gone: every other member is destroyed, or confirmed powered off and kept off.

• You are repairing a failed member of a healthy cluster, and the node you pass as --server belongs to that same cluster. Check on that node first that every endpoint is healthy, apart from the member you are repairing, and that the member list is the cluster you expect:

  sudo /usr/local/bin/stratum etcdctl endpoint health --cluster
  sudo /usr/local/bin/stratum etcdctl member list -w table
  

  The healthy majority holds the cluster's data, so the failed member loses nothing it alone had.

Then run:

sudo /usr/local/bin/stratum snapshot rejoin --server https://<seed-node-ip>:6443 --yes --force

If rejoin fails after it has wiped this node's etcd, run the same command again. The check guards only a wipe that could lose data, so the re-run, which resumes as the table above describes, needs --force only if this node's etcd data directory holds member data again by then.

The MCP snapshot_rejoin tool cannot pass --force. Run the CLI on the node.