High Availability

One address, inside and out

Set a VIP in the config and every kubeconfig targets it, with kube-vip supervised on each controller:


spec:
  vip:
    address: 192.168.1.10

This is one address for workers and for kubectl — not a per-node proxy that each client resolves differently. Losing a controller does not change the address anyone uses.

Growing to three controllers

Start from a running controller, then mint a control-plane token:


sudo stratum token create --control-plane

That prints the token and the CA pin, and writes a certificate key — the extra secret that lets a joining controller decrypt the shared CA material — to a 0600 file, printing its path rather than the key itself, so the key stays out of logs and scrollback. Copy that file to each joining controller (e.g. scp).

Only a control-plane token can read the shared certs

--control-plane puts the token in the one bootstrap group allowed to read stratum-certs, the encrypted shared CA material, so the command prints only the controller join line. Join workers with a separate token from stratum token create. On a cluster upgraded from an earlier release, see HA join tokens and the shared certs.

On each additional controller:


sudo stratum up --join https://<vip>:6443 \
  --token <token> \
  --discovery-token-ca-cert-hash sha256:<pin> \
  --certificate-key-file <path-to-copied-key-file>

The joining node downloads the shared CA set over an encrypted channel, generates its own leaves, and joins etcd as a live member.

Both the token and the shared certs expire

The control-plane token carries the same default 24-hour lifetime as a worker token (--ttl overrides it; --ttl 0 opts out). The encrypted shared-cert Secret (stratum-certs) is owned by that token, so it is garbage-collected when the token expires or is deleted — the shared CA material never outlives the credential that grants read access to it. Mint a fresh --control-plane token for each new join; each one re-uploads and re-owns the shared certs. Use stratum token list to see what is still live.

The certificate-key that encrypts stratum-certs re-keys only as a side effect of two operations — token create --control-plane and certs rotate-ca — so a cluster that never does either keeps its install-time key. On clusters built before the token-ownership change above, the Secret also has no owner and is not garbage-collected, and on the earliest builds that key was printed to stdout. To re-key on demand without issuing a token or rotating the CA, run stratum certs rotate --rekey-shared --yes: it re-encrypts the material under a fresh key (invalidating the old one) and hands the new key over by file. stratum certs check shows how long the current key has been in place.

Three, not two

etcd needs a majority. Two members tolerate zero failures — losing either one loses quorum and the cluster goes read-only. Go from one to three.

Give each joining controller the same spec.vip in its own /etc/stratum/config.yaml before it runs up --join. That is what puts the VIP in its apiserver certificate and starts kube-vip on it, so it can hold the VIP when the node that has it goes away. A controller joined without it serves a certificate that does not cover the VIP.

The join reaches etcd directly, not through the VIP: etcd listens only on each controller's own address. token create --control-plane publishes the etcd members' addresses beside the shared certs, and the joining node adds itself through whichever of them answers. A cluster whose controllers run a stratum older than this cannot publish that list; join such a cluster through a controller's own address (--join https://<controller-ip>:6443), which the join then uses for etcd as well.

If a controller was already running standalone

A cluster that began as a single controller has kubeconfigs pointing at that node rather than at the VIP. Bring the VIP up first, then move the clients onto it.

On the controller, add the VIP to /etc/stratum/config.yaml:
```
spec:
  vip:
    address: 192.168.1.10
```
kube-vip announces it on the interface that holds the node's default route. Set interface under vip if it belongs on another one, for example when the node IP is on a different NIC.
Restart Stratum on the controller:
```
sudo systemctl restart stratum
```
The restart reissues the apiserver certificate with the VIP in it, starts kube-vip, and points the controller's own kubeconfigs at the VIP.
Check that the VIP answers as the cluster's apiserver:
```
curl --cacert /var/lib/stratum/pki/ca.crt https://192.168.1.10:6443/healthz
```
This should print ok.
On each worker that joined before the VIP existed, move its clients:
```
sudo stratum ha-repoint --vip 192.168.1.10
```
Once for the whole cluster, from a controller, move Cilium onto the VIP:
```
sudo stratum ha-repoint --vip 192.168.1.10 --cilium
```

ha-repoint changes nothing until the VIP passes the check in step 3. It refuses unless something at the VIP on port 6443 presents a certificate from this cluster's CA that covers the VIP. A node repointed at a VIP that nothing serves would lose every API client, and Cilium repointed at one would lose the apiserver on every node.

Then join the new controllers as above, each with the same spec.vip.

Removing a member

Removing a controller is an etcd membership change, not just a shutdown. Run member list / member remove so the cluster does not keep counting a departed node toward quorum. Run these on a surviving controller — stratum etcdctl configures itself against the etcd on the node you run it on:


sudo stratum etcdctl member list -w table
# then, with the departed member's ID from the table:
sudo stratum etcdctl member remove <ID>

A dead member still counts

If a node is destroyed without being removed from etcd's membership, it is still counted when quorum is computed. Three members with one destroyed and not removed tolerate no further failures.

When a join is interrupted

A stratum up --join that fails, or whose node dies mid-join, can leave the node between two states: its etcd member may already be in the cluster, while the node's own state (/var/lib/stratum/config/node.yaml) was never written. To make that recoverable, the join writes an intent journal (/var/lib/stratum/config/join-intent.json) before it adds the member. The journal outlives the node state save on purpose: etcd only starts later, and a crash before its first WAL would otherwise leave a published node.yaml that a bare stratum up refuses and a re-run routes to the member restart. The journal ends when a later start confirms this node's local etcd data, or when a failed join removes its added member with quorum confirmed. The journal records the member's name, peer URL, ID, phase, and prepared configuration. The shared trust and runtime configs are made durable before the member-add.

Recovery is a re-run of the same join command. The re-run reads the journal and settles it against the cluster's own member list — the list, not the journal, is the truth about the member:

The recorded member is still listed: the join adopts it and finishes without a second member-add. This holds for a re-run on the same node at the same address; if the node would join at a different address than the journal recorded, the join refuses instead — a member can only be resumed from the address it advertises — and names the removal remedy.
A quorum-backed member list confirms that the recorded member is absent: the join proceeds as a fresh one.
The member-add's reply was lost, so the add may or may not have committed: the member list decides. A committed member is adopted; an add that never landed is retried once only when a quorum-backed member list confirms its absence.

When quorum is unavailable, the join can resume a matching recorded member using the retained local trust and prepared configuration, together with the surviving member's membership view. It refuses a fresh add or stale-journal cleanup if absence cannot be confirmed. Resume requires the recorded CA pin, hostname, address, CIS profile, and VIP. Older journals without prepared configuration still need a functioning API to repeat preparation.

While the journal is on disk and the node has no saved state, a bare stratum up (without --join) refuses to start: bringing the node up as a separate single-member cluster while the joined cluster still counts it toward quorum is exactly the failure the journal exists to prevent. Re-run the join instead. Remove the journal by hand only after a quorum-backed member list confirms that the recorded member is absent. Manual removal requires a functioning quorum. If adding the second voter lost quorum, resume that joining node at its recorded address. If the node or its local recovery material is permanently lost, recover the cluster using Snapshot & Restore.

Overview Upgrades & Rollback