Cluster Administration

Day-2 operations. Every command here is a subcommand of the same single binary that installed the cluster — there is no separate operator CLI to distribute.

Certificates

sudo stratum certs check                                # per-leaf expiry table + shared-cert-key age
sudo stratum certs rotate --yes                         # reissue the leaves from the existing CAs
sudo stratum certs rotate --rekey-shared --yes          # re-key the HA shared cert material
sudo stratum certs rotate-ca --cluster --yes            # the heavier cluster-wide CA rotation
sudo stratum certs migrate-etcd-ca --cluster --yes      # move a pre-existing cluster onto the etcd CA
sudo stratum certs migrate-kubelet-client-ca --cluster --yes  # move a pre-existing cluster onto the kubelet client CA

Every command here except certs check needs --yes. Without it, the command refuses and says what it would disrupt.

rotate-ca, migrate-etcd-ca and migrate-kubelet-client-ca also need an operation selector. --cluster runs every phase across the cluster: on every node for rotate-ca and migrate-kubelet-client-ca, and on every control plane for migrate-etcd-ca, because workers run no etcd. Start a --cluster run from a control-plane node; on a worker, the command refuses. --phase=<name> runs one phase on the current node only. Use --cluster. The --phase form is what the cluster run executes on each node, and you need it only to recover a halted run by hand.

To undo a failed run, use --rollback instead. rotate-ca --rollback reverts the current node only, even with --cluster. migrate-etcd-ca --rollback reverts the current node, or every control plane with --cluster. migrate-kubelet-client-ca --rollback needs --cluster and reverts every node. Without --cluster, --phase or --rollback, the command refuses and lists the phases.

Leaf certificates are issued for 365 days. Each control plane checks its leaves daily and rotates any within 45 days of expiry, so certs rotate is for when you want it to happen now rather than eventually. To turn the automatic rotation off, set spec.certRenew: false in /etc/stratum/config.yaml; it is on by default.

certs check also reports when the HA shared-cert certificate-key was last set. That key encrypts the stratum-certs Secret and only re-keys as a side effect of token create --control-plane or certs rotate-ca, so an install-once cluster can carry its install-time key indefinitely. certs rotate --rekey-shared mints a fresh key and re-encrypts the shared material on demand — after it runs, the previous key (printed to stdout on builds before the file-handover change) no longer decrypts anything. The new key is written to a 0600 file whose path is printed, never the key itself; existing control planes need no action.

certs rotate-ca is phased — it widens trust, reissues under the new authority, then narrows — so a cluster is never in a state where one node trusts an authority another does not.

Restart workloads after rotate-ca

The phases protect Stratum's own components, not your workloads. A pod that talks to the API server through in-cluster config (operators, cert-manager, ingress controllers, CoreDNS, Cilium) can fail in two ways during a rotation:

• From sign-new, a cached CA causes x509 errors. The API server starts serving a certificate from the new CA. The pod's mounted ca.crt already holds both CAs, but most clients load the CA pool once, at startup, and never re-read it. Every API call fails with x509: certificate signed by unknown authority until the pod restarts. This does not clear by itself, and a controller stuck in leader election never exits, so Kubernetes never restarts it for you.
• After finalize, an old-key token causes 401 errors. From sign-new, the API server signs new service-account tokens with the new key, and it accepts tokens signed by either key until finalize drops the old one. A pod restarted after sign-new holds a new-key token and is unaffected. A pod still holding an old-key token at finalize, because it was never restarted or because its client caches the token, gets 401 Unauthorized. rotate-ca refuses to finalize while a pod still mounts an old-key token. See Before finalize: mounted tokens.

Pods that do not call the API keep serving traffic, but controllers, DNS records for new Services, ingress route changes and certificate renewal all stop until the affected workloads restart.

The restart that clears the x509 errors has to happen after the new CA is published and before sign-new. rotate-ca checks for it, and refuses until you choose who does it. See Workloads that call the API server.

• --cluster --restart-workloads: Stratum restarts the workloads between trust-new and sign-new. You restart nothing for x509.
• --cluster --skip-workload-check: Stratum restarts only its own add-ons. Restart your workloads once, when the command completes. They are degraded from sign-new until then.
• --phase: restart the workloads after trust-new has completed on every node and before you run sign-new. Otherwise sign-new refuses.

A pod restarted before sign-new is issued an old-key token. A client that re-reads its token file picks up a new-key token when the kubelet refreshes it, so it needs no restart after sign-new, provided it was restarted before it. A workload left to you with --skip-workload-check still needs its restart, for x509. Two cases still get 401 after finalize:

• Clients that cache their service-account token in memory keep the old-key token they started with, even after a restart before sign-new. Restart them after sign-new has completed on every control plane and before finalize. The finalize check reads mounted token files, so it cannot see these clients. --cluster runs finalize itself, so to restart them in that window, run the phases with --phase. Otherwise restart them when --cluster completes; they get 401 until then.

• Long-lived tokens from kubernetes.io/service-account-token Secrets, and kubeconfigs built from them (CI/CD credentials, Argo CD cluster Secrets), were signed with the old key and never rotate. They get 401 after finalize, and restarting the pod that mounts them does not help. Delete and recreate each Secret so it is issued a token under the new key, then update any kubeconfig built from it. To list them:

  kubectl get secrets -A --field-selector type=kubernetes.io/service-account-token
  

rotate-ca prints a warning before it starts. Which one depends on the path. rotate-ca --cluster prints:

WARNING: workloads that use in-cluster API config (operators, cert-manager, ingress controllers, Cilium)
         keep the CA they started with, so from sign-new they fail with x509 errors until restarted.
         Stratum restarts its own add-ons before sign-new. Your Deployments, DaemonSets and StatefulSets
         are restarted before sign-new with --restart-workloads; with --skip-workload-check they are
         not, and you restart them once, when this command completes.
         A client that re-reads its service-account token needs no restart after sign-new, if it was
         restarted before it.
         Finalize refuses while any node still mounts a token signed with the old key. That scan reads
         mounted token files, so it cannot see a client that caches its token in memory: that client
         holds an old-key token even after a restart before sign-new, and is rejected with 401 once
         finalize drops the key. Restart it after sign-new has completed on every control plane and
         before finalize. This command runs finalize itself, so use --phase runs for that window;
         otherwise restart it when this command completes.
         Tokens from kubernetes.io/service-account-token Secrets do not rotate: recreate those
         Secrets (and any kubeconfig built from them) after finalize; a restart does not help.

rotate-ca --phase prints:

WARNING: workloads that use in-cluster API config (operators, cert-manager, ingress controllers, Cilium)
         keep the CA they started with, so from sign-new they fail with x509 errors until restarted.
         Restart all of them, Stratum's own add-ons included, after trust-new has completed on every
         node and before sign-new; sign-new refuses until they have, unless you pass --skip-workload-check.
         A client that re-reads its service-account token needs no restart after sign-new, if it was
         restarted before it.
         A client that caches its service-account token in memory holds an old-key token even after
         that restart, and is rejected with 401 once finalize drops the key. Restart it after sign-new
         has completed on every control plane and before finalize.
         --phase=finalize on a control plane refuses while that node mounts a token signed with the
         old key. It checks that control plane only, and finalize on a worker does not check: scan
         every node before the first control plane finalizes. The scan reads mounted token files, so
         it does not see a client that caches its token in memory.
         Tokens from kubernetes.io/service-account-token Secrets do not rotate: recreate those
         Secrets (and any kubeconfig built from them) after finalize; a restart does not help.

Under --cluster, each node's Job runs rotate-ca --phase through the same CLI, so every node's Job log also carries the --phase warning. It is expected there, does not mean the phase failed, and does not change what you do: follow the --cluster warning. A successful Job is deleted. A failed Job is left in place, and the error rotate-ca --cluster prints ends with the exact command to read its log, of the form kubectl -n kube-system logs job/stratum-carotate-<phase>-<node>. Job names are cut to 63 characters, so on a long node name the real name is shorter than that pattern. Copy it from the error, or list the Jobs with kubectl -n kube-system get jobs -l stratum.ciq.com/carotate.

To recover, do a rolling restart of every Deployment, DaemonSet and StatefulSet that uses in-cluster config:

kubectl -n kube-system rollout restart daemonset/cilium deployment/cilium-operator deployment/coredns
kubectl -n cert-manager rollout restart deployment
kubectl -n <namespace> rollout restart deployment/<name>   # your operators, ingress controller, API clients

To find the workloads you missed, grep recent pod logs for x509 and 401. In the cluster where this was measured (one control plane, about 40 pods), every restarted pod was Ready within 28 seconds, with no x509 or 401 errors afterwards.

Workloads that call the API server

rotate-ca counts a pod as an API client when both of these are true:

• A container mounts a service-account token: a projected volume with a service-account token source (the kube-api-access-* volume every pod gets, or any other name), or a kubernetes.io/service-account-token Secret. The container can be a regular, init or ephemeral one; a volume that no container mounts does not count.
• The ServiceAccount of that token is a subject of a RoleBinding or ClusterRoleBinding. That is the pod's own ServiceAccount for a projected token, and the account a kubernetes.io/service-account-token Secret was issued for (its kubernetes.io/service-account.name annotation), which can be another's. It is bound by name (as a ServiceAccount subject, or as the user system:serviceaccount:<namespace>:<name>) or through the system:serviceaccounts:<namespace> group. A subject of the cluster-wide system:serviceaccounts group does not count: Kubernetes binds that group itself, so it would select every pod.

Finished pods are ignored. Pods that are being deleted still count, because their containers keep running on the old CA until the grace period ends. Pods are grouped by the controller that owns them, so a Deployment's ReplicaSet pods are reported as the Deployment. Jobs and bare Pods are reported, but Stratum never restarts them: recreate them yourself.

A pod's owner reference is written by whoever creates the pod, so Stratum only believes it when the Deployment, DaemonSet or StatefulSet it names exists with the same UID and has a valid name. A pod whose reference fails that is reported as the pod itself, and never restarted. Names that are not plain are printed quoted, and the commands in the listings are quoted for a shell.

The command refuses before it changes anything if you pass:

• both flags: --restart-workloads and --skip-workload-check contradict each other: pass one
• --restart-workloads without --cluster: --restart-workloads goes with --cluster

Before trust-new (--cluster)

Before the first phase, rotate-ca --cluster lists the API clients. If it finds any and you passed neither flag, it refuses to start and changes nothing:

refusing to start: 1 workload(s) use in-cluster API credentials, and unless they restart after the new CA is published (trust-new) and before sign-new they fail with x509 errors:
  Deployment apipoller/apipoller: stratum kubectl -n apipoller rollout restart deployment/apipoller
  Pass --restart-workloads to have Stratum restart the Deployments, DaemonSets and StatefulSets among them (Jobs and bare Pods are never restarted), or --skip-workload-check to rotate without this check and restart them yourself

The check runs this early because a run that halts while trust-new is still running on some node can only be rolled back. Stratum's own add-ons are not listed here, because the run always restarts them.

Two more checks run before that one, with or without --restart-workloads. They cover Stratum's own add-ons as well as your workloads, so they can refuse even when you have no workloads of your own. Each refuses to start and changes nothing, because a restart cannot fix the cause and the run would otherwise halt after trust-new.

A StatefulSet or DaemonSet that a restart does not reach whole. A rollout restart leaves some or all of its pods on the old CA when:

• a StatefulSet has a rollingUpdate partition, which leaves the pods below it;
• a StatefulSet or DaemonSet has the OnDelete update strategy, which does not recreate pods on a restart.

The refusal lists each workload with its reason, then the fix: one line for Stratum's own add-ons and one for your workloads, as they apply:

refusing to start: 2 workload(s) are not restarted whole by Stratum, so their pods stay on the old CA and the run would halt after trust-new, to be finished by hand:
  StatefulSet db/postgres: a rollingUpdate partition leaves the pods below it
  DaemonSet kube-system/cilium: the OnDelete update strategy does not recreate pods on a restart
  Switch the update strategy of Stratum's own add-ons to RollingUpdate and run this again: Stratum restarts them whatever the flags say, and waits for a rollout that OnDelete never completes
  Set the partition to 0 or the update strategy to RollingUpdate for your own workloads, or pass --skip-workload-check to rotate without this check and restart them yourself

Set the partition to 0, or switch the update strategy to RollingUpdate, then run rotate-ca --cluster again. For Stratum's add-ons, such as Cilium, switching to RollingUpdate is the only fix: Stratum restarts them and waits for their rollout whatever the flags say, and an OnDelete rollout never completes, so --skip-workload-check does not help.

# a StatefulSet with a partition
stratum kubectl -n db patch statefulset postgres \
  -p '{"spec":{"updateStrategy":{"rollingUpdate":{"partition":0}}}}'
# a StatefulSet or DaemonSet with OnDelete
stratum kubectl -n kube-system patch daemonset cilium \
  -p '{"spec":{"updateStrategy":{"type":"RollingUpdate"}}}'

If one of your own workloads needs to keep the partition or OnDelete, pass --skip-workload-check instead, and replace its pods yourself between trust-new and sign-new:

• For a StatefulSet with a partition, restart it, wait for the rollout, then delete the pods below the partition. With a partition of 1, that is postgres-0:

  stratum kubectl -n db rollout restart statefulset/postgres
  stratum kubectl -n db rollout status statefulset/postgres
  stratum kubectl -n db delete pod postgres-0
  

• For an OnDelete StatefulSet or DaemonSet of yours, delete every pod of it.

A kube-root-ca.crt written in the future. Stratum tells a restarted pod from a stale one by comparing its start time with the time kube-controller-manager last wrote the namespace's kube-root-ca.crt, read from the ConfigMap's managedFields. A namespace admin can set those fields. If the time is more than 5 minutes ahead of this host's clock, every pod in the namespace would count as stale however often it restarts. Every namespace with an API client is checked, including one whose only clients are Stratum's own add-ons, such as kube-system:

refusing to start: apipoller/kube-root-ca.crt was written by kube-controller-manager at 2026-10-09T20:00:00Z, more than 5m0s ahead of this host's clock; its managedFields are not to be trusted (a namespace admin can set them), so pods cannot be told apart by start time
  Delete that ConfigMap so kube-controller-manager recreates it, or pass --skip-workload-check to rotate without this check

Delete the ConfigMap. kube-controller-manager recreates it with a current write time. Then run rotate-ca --cluster again:

stratum kubectl -n apipoller delete configmap kube-root-ca.crt

If you pass --skip-workload-check instead, Stratum does not check whether your workloads restarted, so restart them yourself between trust-new and sign-new.

If the ConfigMap is forward-dated after this check, the run still halts before sign-new with the was written by kube-controller-manager at error. Delete the ConfigMap, restart the workloads in that namespace, and finish by hand as described below.

Between trust-new and sign-new (--cluster)

Once trust-new has completed on every node, the run:

1. Rebuilds its admin client to trust old‖new.
2. Waits up to 5 minutes for the kube-root-ca.crt ConfigMap in every namespace to carry the new CA.
3. Restarts Cilium, the Cilium operator, CoreDNS and local-path-provisioner, and with --restart-workloads your Deployments, DaemonSets and StatefulSets. It waits up to 10 minutes for them to roll out. A Job or bare Pod is printed as is not restarted and the run continues.
4. Waits for the old pods to be gone. A rollout is done when the new pods are ready, and the old pods keep running on the old CA until their grace period ends. It also waits out the 2-minute margin for a pod whose containers have not started yet (see below). The wait is up to the same 10 minutes. A pod that is still on the old CA and is not terminating or waiting out the margin (a StatefulSet with a partition, an OnDelete workload, or a workload that appeared since the preflight) ends the wait at once and is listed.
5. Checks, once, that no client is left on the old CA (below).

If a step fails, the run halts before sign-new. Every node already trusts old‖new and nothing has been promoted, so you can finish the rotation by hand:

1. Fix the cause the halt names.
2. Run stratum certs rotate-ca --phase=sign-new --yes on each control plane, then on each worker. Add --skip-workload-check only for workloads you will not restart, such as Jobs. Unlike --cluster, this runs the client check on each control plane and does not wait for the node to be Ready.
3. Confirm every node is Ready, kubectl get csr shows none pending and the local apiserver /healthz responds.
4. Run stratum certs rotate-ca --phase=finalize --yes on each control plane, then on each worker. A worker's finalize reads the new-only cluster-info bundle that the control plane's finalize publishes, so it refuses until a control plane has finalized. A control plane's finalize also refuses while that node mounts a token signed by the old service-account key. Before the first finalize, check every node's tokens with steps 1–4 of Finishing a rotation by hand, which restart the kubelet of a node that has old-key tokens.

To abandon the rotation instead, run stratum certs rotate-ca --rollback --yes on each control plane, then on each worker. You cannot re-run --cluster: it refuses to start on a node that is mid-rotation.

This path is only for a halt here, after trust-new has completed on every node. A halt while trust-new is still running on some node (a control plane or worker whose trust-new failed) leaves nodes on different trust, and the only way out is --rollback --yes. A rotation started with --force is the exception: it cannot be rolled back, and it resumes instead (see Recovering a lost key).

The wait is bounded as a whole, including each API call it makes. When the old pods outlive it, the halt lists each pod with what is left of its grace period:

1 workload(s) still have pods running on the old CA while they terminate, 10m0s after the rollouts; from sign-new they fail with x509 errors:
  Deployment kong/ingress: still terminating on the old CA: ingress-7d9c-x2 (4m12s of its grace period left)
  They go when their grace period ends; sign-new can run once they have

Before sign-new (--phase=sign-new)

On a control plane, rotate-ca --phase=sign-new checks every API client before it promotes the new CA, Stratum's own add-ons included. It refuses while any of their running containers (regular, init, native sidecar and ephemeral), including those of pods that are being deleted, started before, or in the same second as, the write that gave kube-root-ca.crt in its namespace the new CA. For a pod created before that write, a container that started up to 2 minutes after it also counts, because the kubelet may not have synced the new ca.crt into the pod yet. For the same 2 minutes, so does a container of such a pod that has not started yet (Pending, or waiting out a CrashLoopBackOff), because it can still start with the old ca.crt:

sign-new refused: 1 workload(s) use in-cluster API credentials and have not restarted since the new CA was published; from sign-new they fail with x509 errors:
  Deployment apipoller/apipoller: stratum kubectl -n apipoller rollout restart deployment/apipoller
  Restart them before sign-new, or pass --skip-workload-check to sign-new to proceed anyway

Run the listed commands, wait for the rollouts, then run the phase again. A workload whose old pods are still terminating is listed with the grace period they have left: wait for them to go. The new CA stays staged, so nothing needs to be rolled back.

A workload whose pods have not started yet is listed with the time the 2-minute margin ends. It needs no restart: run the phase again after that time.

  Deployment apipoller/apipoller: not settled on the new CA: apipoller-5f8c-k2 (not running yet; it can still start on the old CA until 2026-10-09T00:40:00Z)

It also refuses when:

• kube-root-ca.crt does not carry the new CA yet in some namespace. Finish trust-new on every node and give kube-controller-manager time to republish it.
• It cannot tell when a namespace's kube-root-ca.crt was republished, because no kube-controller-manager write is recorded in the ConfigMap's managedFields. Pass --skip-workload-check to proceed.
• That write time is more than 5 minutes ahead of this host's clock (was written by kube-controller-manager at ..., more than 5m0s ahead of this host's clock). Delete the namespace's kube-root-ca.crt so kube-controller-manager recreates it, restart the namespace's API clients, and run the phase again, or pass --skip-workload-check to proceed.

Under --cluster, the run makes this check itself once, after the restarts and the wait for the old pods, and before the first sign-new, and halts before sign-new if it fails. A rollout can succeed and leave containers on the old CA (a StatefulSet with a rollingUpdate partition does not restart the ordinals below it, and an OnDelete workload restarts none of its pods), and a workload can appear after the preflight. The Jobs and bare Pods that were listed and not restarted are not counted: they were reported, and halting on them would stop a run over workloads you were already told about. Each node's sign-new Job then runs with --skip-workload-check, because the run has done the check, or was told not to.

Before finalize: mounted tokens

finalize drops the old service-account signing key. After that, the API server rejects every token signed with the old key, with 401 Unauthorized. Before any node finalizes, rotate-ca checks that no pod still mounts such a token.

The check reads every file in each projected volume under /var/lib/stratum/kubelet/pods/, because the pod spec chooses the path of the token inside the volume. For each file that holds a token, it compares the key ID (kid) in the token's header with the ID of the old key. It reads the files and does nothing else with them. A file or directory it cannot read fails the check.

--cluster

After sign-new has completed on every node, the run:

1. Scans every node: the node it runs on directly, and every other node with a Job.
2. Restarts the kubelet once on each node that mounts an old-key token, then scans that node again. Pods keep running through a kubelet restart, and the restarted kubelet re-issues their tokens within seconds.
3. Scans again every minute until no node mounts an old-key token, for up to 60 minutes. A token the restart did not re-issue is refreshed by the kubelet on its own schedule, within about 50 minutes. The 60 minutes include the scans themselves, so a scan still running at the limit ends the check.
4. Once every node is clean, waits 2 minutes, so that clients re-read their token files. client-go re-reads its token file once a minute.
5. Checks node health again, as it did before the token check: every node is Ready, the local API server is healthy, and no CSR is pending. The token check can restart kubelets and take up to an hour, so a node can go NotReady in the meantime. A failure halts the run before any node finalizes, with the same finalize refused message.

The 2-minute wait runs on every --cluster rotation, even when no node had an old-key token. A run where one node had stale tokens prints:

==> Phase 3/3: finalize preflight (no node mounts a token signed by the old service-account key)
restarting the kubelet: 4 mounted token(s) are signed by the old service-account key
  no node mounts a token signed by the old service-account key; waiting 2m0s for clients to re-read their token files
==> Phase 3/3: finalize (drop the old CA)

While a node is not clean, the run prints a waiting for the kubelets to re-issue tokens signed by the old service-account key line each minute that lists the node and its pods.

If a node still mounts an old-key token after 60 minutes, or a node cannot be scanned, the run halts before any node finalizes. Every node still trusts both CAs and the API server still accepts both keys, so nothing is broken:

finalize refused (cluster left in the safe old‖new state): after 1h0m0s the old service-account key still could not be cleared for finalize: these nodes still mount them: <node>: <namespace>/<pod> (volume <volume>/<path>) (clients holding a token it signed would get 401)
  the run halted in sign-new. Run …

The second line goes on to tell you to run stratum certs check on every node, not to run a single phase on one node, because the phases must finish on every node in order, and to roll back with stratum certs rotate-ca --rollback --yes on every node to abandon the rotation. For a rotation started with --force, which cannot be rolled back, it gives the way to finish it instead.

When a scan was still running at the limit, the list ends with the scan of <node> did not finish. To continue, follow Finishing a rotation by hand, which finalizes every node in order, or abandon the rotation with stratum certs rotate-ca --rollback --yes on every node (not after --force).

The run also warns about each kubernetes.io/service-account-token Secret that holds an old-key token. These warnings do not stop the run:

  WARN: Secret <namespace>/<name> holds a token signed by the old service-account key; it stops working at finalize. Delete the Secret and create it again to get a new one.

Each node's finalize Job runs with the internal --tokens-checked flag, because the run has already checked every node. A refusal on one node after another had dropped the old key would leave the rotation half finished.

--phase=finalize

On a control plane, --phase=finalize checks the tokens on that node only. It does not restart the kubelet and does not wait. If the node mounts an old-key token, it refuses and lists the pods:

finalize refused: 2 token(s) mounted on this node are signed by the old service-account key, which finalize drops (their clients would get 401): <namespace>/<pod> (volume <volume>/<path>), <namespace>/<pod> (volume <volume>/<path>). Restart this node's kubelet to re-issue them and run finalize again, or pass --skip-token-check

The check covers one node, and workers are not checked at all, so check every node before you finalize the first one. See Finishing a rotation by hand. A control plane that no longer has the old key (/var/lib/stratum/pki/sa.pre-carotate.pub) skips the check only if its sa-bundle.pub holds a single key or does not exist, which means it has already finalized or never rotated. That lets you run finalize again after a failed restart. If the old key is missing but the bundle still holds two keys, or the bundle exists but cannot be read or parsed, finalize refuses, because it cannot tell which tokens the old key signed. Pass --skip-token-check --force only if you accept the 401 errors that follow. The --skip-token-check the message suggests also needs --force. See Skipping the check.

Finishing a rotation by hand

Use this procedure when --cluster halted with finalize refused (cluster left in the safe old‖new state). At that point every node has completed sign-new and no node has finalized. --phase=finalize checks only its own control plane and does not wait, so you do the cluster-wide check yourself:

1. On a control plane, compute the ID of the old service-account key. It is the same on every control plane:

   OLD_KID=$(sudo openssl pkey -pubin -in /var/lib/stratum/pki/sa.pre-carotate.pub -outform DER \
     | openssl dgst -sha256 -binary | openssl base64 -A | tr '+/' '-_' | tr -d '=')
   echo "$OLD_KID"
   

2. On every node, control planes and workers, scan the mounted tokens. Pass the same OLD_KID everywhere, because workers do not hold the old key:

   sudo stratum certs rotate-ca --scan-tokens --old-kid "$OLD_KID" --restart-kubelet --yes
   

   --restart-kubelet restarts the node's kubelet once if it finds old-key tokens, then scans again. The command exits 0 when the node is clean. It exits 10 when it still finds old-key tokens, and prints one stale-token <pod-uid> <volume>/<path> line for each, for example stale-token 3f2c…9a1e kube-api-access-x7k2p/token. To name a pod from its UID:

   kubectl get pods -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,UID:.metadata.uid | grep <pod-uid>
   

   Delete or restart those pods, or wait for the kubelet to refresh their tokens. To scan again, run the command without --restart-kubelet.

3. Repeat step 2 until every node exits 0 in the same pass.

4. Wait 2 minutes, so that clients re-read their token files.

5. Check that the cluster is still healthy on the new CA. The run checks the same things before it finalizes:

   kubectl get nodes          # every node Ready
   kubectl get csr            # no CSR Pending
   kubectl get --raw /healthz # ok, run on each control plane
   

   If a node is not Ready, a CSR is Pending or /healthz does not answer ok, fix that first. It is also what the run halts on when its message names a node that is not Ready or a pending CSR instead of tokens.

6. Check that every kubelet serves a certificate the new CA signed. The run checks this before it finalizes, and these checks all pass while a kubelet still serves an old-CA certificate. Like the run, it checks that each certificate is signed by the new CA and names the node's IP. Run this on a control plane, where sign-new has made ca.crt the new CA:

   for ip in $(kubectl get nodes -o jsonpath='{range .items[*]}{.status.addresses[?(@.type=="InternalIP")].address}{"\n"}{end}'); do
     case $ip in *:*) addr="[$ip]:10250" ;; *) addr="$ip:10250" ;; esac
     echo | openssl s_client -connect "$addr" -CAfile /var/lib/stratum/pki/ca.crt -verify_ip "$ip" -verify_return_error >/dev/null 2>&1 && echo "$ip ok" || echo "$ip not verified against the new CA"
   done
   

   On a --cis cluster, skip the control planes: their kubelets renew their serving certificates in their own finalize. A node that cannot be reached also prints "not verified"; make its kubelet reachable and check again. For a worker that is not on the new CA, run --phase=sign-new on it again. For a control plane, restart Stratum there (sudo systemctl restart stratum). Finalizing while a kubelet serves an old-CA certificate breaks kubectl logs and exec to that node.

7. Finalize the control planes one at a time, starting with the one that ran --cluster. Wait for each to finish before you start the next:

   sudo stratum certs rotate-ca --phase=finalize --yes
   

   Each control plane checks its own tokens again and refuses if it finds one. If that happens, go back to step 2 and do not finalize the other nodes yet.

8. Finalize the workers the same way, one at a time.

9. Run stratum certs check on every node to confirm that each one is on the new CA.

After the first control plane finalizes, do not roll back a node that has finalized: its old CA is gone. Until then, stratum certs rotate-ca --rollback --yes on every node is still the way out, unless the rotation was started with --force, which cannot be rolled back.

Nobody has run this procedure end to end on a cluster yet. The commands match what --cluster runs on each node.

What the check does not catch

• Clients that keep the token in memory and never re-read the file. Restart them after sign-new has completed on every control plane and before finalize, or they get 401 once it drops the old key. The 2-minute wait covers clients that re-read the file, as client-go does.
• Tokens from kubernetes.io/service-account-token Secrets, and kubeconfigs built from them. --cluster warns about the Secrets but does not wait for them. See Restart workloads after rotate-ca.

Skipping the check

The flag is for compromise recovery, when you need the old key gone now and cannot wait for every token to be re-issued. It needs --force, and without it the command refuses before it changes anything:

stratum certs rotate-ca: --skip-token-check is for compromise recovery and goes with --force

The check also tells you to pass the flag when it cannot read the old key:

cannot tell which tokens the old service-account key signed: <reason> (--skip-token-check finalizes without the check)

Recovering a lost key (--force)

If a control plane has lost the root CA's private key (ca.key under /var/lib/stratum/pki), the front-proxy CA's key, or the service-account key, rotate-ca refuses unless you pass --force. With --force the rotation goes ahead without them and ends on a new root whose key is on every control plane:

sudo stratum certs rotate-ca --cluster --force --yes

Workloads are handled as in any rotation (see Restart workloads after rotate-ca). Nothing has to be removed or joined again:

• The old root's certificate (ca.crt) stays in every trust bundle until finalize, as in any rotation, so every node and client, workers included, verifies the new root by it. Only the key is lost; the certificate is public.
• From trust-new until sign-new promotes the new root, the staged root (ca-new.*) signs in place of the lost key: the controller-manager's certificate signer and the operator's webhook certificate. The staged service-account key does the same for a lost sa.key.
• --cluster passes --force to the other control planes' trust-new, so a control plane that lost the same keys goes ahead too.

The keys are the same on every control plane. If only one control plane lost them, you can instead run a plain rotation (without --force) from a control plane that still holds them.

A rotation started with --force cannot be rolled back, on any node. A control plane that lost a key has no old signer to return to, and rolling back the others would leave them trusting only the old root while it can only go forward. rotate-ca --rollback refuses on every node that ran its trust-new, and changes nothing:

pki: refusing to roll back (nothing has been changed): this rotation was started with --force: ...

If the run halts, finish it forward. The halt message says how:

• Before sign-new (in trust-new, or just after it): fix the cause and run sudo stratum certs rotate-ca --cluster --force --yes again from the same control plane, adding --restart-workloads or --skip-workload-check if the first run had it. It resumes the staged rotation: trust-new runs again on every node with the material already staged, and the run carries on.
• In sign-new: run stratum certs check on every node, fix the cause, then run sudo stratum certs rotate-ca --phase=sign-new --yes on each control plane and then each worker where sign-new has not completed. A control plane whose sign-new promoted and then stopped, for example on a restart that timed out, finishes where it stopped. Then follow Finishing a rotation by hand.

The old root's certificate is required, even with --force. Without it nothing could verify the new root, and every component would fail TLS once trust-new restarts it, so rotate-ca refuses before it changes anything. It refuses a ca.crt that is not one of the certificates in ca-bundle.crt too, such as a file copied from another cluster:

certs: refusing to rotate (nothing has been changed): /var/lib/stratum/pki/ca.crt is missing, or is not a certificate in ca-bundle.crt: cannot rotate the CA without the certificate of the root the cluster trusts: ...

Put the certificate back and run the rotation again. Copy ca.crt from another control plane, or, if this node's ca-bundle.crt holds a single certificate, copy that file to ca.crt.

The same holds for the other two public halves. A lost sa.pub is rebuilt from sa.key when that key is still there. Without front-proxy-ca.crt (or with one that front-proxy-ca-bundle.crt does not carry), or without both sa.pub and sa.key, trust-new refuses before it changes anything (cannot rotate the CA without the old front-proxy CA certificate and service-account public key: ...). Copy the file from another control plane, or copy a single-entry front-proxy-ca-bundle.crt or sa-bundle.pub to it.

On a cluster whose kubelets have their own client CA (see The kubelet's client trust), kubelet-client-ca.crt is a fourth public half: without it, or with one that kubelet-client-ca-bundle.crt does not carry, trust-new refuses the same way. A lost kubelet-client-ca.key needs --force like the other keys.

A worker that fetches a bundle sharing no CA with its own refuses it in every phase. No rotation produces such a bundle:

refusing to apply the fetched trust bundle: it shares no certificate with this node's on-disk bundle (...), which is a wholesale trust pivot rather than a rotation: every rotation, --force included, keeps the old root in the bundle until finalize, so do not trust this bundle. If the cluster's control planes were rebuilt on a new root, remove this node (`sudo stratum uninstall --yes`) and join it again with a token from `stratum token create` on a control plane

Phases run out of order

Each phase checks that this node completed the phase before it. A refused phase changes nothing on the node, so you can fix the order and run it again. Every refusal offers two ways out: run the missing phase on this node, or abandon the rotation with stratum certs rotate-ca --rollback --yes (not a rotation started with --force; see Recovering a lost key). Run stratum certs check to see where the node stands.

Without these checks, finalize on a control plane whose sign-new was refused would keep the old CA, delete the staged new one, and exit 0. The cluster stays healthy, but the rotation is silently abandoned. On a worker, it would leave the kubelet with a client certificate the API server no longer accepts.

sign-new records that it finished in /var/lib/stratum/pki/carotate-promoted, after it has promoted the new CA and SA, reissued the leaves under them and restarted the control plane. If that restart fails, the node is refused at finalize, because a static pod may still be serving a certificate the old CA signed. Run sign-new again there once the cause is fixed: its staged material is already used, so it skips the promotion, reissues the leaves, restarts the control plane and records it. Rolling back with stratum certs rotate-ca --rollback --yes also works, except after --force. A control plane passes finalize only with that file and nothing staged left. The ca.pre-carotate.crt backup is not proof, because a failed cleanup can leave one behind after a rollback. sign-new also checks that all the staged files are present before it promotes anything, and refuses, changing nothing, if one is missing.

A control plane that already finalized also passes, so you can re-run a finalize whose restart failed. finalize writes carotate-finalized in the same directory to say so, and removes carotate-promoted. A new trust-new or a rollback removes both files. finalize, rollback and trust-new fail if they cannot delete a rotation file. A control plane that never rotated has neither file, so finalize is refused there too.

The control-plane refusal reads:

certs: finalize refused: this node's sign-new never promoted a new CA, so finalize would collapse trust to the current CA and discard the rotation — run --phase=sign-new first, or `stratum certs rotate-ca --rollback --yes` to abandon it (see `stratum certs check`)

The stratum-carotate key Secret

While rotate-ca --cluster runs, it stages the AES key that decrypts the shared CA material (ca.key, sa.key, front-proxy-ca.key) in the kube-system/stratum-carotate Secret. Anyone with get secrets in kube-system can read it, so it should exist only while a rotation is in flight.

The run deletes the Secret when it ends, whether it succeeded or failed. It first waits up to 60 seconds for the API server to pass its health check, so the delete does not land inside the API server restart. If the API server is still down after that, the run tries the delete anyway. If the delete fails, the run prints a WARN with the manual command.

On a control plane, a successful rotate-ca --phase finalize or rotate-ca --rollback also removes a stratum-carotate left behind by an earlier failed run, but only if it is the key of the rotate-ca --cluster run whose sign-new phase this node completed. The node records that run when it promotes the new signer, and keeps the record until the key is known to be gone. A finalize or rollback that you retry after a failure therefore still removes the key. If the Secret is already gone, they print nothing.

Two markers tie the key to its run:

• stratum.ciq.com/carotate-run: an annotation on stratum-carotate that holds a random ID for the rotate-ca --cluster run that staged it. migrate-etcd-ca stages its key without this annotation.
• /var/lib/stratum/pki/carotate-run-id: a file (mode 0600) on each control plane that holds the ID of the run whose sign-new the node completed. Stratum removes it once that run's key is gone. Do not edit it by hand.

migrate-etcd-ca stages the same Secret, and its followers read it while the migration runs. Finalize and rollback therefore leave the Secret in place and print a WARN when the kube-system/stratum-etcdca-migration-lock Secret exists, or when they cannot read it. The warning names the lock's holder and start time. If that migration run was killed and is not coming back, remove the lock first, then the key:

kubectl -n kube-system delete secret stratum-etcdca-migration-lock stratum-carotate

Finalize and rollback also print a WARN and leave the Secret in place in three other cases:

• They cannot read stratum-carotate. The warning prints the manual delete command.
• The Secret is not the key of this node's run. This covers a node with no carotate-run-id record (for example a rollback from the trust-new phase), a key from a later rotation or from migrate-etcd-ca, and a rotation driven by manual --phase commands. Another rotation or migration may still be fetching the key, and rotate-ca --cluster publishes no lock to check. Once no rotation or migration is running, delete the Secret by hand with the command the warning prints.
• A newer run replaces the Secret while they are cleaning up. The Secret belongs to that run, and you have nothing to remove.

certs check warns when it finds the Secret while this node reports no CA rotation in progress:

WARN: kube-system/stratum-carotate exists while no CA rotation is in progress on this node.
      It holds the AES key that decrypts the shared cluster CA material (ca.key, sa.key,
      front-proxy-ca.key). If no rotate-ca or migrate-etcd-ca run is active, remove it:

        kubectl -n kube-system delete secret stratum-carotate

The check only knows this node's rotation phase. Before you delete the Secret by hand, confirm that no rotate-ca or migrate-etcd-ca run is active on any control plane. If stratum-etcdca-migration-lock exists, follow the lock-first order above.

Migrating an existing cluster onto the etcd CA

Clusters built by any released Stratum already issue etcd's certificates from a dedicated authority, separate from the Kubernetes root — see Hardening. They are split from birth and need nothing here.

certs migrate-etcd-ca exists for a cluster built before that split, from a development checkout predating it. If stratum hardening check reports CIS 2.7 as a FAIL, this is the command it names.

sudo stratum certs migrate-etcd-ca --cluster --yes

It walks three phases across every control plane, completing each one everywhere before starting the next:

The order is not cosmetic. etcd's trust widens before any certificate moves, and narrows only after every certificate has moved, so no member ever presents a certificate its peers do not already accept. drop-root will not start until the command has confirmed, against each member's running etcd, that every control plane completed issue-new.

Only control planes are touched. Workers hold no etcd certificate.

If a run stops partway, the cluster is left in the last phase every node completed, which is a working one. Fix the node and re-run with --resume, or go back with --rollback:

sudo stratum certs migrate-etcd-ca --cluster --rollback --yes

Rolling back a cluster that has already finished drop-root re-opens the hole, so it additionally requires --force:

sudo stratum certs migrate-etcd-ca --cluster --rollback --force --yes

That form runs in two passes — it re-adds the cluster root to every control plane's etcd trust first, and only then re-issues the certificates — for the same reason the forward phases are ordered the way they are. Undoing it one node at a time would move a certificate its peers do not yet accept, and that node would leave the etcd cluster immediately.

Migrating an existing cluster onto the kubelet client CA

A cluster created by this release is split from birth: its kubelets trust only kubelet-client-ca, which signs the API servers' kubelet client certificate and nothing else (see Hardening). A cluster created by an earlier release has its kubelets trust the cluster root, so any certificate the root signs, admin and every node's kubelet client included, can read logs, run commands and forward ports on any node. certs migrate-kubelet-client-ca moves such a cluster onto its own CA. Run it from a control plane once every node runs this release:

sudo stratum certs migrate-kubelet-client-ca --cluster --yes

It runs three phases, each on this control plane, then the other control planes, then the workers, before the next one starts:

The kubelets trust the new CA before any certificate moves, and stop trusting the root only after every API server's certificate has moved, so kubectl logs and exec keep working throughout. Each phase restarts the kubelets or the API servers it changes; running pods are not restarted. Before drop-root the command checks that every kubelet accepts the API servers' new certificate, and afterwards that every kubelet refuses a certificate the cluster root signed. sudo stratum hardening check reports the same on any later day: on a control plane, live.kubelet fails while any kubelet accepts a root-signed certificate, and on every node kubelet.1 fails while that node's kubelet bundle holds the cluster root.

Do not join nodes or run certs rotate-ca while the migration runs. A CA rotation on a node part way through the migration is refused until it finishes or is rolled back.

If a run stops partway, every kubelet still accepts the API servers. Fix the cause and run the same command again: each node skips the phases it has already completed. To go back to the cluster root instead:

sudo stratum certs migrate-kubelet-client-ca --cluster --rollback --yes

The rollback runs the phases in reverse: every kubelet trusts the cluster root again, then every control plane reissues apiserver-kubelet-client under it, then every node drops kubelet-client-ca. On a cluster where any node has completed drop-root the rollback lets root-signed certificates reach the kubelets again, so it also needs --force.

A cluster parked part way

A run that halted, or was stopped, leaves its nodes in one of the migration states. Until drop-root completes on a node, its kubelet still accepts certificates the cluster root signs. sudo stratum hardening check names each node's state in the kubelet.1 detail:

While any node is in trust-both or issue-new, certs rotate-ca refuses to start on it. Finish the migration by running the same --cluster command again, or go back with --rollback. live.kubelet, on a control plane, shows which kubelets still accept a root-signed certificate, whatever their files say.

Re-run --cluster rather than --phase by hand. --phase runs one phase on this node only and is what the cluster run executes on each node; a control plane other than the one that created kubelet-client-ca can fetch it only while a --cluster run has it staged.

Talking to the cluster

sudo stratum kubectl <args>
sudo stratum etcdctl <args>

Both are the versions built into this artifact, so they match the cluster, and on a controller both configure themselves: kubectl against /var/lib/stratum/kubeconfig/admin.conf, and etcdctl with the etcd mTLS credentials and the etcd-ca-bundle.crt trust file. The bundle is the right trust file rather than etcd-ca.crt, because it is what clients verify etcd against in every trust state, including mid CA rotation.

Your own settings win. Anything already in the environment is left alone, so a variable you set is the one that takes effect:

# talk to a different member, still using this node's client certificate
sudo env ETCDCTL_ENDPOINTS=https://<other-node>:2379 stratum etcdctl member list

On a worker there is no admin.conf, so stratum kubectl sets nothing and behaves as it always has — pass --kubeconfig, or copy a config to the node.

admin.conf's rights come from RBAC and can be revoked; the system:masters super-admin.conf beside it is for recovery. See admin.conf and super-admin.conf.

Deploying manifests

The operator applies every *.yaml and *.yml file in /var/lib/stratum/manifests with server-side apply. Drop a file there on a control plane and it is applied on the next reconcile pass, within about 20 seconds. A file is applied again when its contents change, and every 5 minutes otherwise.

How a file is treated depends on where it sits:

A first-level subdirectory is a stack. Every object applied from it is labeled stratum.ciq.com/stack=<stack>. When an object disappears from a stack's files, the operator deletes it from the cluster. Loose files are only applied, never pruned.

Because the directory name becomes a label value, a stack name must be a valid Kubernetes label value: at most 63 characters, made of letters, digits, -, _ and ., and starting and ending with a letter or digit. Stratum does not check the name, so every object in a stack with an invalid name fails to apply.

# the stack's workloads and Services (`get all` skips ConfigMaps, Secrets, RBAC and CRDs)
sudo stratum kubectl get all -A -l stratum.ciq.com/stack=<stack>

# every kind the stack labeled
sudo stratum kubectl api-resources --verbs=list -o name \
  | xargs -n1 sudo stratum kubectl get -A --ignore-not-found -o name \
      -l stratum.ciq.com/stack=<stack>

Only the operator's leader applies the directory, and it reads its own node's copy. In an HA cluster, put the same files on every control plane, or apply them with kubectl instead.

The deployer directory is not /var/lib/stratum/config/manifests, which holds the control-plane static pod manifests that kubelet runs.

The dashboard

Every control plane stages a dashboard stack, and the operator runs it in the cluster: a single stratum-dashboard Deployment in the stratum-dashboard namespace, served over HTTPS on NodePort 30443 of every node, and of the VIP on a cluster that has one. Its image is in the payload, so it starts with no registry access.

stratum kubectl -n stratum-dashboard get deploy,svc,secret

The dashboard pod itself has no cluster permissions. It forwards each request with the token you signed in with, so you see and change exactly what that token's RBAC allows.

Connect to the dashboard

1. Find the address. The dashboard answers on port 30443 of every node, at https://<node address>:30443. A node's address is its INTERNAL-IP:

   stratum kubectl get nodes -o wide
   stratum kubectl -n stratum-dashboard get svc stratum-dashboard
   

   With a VIP, https://<vip>:30443 works too. If you set publicURL, use that instead.

2. Check the path to the port. Cilium answers NodePort traffic before the host's netfilter rules see it, so firewalld neither needs 30443/tcp opened nor can block it: the dashboard is reachable on every node address that routes to you. To limit who can reach it, filter 30443/tcp in a network firewall or security group in front of the nodes, or set nodePort: 0 and use a port-forward (below). If you changed nodePort, that is the port.

   If another Service already holds the port, for example a fleet manager exposed on 30443, the operator serves the dashboard on a ClusterIP only and logs why, and the dashboard Service names the holder in an annotation:

   stratum kubectl -n stratum-dashboard get svc stratum-dashboard \
     -o jsonpath='{.metadata.annotations.stratum\.ciq\.com/nodeport-held-by}{"\n"}'
   

   Set spec.dashboard.nodePort to a free port on every control plane and restart stratum up on each, or move the other Service off the port. Once the port is free, the operator puts the NodePort back on its next pass. Until then, use a port-forward.

3. Trust the certificate. The certificate is signed by the dashboard's own CA, which your browser does not trust yet. Export the CA:

   stratum kubectl -n stratum-dashboard get secret stratum-dashboard-tls \
     -o jsonpath='{.data.ca\.crt}' | base64 -d > stratum-ca.crt
   

   Then add it to the trust store of the machine that runs the browser:

   OS	Command
   macOS	sudo security add-trusted-cert -d -r trustRoot -k /Library/Keychains/System.keychain stratum-ca.crt
   RHEL, Rocky, Fedora	sudo cp stratum-ca.crt /etc/pki/ca-trust/source/anchors/ && sudo update-ca-trust
   Debian, Ubuntu	sudo cp stratum-ca.crt /usr/local/share/ca-certificates/ && sudo update-ca-certificates
   Windows (admin PowerShell)	certutil -addstore -f Root stratum-ca.crt

   Firefox keeps its own store: import the file under Settings → Privacy & Security → Certificates → View Certificates → Authorities.

   You are adding a root to that machine, so it matters what the root can vouch for. The dashboard CA is name-constrained: it can only sign for the dashboard's own names, localhost, the publicURL host, and addresses in the subnets of your nodes and VIP (a /24 for IPv4, a /64 for IPv6). It cannot sign another CA, and it is separate from the cluster CA, so a rotate-ca does not change it. Its key is in kube-system/stratum-dashboard-ca: whoever can read that Secret can sign for those names, so treat read access to Secrets in kube-system accordingly.

   For a quick look you can accept the browser's warning instead. Do not make that a habit: the warning is the only thing that would tell you someone is in the middle. To avoid the step entirely, serve your own certificate with tlsSecretName.

4. Get a token. The stack keeps two in Secrets:

   Secret	ServiceAccount	Access
   stratum-dashboard-admin-token	stratum-dashboard-admin	cluster-admin
   stratum-dashboard-viewer-token	stratum-dashboard-viewer	the built-in view role: read most resources, no Secrets, no changes

   stratum kubectl -n stratum-dashboard get secret stratum-dashboard-admin-token \
     -o jsonpath='{.data.token}' | base64 -d; echo
   

   Or use a token for your own ServiceAccount; it can be short-lived:

   stratum kubectl create token <serviceaccount> -n <namespace> --duration=8h
   

5. Sign in. Open the address, paste the token into Service account token, and choose Sign in. The session lasts 12 hours, survives a restart of the dashboard pod, and ends early if the cluster stops accepting the token. Sign out from the user menu (Sign out).

The stored tokens do not expire. Anyone who can read Secrets in stratum-dashboard can take the admin one, so limit that access, or set tokenSecrets: false and hand out short-lived tokens instead. When the API server stops accepting a stored token, as after a rotate-ca that replaces the service-account signing key, the operator replaces the Secret within about five minutes and the cluster fills in a new token. Get it again and sign in with it.

Without a NodePort

With nodePort: 0, or when the node port is not reachable, forward the Service to your own machine. This needs only access to the API server:

stratum kubectl -n stratum-dashboard port-forward svc/stratum-dashboard 8443:443

Then open https://localhost:8443. The certificate names localhost and 127.0.0.1, so a trusted dashboard CA works here too.

From a node, without the deployment

stratum dash serves the same dashboard as a process on a control plane, with no in-cluster objects. It listens on 127.0.0.1:8443 over plain HTTP, the same trust model as kubectl proxy:

sudo stratum dash

From your own machine, tunnel to it over SSH and open http://127.0.0.1:8443:

ssh -L 8443:127.0.0.1:8443 <user>@<control plane>

If something on the node already holds port 8443 (listen tcp 127.0.0.1:8443: bind: address already in use), pick another with --listen and tunnel to that one:

sudo stratum dash --listen 127.0.0.1:18443
ssh -L 8443:127.0.0.1:18443 <user>@<control plane>

To serve it on another address, give it a certificate with --tls-cert and --tls-key; it refuses plain HTTP off loopback. Sign in with a token as above.

When it does not connect

openssl s_client -connect <address>:30443 </dev/null | openssl x509 -noout -ext subjectAltName

Certificate

The operator writes the serving certificate into the stratum-dashboard-tls Secret. It is signed by the dashboard CA, which the operator keeps in kube-system/stratum-dashboard-ca, out of the dashboard pod's reach. No Kubernetes component trusts that CA, and its name constraints (step 3 of Connect to the dashboard) keep it from vouching for any other site. The certificate names the Service, localhost, every node address, the VIP and the publicURL host. The operator reissues it when a node joins and before it ages out, and the dashboard picks up the new one without a restart. Disabling the dashboard keeps the CA, so browsers that trust it keep trusting the dashboard when it is turned back on.

The operator replaces the dashboard CA, and logs that it did, when it no longer covers the dashboard: when publicURL changes, or a node joins from a subnet the CA does not include. Export and trust the new CA after that. To replace it yourself, for example if its key may have leaked, delete the Secret; the operator creates a new CA and reissues the certificate:

stratum kubectl -n kube-system delete secret stratum-dashboard-ca

To serve your own certificate instead, put it in a kubernetes.io/tls Secret in stratum-dashboard and name it in tlsSecretName.

Configuration

Configure it under spec.dashboard in /etc/stratum/config.yaml, with the same block on every control plane:

spec:
  dashboard:
    enabled: true                         # false removes the stack
    nodePort: 30443                       # 30000-32767; 0 serves on a ClusterIP only
    publicURL: https://dash.example.com   # where users reach it, when that is a proxy or a name; https only
    tlsSecretName: dash-cert              # your certificate instead of the dashboard CA's
    enableNodeShell: true                 # the privileged node shell; default false under cis: true
    tokenSecrets: true                    # keep the admin and viewer sign-in tokens as Secrets

The dashboard needs the operator: with spec.operator: false it is off, and enabled: true is refused. Setting enabled: false and restarting Stratum on each control plane removes the stack file, and the operator deletes the namespace and everything in it. Sessions are signed with the key in the stratum-dashboard-session Secret.

Logs