Hardening

The CIS profile

sudo stratum up --cis

or, declaratively:

spec:
  cis: true
  operator: true      # required — see below

--cis turns on the controls that are off by default because they change cluster behaviour: apiserver audit logging, a PodSecurity admission configuration enforcing the restricted profile, encryption of Secrets at rest, bootstrap-issued kubelet serving certificates on control planes as well as workers, and a seccomp filter for every container that names none (the kubelet's seccompDefault).

The apiserver verifies every kubelet's serving certificate against the cluster CA (--kubelet-certificate-authority) in every profile, not only under --cis.

Files the CIS profile writes

The audit log and the three files under /var/lib/stratum are on control planes only. The policy and admission files are rewritten each time a control plane starts, so a hand edit does not last. encryption-config.yaml is written once, when it does not exist yet.

What the audit log records

The audit policy logs every request at Metadata level: who made it, when, the verb and the resource, but no request or response body. Bodies are kept only for changes to access control and certificates: RBAC, authorization reviews, ServiceAccounts and certificate signing requests. Requests that touch Secrets, ConfigMaps, ServiceAccount tokens and TokenReviews are logged at Metadata too, so Secret values and bearer tokens do not reach the log.

Some requests are not logged at all:

• health and version probes: /healthz, /readyz, /livez and /version;
• authenticated reads of /metrics and the OpenAPI schema (/openapi/*), which metrics scrapers and kubectl make constantly;
• the anonymous read of kube-public/cluster-info that a token join makes.

Every other anonymous request is logged, /metrics and /openapi/* included.

/var/log/stratum is root-only (0700). The log still names every Secret that is read and who read it, so treat it, its rotated copies and anything you forward it to, such as a SIEM, as sensitive.

The API server reads the policy only when it starts. Its manifest carries a hash of the policy, so when an upgrade changes the policy, the API server restarts onto the new one.

Verifying

sudo stratum hardening check

Reports each control as PASS, FAIL or EXCEPTION. It reads /var/lib/stratum/config/manifests/*.yaml — the static pod manifests kubelet actually runs, not a template — so it describes the process rather than the intent.

EXCEPTION means a deliberate documented deviation or a control that does not apply to this node, not a missing check. On a control-plane node, two exist by design, besides the selinux controls on a host without SELinux (see SELinux confinement):

• anonymous auth stays enabled on the apiserver, because the token-join flow reads the cluster-info ConfigMap in kube-public before the joining node has credentials. The node trusts that document through the bootstrap token's signature on it, not through the connection. RBAC scopes anonymous access to that document and health endpoints.
• bootstrap-token authentication stays enabled, because it is how nodes join.

On a worker, which has no control-plane static pod manifests, the controls that apply only to the control plane report EXCEPTION. The kubelet and file permission controls still run.

• cp stands in for the apiserver controls. Run hardening check on a control-plane node to evaluate them.
• 1.1.21 (admin.conf perms 0600), because admin.conf exists only on control-plane nodes. This exception applies only when kube-apiserver.yaml does not exist. If the manifest exists, even empty or unreadable, a missing admin.conf or one that is not 0600 fails.
• The etcd controls (2.2, 2.3, 2.5, 2.6 and 2.7), because the node has no etcd.yaml.
• frontproxy.1, because the node runs no apiserver. On a worker, frontproxy.2 and frontproxy.3 still run and pass only when the keys are absent (see The front-proxy client key).
• live stands in for the runtime probes, live.csr and live.kubelet among them, when the node has no admin credentials to reach the apiserver.

The live.csr control fails only when a kubelet-serving CSR has stayed undecided for more than 10 minutes. A node that has just joined has CSRs that stay pending until the approver runs. The detail line counts those as within grace, and they do not fail the check. If live.csr fails, find out why the CSR is still pending. The approver may not be running, or it may be waiting because the node's reported addresses do not yet match the names and IP addresses the CSR requests.

SELinux confinement

On an SELinux host, enforcing or permissive, the selinux controls check that the node's runtime and containers are confined (see Containers). They are not CIS Kubernetes controls, and a node that deliberately opted out with spec.selinux: false fails them. On a host without SELinux, one selinux control reports EXCEPTION, as it does not apply. Run the check as root: without it, selinux.3 and selinux.4 cannot read other processes' domains and report EXCEPTION.

The CIS profile refuses a new node without the SELinux policy packages and refuses spec.selinux: false. A node brought up before the packages were installed only warns, and these controls fail until it is labelled.

What the check does not cover

SELinux policy beyond Stratum's own files and containers, PAM and auditd are left to the host's own hardening.

Pod security contexts

The fields of a pod's or container's securityContext, as described in the Kubernetes task Configure a Security Context for a Pod or Container, work on Stratum as follows. The integration check tools/inttest/check-security-context.sh asserts each row from inside a pod.

To use a Localhost seccomp profile, put the file on every node that may run the pod, and name it relative to that directory:

sudo mkdir -p /var/lib/stratum/kubelet/seccomp/profiles
sudo cp audit.json /var/lib/stratum/kubelet/seccomp/profiles/

securityContext:
  seccompProfile:
    type: Localhost
    localhostProfile: profiles/audit.json

Under --cis, Pod Security restricted is the default level for every namespace that does not set its own with pod-security.kubernetes.io/enforce labels; kube-system is exempt. Stratum's own local-path-storage namespace sets privileged, because its helper pods mount host paths, and a namespace you label the same way opts out too. Under restricted a pod must run as non-root, set allowPrivilegeEscalation: false, drop all capabilities and set a seccomp profile; privileged pods and host namespaces are refused.

On a CIS cluster, stratum token create adds --cis to the join commands it prints. A worker reads no cluster config, so that flag is how it learns to run the CIS profile; join workers with the printed command.

Every container that names no seccomp profile runs under RuntimeDefault, in any namespace. One that names Unconfined still runs unconfined, as the bundled Cilium agent does. On an existing CIS node, the default applies to every container the kubelet starts or restarts after the node restarts with this release, including restarted containers of older pods; containers already running keep their seccomp state.

etcd's trust domain

etcd has its own certificate authority, separate from the Kubernetes root.

This matters more than it sounds. etcd authorises a client by chaining its certificate to the authority it is configured to trust. If that were the cluster root, then every certificate the cluster ever issued — including every kubelet's — would be a valid etcd client, and any node would have full read/write access to all cluster state, which is every Secret in the cluster.

hardening check verifies this against the running etcd by attempting a connection with a cluster-root-signed certificate and confirming it is rejected (CIS 2.7). That is a live handshake, not an inspection of a config file — a trust bundle on disk records what was intended, and etcd builds its authority pool once at startup, so only a handshake shows what the running process actually enforces.

Alongside it, hardening check reports etcd's TLS switches — client certificate authentication on both the client and peer ports (CIS 2.2 and 2.5), and self-signed auto-TLS off on both (2.3 and 2.6). These are read from the rendered static pod manifest, which carries etcd's arguments and its environment, so a value supplied by an environment variable rather than a flag is seen too.

If 2.7 reports FAIL, the cluster predates the split and the remediation is certs migrate-etcd-ca.

The kubelet's client trust

Every kubelet serves an API on :10250 that can read any pod's logs, run commands in its containers and forward its ports. It accepts a client only if the client's certificate chains to a CA in /var/lib/stratum/pki/kubelet-client-ca-bundle.crt.

On a cluster created by this release that file holds its own CA, kubelet-client-ca, which signs only apiserver-kubelet-client certificates. The apiserver uses them for kubectl logs, exec, attach, port-forward and the node proxy. Nothing the cluster root signs, including admin, kube-vip, every worker's kubelet client and any certificate the controller-manager issues for an approved request, can authenticate to a kubelet.

• Control planes create the CA with the cluster and share it with joining control planes through the encrypted stratum-certs Secret, like the other CAs.
• Workers read the bundle from the stratum-kubelet-client-ca ConfigMap in kube-public when they join. It is published beside cluster-info, readable the same anonymous way, and fetched over TLS verified by the cluster CA the join token pinned. The signed cluster-info kubeconfig records whether the cluster uses the separate CA. On a split cluster, a missing ConfigMap or a denied read stops the join; it never installs the cluster root as kubelet client trust.
• certs rotate-ca rotates this CA with the cluster root, in the same phases. trust-new makes every kubelet trust the new and the old one, workers through the republished ConfigMap. sign-new reissues apiserver-kubelet-client under the new one. finalize drops the old one, and a rollback restores it. The cluster root never enters a kubelet's bundle. A worker with a missing kubelet client-CA bundle refuses a root trust refresh until the original bundle is restored, since its mode can no longer be inferred safely from that file.

Join a cluster created by this release with this release or later. An earlier release's worker writes the cluster root into its kubelet's bundle, so it refuses the apiserver's kubelet client and kubectl logs and exec to it fail. An earlier release's control plane refuses the join outright, because the shared certificates carry a CA it does not know.

A cluster created by an earlier release keeps the cluster root in that file. hardening check reports two controls for it:

• kubelet.1, on every node, worker or control plane, reads the bundle the kubelet's clientCAFile names. It fails when the bundle holds a CA from ca-bundle.crt, a CA that is neither, or no kubelet client CA, and its detail names the node's migration state.
• live.kubelet, on a control plane, connects to every node's kubelet on :10250, once with a certificate the cluster root signs and once with apiserver-kubelet-client. It passes only when every kubelet refuses the first and accepts the second. A kubelet that accepts the first fails it, as does one that refuses the API servers, whose kubectl logs and exec would fail. A kubelet that cannot be reached leaves the control an EXCEPTION, not a pass. It shows what the running kubelets enforce, not what their files say.

Move an earlier cluster onto its own kubelet client CA with stratum certs migrate-kubelet-client-ca --cluster --yes; see Migrating an existing cluster onto the kubelet client CA.

The front-proxy client key

The apiserver believes the X-Remote-User and X-Remote-Group headers of any client that presents a certificate signed by the front-proxy CA with a common name listed in --requestheader-allowed-names. That is how the aggregation layer passes the caller's identity to an extension API server such as metrics-server, so it stays on. It also means that /var/lib/stratum/pki/front-proxy-client.key, and front-proxy-ca.key that can sign another, can act as any user, including system:masters. Treat both as cluster-admin credentials.

Where they live:

• On every control plane, under /var/lib/stratum/pki (0700), owned by root and 0600. The apiserver uses the client pair to reach extension API servers and Stratum's own admission webhook.
• In the stratum-certs Secret in kube-system, encrypted under the certificate-key, so that a joining control plane receives the same CA. Anyone who holds both the Secret and the certificate-key holds these keys.
• Never on a worker. A worker joins with the cluster root certificate and a bootstrap token only.

stratum certs rotate and the renewal timer reissue front-proxy-client with a new key; stratum certs rotate-ca replaces the front-proxy CA as well. A leaf rotation does not revoke the old leaf, which stays valid until it expires. If the key may have leaked, run certs rotate-ca through finalize, which is the step that stops the apiserver trusting the old CA.

hardening check reports the controls below. A key found on a worker, or one that was readable by anyone but root, may already have been copied: remove or tighten it, then run stratum certs rotate-ca --cluster --yes through finalize. Run the check as root: without access to /var/lib/stratum/pki it reports that it cannot inspect the keys.

The apiserver's kubelet client

The apiserver reaches every kubelet's API on :10250 for kubectl logs, exec, attach, port-forward and the node proxy with apiserver-kubelet-client. Its certificate names the user kube-apiserver-kubelet-client and carries no group. Each kubelet authorizes it through the apiserver, and the ClusterRoleBinding stratum:kube-apiserver-kubelet-admin grants it the upstream system:kubelet-api-admin role: the kubelet API and nothing else. Earlier releases issued it in system:masters, which bypasses RBAC, so the key on every control plane was a cluster-admin credential that no RBAC change could limit.

Every controller re-creates the binding when it starts, or restores it if its subjects or role were changed. Deleting it breaks kubectl logs and exec until the next controller start.

A control plane upgraded from an earlier release moves to this certificate on its first start of the new release; see Upgraded clusters and system:masters.

admin.conf and super-admin.conf

Each control plane holds two administrator kubeconfigs, as kubeadm does:

Both are root-only (0600) and stay on the node they were issued on. Use admin.conf for day-to-day work, and keep super-admin.conf for recovery when RBAC itself is broken.

Revoking admin.conf. Delete the binding:

sudo stratum kubectl --kubeconfig /var/lib/stratum/kubeconfig/super-admin.conf \
  delete clusterrolebinding stratum:cluster-admins

Every admin.conf on every control plane is refused from then on, without a CA rotation, and the cluster keeps running on super-admin.conf. Stratum creates the binding once per cluster and records that in the ConfigMap kube-system/stratum-cluster-admins-binding, so a restart or upgrade does not put it back. The first controller to start claims that ConfigMap before it creates the binding, and no other controller creates it. If the claiming controller stops in between, admin.conf stays refused, and the other controllers warn, naming it, until its next stratum up finishes the job. To restore access, create the binding again:

sudo stratum kubectl --kubeconfig /var/lib/stratum/kubeconfig/super-admin.conf \
  create clusterrolebinding stratum:cluster-admins \
  --clusterrole=cluster-admin --group=stratum:cluster-admins

Restoring an etcd snapshot taken before the binding existed creates it again, matching the restored cluster.

A control plane upgraded from an earlier release gets super-admin.conf, and an admin.conf that the binding governs, on its first start of the new release; see Upgraded clusters and system:masters.

kube-vip's identity

On a control plane that holds a VIP, kube-vip elects the VIP's holder with the Lease kube-system/plndr-cp-lock. Its certificate names the user system:kube-vip and carries no group. The Role and RoleBinding kube-system/stratum:kube-vip let that user create the Lease and get and update it by name, and nothing else. Earlier releases issued it in system:masters, so the key read by kube-vip, a pod with NET_ADMIN and NET_RAW, was a cluster-admin credential. Its pod mounts only /var/lib/stratum/kubeconfig/kube-vip/, so it cannot read the admin kubeconfig beside the others.

Every controller with a VIP creates the Role and RoleBinding through its local apiserver when it starts, and checks them every five minutes, restoring them if they were changed or deleted, as an etcd restore from an older snapshot does. Until they exist, kube-vip cannot take the Lease and no node holds the VIP.

A control plane upgraded from an earlier release moves to this certificate on its first start of the new release; see Upgraded clusters and system:masters.

kube-scheduler's files

kube-scheduler reads only its own kubeconfig, /var/lib/stratum/kubeconfig/scheduler/scheduler.kubeconfig, which embeds its certificate and key. Its pod mounts only that directory. Earlier releases mounted all of /var/lib/stratum/pki and /var/lib/stratum/kubeconfig, so a compromised scheduler could read the cluster CA's key and the admin and super-admin credentials.

An upgrade writes the new kubeconfig before it restarts kube-scheduler. The copy at the old path, /var/lib/stratum/kubeconfig/scheduler.kubeconfig, is kept current, because a rollback restores a manifest that names it. The new pod cannot read it.

kube-controller-manager keeps its mounts: it signs certificates and service account tokens with the cluster CA's and the service account keys, so it holds the cluster's root credentials by design.

Upgraded clusters and system:masters

Earlier releases issued admin, kube-vip and apiserver-kubelet-client in system:masters. A leaf is otherwise reissued only when it is missing or lacks a SAN, so an upgrade alone would leave those certificates in place.

The first start of the new release on each control plane fixes that. Upgrades start it themselves. Once its apiserver answers, the controller:

1. finds the leaves whose subject this release no longer issues;
2. makes sure the bindings their new subjects need exist: stratum:cluster-admins, stratum:kube-apiserver-kubelet-admin, and, on a VIP node, kube-system/stratum:kube-vip;
3. reissues every leaf, rewrites the kubeconfigs and restarts the control-plane components, as stratum certs rotate does.

The bindings come first, so admin.conf, kube-vip and kubectl logs keep working while the certificates change. The rotation takes the cluster maintenance lease, so control planes move one at a time and never during a cluster upgrade or another maintenance operation. A controller that cannot take the lease retries every 30 seconds, and logs once that it is waiting. A reissue that fails for any other reason is logged at once and retried after a minute, then after twice as long each time, up to 30 minutes, because every attempt restarts the control plane. kube-vip must be restarted, and its old process gone within two minutes, or the reissue counts as failed, since the running kube-vip would otherwise keep its old certificate. The controller log reports which leaves it found and when it has reissued them. A controller stopped part-way through the reissue finishes it on its next start.

Copies made before the upgrade. An admin.conf copied off a control plane before the upgrade still carries system:masters. RBAC cannot revoke it, and it stays valid until it expires, a year after it was issued. To cut such copies off, rotate the cluster CA with stratum certs rotate-ca --cluster --yes through finalize.

hardening check reports the control below on every node. Run it as root.

HelmChart installs run as the requester

The operator installs a HelmChart's rendered objects with the requester's authority, not its own cluster-admin identity. A mutating admission webhook stamps the writer onto the CR at creation and on spec changes (stratum.ciq.com/requester) and signs the resource identity, desired spec, and requester identity with an HMAC keyed by a cluster secret only the operator can read, so a tenant cannot forge a requester: system:masters annotation — an unsigned or tampered stamp fails verification. The operator then applies the release impersonating that user, and the apiserver's RBAC evaluates every rendered object — namespaced and cluster-scoped — as the requester, so a tenant who can create HelmCharts in their namespace cannot use the operator's credentials to have objects created that they could not create themselves (a ClusterRoleBinding, or an in-namespace Role that escalates their own access). A same-name HelmChartConfig overlay must have the same username, UID, and group set as the chart requester, so its values cannot be applied under another principal's identity.

This layers on the namespace containment (a namespaced CR may install only into its own namespace unless it lives in an operator --privileged-namespaces namespace, default kube-system). Charts in a privileged namespace — the bundled add-ons — still install as the operator.

The webhook uses mutual TLS: the apiserver authenticates with its front-proxy client certificate, and the webhook verifies that identity against the dedicated front-proxy CA. Tenant pods cannot obtain signatures by sending fabricated AdmissionReviews. Static-pod rendering configures these credentials for the apiserver without changing the CIS PodSecurity policy.

The webhook runs failurePolicy: Fail. Tenant CREATE and UPDATE requests fail while it is unavailable; existing workloads keep running. The kube-system add-ons are exempt. This prevents replaying an old stamp on a recreated resource during an outage. Resources must use an explicit metadata.name rather than generateName, since the name is part of the signature. Metadata-only controller writes preserve the existing requester. Unverified tenant releases are never uninstalled as the operator: their finalizer is retained for administrator-assisted recovery.

Requester signing keys and signature rollout

Every operator refreshes the shared kube-system/stratum-operator-requester-hmac Secret every 30 seconds. Admission and reconciliation use immutable snapshots. A transient read error preserves the working snapshot; a failed initial load keeps retrying. Replacing the signing key retains all previous verification keys in the Secret's retained-keys map before the replacement becomes active, so stored stamps remain verifiable after restarts and repeated rotations. Keep that map when managing the Secret. A key identifier always names the same key material; give a replacement key a fresh identifier. Operators reject a reused identifier instead of discarding its previous verifier. Deleting the Secret does not revoke a key: a running operator restores it from its held snapshot. Revocation needs all operators stopped and an administrator-led recovery of affected objects.

New-format signing is disabled by default. Follow this order when migrating:

1. Upgrade every control-plane operator, including possible rollback targets, to a release that verifies the versioned field policy. Verify the running binaries on every instance; elapsed release time does not establish adoption.

2. After adoption, enable the versioned signer explicitly:

   sudo stratum kubectl -n kube-system patch secret stratum-operator-requester-hmac \
     --type=merge -p '{"stringData":{"binding-version":"v2-hmac-sha256"}}'
   

   Allow 30 seconds for every operator to refresh. New admissions then carry the legacy signature, versioned signature, key identifier, and an authenticated list of the spec fields present at admission.

3. Before extending the CRD schema, migrate existing tenant HelmChart and HelmChartConfig objects through authenticated spec-changing updates by their intended requester. Metadata-only updates and identical applies preserve the existing stamp. Check that each object has the stratum.ciq.com/requester-sig-v2 and stratum.ciq.com/requester-fields-v2 annotations before introducing new defaults.

The versioned binding covers every admitted field's exact value, including values equal to the admission-time default. Later schema fields absent from that authenticated list are tolerated only when they equal their served schema default. Changing an already admitted field to a new default does not authorize the change. The policy supports later top-level defaults; a future change that defaults inside an already admitted object or list needs a new binding version and the same verifier-first rollout.

Before changing or removing an existing schema default, migrate every affected tenant object whose authenticated field list does not include that field, including objects that already carry a v2 signature. While the old default is still served, have the intended requester submit an authenticated spec-changing update that preserves the field's intended value. Check that the new stratum.ciq.com/requester-fields-v2 annotation includes the field before changing the schema. Legacy objects must acquire the v2 signature and field list through the same update.

Metadata and status writes can persist a later default without adding it to the authenticated field list. For example, an object created before a field exists can store its later default of 3 through a status write. If that default then changes to 4, the stored 3 is outside both the original field list and the current default policy, and verification refuses the object. Metadata-only updates and identical applies do not perform the required migration. Authenticated spec-changing migration before the schema change binds the stored 3, so it remains valid and a later unauthorized change to 4 is rejected. The verifier does not infer historical defaults from stored values; skipping this migration leaves affected objects failing closed until their intended requester repairs the stamp through a spec-changing update.

Legacy verification remains available while old stamps exist. Its compatibility policy preserves the released schema's fields and tolerates later default-only fields; it cannot reconstruct arbitrary sequences of schema changes for newly created legacy-only objects. Activate and migrate before extending the schema. Keep every key needed by stored stamps; removing a verification key requires authenticated restamping or an explicit administrator recovery first.

What Helm v4 changes for the requester

The operator runs Helm v4. For a tenant HelmChart, that changes only the rights the requester needs for uninstalls:

• Installs keep Helm v3's client-side apply, so they need the same create rights as before. Upgrades and rollbacks follow the apply method the release recorded, so they stay client-side too. The operator also keeps Helm v3's waits: it watches only Job and Pod hooks until they finish, waits for a deleted hook to be gone before creating it again, and waits for the CRDs a chart installs to be established.
• Uninstalls need get on every kind of object the release created, because Helm v4 reads each object to confirm that the release owns it before deleting it. When the chart has pre-delete or post-delete hooks, the uninstall also needs both create and patch on each hook object's kind, because Helm v4 always applies those hook objects with server-side apply.

The operator uninstalls a release when its HelmChart is deleted, and also to clear a release that a crash left pending when the release has no earlier revision that served to roll back to (one that is deployed, or superseded after its install or upgrade completed), or one that a failed recovery left uninstalling. A pending release that has such a revision is rolled back to it instead. A recovery that fails sets the HelmChart to phase Failed with a status.message that starts with recover <status> release:; unlike the hold described below, it needs no action, and the operator retries it about a minute later.

Recovering from an incomplete uninstall

When Helm cannot read an object to confirm that the release owns it, it leaves the object in place but still deletes the release record. The read fails, for example, with a 403 when the requester lacks get on that kind, or with a transient apiserver error. The operator does not treat this as a clean uninstall: it sets the HelmChart to phase Failed with a status.message that starts with uninstall incomplete: and lists only the objects whose ownership Helm could not verify. The message takes one of two forms:

uninstall incomplete: remove these objects by hand, then the stratum.ciq.com/helmchart finalizer: <objects>
uninstall incomplete: remove these objects by hand, then change the chart source (spec.repo, spec.chart, spec.version or spec.chartContent) or the merged values (spec.valuesContent, spec.set, spec.setString, or the same-name HelmChartConfig's valuesContent) to reinstall; other spec fields, and values edits that leave the merged values the same, do not reinstall: <objects>

The first appears when the HelmChart is being deleted, and the second when the operator was uninstalling a pending or uninstalling release to recover it, as described above. Beside the message, the operator records the hold in a status.uninstallIncomplete field: the objects the message lists, which of the two paths it asks for, and the fingerprint of the chart source and merged values (the specHash) current when the hold began. For a hold written by an older operator version, the operator promotes its message to this field before continuing a reconcile. While the recorded specHash still matches, the operator keeps the stratum.ciq.com/helmchart finalizer and does no further Helm work for the HelmChart. This is the hold. A change to the chart or merged values lets the operator record a new install or upgrade attempt and clear the field, even if that attempt fails. A deleted HelmChart stays held until a person removes its finalizer.

To recover:

If the message says any objects of release <namespace>/<name> that remain, the operator cannot list what the interrupted uninstall left. Check earlier operator logs for a report if one was recorded. Otherwise, find remaining objects using the recorded release name and namespace and the chart's resources, including cluster-scoped resources, then verify their Helm ownership metadata before removing them.

1. For each object the message lists, or each remaining object you found for a generic hold, make the check Helm could not make: confirm that it carries the label app.kubernetes.io/managed-by: Helm and the annotations meta.helm.sh/release-name and meta.helm.sh/release-namespace set to this release's name and namespace.
2. Remove by hand each such object that passes the check, and nothing else. If the operator logged Helm's full report, it also names any object that a helm.sh/resource-policy: keep annotation kept and any object that another release owns; leave those objects alone.
3. If the message asks for the finalizer, remove the stratum.ciq.com/helmchart finalizer from the HelmChart (for example with stratum kubectl edit).
4. If the message asks for a spec change, change the chart or the merged values as described below. For a tenant HelmChart, the tenant must make that change.

While the hold stands, the operator reinstalls only after a change that moves the specHash away from the one status.uninstallIncomplete records — a change to one of these:

• The chart: spec.repo, spec.chart, spec.version or spec.chartContent.
• The merged values, which the operator builds by parsing and layering spec.valuesContent, the valuesContent of the HelmChartConfig with the same name, spec.set and spec.setString.

These changes do not trigger a reinstall:

• An edit that leaves the merged values the same, such as adding a comment, reformatting or reordering keys in a valuesContent, or changing a value that a later layer overrides (for example, a spec.valuesContent key that spec.set also sets).
• A change to any other field, such as spec.timeout; these fields are not part of the specHash. A spec.targetNamespace outside a privileged namespace — a target other than the HelmChart's own namespace — is still refused, but that refusal replaces the message without ending the hold.

Deleting the HelmChartConfig, or setting its valuesContent to an empty string, is not refused, because the operator checks the overlay's requester only while its valuesContent is not empty. A valuesContent that holds only a comment, whitespace or {} is still checked. If deleting or emptying the overlay changes the merged values, the operator reinstalls the chart as the tenant, without the overlay's values.

When the hold ends early

The hold ends when the operator records a changed chart for an install or upgrade attempt, or when a person removes the finalizer of a deleted HelmChart. It does not end when another status replaces the uninstall incomplete: message: the hold lives in status.uninstallIncomplete, and while the spec the hold recorded is still the current one, the operator regenerates the message from it. For a HelmChart that is not being deleted, these replace the message while the hold still holds the reinstall:

• A values parse error.
• A refused spec.targetNamespace: outside a privileged namespace, one other than the HelmChart's own namespace.

Two other refusals replace the message only once a change has already moved the specHash away from the one the hold records: the operator reaches them only past the gate that, while the hold's specHash still matches, stops the reconcile first.

• A refused requester stamp.
• An administrator's edit to the HelmChartConfig that changes the merged values but leaves any text in its valuesContent, even only a comment or {}. The operator refuses it, because the overlay's requester no longer matches the chart's.

The hold's field survives both refusals. Undo the change, and the hold holds the reinstall again and regenerates its message; keep the change, and once the refusal clears the operator may attempt to reinstall over the leftover objects.

A change to the chart or to the merged values can end the hold: it moves the specHash away from the one the hold records, so the operator clears the field when it records the install or upgrade attempt. The attempt can fail; the field is still gone, and a later retry may install over leftover objects even if the spec is put back as it was.

The operator retries a failed write of the message and the field recorded beside it, and holds the HelmChart in its own memory until the write succeeds. That memory does not survive the operator restarting, or another operator instance taking over as leader, before the write succeeds. The hold itself survives, over any number of restarts: before each uninstall the operator writes a marker to status.message — uninstalling release for a deleted HelmChart's uninstall, uninstalling release to recover for the uninstall that recovers a pending or uninstalling release. The recovery's marker also records the unresolved recovery in status.unrecordedRecovery; the status writes a reconcile makes before it decides on the release leave that record standing. A reconcile after the restart, finding that evidence with the release gone, cannot tell what the earlier uninstall left, so it records the hold with any objects of release <namespace>/<name> that remain in place of the list Helm reported, and the hold ends the way any hold does: a person removes the finalizer of a HelmChart being deleted, or changes the spec of one that is not — the change permits an install attempt, so whatever of the release remains is best removed first. It stands even when the interrupted uninstall left nothing, since the operator cannot tell. Earlier operator logs may contain Helm's report if the uninstall returned before the restart; a crash immediately after the purge can leave no report.

Encryption at rest

Under --cis, Secrets are encrypted at rest. For a cluster that predates the profile:

sudo stratum hardening encrypt-existing --yes

This rewrites existing Secrets through the encryption provider. New Secrets are encrypted from the moment the profile is on; already-stored ones are not, until this runs.

One key for every control plane

Every control plane's apiserver must hold the same at-rest key. A Secret encrypted through one apiserver can be read only through an apiserver that has the same key. The first control plane started with --cis creates the key, and each control plane that joins after that receives it with the shared certificates. A control plane that joined before CIS was turned on has no key, and starting it with --cis cannot create one that the others share.

So stratum up --cis creates a key only on a cluster's only control plane. When there is no key yet and this control plane's etcd has other members, it refuses to start and changes nothing on the node. It also refuses when it cannot read the etcd membership, because it then cannot tell whether the node is the only control plane. A control plane that already has the key, because it joined a CIS cluster, starts normally.

A join command handed out earlier would also add a control plane without the key. stratum token create --control-plane publishes the shared certificates as the Secret kube-system/stratum-certs, and a bundle published before the key existed carries no key. So stratum up --cis also refuses to create the key while that Secret exists, even on a single control plane. Deleting the Secret needs a running apiserver, and the refused start does not bring one up:

1. Start the control plane without --cis. If spec.cis: true is in its config, remove it for now.

2. Delete the Secret:

   sudo stratum kubectl -n kube-system delete secret stratum-certs
   

3. Stop the control plane.

4. Start it again with --cis (or put spec.cis: true back). It creates the key.

5. Only now issue new join commands with sudo stratum token create --control-plane. Their bundle carries the key.

Deleting the Secret stops every control-plane join command already handed out from working. Worker joins do not use it.

A join started with --cis refuses a bundle that carries no key: join without --cis if the cluster does not run the CIS profile, or turn CIS on in the cluster first and use a new join command. A join started without --cis refuses a bundle that carries one, because that cluster runs the CIS profile: re-run it with --cis. Both refuse before they add anything to etcd, and a joining node removes any key file its refused or keyless join would otherwise keep. stratum up --cis also checks etcd again right after writing the key, and removes the key if a member or a bundle appeared in the meantime. Four cases are left that these checks do not see. Avoid the first three while turning CIS on:

• A join without --cis that downloaded its bundle before the Secret was deleted, and adds its etcd member after the key was created. Do not delete the Secret while a control-plane join is in progress.
• A stratum token create --control-plane that runs on this control plane while it starts with --cis, and publishes its bundle after that second check. Run it only after the control plane is up.
• The same command run a moment later, so that its bundle already carries the new key: if that second check sees the bundle, it removes the key while the bundle still hands it out. The same advice avoids it.
• A crash just before turning CIS on. When etcd is stopped, the checks read its database file, which can trail etcd's write-ahead log by a fraction of a second after an unclean stop. A control-plane join or a token create --control-plane acknowledged in that last instant before the crash can be missing from what the checks read. This is not refused, because refusing --cis after every unclean stop would block recovering from a crash.

For an HA cluster, turn CIS on at install time: start the first control plane with --cis (or spec.cis: true), then join the others.

Turn CIS on for a running HA cluster

Give every control plane the same key before any of them starts with --cis. Writing the key by hand skips the checks above, so this procedure does their work: it never replaces a key that is already in use, and it removes the old join bundle itself.

1. On every control plane, check for an existing key file. The checksum tells copies apart without showing the key:

   sudo stat -L -c '%U:%G %a' /var/lib/stratum/pki/encryption-config.yaml
   sudo sha256sum /var/lib/stratum/pki/encryption-config.yaml
   

   Where the file exists, stat must print root:root 600. If it prints another owner or mode, correct it on that control plane before going on. This changes who can read the key, not the key:

   sudo chown root:root /var/lib/stratum/pki/encryption-config.yaml
   sudo chmod 600 /var/lib/stratum/pki/encryption-config.yaml
   

   • If no control plane has the file, write a new one in step 3.
   • If one or more have it and they all show the same checksum, that file is the cluster's one key. Skip step 3 and copy that file in step 4. Writing a new key instead would leave the Secrets already encrypted under it unreadable.
   • If the checksums differ, stop. Secrets are already encrypted under more than one key, and this procedure cannot keep all of them readable: do not merge the files, rename a key or overwrite any of them, because every one of those changes leaves the Secrets encrypted under some key unreadable. There is no tested recovery for this case yet.

2. Make sure no control-plane join is in progress, then delete the published join bundle. It carries no key, and any control-plane join command already handed out stops working:

   sudo stratum kubectl -n kube-system delete secret stratum-certs
   

3. Only if no control plane had a key in step 1: on one control plane, write a new key file. It is written under a temporary name and synced first, then linked into place: ln refuses to replace a file that is already there, and a write cut short never leaves a partial key where stratum up looks for one:

   sudo sh -c 'set -eu; umask 077; d=/var/lib/stratum/pki; t=$(mktemp "$d/.encryption-config.yaml.XXXXXX"); trap "rm -f $t" EXIT; KEY=$(head -c 32 /dev/urandom | base64); cat > "$t" <<EOF
   apiVersion: apiserver.config.k8s.io/v1
   kind: EncryptionConfiguration
   resources:
     - resources: ["secrets"]
       providers:
         - secretbox:
             keys:
               - name: key1
                 secret: ${KEY}
         - identity: {}
   EOF
   sync "$t"; ln "$t" "$d/encryption-config.yaml"; rm -f "$t"; sync "$d"'
   

4. Copy that file to every other control plane that had no key file in step 1. Leave the ones that already have it alone: theirs is the same key, and a copy over it that fails partway leaves a truncated file in its place. Run this on a control plane that has the file, once for each of the others, with OTHER set to that control plane's address:

   KEY=/var/lib/stratum/pki/encryption-config.yaml
   SUM=$(sudo sha256sum "$KEY" | cut -d' ' -f1)
   sudo cat "$KEY" | ssh "$OTHER" "sudo sh -c 'set -eu; umask 077; d=/var/lib/stratum/pki; t=\$(mktemp \$d/.encryption-config.yaml.XXXXXX); trap \"rm -f \$t\" EXIT; cat > \$t; echo \"$SUM  \$t\" | sha256sum -c --quiet; sync \$t; ln \$t \$d/encryption-config.yaml; rm -f \$t; sync \$d'"
   

   The key travels over ssh and lands under a temporary name, owned by root with mode 0600. It is linked into place only once its checksum matches the source and it is synced, so an interrupted copy never leaves a partial key where stratum up looks for one, and ln refuses to replace a key that is already there. sudo on the other control plane must not prompt for a password: ssh carries the key on the same input.

5. Restart each control plane with --cis, one after another. Until all of them have restarted, a Secret written through a restarted control plane cannot be read through one that has not, so do this in a short maintenance window.

6. Re-encrypt the Secrets stored before the change under that key:

   sudo stratum hardening encrypt-existing --yes
   

7. Only now issue new join commands with sudo stratum token create --control-plane. Their bundle carries the key.

New --cis clusters use the authenticated secretbox provider (AEAD), so a tampered ciphertext is rejected rather than decrypted. A cluster created by an older build keeps the provider recorded in its existing encryption-config.yaml; switching that cluster's provider is a separate migration (the old and new providers have to be listed together while encrypt-existing re-encrypts every Secret), not something regenerating the config does on its own.

Disable the operator and remove its admission webhook

Setting spec.operator: false stops that node from running the operator on its next stratum up.

The operator approves kubelet serving certificate requests. A token-joined worker requests one in every profile. Without a running operator, a new worker cannot get a serving certificate automatically; until an administrator approves its CSR, kubectl logs, exec and top fail for pods on it. Disabling the operator does not remove certificates already issued to existing workers; they continue serving them. If all operators stop, their renewal requests remain unapproved unless an administrator approves them. certs rotate-ca --phase=sign-new refuses when it needs a new serving certificate. Access also fails if an existing serving certificate expires without renewal. Under --cis, where control planes get their serving certificates the same way, Stratum refuses spec.operator: false at startup. In an HA cluster, disable and stop the operator on every control-plane node before removing its cluster-wide webhook. A running instance will recreate the webhook configuration. Normal shutdown deliberately leaves the configuration intact so an outage cannot bypass admission security.

After all operator instances have stopped, use an administrator kubeconfig:

stratum kubectl delete mutatingwebhookconfiguration stratum-operator-requester --ignore-not-found
stratum kubectl -n kube-system delete service stratum-operator-webhook --ignore-not-found
stratum kubectl -n kube-system delete endpointslices -l kubernetes.io/service-name=stratum-operator-webhook

Tenant HelmChart writes then no longer depend on the webhook, but no HelmChart reconciliation occurs while the operator is disabled. Re-enabling it restores admission enforcement; submit a spec-changing update to establish the requester for CRs created while it was disabled.

Serving instances refresh their EndpointSlice heartbeat every minute. Surviving instances prune peers after three missed renewals (up to approximately four minutes including the next sweep). Admission may fail while a crashed endpoint is awaiting removal. If all instances are down, admission remains fail closed until an operator recovers or an administrator intentionally performs the cleanup above. Keep control-plane clocks synchronized for heartbeat expiry.