Networking & Storage

Cilium is the only CNI, and the only service dataplane

Cilium 1.20.3 ships in the payload and is applied by the operator at bring-up. It provides pod networking, network policy, and service load balancing.

There is no kube-proxy — not disabled, not optional, not shipped as a fallback. Cilium's kube-proxy replacement handles Services, and the cluster runs exactly one dataplane.

That is a deliberate choice with real consequences:

• one component to reason about when a Service does not work, not two that can disagree
• no iptables rule explosion on large Service counts
• but anything expecting kube-proxy to exist — a diagnostic script, a DaemonSet that assumes iptables rules, a runbook — will not find it

Services work normally. It is the implementation that differs.

The MTU, the interfaces Cilium attaches to, its BGP control plane and native routing are set under spec.network.cilium. See Cilium settings.

LoadBalancer Services

A type: LoadBalancer Service gets its address from Cilium's LB-IPAM, out of a CiliumLoadBalancerIPPool you create. On a bare-metal network, L2 announcements make that address reachable: one Cilium agent answers ARP for it on the local segment, and the other machines on the segment send their traffic for it to that node. On an L3 fabric, advertise the addresses over BGP instead (bgpControlPlane in Cilium settings).

L2 announcements are on in every cluster, but nothing is announced until you add a pool and a policy. For example, to announce addresses 192.0.2.200 to 192.0.2.220 on interfaces whose names start with eth:

apiVersion: cilium.io/v2
kind: CiliumLoadBalancerIPPool
metadata:
  name: lan
spec:
  blocks:
  - start: 192.0.2.200
    stop: 192.0.2.220
---
apiVersion: cilium.io/v2alpha1
kind: CiliumL2AnnouncementPolicy
metadata:
  name: lan
spec:
  interfaces: ["^eth[0-9]+"]
  loadBalancerIPs: true

The pool must hold free addresses on the same L2 segment as the interfaces the policy names, outside any DHCP range. Without a serviceSelector the policy covers every LoadBalancer Service, and without a nodeSelector any node can answer. A Service is announced if it sets no loadBalancerClass, or sets loadBalancerClass: io.cilium/l2-announcer.

Each address is announced by one node at a time, the holder of a Lease in kube-system named cilium-l2announce-<namespace>-<service>. If that node goes down, another agent takes the Lease once it expires, 15 seconds by default, and starts answering. All the traffic for one address passes through the node holding its Lease. Agents renew the Leases they hold every few seconds, and those requests count against each agent's Kubernetes API rate limit, so test before announcing hundreds of Services from one cluster.

A Stratum upgrade or a Cilium settings change that changes the Cilium configuration restarts the Cilium agents, one node at a time, and the Cilium operator. Existing connections keep flowing while an agent restarts, but no new pod starts on that node until it is back, and on a single-node cluster that is the whole cluster.

CoreDNS

CoreDNS 1.14.7, applied from the payload. Standard cluster DNS at the usual Service address.

Where pods' upstream DNS comes from

CoreDNS answers cluster names itself and forwards every other name to the resolvers of the node it runs on, as the upstream Kubernetes default does. Private zones such as a Route 53 private hosted zone, split-horizon names, and corporate DNS therefore resolve in pods once they resolve on the nodes running CoreDNS.

A pod asks the CoreDNS Service, and the replica that answers may be on another node than the pod. CoreDNS can run on any node, controllers included, so make site DNS resolvable on every node, or answers will differ between replicas.

Only CoreDNS forwards through a resolver Stratum chooses. The kubelet keeps the node's /etc/resolv.conf, so every other pod that uses the node's DNS (dnsPolicy: Default, host-network pods included, the control plane among them) resolves exactly as the node does, loopback resolvers such as a local dnsmasq or systemd-resolved's stub included.

Each start of Stratum on a node chooses what that node's CoreDNS pods forward through:

Every row but the first writes /var/lib/stratum/dns/resolv.conf, which the CoreDNS pod mounts; the first removes it. A CoreDNS pod on a node that has not written one forwards through its own /etc/resolv.conf.

A pod has its own network namespace, so some nameservers are not the node's resolver from inside one, and they are left out. A loopback or unspecified address such as 127.0.0.53 or 0.0.0.0 is the pod itself, and for CoreDNS that is CoreDNS: it would forward to itself and stop. An IPv6 link-local address (fe80::) names one of the node's links, not the pod's. Any other IPv6 address counts only on a dual-stack cluster, since an IPv4-only pod has no IPv6 route. A node tells from the pod and service ranges it recorded; a worker that joined before workers recorded them counts as IPv4-only, and says so when it leaves an IPv6 nameserver out. Rejoin such a worker with the line stratum token create prints if it needs IPv6 resolvers. An IPv4 link-local resolver such as the AWS VPC resolver 169.254.169.253 is routed through the node and is kept.

When the choice is not the first row, Stratum logs it at each start: the file CoreDNS forwards through, its nameservers, and every nameserver left out with the reason, or a file it could not read. The last row is logged as a WARN:.

During a rolling upgrade from a release that forwarded to 8.8.8.8 and 1.1.1.1, the CoreDNS pods are recreated, and one placed on a node not yet upgraded forwards through that node's /etc/resolv.conf. On a node that uses the systemd-resolved stub, that pod forwards to itself and fails until the node is upgraded. Pods on upgraded nodes resolve normally; if every CoreDNS pod is on a stub node not yet upgraded, cluster DNS fails until one of them is.

There is no cluster-level forwarder setting, and the operator restores the CoreDNS ConfigMap if it is edited by hand. To change where pods resolve:

1. Change the resolver on every node that can run CoreDNS.
2. Restart Stratum on each of those nodes (sudo systemctl restart stratum), so it chooses again from the new resolvers. Running pods are not restarted.
3. Delete the CoreDNS pods, since CoreDNS reads its upstream when it starts: kubectl -n kube-system delete pod -l k8s-app=kube-dns.

Skipping step 2 matters when the change crosses a row of the table above: a node whose /etc/resolv.conf becomes a loopback stub would otherwise still hand new CoreDNS pods that file, and they would forward to themselves.

local-path is the default StorageClass

local-path 0.0.37 is applied at bring-up and marked default. It provisions PersistentVolumes from a directory on the node that schedules the pod.

There is no second storage backend in this distribution — no SAN or NVMe-oF driver, no alternative to select. local-path is what ships.

etcd is the only datastore

No kine, no SQL backend, nothing pinned or built for one. etcd 3.6.8, TLS throughout, with its own certificate authority distinct from the Kubernetes root — see Hardening for why that separation matters.

Adding a different datastore later would be a datastore-endpoint change rather than a redesign, which is what makes deferring it cheap.

Network configuration

spec:
  network:
    podCIDR: 10.245.0.0/16      # the default
    serviceCIDR: 10.96.0.0/12   # the default
    clusterDomain: cluster.local  # the default

Set these before the first controller starts. Its first boot records them in /var/lib/stratum/pki/netconfig: the pod CIDR, the service CIDR and, only when it is not cluster.local, the cluster domain. Joining control planes receive that file with the cluster certificates, so every node agrees on them. A joining node does not need podCIDR, serviceCIDR or clusterDomain in its own config. It still reads spec.network.nodeIP.

Once a node has the cluster's recorded CIDRs, they win on every start. Leaving spec.network out, or leaving one CIDR unset, does not reset it to the default. If you edit podCIDR or serviceCIDR to a different value, the node keeps its recorded range and logs a warning on startup:

stratum: WARNING: spec.network (pod "...", service "...") differs from this cluster's recorded CIDRs (pod ..., service ...); ignoring it -- changing them needs a cluster rebuild; if this node's recorded CIDRs are the wrong ones, restore /var/lib/stratum/pki/netconfig from another control plane

Changing the CIDRs of a running cluster is a rebuild, not a reconfiguration.

Before you upgrade an HA cluster with custom CIDRs from an older build, check that every control plane recorded the same ranges. See HA clusters with custom CIDRs.

Host firewall rules that name the pod range have to use the range set here. See Ports between nodes.

Cluster domain

clusterDomain is the DNS suffix of in-cluster names, as in kubernetes.default.svc.<clusterDomain>. It sets the kubelet's clusterDomain, the CoreDNS zone, the apiserver's service-account issuer and audience, and a DNS name on the apiserver certificate. It must be a lowercase DNS subdomain, such as corp.internal. Any other value stops stratum up with an error.

A control plane that joins with --join takes the domain from the cluster. A worker gets it from the --cluster-domain flag. A worker's config file may set only spec.network.nodeIP and spec.selinux, so a worker join refuses a file that sets clusterDomain. On a cluster with a non-default domain, stratum token create adds the flag to the worker join command it prints:

sudo stratum up --worker --server https://<endpoint>:6443 \
  --token <token> --discovery-token-ca-cert-hash sha256:<pin> \
  --cluster-domain 'corp.internal'

Run the printed command as it is. A worker that joins without --cluster-domain uses cluster.local, and its pods get DNS search domains that the cluster's CoreDNS does not serve.

Dual-stack is supported by supplying both families.

Cilium settings

By default Cilium runs a VXLAN tunnel, attaches to every interface it detects, uses its own MTU, and has no BGP control plane. spec.network.cilium changes that. Every field is optional, and a config without the block runs the defaults.

An L3 fabric with BGP to the host and jumbo frames, for example:

spec:
  network:
    cilium:
      mtu: 9216
      devices: [fe-nic+, fblo]
      bgpControlPlane: true
      routingMode: native
      ipv4NativeRoutingCIDR: 10.245.0.0/16

stratum up writes the settings into the Cilium manifest, and the operator applies them. A change to the block restarts the Cilium agents and the Cilium operator, which read the settings only when they start. The settings are kept across the operator's periodic re-apply, stratum up restarts and upgrades.