Data and Control Plane State
In Kubernetes Implementation Design, we discussed the conceptual interplay between data, control, and work planes, as implemented in Kubernetes. In the following we discuss how state plays a crucial role for for backup, restore, migration and pivoting.
State
In Kubernetes the state of the control plane is fully captured within the data plane and persisted primarily with technologies like etcd. This plane serves as the shared repository for all digital twin resources, which function as a virtual representation of the real-world environment.
The data plane state can be backed-up and later restored by
- securing the data and config files of the persistency technology (e.g., etcd or RDBMS),
- capturing the underlying volume or block-storage, potential quiescening the database while using advanced snapshot mechanisms, or
- saving the logical Kubernetes resources via the API level, which makes it independent of the technology and therefore fully portable.
Data Plane State vs. Cluster State
Here, we are only focused on the state of all controllers associated in a control plane. For a full Kubernetes cluster backup and restore, the workloads that have produced state in own block or object storage also need to be considered.
Popular tools in the Kubernetes ecosystem are Velero.io, or the CNCF project k8up.io.
Operations with State
By conceptually separating the control plane from the data plane, we can devise operations on the system, which can replace, migrate, pivot the planes (even while running) without breaking the logic, as long as the two are correctly reconnected.
Live Migration
Control Plane Migration is a key feature for enterprise-grade Managed Kubernetes-as-a-Service solutions, designed to safely move a Kubernetes cluster's control plane from one hosting cluster (source cluster) to another (target cluster). This process is crucial for maintenance, regional failovers, and infrastructure provider transitions.
The migration process is governed by non-negotiable principles aimed at maintaining the continuous availability of the managed Kubernetes cluster and requires (non default) control plane extension maintainers to adhere to them.
Apeiro's default Kubernetes-as-a-Service solution Gardener has specific documentation on this topic for extension maintainers and operators.
Pivoting
Pivoting, similar to migration, is an architectural concept enabled by the recursive nature of the planes and is often required in bootstrapping clusters.
For example, after setting up a data and control plane in a bootstrap cluster (in a donor cloud or on a laptop) connected to a remote work plane, the pivoting process moves the state and planes into the remote work plane, handing over the management to itself (and thereby making it self-reliable).
Pivoting solves the "chicken-and-the-egg" problem for how to utilize a declarative Kubernetes API to create the very Kubernetes cluster that you are managing.
Cluster API (CAPI) applies the core principles that made Kubernetes successful to cluster management itself. CAPI supports the pivoting process with its command-line utility clusterctl, specifically through the clusterctl move command. CAPI provides functional building blocks for teams to build platforms bottum-up.
Gardener, bootstrapped within a prefabricated garden cluster and providing an end user optimized "cluster api experience" from a platform perspective, is enhancing its capabilities to support setting up autonomous clusters with its command-line utility gardenadm. The bootstrap command uses and automatically pivots from a KinD cluster.