Kubernetes Implementation Design

Kubernetes is said to be composed of microservices which form the aforementioned three planes. But an important distinction needs to be made: microservices manage and encapsulate their state with their own API protocol, whereas Kubernetes fully defines a common API framework and manages the state for all its components in the data plane

Persistency part of the management or control plane that hosts the shared repository for all digital twin resources.

.

There is ample external documentation available, which describes the architecture and detailed implementation of Kubernetes (primarily as container orchestrator).

The authors of Kubernetes made early design choices which simplify the efficient bootstrapping and setup of a cluster from zero (for example, instead of introducing a message bus microservice - like many similar management systems - all communication between the components is handled via the API server, which reduces the setup complexity). The same principles used in Kubernetes can be extended to serve in the reference architecture to integrate every type of capabilities, beyond container orchestration within a single cluster.

Scalability and fault tolerance in Kubernetes are implemented entirely in software with the assumption that anything can fail anytime. A typical setup for all planes therefore consists of multiple independent nodes (machines or virtual machines) where components scale and operate redundantly even in cases of failure. Furthermore, Kubernetes provides a universal runtime since almost all workloads can be packaged into containers, including the described planes. As a point of proof, Kubernetes itself (as many other platforms with their own control

Central management system responsible for orchestrating and managing a target environment.

and work planes

Operational heart of a target environment.

) can be run on top of Kubernetes.

Kubernetes Planes

In the following, we will untangle the conceptual building blocks/planes from Kubernetes as interpreted in the reference architecture.

Components of the data plane

a) The persistency: Kubernetes and kcp

kcp

An open source horizontally scalable control plane for Kubernetes-like APIs.

support etcd (other projects like K3s also support SQLite, or a full RDBMS like MySQL, MariaDB, or Postgresql), and b) the API server (in the data plane, we assume the API server to have no use case specific opinion; cf. generic control plane). The goal of the data plane is to provide c) controlled access to a shared repository for all digital twin

A virtual representation of a physical object, system, or process that is continuously updated with real-time or near-real-time data to mirror its real-world counterpart.

resources which function as a virtual representation of the real-world.

kcp

kcp

An open source horizontally scalable control plane for Kubernetes-like APIs.

can be thought of as Data-Plane-as-a-Service (including a generic API server).

Data plane components

Components of the Control Plane

If we conceptually attribute the API server to the data plane

Persistency part of the management or control plane that hosts the shared repository for all digital twin resources.

, then the control plane

Central management system responsible for orchestrating and managing a target environment.

contains a) all needed API Extensions. The Kubernetes core design pattern includes b) scheduler, and c) controller manager which bundles various Kubernetes internal controllers. Furthermore, a typical cluster minimally requires d) cloud controller manager which bundles controllers necessary for dealing with resources from external cloud providers. Administrators can add e) additional, but optional features and their controllers (for example VPA or API Gateways). Kubernetes also requires f) a few controllers to run on each worker node (in the work plane) to bring the nodes under management. Conceptually, it can be argued that they belong to the control plane.

Control plane components

The data plane and control plane can be colocated, or they can run separated. Collectively they function as the orchestration brain. API requests are processed, converted into desired states, which are picked up by the various controllers, some of which translate the desire into imperative instructions that will move the actual state on the work plane. Scheduling is implemented entirely in the virtual representation, using available observability metrics reported back into the digital twins.

Managed Kubernetes-as-a-Service are often built with an abstracted Control-Plane-as-a-Service.

Components of the Work Plane

Its main purpose is to bring the capabilities (compute, network, and storage) of multiple worker nodes under management. The work plane

Operational heart of a target environment.

hence consists of a) physical or virtual machines with a suitable (minimal) operating system, and b) essential actuators/controllers and drivers/plugins which actuate the capabilities on each of the nodes. For compute, Kubernetes utilizes the kubelet controller which interacts with CRI and a fitting container runtime, such as containerd or CRI-O. For networking, the kube-proxy controller (or modern equivalent) and a CNI driver is needed. For storage, a specific CSI controller setup is expected. Kubernetes can be extended to manage further capabilities (for example GPUs, or virtualization) by deploying specialized extensions, controllers, and their drivers in the control and work plane.

Work plane components

These components are quasi part of the control plane, they pull the desired state from the data plane, compare with the local actual state and compute any necessary local instructions: create or destroy containers, manage the associated networking (which enable communication between containers in and outside the cluster), as well as handle externalized storage for the containers.

Connecting the Planes

With the foundational knowledge of the three planes at hand, we can connect them to form a cluster. The controllers map the desired state described by the digital twins to the real world, while reporting back the actual state. The real world for a digital twin in this Kubernetes configuration consists of a work plane for containers, which can be used to implement any desired workload. The work plane can be constructed with manually attached machines, or with automation using an infrastructure provider and corresponding actuators for machines.

Kubernetes planes