Multi-Plane Controller

In the Controller Pattern section, we discussed the general responsibilities of a controller

A control loop that watch the for state changes and makes or requests changes where needed.

and in the Kubernetes Implementation Design section, we seem to assume that the context for a controller always has to be a single Kubernetes cluster (assembled with the three planes).
However, a reconciling controller can be executed anywhere suitable and be implemented to be multi-cluster, or more precisely, multi data plane aware.
This becomes self-explanatory during the development of a controller. Typically, development occurs on a laptop, where the controller is run outside the cluster (the laptop serving as work plane) and debugged against different clusters (for example, connected to a local cluster on the laptop first, then a cluster in the cloud).

By making the controller multi-plane, or more precisely, multi data plane aware, we a) establish the necessary isolation boundaries required by cloud services, and b) enable the design for Multi-Cluster Federation:

1. Runtime Cluster:
   The controller is resiliently executed as a container in a Pod in the work plane

   Operational heart of a target environment.
   of a runtime cluster. The controller may be instrumented with access to the data plane
   Persistency part of the management or control plane that hosts the shared repository for all digital twin resources.
   of the runtime in order to manage and scale (inside-out) its own service requirements. Alternatively, an outside-in approach with other controllers, like the VPA or HPA, which monitor and manage a desired objective, that is supplying the controller with adequate resources, is possible as well.

2. Digital Twin API Layer:
   The business users interact with a dedicated API hosted by a separate data plane

   Persistency part of the management or control plane that hosts the shared repository for all digital twin resources.
   . It hosts the source of truth (the digital twins and their respective desired state) for the controller. It is separate from the data and control plane of the runtime cluster. This design encourages an indispensable isolation boundary by separating the user accessible API layer from any internal implementation concern.

3. Multiple Targets:
   The multi-plane controller practice also encourages to utilize separate Kubernetes clusters as target for its workload, using available resource primitives (see Multi-Cluster Federation for a more detailed discussion), or order the desired target outcome on any type of platform¹ with an API. This practice splits and isolates the controller & runtime concern from the workload concern and enhances the security posture. Although, in the case of clusterlet (or servicelet) controllers, the runtime cluster is pragmatically used as work plane.

Multi-plane aware controller

Caveat: the popular controller-runtime allows writing controllers against one cluster only. The multi-plane (or multi-cluster) design is being discussed in the Kubernetes controller-runtime Projects Issue #2746. As consequence, projects that operate with multiple clusters/planes commonly implement own special libraries for the puprose².

Footnotes

1. Crossplane is a CNCF incubating project and prominently enables the KRM

   A framework used in Kubernetes to define and manage the various entities or resources.
   translation to any type of API. Its fitting tagline is:

   "Crossplane lets you manage anything, anywhere, all through standard Kubernetes APIs. Crossplane can even let you order a pizza directly from Kubernetes. If it has an API, Crossplane can connect to it."

   ↩

2. For example, Gardener

   Gardener

   The Gardener is an open source project that provides Kubernetes-based cluster management and automation at scale.
   has a controller-manager-library, allowing to work with several logical clusters that are finally mapped to effective clusters when instantiating controllers. ↩