Author: Svenja Michal
Bio: Svenja Michal is a technical writer at the cloud hosting provider Hetzner Cloud based in Germany. She has spent several years writing and reviewing tutorials and contributing a broad variety of technical content to the Hetzner Community.

⎯⎯⎯⎯ when one cluster isn’t enough ⎯⎯⎯⎯

Introduction

Ever since the release of Kubernetes, the open-source container orchestration system has become more and more popular among developers. Today, there is hardly anyone in the IT industry who wouldn’t recognize the seven-spoked wheel at first glance. But in parallel to its growing popularity, expectations and demands for solutions for more and more complex use cases have increased just as much. In cases in which one Kubernetes cluster isn’t enough anymore, multi-tenancy is often used to foster conditions ideal for wide-ranging needs, while saving money on the go by not having to fund several Kubernetes environments with dedicated servers and dedicated teams to maintain those servers. Because of the popularity of Kubernetes and its widespread use, there are now even cases in which a seemingly simple cluster can in fact involve a second Kubernetes cluster in one way or another — possibly without the user even realizing it. The following presents an overview of different multi-tenancy solutions and two special cases in which Kubernetes meets Kubernetes.

Kubernetes namespaces

The first multi-tenancy solution to look at is of course the official solution provided by Kubernetes itself: Kubernetes namespaces. These are not to be confused with Linux namespaces, which also play an important role in Kubernetes. Kubernetes namespaces are used to isolate selected Kubernetes objects from other Kubernetes objects. This is possible for resources such as services, pods, and deployments. The isolation can be based on user permissions, on the purpose of the objects (e.g. testing, production), or on other specifications. However, not all resources can be “namespaced”. This restriction makes isolation via Kubernetes namespaces relatively limited and not very suitable for those who depend on fully independent clusters and a high security level.

Architecture:

Another solution that will be introduced below builds on the architecture of Kubernetes namespaces, which is why this section includes a quick review of the original concept of Kubernetes namespaces. As previously mentioned, Kubernetes uses both Linux namespaces and Kubernetes namespaces. Before looking at Kubernetes namespaces, let’s first highlight the difference between those two namespaces.

Linux namespaces are used to partition kernel resources.

Kubernetes namespaces are used to isolate groups of Kubernetes resources.

In Linux, there are different types of namespaces. Containers that form one Kubernetes pod often share the same Linux namespaces. Containers within one pod usually have different namespaces than containers in another pod, which isolates them from each other. However, the pods themselves are all Kubernetes resources which by default can access each other on a network level.

Example: Linux namespaces

In Kubernetes, there is only one type of namespace. But when adding an entity of a Kubernetes cluster to a namespace, you do have to distinguish between two types of entities as they can be either “namespaced” or “non-namespaced”. Non-namespaced entities are used for the entire cluster across all namespaces and they cannot be added to a Kubernetes namespace themselves. This is why they are also referred to as “cluster-scoped”. Examples of cluster-scoped entities are nodes, RuntimeClass, ClusterRoles, and CustomResourceDefinitions (CRDs). Namespaced entities, on the other hand, are added to a namespace at creation. If no namespace is specified at creation, they are automatically added to a namespace called “default”. Examples of namespaced entities are pods, services, and deployments. Just like cluster-scoped entities, some namespaced entities are also used “cluster-wide”. Examples for this are the control plane components: the API, controller module, and etcd. These components are managed in the cluster-wide used namespace “kube-system”.

Example: Kubernetes namespaces

This means that information about all pods is saved in the same data store (etcd), for example. It is possible to increase the general isolation by restricting users / groups to certain namespaces via Kubernetes RBAC, or by using policies such as resource quotas and limit ranges to set certain limits for and within each namespace. However, all those additional security settings need to be set up manually and, usually, traditional Kubernetes namespaces do not provide “full” isolation. Another disadvantage of a single cluster is that users or groups that are restricted to individual namespaces cannot use or control cluster-scoped resources such as CRDs.

Virtual Cluster

One of the attempts to overcome the limits presented by Kubernetes namespaces are virtual clusters. As mentioned above, virtual clusters build on the concept of Kubernetes namespaces. The idea is to advance isolation, so that namespaced resources no longer have to share the same control plane.

Architecture:

One open-source solution for virtual clusters is vcluster by Loft, which makes use of two main components: the Kubernetes namespaces and the data store, which saves cluster-wide information, such as namespaces and users. To create a virtual cluster, a new pod is added within a Kubernetes namespace. This pod simulates an “additional” control plane. It consists of two containers with an API, a controller manager, an independent data store, and a syncer which will be explained in more detail below. When a new namespace or a new user is created, this information is processed and saved in the new control plane within the original Kubernetes namespace. These namespaces and users that were created in a virtual cluster do not exist outside of the Kubernetes namespace they are in. This means that these virtual clusters are completely isolated on a Kubernetes resource level. However, the pods still share the same nodes.

Example: Virtual cluster namespaces

Even though this level of isolation is great for managing multiple tenants while maintaining a high security level for Kubernetes resources, there is some information that needs to be forwarded to and processed by the original control plane in the underlying Kubernetes cluster. Mainly, this applies to information about pods that need to get created and started on the worker nodes. The original scheduler coordinates this for all virtual clusters. In order for this to work, the syncer of the virtual cluster has to forward all the relevant information to the scheduler.

Names have to be unique per resource group and per namespace. Two pods from different virtual cluster namespaces could have the same name. However, as mentioned before, those virtual cluster namespaces do not exist outside of the Kubernetes namespace they are in. From the perspective of the scheduler, it would be two pods from the same Kubernetes namespace with the same name, which is not possible. The syncer, therefore, has to rename the pods for the scheduler.

Example:

Vcluster uses StatefulSets to manage the control plane pod. It is theoretically possible to create several StatefulSets with their own control planes in a single Kubernetes namespace.

When the syncer renames a pod for the scheduler, it adds the name of the virtual cluster namespace in which the pod was saved and the name of the respective StatefulSet.

Kubernetes in Kubernetes

The idea of this architecture was already introduced at Kubecon Europe 2017 in a talk called “How We Run Kubernetes in Kubernetes, aka Kubeception”. Generally speaking, Kubernetes in Kubernetes could be considered any architecture in which an entire Kubernetes cluster only works because of an underlying, second Kubernetes cluster. Additionally, this Kubernetes in Kubernetes principle could theoretically be continued recursively.

Architecture:

One way to accomplish this Kubeception effect is to have one Kubernetes cluster that runs entirely in another underlying Kubernetes cluster. This could be achieved by having one Kubernetes cluster which runs virtual machines and to then have these virtual machines run separate Kubernetes clusters themselves. This option makes it possible to dynamically manage several Kubernetes clusters that are completely independent of each other. A similar example is the Kubernetes Cluster API. It uses Kubernetes to run and manage other Kubernetes clusters. At the same time, the Cluster API actually runs on Kubernetes itself.

Another way to achieve this Kubeception effect is by creating and managing a control plane via a Kubernetes cluster. As opposed to the solution with the virtual machines, this approach is a little bit less complex and is already in use by some large companies that offer Kubernetes services. Two examples of this concept in use are Google Anthos clusters and DigitalOcean’s managed Kubernetes service which both work with containerized control planes that are managed in a separate underlying Kubernetes cluster.

In general, the need for these Kubeception solutions is quite niche and relatively limited to large-scale companies that offer Kubernetes related services, for example.

Kubernetes meets Kubernetes

The final architecture to look into is when an application from one Kubernetes cluster is used in another Kubernetes cluster. Kubernetes clusters usually rely on Kubernetes native resources. However, the optional LoadBalancer Service makes it possible to use external load balancers as an additional cluster component. What is important to emphasize here is that it is “optional” to add, which essentially means that the Kubernetes cluster could still work perfectly fine even without the additional LoadBalancer Service. This is why this example cannot be considered to be another case of “Kubeception”.

Architecture:

The effect of Kubernetes meets Kubernetes does not automatically happen with any external load balancer as it really depends on the provider and on the way that they create and manage their Kubernetes-supported load balancers. One example in which Kubernetes is in fact used, are the Load Balancers provided by Hetzner Cloud. Customers that buy a Hetzner Cloud Load Balancer are essentially allocated one pod from a running Kubernetes cluster. If they then use this Load Balancer in their own Kubernetes environment, it will look like this:

Conclusion

All in all, the increase in popularity and more wide-spread use of Kubernetes does not seem to be slowing down any time soon. With this growing trend, Kubernetes clusters will probably continue becoming more and more complex, and the important role of isolation will likely also become more crucial.

Sources:

DigitalOcean (Kubeception)         https://www.digitalocean.com/blog/digitalocean-kubernetes-high-availability-control-plane-sla

Google Anthos (Kubeception)      https://cloud.google.com/anthos/clusters/docs/on-prem/latest/how-to/create-user-cluster-controlplane-v2