Author: Adnan Abdulhussein

As more teams adopt Kubernetes in production, specific use cases and needs have emerged that build on the core feature set of the project. Rather than attempt to fit every requirement in Kubernetes itself, the community has worked towards building an extension framework to enable developers to build support for these different scenarios. Examples of customizing Kubernetes include configuring different network or storage plugins, restricting what container images can be run inside Pods and other admission policies, or creating API extensions for automating common cluster operations. Let’s take a deeper look at the latter type of extension.

An introduction to Operators

Certain operations inside a cluster are great cases for automation. Other than reducing toil, automation enables access control, logging, and auditing. As an example, Kubernetes could be extended to support new primitives, such as a serverless function or a database. To do so, Kubernetes allows the creation of custom API resources to represent this information. With Custom Resource Definitions (CRD) it becomes possible to interact with extensions the same way users interact with the core APIs. Below is an example of what a custom API for the serverless function might look like:

apiVersion: myextension.com/v1alpha1
kind: ServerlessFunction
metadata:
  name: my-function
spec:
  runtime: nodejs
  function: |
    console.log("Hello")

Like the core API, custom resources can be watched and acted upon using the controller model. Controllers are programs that interact with the Kubernetes API to monitor changes to resources and ensure the system state is updated to match the desired state. For a more in-depth look at the controller model, check out this deep-dive. Alongside a CRD, an extension might include a controller to implement some new functionality. Going back to the Serverless Function example, a controller could handle the following functions:

• When a ServerlessFunction resource is created
  • Create a Pod to run the Serverless Function
  • Create a Service to expose the Serverless Function
• When a ServerlessFunction resource is updated
  • Update the related Pod to reflect changes to the function code or runtime
• When a ServerlessFunction resource is deleted
  • Delete the related Pod and Service

Controllers and CRDs can encode specific knowledge about a service or application, and automate the provisioning, scaling and backup/restore operations. Extensions of this class are typically referred to as Operators. Other examples of Operators available in the ecosystem can be found in this list.

Developing custom controllers or Operators

As Kubernetes is written in Go, the low-level libraries used to interact with the API are also written in Go. Though there are clients for Python, JavaScript and other languages, the Go client (client-go) is the earliest and most well-supported. As a result, most custom controllers and Operators are written in Go and make use of client-go.

To get started developing a custom controller, a framework or SDK can help scaffold a new project. There are a handful of different tools available in the ecosystem, this article gives an overview of Kubebuilder and the Operator SDK as two of the most popular toolkits, and metacontroller as a more interesting take on building controllers.

Kubebuilder

Kubebuilder is maintained by the Kubernetes API Machinery SIG (Special Interest Group) and enables developers to easily bootstrap a new API extension. It has support for defining multiple custom APIs using CRDs, and scaffolds out the controller code to watch the custom resources. With Kubebuilder, developers can just focus on implementing the reconcile function to update the system state with the desired state.

Kubebuilder also sets up a test system that can be run against a cluster, which allows developers to write integration tests for their custom APIs and reconcile logic. Once ready to deploy the custom controller, Kubebuilder provides a Dockerfile and manifests to build and deploy to a cluster.

Operator SDK

The Operator SDK comes from the CoreOS team that originally introduced the Operator model. It uses the same underlying libraries as Kubebuilder, but builds on them by adding support for generating Operators from existing Helm charts or using Ansible playbooks. It also integrates with the Operator Lifecycle Manager for added management of Operators inside the cluster.

The Operator SDK focuses on supporting the Operator concept, to provision and operate the lifecycle of a specific application. Therefore, the Operator SDK is a good fit for codifying the operational knowledge of a specific complex application (e.g. the etcd-operator that manages instances of etcd). Alternatively, Kubebuilder is a great choice for more general purpose automation tasks or abstractions (e.g. the sealed-secrets controller which handles decryption of encrypted Secrets).

Metacontroller

The Metacontroller simplifies custom controller development by completely abstracting the machinery of controllers and allows developers to just focus on the logic of the API extension. The Metacontroller is deployed inside clusters and calls out to webhooks defined by developers when changes to the custom API happen. The webhooks can be implemented in any language and can return responses to instruct the Metacontroller what changes to apply.

Since a webhook can be implemented by any HTTP server, Metacontroller makes the development of custom controllers and Operators much more accessible as developers don’t need to learn Go.

Conclusion

From automating ad-hoc operational tasks, such as ensuring all Service objects hold a label, to orchestrating instances of a complex stateful application, Kubernetes API extensions are a great way to extend the core functionality. The Operator ecosystem is growing, both in terms of tooling to help build and distribute Operators, and the availability of off-the-shelf Operators that are ready to deploy and use. This ultimately makes Kubernetes a very powerful platform for running and operating cloud-native applications.