Context

At one of our clients, we have multiple development teams all creating microservices. These microservices are deployed with a CI/CD (Continuous Integration and Delivery) pipeline to a shared Kubernetes cluster. We strive to give much responsibility and flexibility to these development teams in order to speed up the software delivery process. Lots of settings, in the past, the exclusive realm of the operation team, can now be configured by our developers within Kubernetes manifest files. Examples:

  • Resource needs
  • Public IP vs Private IP for ingress service
  • SSD usage vs HDD usage
  • Running containers in privileged (root) mode
  • …etc.

How can we control whether our applications run within the boundaries we define? We need a tool to create policies/rules which define those boundaries and a tool to enforce them.

OPA

Enter OPA.

OPA, short for ‘Open Policy Agent’ is a ‘General Purpose Policy Engine’. OPA gives us a higher-level, declarative, language to author our policies and to enforce them within our environment, leaving our cluster at all times compliant with the company policies. OPA provides a high-level declarative language for specifying these policies as code which is called Rego. More information on Rego can be found here.

Example of a Rego policy:

deny[msg] {

                input.kind = “Deployment”

                not input.spec.template.spec.securityContext.runAsNonRoot = true

                msg = “Containers must not run as root”

}

The above Rego policy shows a deny message in case a Kubernetes Deploymentis configured to run as root.

OPA is an open-source policy engine (part of CNCF) not bound to Kubernetes and it can be integrated into a myriad of systems and platforms, leveraging use case which reach far beyond Kubernetes governance.

OPA & Kubernetes

OPA is integrated within Kubernetes as an Admission Controller. An admission controller is a piece of code that intercepts requests to the Kubernetes API server before an object is persisted. For example, if you send a request to the API server to create a Deployment resource, the admission controller may intercept this request, mutate or validate it. When we use OPA as an admission controller we can enforce OPA policies on these requests send to the API server before they are processed by the API server. This gives us many benefits like:

  • Make sure our containers do not run in ‘privileged’ mode
  • Make sure developers can only create internal load balancers!
  • Make sure applications cannot use expensive SSDs
  • Make sure configurations have the proper labels attached
  • All PODs must have an upper bound for resource usage
  • Used Container Registry must be in allowed list.
  • Check for Kubernetes API deprecations.
  • Etc….

The OPA admission controller we use within Azure Kubernetes Service is called Gatekeeper. Gatekeeper makes from OPA, the “General Purpose Policy Engine”, a first-class Kubernetes citizen. It evaluates incoming requests based on Rego policies we define and answers the API server what to do with these request (deny/allow). In practice we can deny requests made to the API server if these requests are not compliant with the policies we define.

Gatekeeper is a native Kubernetes initiative and can be easily installed into Azure AKS (or Amazon EKS or Google GKE)

Azure AKS and Azure Policy

Many companies today use Azure Policy as the central governance service within Azure cloud where we can define policies and enable them on Azure resources. Azure Policy for Azure Kubernetes Service integrates Azure Policy with Azure AKS. It extends Gatekeeper to enforce policies within your Azure AKS cluster. These policies are provided out of the box by Azure Policy and managed centrally so you can report the compliance of all your clusters from one central place.

You can install it in your Azure AKS cluster by simply enabling the add on for Azure Policy:

Figure 1: Enable Azure Policy add-on on Azure AKS

Enabling the add-on will automatically install Gatekeeper (in namespace ‘gatekeeper-system’) and the Azure Policy for Kubernetes Add-On (in namespace ‘kube-system’).

Figure 2: Gatekeeper and Azure Policy in Azure AKS

Some important limitations of the add-on:

  • Today only built-in policy definitions are supported.
  • Only available for Linux node pools.

Azure Policy for Kubernetes contains a set of built-in policies which you can assign to your cluster. This makes applying policies very user friendly compared to writing your own Rego policies.

The existing Azure policy language contains extensions (constraintTemplate) to provide Custom Resource Definitions (CRD’s) containing the policy in the Rego language. This usage of Rego would allow us to reuse existing rules and apply them via Azure Policy within a unified cloud governance and compliance reporting environment.

Figure 3: Azure Policy – Built-in policies for Kubernetes

For example, the ‘ConstraintTemplate’ which contains the Rego policy to deny all Kubernetes deployments in the default namespace looks like this:

apiVersion: templates.gatekeeper.sh/v1beta1

kind: ConstraintTemplate

metadata:

  name: k8sazureblockdefault

spec:

  crd:

    spec:

      names:

        kind: K8sAzureBlockDefault

  targets:

    – target: admission.k8s.gatekeeper.sh

      rego: |

        package k8sazureblockdefault

        violation[{“msg”: msg}] {

          obj := input.review.object

          is_default_namespace(obj.metadata)

          msg := sprintf(“Usage of the default namespace is not allowed, name: %v, kind: %v”, [obj.metadata.name, obj.kind])

        }

        is_default_namespace(metadata) {

          not metadata.namespace

        }

        is_default_namespace(metadata) {

          metadata.namespace == “default”

        }

Figure 4: Source: https://raw.githubusercontent.com/Azure/azure-policy/master/built-in-references/Kubernetes/block-default-namespace/template.yaml

Conclusion

Validating Kubernetes manifests with OPA policies in production, now for almost 6 months, gives us much more control while keeping a fast (automatic) software delivery process. OPA is one of the cornerstones of our Kubernetes governance initiatives. Enabling Azure Policies for Kubernetes tremendously facilitated our governance approach by providing a user-friendly centralized management of policies and reporting of policy compliance.

Author: Bert Meuris