Hands-on Kubernetes Security

By Pi Unnerup

Kubernetes misconfiguration mistakes are easy to make. ControlPlane has a free Kubernetes Simulator to help engineers and managers learn about Kubernetes security by hacking and fixing clusters.

Kubernetes: The Threat of Misconfiguration

A StackRox report from this year found that 94% of respondents had experienced a Kubernetes related security incident in the last year. Misconfigurations were by far the most common type of Kubernetes vulnerability, reported by seven out of ten companies.

Note. Reprinted from The State of Container and Kubernetes Security, Winter 2020, by StackRox, retrieved from https://security.stackrox.com/rs/219-UEH-533/images/State_of_Container_and_Kubernetes_Report.pdf

DevSecOps can bring operational advantages when leveraging the security abstractions provided by Kubernetes. However, it would seem that the knowledge of how to implement the processes, tooling and configuration lags behind the wave of enthusiasm for containerised technology. Tools such as the Kubernetes Simulator hope to breach this gap by guiding users through specific attack scenarios.

The Kubernetes Attack Surface

The Kubernetes control plane is a juicy target for cluster attacks. The perimeters and content of nodes, masters, core components, APIs, and public-facing pods all require inspection and hardening to properly defend against existing or potential vulnerabilities. It’s no surprise that out of 540 IT professionals surveyed in the report, 44% recorded having to delay deployment due to security concerns.

Note. Reprinted from The State of Container and Kubernetes Security, Winter 2020, by StackRox, retrieved from https://security.stackrox.com/rs/219-UEH-533/images/State_of_Container_and_Kubernetes_Report.pdf

While the default configuration of Kubernetes is not insecure, effort is needed to harden it for workloads; including the initial best-practice recommendations that need to be explicitly enabled and configured:

Network Policies (firewall rules): By default, all deployments on the Kubernetes’ flat network can reach all other deployments even across namespaces. Missing isolation between pods means that a compromised workload can be the starting point of an attack on other components in the network. This is what happened in the infamous 2017 Equifax breach where attackers used a known vulnerability to compromise a web portal, and from there escalated to back-end servers which were not appropriately segmented. Network policies prevent this exact type of escalation by defining the networking rules for pod-to-pod communication. For example, a front-end may only need to talk to the API which in turn can access the back-end.

Pod Security Policy: Kubernetes by default allows pods to be run with a number of insecure configurations. This includes running privileged containers which have root permissions on the host, usage of the host’s namespaces and filesystem, including sharing the host’s networking, and other configurations that are not necessary for most workloads. Pod security policies allow administrators to limit the privilege and permissions a pod has to adhere to before being allowed deployment into the cluster.

Secrets encryption: Surprisingly to many, secrets are not encrypted at rest by default in base distributions of Kubernetes (managed services such as GKE do encrypt secrets at rest). If anyone gets hold of the key-value store, usually Etcd, they will have access to everything in the cluster including all cluster secrets in plain text. Encrypting the cluster state store secures a cluster against data-at-rest exfiltration.

Role-based access control: RBAC is not unique to Kubernetes, but as with the other items on this list, needs to be configured correctly to avoid cluster compromise. Both users and pods can be restricted to what they can view, update, delete and create within the cluster. RBAC allows granularity over what components in the cluster a pod or user can access and, like networking policies, help limit the blast radius of a compromise.

What can we do?

Learn more! Cluster hardening is a wide-ranging and often neglected part of a cluster deployment. And it can be difficult — gaining expertise requires hands-on security experience, often only acquired by troubleshooting on live production clusters.

There are a few options including Katacoda and Instruqt, which takes users through a step by step guide on how to set up and configure Kubernetes security. ControlPlane’s free Kubernetes Simulator takes this one step further: it allows users to provision sandboxed Kubernetes clusters with intentional misconfigurations that can be explored, exploited and fixed in a gamified environment.

Tasks involve breaching and consequently fixing a Kubernetes cluster using security policies and industry best practices (including CIS benchmarks, security testing, and open source tooling). These are based on real examples gained through threat modelling, post-mortem consulting and implementation of secure Kubernetes clusters for high compliance clients in financial services and critical national infrastructure.

Of course, no cluster is more secure than the infrastructure it runs on. There is no one-size-fits-all for configuring a Kubernetes cluster, but by getting familiar with the most common misconfigurations, security teams can lessen the most common attack vector.