11 Ways (Not) to Get Hacked

This article was originally published on the Kubernetes Blog (opens in a new tab)

Kubernetes security has come a long way since the project's inception, but still contains some gotchas. Starting with the control plane, building up through workload and network security, and finishing with a projection into the future of security, here is a list of handy tips to help harden your clusters and increase their resilience if compromised.

• Part One: The Control Plane
• 1. TLS Everywhere
• 2. Enable RBAC with Least Privilege, Disable ABAC, and Monitor Logs
• 3. Use Third Party Auth for API Server
• 4. Separate and Firewall your etcd Cluster
• 5. Rotate Keys
• Part Two: Workloads
• 6. Use Linux Security Features and PodSecurityPolicies
• 7. Statically Analyse YAML
• 8. Run Containers as a Non-Root User
• 9. Use Network Policies
• 10. Scan Images and Run IDS
• Part Three: The Future
• 11. Run a Service Mesh
• Conclusion

Part One: The Control Plane

The control plane is Kubernetes' brain. It has an overall view of every container and pod running on the cluster, can schedule new pods (which can include containers with root access to their parent node), and can read all the secrets stored in the cluster. This valuable cargo needs protecting from accidental leakage and malicious intent: when it's accessed, when it's at rest, and when it's being transported across the network.

1. TLS Everywhere

TLS should be enabled for every component that supports it to prevent traffic sniffing, verify the identity of the server, and (for mutual TLS) verify the identity of the client.

Note that some components and installation methods may enable local ports over HTTP and administrators should familiarize themselves with the settings of each component to identify potentially unsecured traffic.

Source (opens in a new tab)

This network diagram by Lucas Käldström (opens in a new tab) demonstrates some of the places TLS should ideally be applied: between every component on the master, and between the Kubelet and API server. Kelsey Hightower (opens in a new tab)'s canonical Kubernetes The Hard Way (opens in a new tab) provides detailed manual instructions, as does etcd's security model (opens in a new tab) documentation.

{{< hidden >}} medium article was deleted. Consider replacing the article. For now we will use an archive to fix the 410 {{< /hidden >}}

Autoscaling Kubernetes nodes was historically difficult, as each node requires a TLS key to connect to the master, and baking secrets into base images is not good practice. Kubelet TLS bootstrapping (opens in a new tab) provides the ability for a new kubelet to create a certificate signing request so that certificates are generated at boot time.

2. Enable RBAC with Least Privilege, Disable ABAC, and Monitor Logs

Role-based access control provides fine-grained policy management for user access to resources, such as access to namespaces.

In Kubernetes, ABAC (Attribute Based Access Control) has been superseded by RBAC (opens in a new tab) since 1.6, and should not be enabled on the API server. Use this flag to disable it:

There a plenty of good examples (opens in a new tab) of RBAC policies for cluster services (opens in a new tab), as well as the docs (opens in a new tab). And it doesn't have to stop there - fine-grained RBAC policies can be extracted from audit logs with audit2rbac (opens in a new tab).

Incorrect or excessively permissive RBAC policies are a security threat in case of a compromised pod - maintaining least privilege and continuously reviewing and improving RBAC rules should be considered part of the technical debt hygiene that any good team has built into their development lifecycle.

Audit Logging (beta in 1.9) provides customisable API logging at payload (request and response), and metadata levels. They can be tuned to your organisation's security policy. GKE (opens in a new tab) provides sane defaults.

For read requests such as get, list, and watch, only the request object is saved in the audit logs; the response object is not. For requests involving sensitive data such as Secret and ConfigMap, only the metadata is exported. For all other requests, both request and response objects are saved in audit logs.

Don't forget: keeping these logs inside the cluster is a security threat in case of compromise. These, like all other security-sensitive logs, should be transported outside the cluster to prevent tampering in the event of a breach.

https://kubernetes.io/docs/tasks/debug-application-cluster/audit/ (opens in a new tab)

3. Use Third Party Auth for API Server

Centralising authentication and authorisation (aka Single Sign On) across an organisation helps onboarding, offboarding, and consistent permissions for users.

Integrating Kubernetes with third party auth providers (like Google or Github) uses the remote platform's identity guarantees (backed up by things like 2FA) and prevents administrators having to reconfigure the Kubernetes API server to add or remove users.

Dex (opens in a new tab) is an OpenID Connect Identity (OIDC) and OAuth 2.0 provider with pluggable connectors. Pusher takes this a stage further with some custom tooling (opens in a new tab), and there are some other (opens in a new tab) helpers (opens in a new tab) with slightly different use cases.

4. Separate and Firewall your etcd Cluster

etcd stores information on state and secrets, and is a critical Kubernetes component - it should be protected differently from the rest of your cluster.

Write access to the API server's etcd is equivalent to gaining root on the entire cluster, and even read access can be used to escalate privileges fairly easily.

The Kubernetes scheduler will search etcd for pod definitions that do not have a node. It then sends the pods it finds to an available kubelet for scheduling. Validation for submitted pods is performed by the API server before it writes them to etcd, so malicious users writing directly to etcd can bypass many security mechanisms - e.g. PodSecurityPolicies.

etcd should be configured with peer and client TLS certificates (opens in a new tab), and deployed on dedicated nodes. To mitigate against private keys being stolen and used from worker nodes, the cluster can also be firewalled to the API server.

5. Rotate Keys

A security best practice is to regularly rotate encryption keys and certificates, in order to limit the "blast radius" of a key compromise.

Kubernetes will rotate some certificates automatically (opens in a new tab) (notably, the kubelet client and server certs) by creating new CSRs as its existing credentials expire.

However, the symmetric encryption keys (opens in a new tab) that the API server uses to encrypt etcd values are not automatically rotated - they must be rotated manually (opens in a new tab). Master access is required to do this, so managed services (such as GKE or AKS) abstract this problem from an operator.

Part Two: Workloads

With minimum viable security on the control plane the cluster is able to operate securely. But, like a ship carrying potentially dangerous cargo, the containers must be protected to contain that cargo in the event of an unexpected accident or breach. The same is true for Kubernetes workloads (pods, deployments, jobs, sets, etc.) - they may be trusted at deployment time, but if they're internet-facing there's always a risk of later exploitation. Running workloads with minimal privileges and hardening their runtime configuration can help to mitigate this risk.

6. Use Linux Security Features and PodSecurityPolicies

The Linux kernel has a number of overlapping security extensions (capabilities, SELinux, AppArmor, seccomp-bpf) that can be configured to provide least privilege to applications.

Tools like bane (opens in a new tab) can help to generate AppArmor profiles, and docker-slim (opens in a new tab) for seccomp, but beware - a comprehensive test suite it required to exercise all code paths in your application when verifying the side effects of applying these policies.

PodSecurityPolicies (opens in a new tab) can be used to mandate the use of security policies and other Kubernetes security directives. They provide a minimum contract that a pod must fulfil to be submitted to the API server - including security profiles, the privileged flag, and the sharing of host network, process, or IPC namespaces.

These directives are important, as they help to prevent containerised processes from escaping their isolation boundaries, and Tim Allclair (opens in a new tab)'s example PodSecurityPolicy (opens in a new tab) is a comprehensive resource that you can customise to your use case.

7. Statically Analyse YAML

Where PodSecurityPolicies deny access to the API server, static analysis can also be used in the development workflow to model an organisation's compliance requirements or risk appetite.

Sensitive information should not be stored in pod-type YAML resource (deployments, pods, sets, etc.), and sensitive configmaps and secrets should be encrypted with tools such as vault (opens in a new tab) (with CoreOS's operator), git-crypt (opens in a new tab), sealed secrets (opens in a new tab), or cloud provider KMS (opens in a new tab).

Static analysis of YAML configuration can be used to establish a baseline for runtime security. kubesec (opens in a new tab) generates risk scores for resources:

And kubetest (opens in a new tab) is a unit test framework for Kubernetes configurations:

These tools " shift left (opens in a new tab)" (moving checks and verification earlier in the development cycle). Security testing in the development phase gives users fast feedback about code and configuration that may be rejected by a later manual or automated check, and can reduce the friction of introducing more secure practices.

8. Run Containers as a Non-Root User

Containers that run as root frequently have far more permissions than their workload requires which, in case of compromise, could give an attacker far more opportunities.

With containers, we still rely on the traditional Unix security model (called discretionary access control (opens in a new tab) or DAC) - everything is a file, and permissions are granted to users and groups.

User namespaces are not enabled in Kubernetes. This means that a container's user ID table maps to the host's user table, and running a process as the root user inside a container runs it as root on the host. Although we have layered security mechanisms to prevent container breakouts, this is still not recommended.

Many container images use the root user to run PID 1 - if that process is compromised, the attacker has root in the container, and any mis-configurations become much easier to exploit.

Bitnami has done a lot of work (opens in a new tab) moving their container images to non-root users (opens in a new tab) (especially as OpenShift requires this by default), which may ease a migration to non-root container images.

This PodSecurityPolicy snippet prevents running processes as root inside a container, and also escalation to root:

Non-root containers cannot bind to the privileged ports under 1024 (this is gated by the CAP_NET_BIND_SERVICE kernel capability), but services can be used to disguise this fact (here the application inside the container is bound to 8443, but the service exposes it on 443):

Having to run as a non-root user is not going to change until user namespaces are usable, or the ongoing work to run containers without root (opens in a new tab) lands in container runtimes.

9. Use Network Policies

By default, Kubernetes networking allows all pod to pod traffic; this can be restricted using a Network Policy (opens in a new tab) .

Traditional services are restricted with firewalls, which use static IP and port ranges for each service. As these IPs very rarely change they have historically been used as a form of identity. Containers rarely have static IPs - they are built to fail fast, be rescheduled quickly, and use service discovery instead of static IP addresses. These properties mean that firewalls become much more difficult to configure and review.

As Kubernetes stores all its system state in etcd it can configure dynamic firewalling - if it is supported by the CNI networking plugin. Calico, Cilium, kube-router, Romana, and Weave Net all support network policy.

It should be noted that these policies fail-closed, so the absence of a podSelector here defaults to a wildcard:

Here's an example NetworkPolicy that denies all egress except UDP 53 (DNS), which also prevents inbound connections to your application. NetworkPolicies are stateful (opens in a new tab), so the replies to outbound requests still reach the application.

Kubernetes network policies can not be applied to DNS names. This is because DNS can resolve round-robin to many IPs, or dynamically based on the calling IP, so network policies can be applied to a fixed IP or podSelector (for dynamic Kubernetes IPs) only.

Best practice is to start by denying all traffic for a namespace and incrementally add routes to allow an application to pass its acceptance test suite. This can become complex, so ControlPlane hacked together netassert (opens in a new tab) - network security testing for DevSecOps workflows with highly parallelised nmap:

Cloud provider metadata APIs are a constant source of escalation (as the recent Shopify (opens in a new tab) bug bounty (opens in a new tab) demonstrates), so specific tests to confirm that the APIs are blocked on the container network helps to guard against accidental misconfiguration.

10. Scan Images and Run IDS

Web servers present an attack surface to the network they're attached to: scanning an image's installed files ensures the absence of known vulnerabilities that an attacker could exploit to gain remote access to the container. An IDS (Intrusion Detection System) detects them if they do.

Kubernetes permits pods into the cluster through a series of admission controller (opens in a new tab) gates, which are applied to pods and other resources like deployments. These gates can validate each pod for admission or change its contents, and they now support backend webhooks.

These webhooks can be used by container image scanning tools to validate images before they are deployed to the cluster. Images that have failed checks can be refused admission.

Scanning container images for known vulnerabilities can reduce the window of time that an attacker can exploit a disclosed CVE. Free tools such as CoreOS's Clair (opens in a new tab) and Aqua's Micro Scanner (opens in a new tab) should be used in a deployment pipeline to prevent the deployment of images with critical, exploitable vulnerabilities.

Tools such as Grafeas (opens in a new tab) can store image metadata for constant compliance and vulnerability checks against a container's unique signature (a content addressable (opens in a new tab) hash). This means that scanning a container image with that hash is the same as scanning the images deployed in production, and can be done continually without requiring access to production environments.

Unknown Zero Day vulnerabilities will always exist, and so intrusion detection tools such as Twistlock (opens in a new tab), Aqua (opens in a new tab), and Sysdig Secure (opens in a new tab) should be deployed in Kubernetes. IDS detects unusual behaviours in a container and pauses or kills it - Sysdig's Falco (opens in a new tab) is a an Open Source rules engine, and an entrypoint to this ecosystem.

Part Three: The Future

The next stage of security's "cloud native evolution" looks to be the service mesh, although adoption may take time - migration involves shifting complexity from applications to the mesh infrastructure, and organisations will be keen to understand best-practice.

11. Run a Service Mesh

A service mesh is a web of encrypted persistent connections, made between high performance "sidecar" proxy servers like Envoy and Linkerd. It adds traffic management, monitoring, and policy - all without microservice changes.

Offloading microservice security and networking code to a shared, battle tested set of libraries was already possible with Linkerd (opens in a new tab), and the introduction of Istio (opens in a new tab) by Google, IBM, and Lyft, has added an alternative in this space. With the addition of SPIFFE (opens in a new tab) for per-pod cryptographic identity and a plethora of other features (opens in a new tab), Istio could simplify the deployment of the next generation of network security.

In "Zero Trust" networks there may be no need for traditional firewalling or Kubernetes network policy, as every interaction occurs over mTLS (mutual TLS), ensuring that both parties are not only communicating securely, but that the identity of both services is known.

This shift from traditional networking to Cloud Native security principles is not one we expect to be easy for those with a traditional security mindset, and the Zero Trust Networking book (opens in a new tab) from SPIFFE's Evan Gilman (opens in a new tab) is a highly recommended introduction to this brave new world.

Istio 0.8 LTS (opens in a new tab) is out, and the project is approaching 1.0. Its stability versioning is the same as the Kubernetes model: a stable core, with individual APIs identifying themselves under their own alpha/beta stability namespace. Expect to see an uptick in adoption of 0.8 soon!

Conclusion

Cloud Native applications have a greater, more fine-grained set of lightweight security primitives to lock down workloads and infrastructure. The power and flexibility of these tools is both a blessing and curse - with insufficient automation it has become easier to expose insecure workloads which permit breakouts from the container or its isolation model.

There are more defensive tools available than ever, but caution must be taken to reduce attack surfaces and the potential for misconfiguration.

However if security slows down an organisation's pace of feature delivery it will never be a first-class citizen. Applying Continuous Delivery principles to the software supply chain allows an organisation to achieve compliance, continual audit, and high security without impacting the business's bottom line.

The only way to iterate quickly on software and security is when it is supported by a comprehensive test suite. This is achieved with Continuous Security - an alternative to point-in-time penetration tests, with constant pipeline validation ensuring an organisation's attack surface is known, and the risk constantly understood and managed. This is ControlPlane's modus operandi: if we can help kickstart a Continuous Security discipline, deliver Kubernetes security and operations training, or co-implement a secure cloud native evolution for you, please get in touch.

Andrew Martin is a co-founder at @controlplaneio (opens in a new tab) and tweets about cloud native security at @sublimino (opens in a new tab)