Securing Kubernetes Clusters: Lessons and Best Practices from the Field

Kubernetes has become the backbone of cloud native infrastructure, offering scalability and resilience for modern applications, but with its growing popularity comes an increased risk of attacks. In a presentation at KubeCon EU 2023, cybersecurity experts from Control Plane demonstrated real-world attack scenarios and how to defend against them. This blog post will summarise key lessons from their “Hacking and Defending Kubernetes Clusters” talk, focusing on threat modeling, exploit scenarios, and critical security practices.

Why Threat Modeling is Essential for Kubernetes Security

Before diving into the attack scenarios, the presenters emphasised the importance of threat modeling in securing Kubernetes environments. Threat modeling helps you identify and address potential risks before they become vulnerabilities in production systems. The goal is to evaluate how your system might be attacked and implement security controls to mitigate those risks pragmatically.

Key Takeaways

• Collaboration is Key

  • Involve developers, security experts, SREs, and domain experts in threat modeling sessions.
  • Each participant provides unique insights into potential risks and mitigation strategies.

• Timing Matters

  • Perform threat modeling early in development.
  • Ideally, conduct sessions when the architecture is still being drafted.
  • Catch vulnerabilities before deployment to improve security and reduce costs.

By leveraging frameworks like the Microsoft Threat Matrix for Kubernetes, teams can systematically evaluate the attack vectors and security controls specific to their Kubernetes setup.

Scenario 1: Discovery and Initial Access

In the first scenario, the ControlPlane presenters, Fabian Kammel and James Cleverley-Prance, illustrated how an attacker could gain initial access to a Kubernetes cluster through basic misconfigurations.

Attack Summary

• A Kubernetes admin deployed JupyterHub using a Helm chart. However, the admin exposed a node port to the internet and left the default credentials unchanged.
• Attackers discovered the open node port via port scanning and accessed the Jupyter Hub instance using default credentials. From there, they launched a terminal within the pod, gaining execution access.

Mitigation Strategies

1. Network Segmentation

   • Restrict node port services to minimise exposure.
   • Apply network policies to prevent unauthorised access.

2. Strong Authentication

   • Always change default credentials.
   • Implement multi-factor authentication (MFA) for critical services.

3. Load Balancer Configuration

   • Use load balancers to mitigate risks.
   • Avoid exposing node ports directly to the internet.

Scenario 2: Persistence Through Leaked Secrets

Attack Summary

• A sensitive Kubernetes token was leaked via a Git repository.
• The attacker used the token to authenticate against the Kubernetes API server and executed commands within the cluster.
• They deployed a persistent workload to maintain access, even if certain pods were deleted.

Mitigation Strategies

1. Restrict API Access: Limit access to the Kubernetes API server through IP allowlists or private networks.
2. Policy Enforcement: Use Kubernetes policy frameworks like OPA (Open Policy Agent) or Kyverno to restrict the creation of unauthorised workloads.
3. Secrets Management: Use short-lived secrets and audit secret access regularly to detect leaks early.

Scenario 3: Privilege Escalation via Privileged Containers

Attack Summary

• The attacker gained access to a privileged container and discovered unmasked host devices.
• They inserted a malicious SSH key by mounting the host’s root file system, allowing direct SSH access to the host machine and bypassing container isolation.

Mitigation Strategies

1. Limit Privileged Containers: Privileged containers should rarely be used in day-to-day operations. Audit your cluster for any unnecessary use of privilege.
2. Pod Security Standards: To prevent container breakout attempts, enforce strict pod security policies through Kubernetes’ Pod Security Standards.
3. Host Hardening: Implement OS-level security measures to protect host resources from container exploits.

Scenario 4: Unauthorised Access via Service Accounts

Attack Summary

• The attacker deployed a pod and accessed Kubernetes API resources using the default service account.
• They enumerated cluster components and discovered valuable information, such as secrets.
• The attacker leveraged these permissions to escalate their attack further.

Mitigation Strategies

1. Auto-Mount Service Account Token: Set autoMountServiceAccountToken to false to disable automatic mounting of service account tokens for pods that don’t require API access.
2. Least Privilege Principle: Create and assign service accounts with the least privileges necessary for each workload. Define custom roles to limit access scope.
3. Secrets Protection: Store sensitive information securely and ensure secrets are not unnecessarily accessible.

Scenario 5: Poisoned Container Images

Attack Summary

• The attacker used stolen Docker credentials to modify an image in the container registry.
• They overwrote the “latest” tag with a compromised version.
• The pod was redeployed, pulling the poisoned image and executing malicious code within the cluster.

Mitigation Strategies

1. Pin Images by Hash: Always pin container images to a specific digest (hash) instead of using mutable tags like “latest.”
2. Container Signing: Use container image signing to deploy only trusted and verified images.
3. Short-Lived Secrets: Use short-lived secrets for container registry access to reduce the risk of stolen credentials being exploited.

Scenario 6: Evading Detection by Deleting Events

Attack Summary

• After compromising the cluster, the attacker deleted Kubernetes event logs to avoid detection.
• Since event logs are critical for monitoring, this tactic could slow incident response.

Mitigation Strategies

1. Log Aggregation: Use external logging solutions (e.g., ELK, Splunk) or cloud provider logging tools to ensure logs are preserved even if deleted within the cluster.
2. Restrict Event Deletion: Limit the ability to delete events to cluster admins and audit those permissions regularly.
3. Monitor for Anomalies: Implement monitoring to detect unusual log deletion or other suspicious behavior.

Key Lessons and Takeaways

Securing Kubernetes is complex, but understanding how attackers operate can help defend your clusters better. The six scenarios highlight common attack vectors, from initial access to privilege escalation and evasion. By adopting a proactive security posture through threat modeling and implementing key defenses, organisations can greatly reduce the risk of compromise.

Summary of Best Practices

1. Threat Modeling: Use tools like the Microsoft Threat Matrix for Kubernetes to map attack paths and prioritise security controls.
2. Network and API Security: Segment your network, restrict API access, and ensure sensitive credentials are stored securely.
3. Policy Enforcement: Use Kubernetes-native policy frameworks to prevent unauthorised actions and enforce least privilege access.
4. Monitoring and Logging: Implement centralised logging and monitoring to detect malicious activity and respond quickly.

By understanding attacker tactics and implementing proactive security controls, you can harden your Kubernetes environments and stay ahead of potential threats. 🚀

To watch these scenarios for yourself, click this link. If you’re looking to secure your Kubernetes clusters, Control Plane offers penetration testing, threat modelling, and workshops to help you build a secure-by-design infrastructure. Contact us to learn more!