Kubernetes (k8s) enables you to efficiently orchestrate container management, in the cloud or on-premises. As a whole, k8s provides many benefits, including features for self-healing, automated rollouts and rollbacks, load distribution, and scalability. However, k8s is a highly complex platform and requires extensive configuration. To ensure your k8s operations are secure, you need to address key challenges related to architecture vulnerabilities and platform dependencies, and implement relevant Kubernetes security best practices.
Kubernetes (k8s) is an open-source orchestration platform that you can use to automate the deployment and management of containers. It enables developers and IT professionals to maintain, monitor, and scale container deployments on-premises or in the cloud.
K8s is based on a client-server architecture that uses clusters of nodes (servers or VMs), each hosting a number of pods (containers). Each server and client includes the following components:
Kubernetes is operated through a REST API that enables programmatic management of the platform. It includes features for self-healing, automated rollouts and rollbacks, load distribution, and scalability.
If you are using or plan to use Kubernetes it’s important that you have monitoring in place that can scale with your projects. Stackify Retrace supports containers like Kubernetes and enables continuous application improvements. Learn more here.
When using Kubernetes to manage container deployments there are a few challenges you may face.
The architecture of Kubernetes creates a large and dynamic surface area for attack. Each node and pod is an additional possible entry point for attackers and must be carefully configured.
Additionally, pods are constantly being created and destroyed to meet workload demands and to maintain deployment health. This makes monitoring and detection of threats a significant challenge. Systems must incorporate service discovery features to ensure that all assets are known – otherwise components are missed.
The dynamic nature of k8s deployments also create a challenge for maintaining regulatory compliance. Compliance requires logging and auditability that are difficult to achieve with ephemeral containers. For example, to maintain a history of events, you need to export logs from your pods to a persistent data store. However, IP addresses may be reused for pods so your logs may identify two different containers as the same because of a shared IP.
Kubernetes is not a freestanding platform. It requires numerous dependencies and often teams incorporate a range of third-party tooling to help ease management.
Some of these tools are designed specifically to increase Kubernetes security by centralizing monitoring and detection. However, others may introduce vulnerabilities. This means you need to carefully assess which components k8s relies on and ensure that those components are internally secure and externally monitored and protected.
Although containers do have some protection from attacks due to being isolated instances, these assets are not invulnerable. The biggest threats to container security are infected images, improper access controls, and root privileges. These threats remain as long as containers are running.
Infected images may contain malware that is used to override container configurations or to spread infection to persistent systems. Improper access controls may enable services or users to either access restricted data in the container or to leverage containers to access system data.
Meanwhile, root privileges enable successful attackers to perform just about any action they want with a container. This includes modifying system settings or deploying new containers.
Below are a few examples of how the above vulnerabilities can be used in an attack:
When deploying Kubernetes, there are several best practices that can help you ensure that your deployment is as secure as possible. Below are a few to start with.
RBACs enable you to define permissions by role, making management simpler, and reducing the chance of misconfiguration. These controls are enabled by default in k8s v1.6 and up. However, if you are using an older version of Kubernetes, managed services, or have recently upgraded, you should double-check your settings.
For RBAC to function correctly, you need to have it enabled and the legacy attribute-based access controls (ABACs) disabled. Once you are sure your controls are enabled correctly, you should prioritize defining namespace-specific permissions over cluster-wide permissions. This helps reduce the chance that attackers are able to move laterally through your clusters if credentials are compromised.
Hardening your k8s nodes restricts attacker access to assets and reduces the chance of compromise. Generally, this involves tightening your configurations and limiting the traffic allowed to or between pods.
To harden your nodes, you can start by comparing your configurations to the Center for Internet Security (CIS) Benchmarks. These benchmarks provide a step-by-step checklist of Kubernetes security best practices.
Next, ensure that you are controlling network traffic to sensitive ports. For example, those used by kubelet. You should also consider limiting k8s API access to only trusted networks or requiring authentication and authorization before access is given.
Properly monitoring events in your deployment is vital. You should ensure that you have audit logging enabled and that you are analyzing these logs continuously. In particular, you should be looking for suspicious or unexpected API calls and authorization failures.
If your deployment is hosted on a public cloud, your provider should make this data available through their console or API. They should also include the ability to alert on certain events. If you are self-hosting, you need to ingest this data into a third-party tool for analysis and alerting.
Your etcd cluster is one of the most sensitive assets in your k8s deployment. It contains all of the information on your cluster states, including configurations and secrets. This information makes it an appealing target for attackers.
When protecting etcd, you need to be mindful of both read and write access permissions. Write access can provide attackers full control over your clusters and read access can be used for reconnaissance or privilege escalation.
To protect your cluster, you should enable TLS for both client-to-server and server-to-server communications. You should also configure a firewall between your etcd cluster and API server.
Kubernetes can be incredibly useful for managing containers, but you should properly understand Kubernetes security challenges before adopting Kubernetes. To ensure the continual health of your operations, you need to secure your architecture, platform dependencies, and containers.
There is a wide range of techniques you can use to secure Kuberenetes workloads. You can start by enabling role-based access controls, to prevent exploitation of privileges. You should also harden Kubernetes nodes, to restrict attacker access to assets. To protect your etcd cluster, you can enable TLS and configure a firewall. Finally, you should turn on audit logging to ensure you have continuous visibility.
Stackify's APM tools are used by thousands of .NET, Java, PHP, Node.js, Python, & Ruby developers all over the world.
Explore Retrace's product features to learn more.
If you would like to be a guest contributor to the Stackify blog please reach out to [email protected]
