Kubernetes Security Standoff

While working at a tech startup in the Netherlands, I saw firsthand how crucial it is to have strong security measures. The company provided vital services to thousands of users with a cutting-edge application powered by a Kubernetes cluster consisting of many containerized microservices.

Everything changed when the news of the Shellshock vulnerability reached us. This critical flaw in the Unix Bash shell allowed attackers to execute arbitrary code remotely by exploiting how Bash processed environment variables. The tech community was sent into a frenzy, and we knew we had to act fast.

Our security team discovered that one of our containers ran a vulnerable version of Bash. Meanwhile, a cunning hacker from a notorious cybercriminal group identified our exposed weakness and devised a plan to exploit it.

The hacker launched a well-coordinated attack, exploiting the Shellshock vulnerability to infiltrate our container. They navigated the container’s environment, hunting for sensitive data and probing for ways to compromise other containers and services within our cluster.

Fortunately, our company had already implemented AppArmor profiles for all our containers. This extra layer of security limited the actions and resources available to the processes running inside the containers, effectively hampering the hacker’s attempts.

As the attacker struggled to access sensitive data and perform malicious actions, they faced barrier after barrier, thanks to our AppArmor profiles. Overcome with frustration and unable to achieve their objectives, the hacker finally abandoned their efforts, and our data and infrastructure remained secure.

This close call served as a stark reminder to our company about the importance of implementing security best practices, like AppArmor, in Kubernetes environments.

Following the incident, we conducted a thorough review of our security measures and further bolstered our defenses, ensuring the continued trust of our users and the preservation of our reputation.

Join me as I share in this article how you can leverage the power of AppArmor within your Kubernetes cluster to protect your containers from potential threats.

In this article, we’ll use the Shellshock scenario outlined in the introduction as a guiding framework to demonstrate the importance of strong security measures and AppArmor's role in safeguarding containerized applications.

AppArmor

AppArmor, short for “Application Armor,” is a Linux kernel security module that provides a simple and effective way to enforce access controls and enhance application security.

In simple terms, AppArmor is a security tool that helps protect applications in Kubernetes by defining a set of rules called a profile. This profile outlines the allowed and restricted actions for an application.

Once created, you load the profile into AppArmor and associate it with a container inside a Kubernetes pod, ensuring the application runs within the specified security boundaries.

AppArmor profiles are written in a human-readable language, allowing administrators to easily create, modify, and understand them.

When combined with other security measures, AppArmor can significantly enhance the overall security of a system, reducing the attack surface and mitigating the impact of potential threats.

Our scenario, protecting an ancient CGI application

To make this article more interesting, we will use the scenario described in the introduction.

Our business relies on a critical legacy web application, but unfortunately, we’ve lost the source code and now only have the container image to operate and maintain it.

Unfortunately, it was unclear that this containerized legacy CGI application, which operates on an Apache web server, contained a version of Bash susceptible to the widely-recognized Shellshock vulnerability.

The Shellshock vulnerability enabled attackers to execute commands following function definitions within environment variables. This posed a risk to systems that stored user input in such variables.

Web servers running CGI programs using the vulnerable Bash version were particularly susceptible, as CGI programs store request information, including user agent data and query parameters, in environment variables.

Thus, an attacker could exploit this vulnerability by crafting a user agent string and sending it to the CGI application.

Despite being unaware of the vulnerability, we decided to employ AppArmor to ensure that the CGI application only has access to the resources and actions we explicitly permitted.

Therefore we first needed to create an AppArmor profile.

Creating an AppArmor profile

To generate an AppArmor profile, we had to follow these steps.

1. Install the tools

Install Docker, AppArmor, and AppArmor utilities. The utilities contain aa-genprof, which is used to generate AppArmor profiles.

2. Run the Docker container

Run the container with the Apache web server and the CGI application. We had to add the --security-opt apparmor:unconfined flag to disable AppArmor confinement for this container.

docker run -d --name=cgi-app --security-opt apparmor:unconfined cgi-app-image

3. Generate the AppArmor profile

Start the aa-genprof utility for the running container. Replace <CONTAINER_ID> with the ID of the container running your CGI application.

This command will start the aa-genprof in interactive mode. It will generate a new AppArmor profile based on the activity it observes.

aa-genprof -f /proc/<CONTAINER_ID>/root/etc/apparmor.d/usr.lib.apache2.mpm-prefork.apache2

4. Interact with the application

While aa-genprof was running, we had to interact normally with the application. This allowed aa-genprof to observe the application's behavior and generate an appropriate profile. We had to make sure to test all functionalities to ensure comprehensive coverage.

5. Complete profile generation

Once we finished interacting with the application, we returned to the terminal where aa-genprof was running. We pressed 'S' to scan the logs and review the suggested rules. For each rule, we had to choose to allow, deny, or edit the rule as needed. In the end, we pressed ‘F’ to finish generating the profile.

6. Save and load the profile

The generated profile was saved in the /etc/apparmor.d/ directory on our host system. We had to copy this profile to all the worker nodes of the Kubernetes cluster.

7. Apply the profile to your Kubernetes pods

metadata: 
  annotations: 
    container.apparmor.security.beta.kubernetes.io/my-cgi-app-container: localhost/usr.lib.apache2.mpm-prefork.apache2

Apply the updated manifest using kubectl:

kubectl apply -f my-cgi-app-deployment.yaml

AppArmor Prerequisites for Kubernetes Integration

This discusses the prerequisites for successfully integrating AppArmor with Kubernetes. These requirements cover container runtime compatibility, node-level installation, profile distribution, and profile specification through annotations.

1. Container runtime compatibility: Ensure that your Kubernetes cluster's container runtime supports AppArmor. Docker, Containerd, and CRI-O all support AppArmor.
2. Node-level installation: Make sure AppArmor is installed on every node in the Kubernetes cluster.
3. Profile distribution: AppArmor profiles must be available on every node in the cluster.
4. Profile specification: AppArmor profiles are assigned to containers using Kubernetes annotations.

Conclusion

AppArmor has proven to be an invaluable security measure for applications operating within our Kubernetes cluster. As highlighted in the introduction, it has thwarted many hacking attempts, reinforcing the security of our systems.

But, we must acknowledge the considerable effort required to craft profiles for each application and configure them within the pods. This labor-intensive process can be a barrier to broader adoption and may hinder efficient implementation.

Moving forward, we expect advancements in tooling and streamlined methodologies will reduce some of these challenges, making AppArmor configuration more accessible and efficient.

As the technology evolves, we hope AppArmor’s growing ease of use will encourage even greater adoption, leading to a more secure and resilient digital landscape for all.