Mastering Kubernetes Audit Logs

Kubernetes offers a robust feature that facilitates the creation of audit logs for every action requested and executed through the Kubernetes API.

Kubernetes Audit logs fall under the Monitoring, Logging, and Runtime Security category in the official curriculum of the Certified Kubernetes Security Specialist (CKS) exam.

Given the examination's heavy emphasis on the importance of audit logs, you may encounter several questions on this subject.

Understanding how to configure, manage, and interpret these logs is critical for maintaining the security and compliance of Kubernetes environments.

This article continues my exploration into Kubernetes security, building on my previous discussion about securing Kubernetes with OPA Gatekeeper.

As part of my journey to prepare for the CKS exam, I aim to share insights that enhance my understanding and assist others in studying for their certification.

Securing Kubernetes With OPA Gatekeeper

Ensuring Compliance and Security Across Your Kubernetes Cluster with the Open Policy Agent Gatekeeper

Configuring Kubernetes Audit Logs

To configure and enable Kubernetes audit logs, you add specific arguments to the kube-apiserver. The most important is the audit-policy-file argument to the kube-apiserver configuration, which points to an audit policy file. This policy file specifies the events to log and the level of detail for those logs.

Here are the necessary arguments to include:

• --audit-policy-file=/etc/kubernetes/audit-policy.yaml: This argument points to the audit policy file, which outlines the logging rules and the details of events to capture.
• --audit-log-path=/var/log/kubernetes/audit/audit.log: The audit-log-path argument specifies the location for storing the logs. In this file, logs are maintained as JSON structures, offering a structured and accessible format for analysis.

By setting up these arguments, you enable logging of the interactions with the Kubernetes API, creating an audit trail for security and compliance purposes.

Mounting the policy and log file

Suppose your kube-apiserver is configured to run as a static pod, often in environments used in the CKS (Certified Kubernetes Security) exam. In that case, ensuring the audit policy and log files are correctly mounted is crucial.

This step guarantees that audit records are not only generated but also retained beyond the lifespan of the pod.

To achieve this, you must define volumeMounts within your pod configuration to link the pod's file system with the host's file system, ensuring the audit records are preserved. Here's how you set it up:

volumeMounts: 
  - mountPath: /etc/kubernetes/audit-policy.yaml 
    name: audit-policy 
    readOnly: true 
  - mountPath: /var/log/kubernetes/audit/ 
    name: audit-log 
    readOnly: false

Additionally, you need to define volumes to specify the exact location on the host system where these files are stored. This is done by configuring hostPath for both the policy file and the log directory, as can be seen below.

volumes: 
  - name: audit-policy 
    hostPath: 
      path: /etc/kubernetes/audit-policy.yaml 
      type: File 
 
  - name: audit-log 
    hostPath: 
      path: /var/log/kubernetes/audit/ 
      type: DirectoryOrCreate

The readOnly: true setting for the audit policy file ensures that the pod does not accidentally modify it, maintaining its integrity.

Conversely, the audit log directory is marked readOnly: false to enable the kube-apiserver to write audit logs to the file system.

The Audit Policy File

As previously discussed, the audit policy file is a YAML document that outlines the rules governing logging events within Kubernetes. It specifies which events are logged and the level of detail recorded, including the request stage.

There are four logging levels available for categorizing events:

• None: Events matching this rule are not logged.
• Metadata: Logs request metadata such as the requesting user, timestamp, resource, and verb but exclude the request or response body.
• Request: This level logs event metadata and the request body, not the response body. It does not apply to non-resource requests.
• RequestResponse: Logs event metadata, request body, and response body. This level does not apply to non-resource requests.

Here is an example of a policy file created to keep a log of all operations related to secrets at the Metadata level. Once you install this policy, any time you create, modify, or delete a secret within the cluster, the metadata associated with that operation will be captured in the audit log file.

apiVersion: audit.k8s.io/v1 
kind: Policy 
 
omitStages: 
  - "RequestReceived" 
 
rules: 
  - level: Metadata 
    resources: 
      - group: "" # Indicates the core API group 
        resources: ["secrets"]

Using the omitStages field with the value "RequestReceived" indicates that events are not logged at the initial request receipt stage, focusing instead on the substantive handling of the request.

The flexibility of the audit policy file extends far beyond the basic configuration outlined above. With many customization options, you can tailor your audit logs to meet particular requirements.

Configuring logs for particular types of actions, known as “verbs,” or restricting logging to specific actions performed on resources within a certain namespace is possible. Essentially, the possibilities are endless.

For example, you can fine-tune your audit policy only to log “create” and “delete” operations for pods within a particular namespace. This level of granularity ensures that you can focus on the most relevant audit data for your security and operational needs without being overwhelmed by extraneous information.

Here’s an example that shows how such logging could be configured:

rules: 
  - level: Metadata 
    verbs: ["create", "delete"] # Focus on create and delete operations 
    resources: 
      - group: "" # Targeting the core API group 
        resources: ["pods"] 
    namespaces: ["production"] # Restricting to the 'production' namespace

This snippet demonstrates one of the many ways you can customize your Kubernetes audit logging to align with your specific security, compliance, or operational requirements.

Additional Options for Managing the Audit Log File

Managing the size and quantity of audit log files is crucial for ensuring that logs are available and manageable over time.

To this end, the kube-apiserver provides three additional options to help administrators control log file retention and rotation:

• --audit-log-maxage: Specifies the maximum days to retain old audit log files. This setting helps manage the logs' lifecycle by automatically purging files older than the specified days.
• --audit-log-maxbackup: Defines the maximum number of audit log files to retain. This option limits the number of old log files on the system, ensuring that storage usage is within reasonable limits.
• -—audit-log-maxsize: This argument determines the maximum size (in megabytes) of an audit log file before it is rotated. Once a log file reaches this size, it is closed, and a new file is started, preventing any single file from becoming too large.

Managed Kubernetes Environments and Audit Logging

When operating within managed Kubernetes services like Azure Kubernetes Service (AKS), Google Kubernetes Engine (GKE), and Amazon Elastic Kubernetes Service (Amazon EKS), administrators can control audit logging with a high degree of flexibility.

These services offer the option to toggle auditing features on or off and seamlessly integrate audit logs with their respective managed logging solutions.

For instance, you can configure your system to automatically forward audit logs to the Azure Log Analytics workspace or equivalent services in GKE and Amazon EKS.

Turning audit logging on or off varies slightly across platforms, but it generally involves setting the appropriate options within the Kubernetes service's management interface.

Here’s a brief overview of where you can find more detailed guidance for each service:

• Azure Kubernetes Service (AKS): Look into the AKS documentation on integrating with Azure Monitor for containers. This guide provides detailed steps to enable audit logging and send logs to Azure Log Analytics. Azure Kubernetes Service (AKS) documentation
• Google Kubernetes Engine (GKE): GKE offers integration with Cloud Operations (formerly Stackdriver) for logging and monitoring. Detailed instructions for setting up audit logs can be found within the GKE documentation. Google Kubernetes Engine documentation
• Amazon Elastic Kubernetes Service (Amazon EKS): EKS allows audit log configuration by integrating with AWS CloudWatch Logs. The EKS documentation guides enable audit logging and managing log retention. Amazon Elastic Kubernetes Service documentation

CKS Exam Focus: Kubernetes Audit Logs

Kubernetes audit logs are crucial to the Certified Kubernetes Security Specialist (CKS) exam. You should expect to encounter questions that require proficiency in creating and managing Kubernetes audit logs using audit policies.

It is crucial to be familiar with configuring audit policies, setting parameters such as the maximum number of retained audit logs, and determining how long a log file should be preserved.

Additionally, you must understand how to properly mount both the audit policy file and the audit log file within the static pod definition of the kube-apiserver.

Resources like Killer Shell and Kodecloud offer practice exams that include multiple scenarios about audit policy configuration.

Conclusion: Harnessing the Power of Kubernetes Audit Logs

Kubernetes audit logs are an indispensable tool for administrators aiming to enhance security and compliance within their Kubernetes environments.

You can achieve a detailed and proactive security posture by configuring, managing, and understanding audit logs.

This article guided you through the essentials of setting up and maintaining effective audit logs, from the basic configuration steps to more advanced practices for managing log file sizes and lifecycles.

For those preparing for the Certified Kubernetes Security Specialist (CKS) exam, mastering audit logs is not just about passing a test — it’s about cultivating a skill set that will significantly contribute to securing Kubernetes deployments in any environment.

Configuring and interpreting these logs is a crucial skill for anyone responsible for the security and operational integrity of Kubernetes clusters.

As Kubernetes continues to evolve as a critical component of cloud-native infrastructures, the role of audit logs will only grow in importance.

Whether managing a local cluster or navigating a managed Kubernetes service, audit logs provide vital insights for identifying, responding to, and mitigating security threats.

Happy studying!