Author: Nitzan Niv

As the number and complexity of Kubernetes deployments grow, a critical gap is coming to light: Kubernetes audit logs are difficult to parse and analyze for critical security breach evidence without deeply specialized Kubernetes knowledge and large time investment.

The dynamic nature of Kubernetes deployments, especially their distributed and ephemeral nature, makes it difficult for security professionals to understand what might have transpired within an application at a given time, including how a security breach occurred and propagated through the system. For example, access to credentials stored in K8s configuration objects is an attacker’s first step towards accessing sensitive resources and critical databases.

Identifying such potential risks has been a growing challenge for organizations using Kubernetes in production. Security teams need a quick way to identify which users and roles access sensitive K8s resources at a particular time without legitimate reasons, which resources were maliciously or erroneously accessed by unexpected users, and what external or internal tools are seeking unprotected access to clusters’ resources. Luckily, all such transactions are recorded in K8s audit logs.

Close examination of K8s audit logs by expert security professionals can reveal many potential risks:

• Stolen credentials, enabling hackers to gain access to K8s-based clusters or pods.
• Stolen tokens or misconfigured Rules Based Access Control (RBAC), enabling lateral cluster or pod movement, privilege escalation and unauthorized data access or manipulation.
• Exploited vulnerabilities in the Kubernetes API Server, enabling bypassing of authentication, authorization, admission control or validation of cluster administration requests. This lets users to gain access to privileged and sensitive resources.
• Violated security policies divergent from compliance best practices.

Yet there is still a timing challenge. When security experts pore over logs to discover security breaches, it’s generally done retroactively in the days or weeks after a breach has been detected. That leaves a vast window where the cluster’s applications continue to be compromised. In much the same way that DevOps has shifted left and applied automation to increase the speed of software development and deployment, DevSecOps can shift the usage of audit logs from a monitoring and response paradigm to one of proactive observability.

Administrators seek to increase their ability to see inside the operation of their software or infrastructure. Taking this to the next level, DevSecOps should aim to see potential security breaches as they occur. In this way, damage can be limited and the vulnerabilities remediated sooner.

Given the complexity of K8s audit logs, security professionals have trouble effectively identifying emergent risks without automated tools to aid them. It is impractical and inefficient to assign a security expert to repeatedly watch and investigate audit logs for signs of security breaches, so instead, an automated system must continuously scan logs and alert administrators if it detects a possible security violation.

By simplifying the reporting to highlight detected risks, an automated system could make audit logs accessible to regular security professionals, not just experts. Furthermore, the scanning tool could be augmented with machine learning algorithms to identify complex, non-trivial threats related to multi-step attacks. Such a tool would be a huge boon to security professionals seeking to shorten the time between breach, detection and remediation.

There are also compliance ramifications to consider. Companies that have shareholders and government regulations to adhere to should consider proactive audit log monitoring as a way to identify breaches earlier, to limit their impact sooner and to record related data for later legal inquiries more accurately, with all the details that are specific and time-relevant to the incident. Focused audit logs investigations will be vital in determining what happened and steer investigators toward the source of the break-in.

Leverage your K8s Logs for Real-Time Security Observability

Kubernetes logs have been underutilized for the security benefits they provide. Yet now, with 71% of the Fortune 100 using Kubernetes as their main container orchestration tool, the time has come to not only leverage K8s audit logs, but to do so in a proactive real-time fashion to minimize the time from breach to detection and remediation. A thoughtful strategy for leveraging the security value of logs will go far to protect your K8s deployments from harm.