Guest: Brad Geesaman (LinkedIn, Twitter)
Company: Aqua Security (LinkedIn, Twitter)
Show: Let’s Talk

Earlier this year, the NSA released its guidance for Kubernetes security. This was necessary because, as Brad Geesaman, Director of Cloud Security at Aqua Security, puts it, “Kubernetes by default was not leaning towards security by default in mind. It was aimed for developer happiness and velocity and operational consistency.” Geesaman continues, “There’s a lot of decisions that had to be changed over time to make it more secure by default. It’s still not all the way where we should probably be, but a lot of work is being done in that regard.”

This new guidance, according to Geesaman, “toes the line decently between the cluster administrators, the system operators, and the security teams or the blue teams, the defenders who are in charge of operating these clusters.” He gets a bit more specific on this when he says, “But the first couple of paragraphs where they’re saying, ‘Hey, here’s the threat model.’ That’s something a CISO would want to be aware of and understand why the cluster owners may be asking for time to implement certain features and how that relates to compliance or the overall security posture.”

As to core features of the guidance, Geesaman touches on it when he says, “I think it’s important to understand that what they’re calling out are the specific things you want to restrict in a pod configuration and they’re done with a different implementation between the two.” Geesaman adds.

Geesaman believes people should care about this guidance because “whenever somebody like the NSA puts out guidance like this, it tends to be well trusted and well leveraged.” He adds that those who follow this guidance can be rest assured that they implemented things in a way recommended by NSA and they are covered.

Geesaman concludes with, “I think there’s a good balance there of high-level guidance and things you should be aware of. But there’s a lot of implementation-specific details. And I think that’s where folks like Aqua can help out.”

This summary was written by Jack Wallen.

Topics we discussed include:

• How different were early days of Kubernetes as compared to Docker containers? Were a lot of sensitive industries apprehensive about using docker containers from a security perspective?
• There are two aspects of security in the Kubernetes world: People and Technology. Which one is more challenging to deal with? People or Technology?
• Let’s talk about what is this guidance is all about? What are they trying to do here with this guidance?
• Who is this guidance meant for – government usage of anyone who is dealing with Kubernetes?
• Geesaman talks about some of the key features of the guidance.
• Who is the target audience? CISOs or Developers?
• Is this guidance complete or there is still see some missing things or room for more information?
• Are there certain things that are at a different stage in the priority list even in terms of Kubernetes security, depending on the industry a company operates in? Or are there certain baselines common to everyone?
• Do you see the industry leveraging this report and based on it creating its own recommendation as not everyone lands up on NSA’s guidance page?

[expander_maker]

Swapnil Bhartiya: Hi, this your host Swapnil Bhartiya and welcome Let’s Talk. Earlier this year, NSA and the cybersecurity and infrastructure security agency released their Kubernetes Hardening Guidance to develop and issue cybersecurity specifications and mitigations. To deep dive into this guidance, today we have with us Brad Geesaman, Director of Cloud Security at Aqua Security. Brad, it’s good to have you on the show.

Brad Geesaman: Yeah, thanks for having me. Appreciate it.

Swapnil Bhartiya: Let’s talk about this Kubernetes Hardening Guidance that was released by these two agencies. But before we go there, if I ask you, if you look at the early days of Docker Containers, there were a lot of apprehensions, a lot of industries were not deploying them for security. What did you see in Kubernetes space from the early days? Of course, it was a Google technology that was already used in production so there was a bit difference there. But what kind of apprehensions there were? Or there were none?

Brad Geesaman: Kubernetes by default was not leaning towards security by default in mind. It was aimed for developer happiness and velocity and operational consistency. There’s a lot of decisions that had to be changed over time to make it more secure by default. It’s still not all the way where we should probably be, but a lot of work is being done in that regard.

Swapnil Bhartiya: When we talk about security in these kind of cloud native technologies, I look at it from two aspect. One is, of course, the built in security features that are purely technical in nature. And second is human. Human errors are there, permissions that you can have are there. Of course misconfigurations are there. When you do look at Kubernetes, where do you see the gray areas?

Brad Geesaman: That’s a great question. Kubernetes is flexible and powerful and it runs in tons of different ways and different variations, different distributions of Kubernetes, as you’ll call it. And every single one of them brings their own unique challenges. Some things the cloud provider will handle for you like protecting SED and the master nodes for you and taking care of them versus you just have to take care of the nodes. And sometimes if you’re rolling it yourself, you have to take care of literally everything operationally from the ground up. The gray area really stems from there’s no one way to do it and therefore there’s no one set of guidance that always applies evenly. You have to match the guidance to your specific implementation. And because it touches all layers of the stack, that touches all layers of things that you would need to secure.

Swapnil Bhartiya: Excellent. Now having the basic foundation for [inaudible 00:02:40] can lay it out there, let’s talk about this guidance. What is the scope? What are they trying to do here with this guidance?

Brad Geesaman: I think what they’re trying to do is recognize that there’s a lot of folks that are using Kubernetes. And it’s gotten to a point of stabilization for the core features. And I think the timing says, “Hey, this is the time where we can put out guidance that hits a broad number of the distributions. There’s a lot of common things we can articulate, the why, the threat model.” And I think that right now is a good time to put that out there and say, “hey, if you’re going to run Kubernetes, these are the best practices that apply to the most [appointments 00:03:19]. You want to be thinking about X, Y, and Z, so that you’re not missing anything obvious or things.” Because it’s a very complex system and there’s a lot to be right about to get things right.

Swapnil Bhartiya: Yeah, Kubernetes is more or less like a cockpit of an airplane. That’s how complicated it is.

Brad Geesaman: It’s a good analogy.

Swapnil Bhartiya: If you look at this report, this guidance, is it specific to government agencies? Or it’s also, it’s a guidance, anybody can benefit from it? Because if you look at government agency or let’s look at DOD, they have, when it comes to container images, they have Iron Bank is there. They have Platform One is there where they do do a lot of things. Let’s try to understand, this is for the government agencies or is for public as well? And do you think on the lines of Platform One or Iron Bank there will also be a hardened Kubernetes distribution purely for government agencies?

Brad Geesaman: I think it’s always good to check assumptions. And I like to think at that high bar of what we can and should be doing and not necessarily dividing on how it’s being used. If you’re running critical workloads, if you have sensitive data, if you have payments going through, anything along those lines all this applies. You have to do all these things well. So I think it’s important that yes, there might be hardened distributions that already solve this for you. Then you still have to check those assumptions anyway and make sure that they’re being solved the right way. Because you might change something over here that changes what you would want to then mitigate over here. And that’s with all the layers of the stack, you can do that easily. You can make a CNI network change that changes how things work inside the application layer and things like that.

Swapnil Bhartiya: Let’s go a bit deeper into this guidance. What are the things that are there when you look at them and you’re like, “this is really a good advice not only for government agencies, for everybody. These are the best kind of practices”?

Brad Geesaman: Yeah, what’s really great about this guidance is it starts out stating as a fact that the posture is probably not where you want it to be for critical workloads from a security standpoint. But then the thing that I like about it is it starts with a basic threat model. So they talk about supply chain, where all the code executing inside containers are coming from as an attack vector, malicious threat actor, adversaries attempting to compromise the system from the outside, from the inside. And then there’s the third one which isn’t often covered, which is the insider threat. Somebody who’s got a trusted level of access, maybe not complete access, what can they do to expand their scope and attempt to escalate privileges? I like that they start out with that basic threat model because it talks about the why and who might be doing the things.

And that sets the stage for all of the listed items that they talk about from network separation, authentication, authorization, logging, and life cycle and maintenance issues. All of that stuff, it looks like a checklist until you see why. You’re doing this, this and this because a specific insider threat might be leveraging that. And that gives you the context you need. I really like that. The last thing I’ll say is I really like how it references the CIS benchmarks and other frameworks as a, “Hey, this is a combined set of knowledge. This isn’t trying to supersede or be the one set of guidance for everybody.” It it defers to a lot of hard work that’s been done in the as well.

Swapnil Bhartiya: Who do you think is the target audience for this? Is it, as you earlier mentioned, for developers or CISOs?

Brad Geesaman: I think this sort of toes the line decently between the cluster administrators, the system operators and the security teams or the blue teams, the defenders who are in charge of operating these clusters. You talking about pod security policy, pod security standards now. Those are things that are kind of down in the implementation details. But the first couple paragraphs where they’re saying, “hey, here’s the threat model.” That’s something a CISO would want to be aware of and understanding why maybe the cluster owners are asking for time to implement certain features and how that relates to compliance or the overall security posture.

Swapnil Bhartiya: When you looked at the report, of course, it just came out in August, you’re like, “there are still a lot of areas that they should work on.”? Or you’re like, “hey, this is a complete report. There’s no further work needed on it.”? Actually nothing can be like that, especially if you look at Kubernetes. What are your thoughts from that perspective?

Brad Geesaman: I think they did a great job but things move fast in the Kubernetes world and the cloud native world, as you can imagine. I mentioned the pod security policy deprecation, it’s being superseded by the pod security standards, which the Kubernetes :6 security group put out. I think it’s important to understand that what they’re calling out are the specific things you want to restrict in a pod configuration and they’re done with a different implementation between the two. But what they’re calling out is solid, host network, privilege pods, capabilities. Those are all things that are very applicable and they do a good job explaining those. It’s not all loss, it’s just converting the implementation to the newer feature.

One of the things that I would like to see out of maybe the next release is it covers so much territory. I think it’d be helpful to help prioritize the guidance so that those who are implementing can focus on the best bang for buck, the key areas. So that in the widest range of deployment scenarios. You might not put set comp restriction at the same level of ensure that you have network exposure limited. It might be a later stage thing because there’s 20 or 30 things that you need to do. You probably want to get those in order and help folks make sure that they’re not missing something simple and focusing all their time and effort on something that’s all the way down in the weeds.

Swapnil Bhartiya: So when you do talk about prioritize things, it might vary from industry to industry as well or, in this case, maybe government agencies. But you also [inaudible 00:09:07] some broad. If I ask, you did give some example, but how would you ensure that they would consider, “hey, this could be the priority for X industries versus this industry.”?

Brad Geesaman: I think prioritization starts with that threat model. If you go through those things, you’re probably an existing Kubernetes cluster administrator or security team. You probably have something deployed to run through that threat model and understand your gaps. That’s probably going to be the quickest path to, “hey, what are we missing?” But if you’re starting clean greenfield, I think sort of a bottoms up or an outside-in approach tends to resonate better with system owners who have, we’ll say growing understanding and maturity of Kubernetes. Things that you understand are the external network exposure are the cluster, the nodes, and the services solid from a layer four perspective. You kind of move up the stack and go, “am I distributing credentials or doing authentication correctly to gain access to the cluster?” Then you work on upgrades and life cycle and maintenance of the entire thing.

I personally place that earlier in the setup. Because if you wait till later, you might be running before you’re walking, so to speak. You want to be able to make sure that you can patch everything at speed that you can take advantage of all the benefits of Kubernetes. And then going into things like audit logging so you’re not blind, system logging so you know what your applications are doing. And then into the things that I think people tend to focus on when they say, “these are the top 10 things you need to do with your cluster.” It’s [r/back 00:10:43], admission control, handling of secrets and credentials in general with cloud credentials and metadata, runtime detection and prevention, and sort of from there. And once you sort of mature through those stages and you get to that point, you’ll know based on where your workloads are and what your security needs are what thing to go from there next. I think that’s where it gets situationally dependent.

Swapnil Bhartiya: Of course the guidance is there. Now it is released by NSA and what I’m trying to understand is that what is the visibility of this guidance? A lot of folks who are using Kubernetes may not be even aware of it. Sometime they look up at [inaudible 00:11:26] Aqua when they are looking at the security posture. And of course we live in this open world. Do you see there will be some work that will be done leveraging this guidance so that companies can easily follow those best practices? Or you think this will remain the place where people should go?

Brad Geesaman: I think there’s a lot of overlap in a good way with what is out there. But whenever somebody like the NSA puts out guidance like this, it tends to be well trusted and well leveraged. So for a vendor like us, adapting our best practices and how we implement or how we protect or how we detect these activities, that’s something that every vendor can do and should do to make sure that it’s an easy process. So if somebody’s following this guidance, there’s a mapping to say, “oh, I implement it this way and I’m covered. And I’m taken care of in that way.” I think there’s a good balance there of high level guidance and things you should be aware of. But there’s a lot of implementation specific details. And I think that’s where folks like Aqua can help out.

Swapnil Bhartiya: Brad, thank you so much for taking time out today and not only talk about this Kubernetes Hardening Guidance, but also share some of best practices and how people can benefit from this guidance. Thanks for those insights and I would love to have you back on the show. Thank you.

Brad Geesaman: Yeah, really appreciate you having me. Thanks so much.

[/expander_maker]