Guest: Nati Shalom (LinkedIn, Twitter)
Company: Cloudify (Twitter)
Keywords: Environment as a Service, Cloud, Terraform
Show: Let’s Talk

Summary: Cloudify is an open-source DevOps automation platform that aims at simplifying end-to-end cloud services and network automation with its intent-based orchestration platform connecting any cloud, device, or third-party automation tool. The company has released a free Terraform UI experience to help DevOps professionals import unlimited Terraform modules to Cloudify without writing blueprints or glue code. In this episode of Let’s Talk, we sat down with Nati Shalom, Founder and CTO of Cloudify, to talk about why the company is focusing on Terraform, some of the key features of the new Terraform UI experience and more.

“There is no state that represents an environment. In fact, it represents components of the environment but not the end environment,” said Shalom. “That’s what we’re trying to change. We want to look at the end-to-end environment as an end-to-end state so we can interact with that state and therefore continuous update and Day 2 operation on that environment would become much easier than it is today.”

Highlights of the show:

• What is Environment-as-a Service and what problem does it solve for DevOps teams?
• How does Cloudify’s free Terraform UI experience help developers?
• What are some of the pain points that DevOps teams face when it comes to Terraform?
• Key features of the new Terraform UI experience

About Nati Shalom: Nati Shalom is the CTO and founder of Cloudify.io. He is a serial entrepreneur and widely published thought leader and speaker in open source, multicloud orchestration, network virtualization, DevOps, and edge computing. Nati has received multiple recognitions including YCombinator and is one of the leaders of the Cloud Native and DevOps Israel Meetup groups.

About Cloudify: Founded in 2016, Cloudify is an open-source DevOps automation platform delivering an ‘Environment as a Service’ technology that packages infrastructure, technologies and existing automation tools into certified blueprints. Cloudify is designed to manage heterogeneous cloud environments at scale and to bridge the gap between DevOps and IT Service Management processes.

Here is the full unedited transcript of the show:

• Swapnil Bhartiya: Hi, this is your host, Swapnil Bhartiya, and welcome to TFiR Let’s Talk and today we have with us, once again, Nati Shalom, founder and CTO of Cloudify. Nati, it’s great to have you on the show.

Nati Shalom: Hi, Swapnil. Nice to be here again.

• Swapnil Bhartiya: Today we are going to talk about Cloudify’s new free Terraform UI experience. But before we go there, since we spoke to you folks on a regular basis, you often use the term environment as a service platform. I want to understand what this platform is all about and what problem it solves for DevOps teams?

Nati Shalom: I think that the term environment, I think, is overloaded and it’s used in many ways, so just to clarify what it means, when we talk about environments it’s not just Terraform, it’s usually an environment is, especially in development type of stack, it would be a combination of Kubernetes, database, storage, network, custom scripts, some legacy stuff, that is an environment.

And usually when we are running in a DevOp production environment, we’ll have the same environment in different versions. Obviously in production, it’s an environment that is optimized for availability, for scalability, and therefore it includes a lot of redundancy, a lot of security features.

When we look at a development environment, it needs to be very agile, something that could run for a certain time, and we don’t want it to run infinitely, so we want to run it and kill it. It doesn’t have to have the same abilities. It doesn’t have to have redundancy at the same level that we have in production. And there’s going to be a lot of versions of that, if you like, per environment, because for different testing we’ll probably need different environments for that.

Environment can also be a customer environment. I’m a SaaS provider and for me, an environment is customer X or customer Y, or data customer A and data customer B. Another environment would be a location-based environment. I have an environment that is spread across multiple availability zones, multiple regions, each region would be an environment. Basically it represents a stack of things that can be spread and organized in different ways, depending on the use case, and that is an environment.

• Swapnil Bhartiya: When we look at these different environments, how easy it is jumping from let’s say, experiment or test environment to production environment, or you mentioned, customer environment. I just want to understand the sync or synergy between these environments, or they’re isolated?

Nati Shalom: I think if you look at the way we are doing it right now in the development experience, I would say it’s a very task-based system running through CI/CD. We have certain tasks that we’ll call Terraform, we’ll call whatever, to provision an environment. And there are many ways to provision an environment, that’s the “easy part” I would say.

The problem remains is that when we’re spawning an environment, then we need to do day two operation on that environment, like continuous updates, changing the new environment. So let’s say that we have that Kubernetes and database, now we want to add the messaging service, we want to add the launder service. It’s a living organism and we need to continuously manage it.

When we are doing it in this task-based approach, there is no state to the environment. There is state for Terraform, there is state for Kubernetes, but there is no state for the environment itself. Therefore, there is no place that we can say the environment is living. What is running? Is it running correctly or not? And if I want to add something, how is that related to the current state and what’s going to be the effect of it on the other systems that are currently running?

So there is no state that is representing an environment. There is a state that represents components of the environment, but not the end environment. And therefore, the main problem with that approach is that running data operation is almost impossible, because every operation that you’re going to do is making a change to the environment, which means that you have to change that operation over and over again. It’s not going to be a repetitive task, it’s going to be a very custom task if every change.

And that’s what we’re trying to change. We want to look at the end to end environment as an end to end state and we can interact with that state and therefore continuous update and day two operation on that environment would become much easier than it is today.

• Swapnil Bhartiya: You folks recently announced a free Terraform UI experience. Once again, I would love to know a bit about why Cloudify is focusing on Terraform?

Nati Shalom: As I mentioned, when I describe what an environment is, obviously, as I mentioned, Terraform is a big component of that stack. And so we had to have very good story around an integration with Terraform because in many cases that’s a starting point. When people developing an environment, that’s a starting point.

So we wanted to have a very crisp story around that and we wanted to make sure that Terraform users would feel at home when they’re running Cloudify or they’re managing their environments with Cloudify. That’s why Terraform is very important in that strategy, in that thing. Obviously we’re doing things beyond that, but Terraform is really big components of that.

• Swapnil Bhartiya: Now, if I ask you that, what are some of the highlights or key features of this Terraform UI experience? You touched upon some in the previous answer, but I want to hear some of the specifics.

Nati Shalom: I would start with the obvious things. It’s managing state file, remote execution integration with GitOps as the out of the box. I would say there is more into it, which is the dependency management. Let’s say that I have more modules that are now interlinked and I cannot really put them under one hierarchy, they’re basically independent of one another, in some cases they will be running together, in some cases they won’t.

So we have a lot of that relationship and dependency management between Terraform modules that you could actually add that. We have also a way to import those modules in a easy way so developers can actually plug in their existing modules and just say, what are the inputs and outputs? And it becomes part of the system. There’s obviously multi-tenancy. So especially when I’m running in team, I can structure who have access to what type of environment.

And the point here is that it’s not just for single users, it’s for multiple users, meaning that I can actually organize my Terraform modules in a way that fits the team and responsibilities around the team. There is a catalog service for certain things.

Actually, I had an interesting discussion with one of the, if you like, Webscale companies around that. I was asking them, “How did you came to the reason… Why did you even develop something like that yourself?” They basically said that if you look at that, even from a single developer, so certain things that went through a maturity cycle, in which they become repetitive. For those things, I don’t want those users to actually go and touch Git and Terraform, I just want to click about it or just call a single API and just get it running.

It’s repetitive, we already know that it working, there’s no reason why I should mess with the low level stuff. There are certain things still that are still in development, it’s not in immaturity. The process itself is still evolving.

And for that, I want to have access to that. It’s not just that we are talking about developers that don’t want access to Terraform or to infrastructure, we’re talking about the same developer that would want access to Terraform for certain things, but would want the repetitive part to be much simpler. And that’s, I think, the right way to look into that and obviously spread between developers and teams. Certain teams wouldn’t want it, certain teams would want it and we want that flexibility.

So if I want to touch Terraform, go ahead and do it. And then you could expose it to the rest of the users. If there are certain repetitive stuff, you could go ahead and just wrap them up in a nice way and just call an API to do it and don’t have to mess with that. That’s the general things that you’ll get almost out of the box.

• Swapnil Bhartiya: Since you brought this up, I’m also curious about number one is, what was the driving inspiration behind this at this time? Second part of the question could also be, what are some of the pain points that you see DevOps teams face when it comes to Terraform so you are trying to solve those problems for them?

Nati Shalom: I think the main problem, I would say, number one is timing and maturity of the market. Number two is scale. And obviously they are related. When I’m talking about timing, when we started the journey two years ago, most people were really starting their infrastructure journey. They were moving from the monolithic application to microservices, not really worried about that.

And today everyone have Terraform, they have Kubernetes, they are, I would say in a more advanced stage in that journey. Now they’re starting to look, “Okay, I can see that it works. I can see that I can automate my infrastructure. I can see that I can run things as code, but how do I do it across the team? How do I make sure that all of them are accessing infrastructure in a consistent way that this individual is using it in this way, this individual is using it in this way, this one left the machine up and running, and didn’t even know that he needs to check that. This one was actually running it in the wrong data center, because he didn’t even know where things are running.”

How do we create consistency across teams? How do we provide agility across teams so that they can actually focus on what their day job is meant to be, and not worry about things that they’re not supposed to be dealing with? And those are the type of questions that people are asking themself right now. And what we’re finding is that a lot of those people are building a lot of that solution themself. And the reason why is because there is a void right now.

I mean, Terraform is a great tool, Kubernetes is a great tool, but the world have changed, the challenges have grown, the workload have changed, the maturity have changed, and now we’re looking at the next wave of problems. And right now there’s a void, and therefore they’re finding themselves having to put DevOps into the problem, meaning developers that will actually augment what they have in their CI/CD pipeline and build it themself.

And we’re basically coming and saying, very soon if you didn’t find it yet, you’ll find that developing it yourself it’s a pretty generic thing. There’s no reason why you would develop it from scratch. We do a lot of the things that you would otherwise do out of the box. It will save you a lot of time and money and will free your developers and DevOps team to do the right things rather than developing a platform.

And that was the case with [inaudible 00:10:38] that I think I touched on later on. That enabled them to really change the focus of the team from developing a platform to grow between four to 200 environments, which means that they’re now bringing much more value to the users and the users are actually much happier right now.

• Swapnil Bhartiya: And you touched upon certain things, which I want to just go a bit. Number one is that with that option of all these technologies, there is also a need for lowering the barrier of entries because every company wants to. So that’s why we also see a lot of trends in the low-code/no-code space as well. But of course there are complexities as well. And then you also talked about developers’ time, why should they waste time in building something that is not adding any value to the business? Basic plumbing never adds any basic value to the business. Developers’ time is very, very valuable. Can you talk about these two aspects, number is to make it easy, so more people can embrace and adopt it. And second is that it also makes it even easier so developers can focus on solving the business problems and not building the platform.

Nati Shalom: Excellent. As I mentioned, I think before when we started this infrastructure code thing, I think that was a reaction to a previous state when we had something that is called platform as a service, which was taking the subtraction to the extreme, which was a black box, hiding everything, and the reaction to that was we needed flexibility. And because we need flexibility, let’s go to the other extreme, which is infrastructure is code.

And so it gives us a lot of flexibility in how we manage infrastructure. But as I mentioned, when we look at the dimension of flexibility versus productivity, there are certain things in which I need that flexibility. There are certain things that are repetitive and I don’t necessarily need that. And the same thing is with S code and no-code.

There are certain things that the flexibility of running things as code is important, there are certain things that are relatively mature and repetitive, and I just want to stick them together and stitch them together in a simple way. I don’t have to go to the full extreme of writing code to do those type of things. And there are certain things that will be complex business logic, complex workflow, networking, whatever, that you would still need code.

And right now you have to choose between the two. You have to choose I’m either in this S code camp or in no-code camp. And the reality is a mix of those. We are trying to harmonize between the two and saying, okay, when you want to run no-code, here it is, we’ll actually generate the code for you.

But at any point in time, you could dig in and see the code and develop it as code and go back into this canvas thing and just write it in this way. So we’re trying to harmonize those two, if you like, camps and saying it’s a continuum. It’s not something dichotomic that you have to choose from, that’s one thing.

And the same thing applies to your second question, which is how do I expand it to the other teams? And then how do I actually expose it to more environments and more tools and things on that line. And there is, I would say, a straight line connecting between the two. In one case, I want to simplify things that are repetitive.

Similarly, when I’m managing things in my infrastructure, I want to make them repetitive as well and don’t deal with repetitive teams. And when we look at doing it yourself versus taking a platform, obviously the ROI for developing it yourself, the numbers are not going to match because everything that you will be developing yourself is going to be much costly than taking a license off the shelf product.

• Swapnil Bhartiya: And that leads to another question that is, if I ask you, it’s the beginning of the 2022, I think it’s almost February, but what kind of trends you are seeing, when I say trends, I’m not talking in journal, but I’m talking about adoption of low-code/no-code automation is going to drive a lot. AI/ML is going to play a very big role in scanning and helping things. So what do you see is happening in this space that first of all aligns very well with what you folks are doing and it also aligns very well with where the world is moving with the adoption of all these cloud data technologies?

Nati Shalom: I think the couple of things that are becoming very clear, and for me, the surprise that I saw is, when you look at the rockstar of DevOps, the Spotify of the world, the Airbnb of the world, all those guys, I was surprised to see that they also got to that realization. I was expecting that the shell, the banks, whatever will get to that point, because not necessarily they have the right skills or the right people, the right team, or the right culture.

But even the organization that, as I said, the rockstar of that world got to that and Spotify went as far as developing a complete platform, open-source that is called Backstage. It is gaining a lot of momentum right now. That indicates two things, one of them that the problem is not just for, I would say, legacy enterprise type of problem, it’s everyone’s problem.

Second, it’s not a tactical problem, it’s a strategic problem as we’ve seen with this $1 trillion products report that I think I talked about last time. It’s not a tactical problem. If you don’t control how your users are going to use your infrastructure, you’re going to end up hitting your own margins and therefore your evaluation, and it’s going to be a huge issue for you.

The third thing is if it hits the rockstars, there is no reason why it would hit all the rest of the industry. And obviously this realization is becoming more clear and COVID was just an accelerator of that. This realization is becoming very clear and I think it’s a maturity thing, as I mentioned earlier, and more and more people are realizing, “Okay, what I have right now doesn’t scale. It works, but it doesn’t scale. And now how do I scale?” That’s what would bring them to that realization.

And between us, there’s not too many options, the options of how to solve it will have to go through this type of platform. Another question becomes the choice of that platform. Do I go with Spotify Backstage? Do I go to Cloudify? Do I go to something else? That’s going to be the main question that I think people will ask themselves rather than do I need it or not?

• Swapnil Bhartiya: One more thing that I want to talk to you about is that the importance of open-source. Open-source gives access to a lot of technologies, very easily folks who are very well-versed or who want to play with they can go and try it out. But the challenge with open-source is not day one is day two, is scalability, adding more features, functionalities, maintenance updates. So let’s just talk about this announcement. You mentioned earlier there’s an open-source version. Can you talk about the open-source version and commercial version, how it helps DevOps teams to evaluate, play with, experiment. Some teams are capable enough to do it by themselves, some teams need commercial help. And then what kind of relationship between the open-source version and commercial versions, so they can play with it, and when they want, they can move to the commercial version.

Nati Shalom: That’s actually very, I’d say, dear to my heart, that topic specifically. I have a podcast of my own that I’m talking a lot about that open-source trend in the world. The philosophy is actually pretty simple and in DevOps, I think, it’s even more important. Because DevOps is about tool chain, about integration of tools, all those things. It’s very hard to accomplish that without an open-source capabilities.

Because with open-source, you have much easier way to understand what’s going on when something went wrong, to understand how to do the integration. You could also hack things if you need to. And it’s very common in DevOps that you would need to get to that level of things versus if you’re taking a black box approach, you’re going to get stuck very quickly at some point and it’s going to hit you as a cliff, and it’s a real problem. So I think in the DevOps world, open-source is, in my view, critical. It’s not an option, it’s critical.

Now, as you said, it comes with a cost. How do I operate it and manage it will be a cost. And so the combination of having in one end, an open-source version that gives me that flexibility, that I can understand what’s going on, that I can take it into my own environments, play with it and understand it, and at the same time, have a managed version of it that fully managed someone take care of the habitability of scalability, of security, of upgrades, all those type of things can be taken away from me, that I think is becoming the right combination. And I’m talking general, not specifically to Cloudify. We’re just following through best practices of how open-source companies are actually operating themself.

And that’s really the separation between the open-source and commercial versions. So we keep all the API plugins, all the developer, if you like, features free all the time for life and all the things that are related to the operation side, like the scalability, security, availability. That’s where we monetize the product. I think that’s going to becoming a fair, if you like, separation between them. And the important thing is that developers would not be blocked on APIs, they would not be blocked on features, that’s always going to be open for them.

• Swapnil Bhartiya: Nati, thank you so much for taking time out today, and of course, talk about this new feature. We also talked about all the problems that DevOps folks are facing, especially Terraform. So thanks for sharing those insights. And as usual, I would love to have you back on the show. Thank you.

Nati Shalom: Thank you very much Swapnil.