Pulumi‘s Cloud Engineering Platform unites infrastructure teams, developers and security engineers around one software engineering process for delivering modern cloud applications faster and speeding innovation. Its open-source Modern Infrastructure as Code helps teams tame the cloud’s complexity using the world’s most popular programming languages and communities, including Python, Node.js (JavaScript, TypeScript), Go and .NET (C#, F#). In this episode of Let’s Talk, we sat down with the CEO of the company, Joe Duffy to continue our discussion around cloud engineering and solving hard problems users face in the cloud.

Joe Duffy is co-founder and CEO of Pulumi. Prior to founding Pulumi, Joe was a long-time leader in Microsoft’s Developer Division, Operating Systems Group, and Microsoft Research.

Topics we covered include:

• Pulumi recently organized its second Cloud Engineering Summit. We talked about the focus of this summit.
• How has the state of cloud engineering changed since we last talked and what kind of emerging challenges are there that need to be solved?
• How different is cloud engineering from Infrastructure as Code? Is there any overlap between the two?
• The DevOps movement was supposed to break old silos; they did but created new ones. So are many of these actually in practice?
• Answering that question, Duffy brought up the point of need for specialization, which might be a reason for new ‘silos’, but you will always need people who are ‘good’ at what they do.
• There is a huge demand for skilled engineers. What tips does Duffy have for those who want to build a career in the cloud space?
• We concluded the interview with a discussion around what is Pulumi Registry and what’s the need for it?

Guest: Joe Duffy (LinkedIn, Twitter)
Company: Pulumi (Twitter)

[expander_maker]

Swapnil Bhartiya: Hi, this is your host, Swapn il Bhartiya, and welcome to another episode of Let’s Talk. And today we have with us, once again, Joe Duffy, CEO of Pulumi. Joe, it’s great to have you back on the show.

Joe Duffy: Swapnil, it’s great to be here. Thank you.

Swapnil Bhartiya: Today, we have so many things to talk about. First of all, is cloud engineering summit. I think this is the second summit. It’s tomorrow. Depends on when we are going to release this video, but if I quickly ask you, what is going to be the focus of the summit, how different it is from the previous summit?

Joe Duffy: Yeah, It’s great to have so many thought leaders in the industry, but really focusing on more practitioners, what are the lessons learned for doing cloud engineering? And to us cloud engineering is developers and infrastructure teams coming together to build great cloud software.

And so the topics sort of are organized in three tracks, build, deploy, and manage. So build it’s all about, hey, how do we build these modern architectures? Deploy is how do we do continuous delivery? How do we bring that level of productivity to the teams? And manages, how to do all this while remaining secure, cost effective, and all those things.

So we got some great keynotes, you know, Emily Freeman, also Kelsey Hightower. I think the difference this year is we really are focused on the practitioner. I think we had more sort of, you know, executive level participation last year, and we really want bring it down to practitioner level. Really take, you know, best practices you can take away and be productive with instantaneously. Helping to solve some hard problems that we’re seeing today in the cloud.

Swapnil Bhartiya: You used the word hard problems that you’re trying to solve in the world. So I wonder, first of all, how much has the state of cloud engineering changed since the summit last year? What kind of things have you seen? Because we also went through this whole pandemic phase, which affected cloud deployments a lot. Plus we have also been seeing a lot of concerns around security and other things happening. So if I ask you, what are the hard problems that you are seeing today that you want to address?

Joe Duffy: Yeah, I think there’s really a couple. So one is, as you mentioned, we’re in this all remote world, and so we have to collaborate in new ways. We have to be shipping code online, you know, things like GitOps, you know, really infrastructures code.

Some of these are key enablers, where we can really just collaborate at the level of code. And what we’re seeing is, that’s unlocking a lot more developers getting into the driver’s seat and infrastructure teams wanting to empower developers to do that. We’re calling this developer first infrastructure, and this is a new, emerging thing that we’ve seen over the last year. You know, honestly, teams like, you know, at AWS, and some of the most innovative teams have been doing this for a while now, but we’re seeing much more broad scale adoption. And I think once you do that, the challenge is, okay, how do we move really fast, but not introduce some of these security challenges that you’re alluding to?

You know, the cloud is very complicated. I think CNCF, how many projects do we have now? Over a thousand probably. And so staying on top of that and knowing what the best practices are, that is really a key challenge. And we launched a registry, Pulumi did. And that’s all about bringing best practices to practitioners so that you don’t have to go make the mistakes to find out that you did it wrong. You can just learn from tried and true best practices. I think that’s the next phase, is really up leveling the level of abstraction that we’re all working at.

Swapnil Bhartiya: We have discussed cloud engineering term, early role. So I want to talk about it once again, basically in the light of all the things that you talked about. And can you also just kind of reaffirm that, what exactly it means and how different it is from when we do talk about cloud-native? Because if you look at cloud-native, cloud-native is not a thing, it’s, once again, the way we do things. And also if we also talk a lot about infrastructure as a code, so is there any difference between these two? What are the overlaps? How would you define it?

Joe Duffy: Yeah, I think cloud engineering to me is the next phase after DevOps. DevOps has really laid this amazing foundation over the last decade of bringing a lot more engineering practices to the way we do operations. But I think of cloud engineering as two things. One, bringing the cloud closer to developers, so developers can think of the cloud as really an extension of their application architecture. Really think of managed services as exciting components and building blocks for building these powerful distributed applications that are sort of born in the cloud.

While at the same time, bringing software engineering best practices to the way we manage infrastructure, to help tame the complexity. Things like sharing and reuse of best practices, testing, continuous delivery of infrastructure alongside the application code. And really by doing that cloud engineering, allows both developers and infrastructure teams to work closer together.

Now I will say security is sort of in the loop as well. And increasingly we see that becoming part of the inner loop of how folks are doing cloud engineering. So that’s what it is to me. You know, I think cloud-native… Everybody has a sort of different definition of cloud-native, but I think of cloud engineering as more the practice, the discipline, very similar to sort of DevOps.

Swapnil Bhartiya: You mentioned DevOps. We… The whole idea of this movement was, and I may be totally wrong about it, is to break the old silos from the traditional legacy infrastructure where you had networking people, security people, storage folks. But if I look at all these paradigms that we talk about, DevOps, DevSecOps, NetOps, you know, SREs, we are creating new silos in a way. And that has more to do with the skill sets. You know, there are folks who are more inclined to our security, so they join security team. Because even today’s world, we talk about, hey, the security teams and developers team, even if we use the word DevSecOps, you know, or we talk about SREs, but then also reliable [tito 00:05:44], where the bug stopped. So can you also talk about, since your focus is going to be on practitioners, so can you talk about how much of these things are actually translating into actual practice versus what we keep hearing about or what we preach?

Joe Duffy: Yeah. I think honestly the interesting thing in the last year, and this is different, you know, we’re actually seeing our customers that we work with at Pulumi, using the same phrase, you know, Snowflake, who we work very closely with. They actually called their organization the cloud engineering team. Atlassian, another set of folks we work with, also use the term cloud engineering. So we’re actually seeing this in practice.

And to your point on silos, it’s absolutely true that we need specialization. I think one of the anti-patterns is, we’re not saying that hey, developers are going to become experts in networks and subnets, and how to cost-effectively run a Kubernetes cluster. Like that, that is really requires deep infrastructure, domain expertise. And that is incredibly valuable and requires specialization. It’s about embracing the specialization, but enabling people to collaborate. And I think that’s where the world is headed.

Swapnil Bhartiya: I think one of the reasons we wanted to break silos were those silos used to create bottlenecks. You know, for security, you have to go on top. So it was moving everything to blur those lines. But as you rightly said, those, as specialization, will remain. And that’s where the real value come from. Right? You cannot have everybody doing everything, then you will have, nobody’s doing nothing. Right?

Joe Duffy: Yeah. There’s actually a story I like to tell sometimes. I used to work at Microsoft, and back in the day we had a test organization and developer organization. And developers would write the code, then the testers would test it. And the testers would find all sorts of bugs.

One day we decided, hey, we’re going to combine these, right? Because that workflow is very waterfally. It doesn’t make sense in this agile world. When we did that we made a critical mistake, which is the team heard the message that, hey, testing is not important. And testing of course is important, but it’s a skillset that we all need.

And what we found was when you have developers thinking about testing their code, they write better code because they’re thinking through the end in quality. But then you still need those test experts, because really, test is enough of a specialization of a discipline, that you want people to think deeply about how best to test code and what systems to put in place. There’s a lot of analogies I see with where we’re heading with infrastructure and cloud engineering.

Swapnil Bhartiya: Let’s talk about folks who do want to build career in this space. And we have seen so many reports. There is a huge gap in supply and demand of [tenen 00:08:16] people who understand that. So what should, you know, a Greenheart… Whatever the right, appropriate, you know, politically correct terms there is, when they do look at a career in that, what they should look at themselves as developers? DevOps, cloud engineers, DevSecOps, SREs, what is the right term? Because it’s, in the end, it’s all about, as you mentioned, practice. You have to bring those practices. Tools are secondary.

Joe Duffy: Yeah. I think there’s a lot of… I don’t view the world in these silos. I really think of them as problems that we want to solve as engineers. Right? And so people naturally gravitate to problems that are more interesting to them. Some people really enjoy networking. Some people may be excited about the world moving to Kubernetes. Some people may be more excited about applications and maybe distributed microservices.

So I’d say, first pick that area that really excites you the most, but there’s a commonality amongst all of those. And to me, things like infrastructure is code, is common to all of these disciplines increasingly. If you’re just going to go be a full stack developer, 100% focused on JavaScript, maybe that’s not as exciting to you, but most folks that are building cloud software are going to care about infrastructures code.

And so interesting, I was talking to a friend of mine, who’s been in Ops for 20 years now. And he was saying, hey, I feel like I’m stagnating a little bit, so I’m going to go learn Python. And he’d never learned a programming language before, and that just unlocked a world of possibilities for him. So I would say, learn a programming language. I think that’s a key skill that no matter where you go, is going to serve you well.

Swapnil Bhartiya: Going back to Pulumi, and you earlier mentioned Pulumi registry. You briefly touched upon that, but I want to go a bit deeper into that. Explain a bit, you know, what it is, why you folks came out with it, and how people can consumer use it.

Joe Duffy: Yeah, absolutely. You know, in Pulumi, it’s an infrastructure’s code technology, we, but different from a lot of other tech, you know, we didn’t do YAML or domain-specific language. We said, hey, let’s use general purpose languages like Python, JavaScript, Go, et cetera.

So you can use the language you’re familiar with, get powerful tools in IDEs, and great programming constructs. And one of the things you get is sharing and reuse. You know, standing on the shoulders of giants of programming languages, the way we’ve tamed complexity in a bunch of different domains, is abstraction, modularity, sharing and reuse, building bigger things out of smaller things.

And what excited us with Pulumi is now with infrastructures code, in general purpose languages, you can apply that to infrastructure. So you’re not always copy and pasting YAML all over the place, relearning the same mistakes, making the same security mistakes, time and time again. You can benefit from real sharing and reuse.

So the registry is a place ago to find packages that you can basically get access to building blocks across, you know, 60 plus cloud providers. AWS, Azure, GCP, Kubernetes, a lot of great things, but also go from building blocks to architectures. So if you want to do like a full EKS cluster, that’s fully configured with Prometheus, and you’ve got an off-the-shelf component that you can use and don’t need to be an expert in literally everything about that hard problem.

And this is what I was alluding to earlier. I think the next phase is, hey, let’s stop all of us doing the same grunge work that is undifferentiated and easy to get wrong, and frankly, no fun. And let’s move to building bigger things out of smaller things. And really that’s that’s the way I think we take the whole cloud space to the next level.

Swapnil Bhartiya: Joe, thank you so much once again, for taking time out and talk about of course upcoming summit or the summit that is underway, depending on when we release the video, but that, most importantly, thanks for sharing those insights. You know, that is really matters today. And once again, it’s always a pleasure to have you on the show and I would love to have you back on the show. Thank you.

Joe Duffy: Absolutely. Always fun to be here. Thank you so much.

[/expander_maker]