There is a growing trend of software developer shortages leading to developer burnouts, gaps in skills, and security risks. In this episode of TFiR Let’s Talk, Glenn Russell, Head of Delivery at Qarik Group, discusses the security-related challenges companies face and how they can overcome them in order to better retain developers.

“By far and away, the biggest cause that we see for attrition at many different enterprises is that a company cannot get out of the way of its developers, and security is a big part of that,” says Russell.

While open source software has provided opportunities for everyone to access free software, it has also made it complicated for developers to see exactly what is going into the software, and securing the supply chain has become a critical consideration. Russell shares the key takeaways organizations can do to better tackle these risks and retain their developers.

Key highlights from this video interview are:

• There can be many reasons why developers burn out and quit. Russell shares his insights into what factors can contribute to this and what the biggest cause for attrition in enterprises is. He goes into some of the key trends he is seeing.
• The state of challenges application teams face has worsened since the advent of cloud and public cloud. Russell explains why the infrastructure amplifies typical software vulnerabilities and why securing the software supply chain is so critical.
• Although the purpose of open source is to make software freely available to everyone, there is a flip side where the attack surface has now been expanded. Russell talks through his views on the risks of open source software and securing the supply chain.
• Russell shares his three key points for how companies can navigate these security risks so that developers can then get on with their jobs: adopting a breach-first mentality, considering the minimal set of permission developers need to use to do their job, and adopting a defense in depth posture.
• While DevSecOps teams embrace security, this is not always the case for developers. Russell explains how we can find a balance between security, productivity, and the developer experience and practical ways to tackle these challenges.
• Russell believes that these security-related challenges do not necessarily require a technical solution or initiative but instead a cultural change. He discusses what organizations can do to bring about cultural change.

Connect with Glenn Russell (LinkedIn)

The summary of the show is written by Emily Nicholls.

[expander_maker]

Here is the automated and unedited transcript of the recording. Please note that the transcript has not been edited or reviewed. 

Swapnil Bhartiya: Hi, this is your host Swapnil Bhartiya and welcome to another episode of TFIR Let’s talk. Today we have with us Glenn Russell, Head of Delivery at Qarik Group. Glen, it’s great to have you on the show.

Glenn Russell: Thank you. Glad to be here.

Swapnil Bhartiya: Yeah. And today’s topic is more or less about how to retain or keep your developers. There are so many reasons why folks are switching jobs. A lot of folks are quitting as well in the tech space. There are so many reasons head of AI and ML that apple could, because they’re forced to go back to the office. People want to work remotely from home. Burnout is also happening because of COVID, everybody’s moving to the Cloud. What are the other factors that play a role where people get burned out and they want to move or quit and what companies can do to keep them?

Glenn Russell: Sure. Well, it’s a great question. And I think that there’s many facets to an answer, but ultimately when it boils down to it is, are you actually letting your developers do their job? And I think what we see frequently, a [inaudible 00:01:09] is we go into many companies who are successful companies. Let’s not misrepresent, but they have higher attrition and in many regards, I think there’s something of a Stockholm syndrome in place. Where, when you initially engage with these developer teams, there’s almost an acceptance that these are the… this is the way things have to be. So let me talk about the biggest example, for example, VDIs. This is a mechanism by which, developers are typically onboarded to a large company. It’s a highly restricted environment, doing anything in that sandbox might be safe from a security point of view, but from a developer productivity point of view, it’s something that you’re instantly fighting against from the moment you work for a company.

And by far and away, the biggest cause that we see for attrition at many different enterprises is, a company cannot get out of the way it’s developers and security is a big part of that. And let’s talk about the other significant challenge. It’s no secret that cyber specifically Cyber Security and in all its facets, not just OPSEC or InfoSec or NetSec is suffering from a chronic talent shortage. Even in a study like Belfast, where I’m based, where we have many security companies, there are many more open positions and they actually are people. And so what’s the obvious answer here really is to have people who have a company that gets out of its developers, but at the same time development teams need to start to accept more responsibility for the security of their applications.

Swapnil Bhartiya: Right. You touched upon it initially though that let developers do their job. Now, there are a couple of things. It comes when either you let developers do their job or developers feel that, Hey, they can do their job without [inaudible 00:02:55] so it’s about it’s mutual trust, right? And it’s also kind of freedom of being able to do, it’s about either you put the gates or you put guard rails, it’s about control as well. So talk about those aspect.

Glenn Russell: So I think let’s be clear. The challenges facing application teams have got worse since the advent of Cloud and Public Cloud. Really, I’m going to focus on the infrastructure and I know this isn’t strictly the application teams themselves, but it, the fact is that you take the typical software vulnerability, like Log4Shell or Heartbleed. The infrastructure really amplifies those vulnerabilities more and more rather than dampen them.

What I mean by that is if you were to SOME…maybe cynically, look at how we distribute software today, really what we have is a really potent vulnerability delivery mechanism. I remember for example, earlier in my career, burning software on a CD and traveling on a plane to a customer to install a piece of software. Now I just have to make it available on GitHub and I can somewhat guaranteed that someone’s going to download it, not knowing what’s in that piece of software. So increasingly we have to look at the Supply Chain itself and say, well, how can we secure this? How can we make that safer while getting out of the way of development teams?

Swapnil Bhartiya: When we look at software Supply Chain, why it matters today? Of course, in the old days you had one Monolith, everything was coming from one source. Today, you have so many dependencies, so many libraries, Open Source technologies are being used. So, I want to first talk about why we need to understand, second would be that how much awareness is there about software Supply Chain. Sometimes what happened is folks will say, Hey, we just downloaded that image, Docker image or Container image from GitHub and they’re using it. They have no clue what is in there.

Where there’re links that can be changed. Where was the source is coming from? They, it may not even just be about security. It could also be about compliance about the Open Source code base that is used there. You may be safe now, but when you build a company and five feet down the lens, somebody’s going to acquire you and they look at the code base and they’re like, Hey, you know what? You’re not even compatible. So, so talk about these two aspects.

Glenn Russell: So let’s talk about that my view here, this is a perfect storm of an inability to manage risk. And let me talk about that. I know it sounds very grandiose, but we can bring it down to actual cases. So you’re perfectly right in what you say more and more. It’s not just, you’re not just bringing together software from one company or one entity it’s many.

And although we kind of maybe understand that the under that the mantra of Open Source to next software freely available to everyone. That might be true to some regard, but now what we’ve done is we’ve expanded our attack surface to be, to encompass many, many different people, with many different opinions, many different political backgrounds. And I don’t want to get into that. But the reality is that we’ve seen a number of documented cases where a developer has decided, they no longer want to support a piece of software and they’ve actually sabotaged it.

And even I’m sure you are well aware of the left-pad library and MPM. That one change for really innocuous piece of software killed perfect productivity at many, many different companies. And it’s not a traditional software vulnerability. That’s why I say risk. It’s not like somebody trying to pop a shell on a VM or perform a sequel injection attack. It’s very much, it’s almost a vulnerability in people if you know what I mean? But now, we’re more subject to it. And that’s why I say current software development methods have become an amplifier for all kinds of risk, rather than helping us, hopefully that answers your questions. So I don’t know how to expand any other points.

Swapnil Bhartiya: Excellent. Thanks for taking that question. Now, one more thing that you were talking about, you gave an example for political reason. Folks can just one line and it could have [inaudible 00:06:59] effect, but the fact is that we have not fully understand the risks that are there. Okay. And once again, it doesn’t matter what your political leaning is. If you block access to X country or region, I don’t know, government agency, whatever it is, it could not be just deception and access to the software. What if it was a library or package that was used for security, like you will expose, user data to every. So we don’t even know the risk. So can you just quickly give us a picture about what are the risks there? And then we’ll talk about what companies can do to put some SafetyNet there? But let’s talk about the risks.

Glenn Russell: Well, the risks are that, you could have an artifact repository, which if we talk about the broader topic of the secure software Supply Chain, and that’s really the discipline of how you take a piece of software and you build it. You basically are super specific about which patents you build into that library. Then when you actually build up piece of software, which could be a hermetic build, something like which basil does for you. And then you could even introduce, if you’re more advanced, you could do something like a Binary Authorization that says, I have to sign this piece of software to run it, even after all that. And that’s a big assumption by the way, you could still be in a scenario where you’ve installed a piece of software, which may even have a secure library.

For example, you could still be in a position where you have a Zero-Day of vulnerability in your infrastructure and you didn’t know about it. Now, Log4Shell is a prime example. Heartbleed was in the OpenSL code for years before anyone detected it. So let’s talk about how we should then think about that. What, and again, back to my comment, our way of building software right now has become a way to amplify those vulnerabilities. So in my mind, there’s three things that really we need to talk about. And it’s going to come back to putting the responsibility at the fate of developers.

Number one: a breach first mentality, build your software and deploy your software that assumes that it’s under attack immediately. And that might be purely an internal attack. But the reason we say that is because that gets us into the mindset of saying, how would I recover from Bad Data? How would I recover from a DDoS attack? for example.

Even internally because as it’s been proven by many breaches over the past few years, that where a hacker gets access to a piece of software, they’re traversed throughout the entire infrastructure. But more often than not, it’s a valid set of credentials, which is from an internal attacker, which brings me onto the second point and that’s the principle at least privilege. Application teams, they need to think about, what’s the minimal set of permissions I actually need to use to actually do my job? No, bringing this back to the broader topic of developer attrition. I think that these kinds of guard rails are things which can be really done once and replicated across application themes. But I think there’s challenges around the heterogeneity of environments overall.

But my last point in this Swapnil is, it’s adopting a defense in depth posture, and that is okay. I’m protecting the application here, but what if this control fails? What’s my backup ? and that could be as simple as well, close all my open ports. But you know, what, if an attacker does find a way through the network, well, how can I protect that data? How can I provide access to that data so that it’s not compromised under many different what we call compensating controls and compensating controls are things which rarely I think application teams think about.

But again, this is why it comes back to an education mindset first and foremost, because we are not going to find 500,000 cyber specialists in the next year or two. The meanwhile, NationStates down to a script [inaudible 00:11:06] who finds some of the internet are attacking applications. Now I’ll finish off if I may by saying, and this is a hard problem. And a security vendor that’s telling you that they have this magic box, which fixes all this for you is either ill-informed or being deceptive. But if we solve this core problem of me, give us guardrails, give us sensible defaults and a sensible configuration. So we can get out of the way of our developers that can write code. I think we’ll make big inroads to develop or address it, the developer attrition problem.

Swapnil Bhartiya: As you’re talking about, they’re not enough cyber security expert and also security folks, DevSecOps folks. They love security, that’s their passion, but not everybody, not developers. Sometimes, this could be a drain on their resources, energy, it affects productivity aswell. So can you also talk a bit about, you did touch upon in great details, but if I can ask you just to summarize. Share some points, not that essentially a whole playbook that what can be done to kind of, so that the…there’s a fine balance between security, productivity and also developer experience.

Glenn Russell: Well, I think that starts with just understanding where any given application stands today. And that could be, for example, the number of open security issues. It could be another good metric for example is, how long it takes to close any given security issue. And I think the suggestion meant to any team. And again, I’m talking to an average application team here. I’m not talking to one that’s running COBOL or on the other hand, one that’s running Kubernetes all day.

It’s just understanding where you are right now. So that helps you understand where you have to make the biggest impact, because obsessing a bit of vulnerability, which in reality is not exploitable or at least as a lesser risk is a waste of time. And using metrics like that, the intent would be that you focus on the security issues that matter most to any given application team. And that’s not a technical answer. I realize that, but because it’s not a technical solution, it’s a mindset in the team itself that needs to change in order to help achieve that balance. Other than a… otherwise teams will be faced with the constant brick wall of the security operations teams in node, everything. And that only goes one way as we’ve discussed already.

Swapnil Bhartiya: Since you’ve, folks work very closely with your customers and of course clients, you have tech solutions everywhere. What do you do to bring cultural change there?

Glenn Russell: Well, it has to start at the top. I think it goes with any kind of technology initiative or overall change in how team works. Even let’s talk about transition agile, is going to fail out of the gate. If you don’t have buy in from an exact team down because everything less basic comes down to money and you need the funding and you need the backing. So I think for me, if I’m not talking to a say-so for example, and that can be a technical say-so or an executive say-so, I need to be able to pitch to them and say, these…this is the problem I’m trying to solve.

Realistically, this is where we can get to. And then having an honest conversation, because unfortunately, I’m sure as you well know nobody…everybody wants security, but nobody really wants to pay for it. And part of the problem is that the average C-suite doesn’t know where to spend money to get the maximum impact of that investment. And that is really the conversation that we have with our customers. Now that does that require deep tech expertise of how GCP works, how appsc works, how networks work. Yes, it does. But ultimately it all comes back to a business problem of…and it relates to one of managing risk of course, but that has, is a story. It has to come from the top down again, a people problem.

Swapnil Bhartiya: Glenn, thank you so much for taking time out today and talk about this topic. We don’t talk about this aspect of attrition and of course selling retention, but it was really incredible that you touch upon those things and also share some tips what companies can do, developer team can do. So that was incredible. And once again, I would love to have you back on the show to discuss another aspect of developer productivity and how companies can more efficient, but I really appreciate your time today. Thank you.

Glenn Russell: Thank you. It was a pleasure.

[/expander_maker]