Understanding the code powering one’s services and products has always been critical to the security, reliability, and reputation of that company. There is now a renewed interest around understanding the software supply chain and bill of materials, thanks to the Biden Administration’s Executive Order, which Dennis Zimmer, CTO of Codenotary, believes will make the software supply chain and bill of materials a global phenomenon. But what exactly is the software supply chain, what is the bill of materials and why does it matter to any company that offers software-based products and services? Here is an in-depth discussion on the topic with Zimmer. I hope you will enjoy the show.

Here are some of the topics we covered in this show:

• What does Codemotary do?
• What is the software supply chain and bill of materials?
• Risks of not understanding your software supply chain?
• How much awareness is already there about the software supply chain and bill of materials?
• What kind of solutions does Codenotary have for organizations?
• What is the impact of the Biden administration’s EO on the cultural shift towards the software supply chain?
• Global impact of Biden’s EO.

Guest: Dennis Zimmer (LinkedIn)
Company: Codenotary (LinkedIn, Twitter)
Show: Let’s Talk

[expander_maker]

​​Swapnil Bhartiya: Hi, this is your host Swapnil Bhartiya, and welcome to other episode of [inaudible 00:00:09] talk. And today we are here with Dennis Zimmer, CTO and Co-founder of CodeNotary. Dennis, it’s great to have you on the show.

Dennis Zimmer: Thanks for having me.

Swapnil Bhartiya: Excellent. Before we get started, I would love to know a bit about the company, you’re the co-founder. So what does CodeNotary do?

Dennis Zimmer: So CodeNotary has been founded in 2018 and we had from the start, the goal that we were able to verify every digital asset. And we focused pretty soon also on artifacts. So whatever artifacts are in software pipeline, be it the build, the file, source code, or then also DACA container image. Everything should be a verifiable and everybody should be able also to get an idea who gave trust… so who trusted a certain asset. And since then, actually came a long way. We also started to develop our own immutable database that is open source. It’s called Immudb.

And we just also released community at station service, that is also completely free. And we have a commercial product called CodeNotary Cloud, that covers all needs from source to production when it comes to followed digital artifacts, the providence of it. And of course, and also if you trust or maybe untrust something. What is special about us is we don’t use any digital certificates for it. So everything is being done using a mutable database. So depend only on database where everything that is stored can be verified and cryptographed, be verified by every client.

Swapnil Bhartiya: I want to understand from you as the importance of understandings open source or just software supply chain, then only we can mitigate it. But talk about what is software supply chain, what is bill of material? Then we’ll talk about the other aspect of it.

Dennis Zimmer: From a software supply chain perspective, it can be very simple. So a lot of companies are still in a very early phase, I would say. So they have some scripts. Maybe they have some continuous integration tooling. So they have sources. They build a certain software, they test it, and then they ship it independently to production. So it’s not an automatic process from building to production. And when we created all of our tooling and all of our software, we always had in mind also the non-automatic fashion and also a supply chain that maybe is cut in half. So you have an automatic part and you have a manual part and still everything needs to work. So when you combine these information, it’s always important that you have a single source of truth, where you can connect to where you can verify, “Did I already touch this source code? Did I touch this binary? Did I touch this script? Is it mine? If it came from an external consultant or a third party vendor?”

And that is more the bill of materials part. I want to know what is inside. So in an open source fashion as mentioned, it’s pretty easy. You have some requirements file, or you have some information that you can use during build time. But they are also in the meantime, open standards, like SPDX or Cyclone DX, where a vendor could provide you with a binary as well as the software bill of materials file. And as long as it follows the standard, you could either attach it as evidence inside of your supply chain or software supply chain, or you could use this information to link to the artifacts that you know. And that way of you have a full transparency from either you create your own source, or you have it ship by yourself party vendor, and what is running in my production right now.

So for us, it’s always important for our customers, how can we connect the dots? So how can I also traverse from my production back to the binary that has been tested and built? When was it built? And when it comes to software bill of materials, I also think that information like vulnerabilities scanner report or compliance scanner report, needs to be attached as well because these situations can change. So it was okay to ship something in production on Monday, but on Wednesday there’s a vulnerability and now it’s not okay anymore. How do I proof that back on Monday, it was okay? So the software supply chain’s much more than just the code, just some attestation capability. It’s also about storing evidence, why something happened, and of course the way back and forth. So I need to be able form source to production, but also from production back to source to understand who developed it, where did it came from, when was it tested, is it still compliant or not, and eventually being able also to pull out something out of production manually, as well as, fully automated fashion.

Swapnil Bhartiya: Excellent. There are a couple other things that I want to discuss. You mentioned compliance and the second thing is that with open source, if you just look at Myskill or modern DB database, there are so many maintainers who maintain it. So you can add [inaudible 00:05:44]. Doesn’t matter what you’re doing. And then there may be, as you mentioned, changes that you made or changes made in the upstream of the project as well. Sometimes we can also talk about containers where they may be hard links to the project, and then those hard links can be changed by the one who ever created that image that you’re pulling into it. Can you talk about the risks that are there by not knowing what’s going in your supply chain? It could be security, or it could also be compliance for example. If you are doing a lot of internal coding, you might be avoiding GPL license, or you may be exposing your own IP. And so you sue or if you are going for an acquisition, that can become a big challenge as well. So talk about the risks of not knowing what isn’t there?

Dennis Zimmer: I actually can give you a real customer example. The customer was using a component where they signed a conflict and an agreement of to use the software for a certain period of time. And now suddenly the terms were not good enough anymore. So the parties actually wanted to terminate the agreement and, of course, to make sure the software has not been used anymore from this day on. If you don’t know where it’s being deployed right now and if you don’t know where it has been used, you automatically violate from the very same day, the agreement that just has been terminated. And you are automatically up for a legal fight in the worst case. So that is just a real example that can happen to pretty much anyone, any company, any kind of software. But you mentioned also GPL or other more restrictive licenses.

If you don’t know when you deployed something, because very often not a lot of licenses change over time. If you don’t know what has been and deployed and when, and is it still the version with a better license for my enterprise, or now a license that I don’t want in my environment anymore, you cannot revoke it. So you cannot really get back and pinpoint, is it still in my production, or maybe it’s just part of a backed up application but we are not using it anymore. And that is a huge risk of course, for any company. But when you take the [inaudible 00:08:09] means cases of course they are very popular one, but in this case, a lot of companies still don’t know if they’re affected or not, because they don’t have a inventory or catalog. They don’t have an idea what DLLs and libraries are being used for certain software pieces.

So if you would be able to search for the unique fingerprint and pretty much everything has a unique fingerprint file, a container image, even a base layer of a container image, if you could find it in and revoke immediately, that is exactly what we provide. So you don’t need to have physical access to this file or to this container image anymore. You just need to know what part of it has a following fingerprint. And on this information you can revoke or just unsupport for a temporary amount of time, a certain container, so you’re not sure. It could also happen, that’s also a real customer case, there was a vulnerability scanner cluster, and one of the cluster nodes was outdated for a couple of days. So how do we know afterwards what is now running in my production and was scanned by an outdated scanner? In our case, the scanner has an identity. The digital assets has a fingerprint and we created transaction based on it. So you can always follow the prices and you can know exactly what an outdated scanner scanned, and if it’s still out there now.

Swapnil Bhartiya: Now let’s just talk about the basic rules again. We are talking about software supply chain, we are talking bill of material, there are a lot of solutions out there. But how much awareness is there within the users or industry? How much they are creating bill of offers because of some government requirements. They do have to do that. But in general, what has been your perspective that companies are fully aware of? If they’re taking steps or you’re like, “We still have to do a lot of work to actually educate them about it.”

Dennis Zimmer: There’s definitely a lot of work to do. So what I mentioned in the beginning, there are so many customers at different stages. Some especially companies that just created software or started to create software for them, it’s so much easier. They can use all the tools around to automate the whole CICD journey, but when it comes to heritage or legacy applications that have never seen any CICD pipeline, it’s extremely hard. But for a complete success of protecting a supply chain or software supply chain, we need to have a combination of both. So you cannot just exclude one of the legacy applications and hope for the best. If this legacy application has an issue, then your whole software stack is at risk. And so to answer your question, it’s really very, very, very mixed.

So we see very advanced customers. I would say everybody cares about it, but for some customers, it’s just a very long way to go from the account development and how they develop today to a fully secured software pipeline, including a bill of materials. So I think the first step is also not the bill of materials. It’s being sure having a Providence in place, having an attestation in place for the already built application and the software bill of materials is then the next step. And that is when it comes to standards, it’s not really a standard, but SALSA framework from Google. So for software supply chain levels for software artifacts, it’s already very good guideline, where you see exactly where you are as a company today, are you at the lowest level, level one where you just have an idea of your current artifacts and what you’re using, or are you already at a higher level.

And level four becomes extremely hard because you need to store immutable, your software bill of materials. You need to store immutable, a lot of results of your software pipeline. And I would say it’s nearly impossible to achieve without automation, therefore, the automation and then solutions like ours that providing mutability and provide also client verification and automation is extremely important to implement into any of the journeys towards the SALSA framework as well.

Swapnil Bhartiya: Yeah. Since it’s a very good segment to that, let’s talk about new solutions. How do you really help these customers either, as you said, new customer, new players, who are still writing, or we look at a broad feeling of they have a long legacy quote up there. So talk about how do you help them? What kind of services or products you offer?

Dennis Zimmer: So, first of all, our solution is always a combination of APIs, CLIs and then our backend who stores all the data immutably. And the CLI part can be run, of course, manually. It can be run in this script in the CLI or the full blown CICD pipeline. From the beginning, we always designed it that a human can run it as well as a machine. And we take into account also the different stages of a company, because when you start using us for the first time, you need to trust everything you have. So there’s a kind of a baseline that you need to set. You more or less say, “Okay, everything that is currently in my environment, I need to trust.” But compared to digital certificates, we can change this status.

So if you encounter weeks later, a situation where you think, “Oh, I cannot trust this software.” You cannot revoke trust for this specific artifact. So you don’t need to revoke a whole certificate or everything that is attached to it. You can really precisely revoke a certain piece of software, a certain library, or a certain container image. So that makes onboarding so much easier for customers because they can start with a baseline and then they can change over time and change also their opinion over time when implementing also vulnerability scanners, or implementing compliance scanner. And then they can use these new functionalities and these new information to change the trust they give to any of their artifacts or applications. And one thing that became very obvious to us is if it’s only a commercial service, you will never reach the majority of people. So there needs to be something where everybody, a small or single maintainer of a tiny software, or also larger open source projects, they need to be [inaudible 00:15:33] operating system providers.

They need to be able to use something that everybody can use free of charge without any limitations when it comes to attestation, for example. And therefore we released about a week ago, the community attestation service, where you just register with your email address, and based on your identity that you get in a return, you can start drafting files, source code repositories, container images, and even the software bill of materials of container images. And you can provide this information to others like some open source projects do. So I’m going to send you my assign ID and starting from them, you can authenticate and verify if this software is really coming from me. And I can untrust at some point in time and say you should go away from this version because I’m not supporting it anymore. And that platform we think is a major milestone to really achieve and get into the whole software attestation software supply chain protection, because you can start in a very simple way just with a command line, but you can also fully automate it using GitHub actions or whatever CICD application you want to use.

Swapnil Bhartiya: The recent Biden administration’s executive order, is that claim in encouraging, though it restrict it to the companies who are serving federal agencies or other agencies, but how much it is helping you folks the cultural shift?

Dennis Zimmer: Actually, a very important point. Basically it’s a presidential executing order for cybersecurity. I think the awareness changed completely. That is also what you see with the latest [inaudible 00:17:32]. The topic was software supply chain protection, how can you make sure that you have a station in place and so on. So I think the priority massively increased from a year ago to today and we are just getting started. Also solutions are getting left and right. People have other ideas how to create the standards, of course, but also how to create a software bill of materials in the simplest way. But from a technology perspective, I also think we have technology in place. There are solutions that are digital certificate based. There are solutions like ours that are more cryptographic verification that lives without digital certificates.

And honestly, from an adoption perspective, I’m a absolute believer that it all boils down to simplicity. So if a solution is not easy to use, people are not going to use it. If it can’t be integrated extremely easy into existing processes, it’s not going to be adopted. And of course, if it cannot be automated at all, then you are stuck very quick and you won’t ever reach the goal of having a full integration into your… no matter if it’s a software pipeline or whatever kind of process. The analysis is there, people have the topic on their agenda. Probably in 2022, it’ll be much more focused. And I would say most of the technology, even if some of the ideas are still in the alpha one in the early stage, they are already tested. So for example, Google has it in place already for many years.

So they are already tested in large scale environments. So now it’s more about how can smaller companies, how can the average company adopt similar or at least a portion of this protection. And we do our best offering very simple to use solutions where there’s no burden to get it up and running. So no friction at all. You can start with it, you can easily create bill of materials, you can easily attest, notarize, verify. And I think over time, people will find out how simple it is also to integrate into existing pipelines. And that is my firm belief that the simplicity is going to change it.

Swapnil Bhartiya: Right. Since we were talking about the bill of material, which is more or less a US centric order, but what trends are you seeing globally in Europe? In some cases they are way ahead when they come to privacy or all these. So talk about whether this is going to be, when you said in 2022, you’ll be taking it more seriously. It’s going to be a global phenomenon or is it going to be US only phenomena?

Dennis Zimmer: I think it’s already starts to be very global. So I had a meeting a couple of weeks ago in Switzerland, in Germany, in the UK, and everybody knows about the executive order. It’s important that we get a solution in place that we know what we currently own, what we currently run. And the biggest fear for the companies is what happens if I see on the news, there’s another kind of a [inaudible 00:21:26] tech. And my boss is asking me for the second time, “Hey, are we affected?” And I need to say, “I have no idea. Give me a couple of weeks and I’ll find out.” I think that is one of the biggest fears that we currently see. And to be honest, it makes a ton of sense to have an answer to that in a… the best way would be instantly. But at least a very short amount of time because otherwise, how can I really mitigate risk fast enough if I don’t even know if I’m affected or not.

Swapnil Bhartiya: Right. Since we are talking about the bill of material, which is more or less a US centric executive order, but what trends are you seeing globally? For example, Europe, not in many cases, they’re way ahead of us when it comes to things like privacy and many other topics. So when you said that in 2022 you’ll be taking it more seriously, is it going to be a global phenomenon or it’s going to remain US only phenomena because of this executive order?

Dennis Zimmer: No, I honestly think it’s the main driver. It’s the main driver. In Europe, of course, GDPR and privacy was one of the first topics. But when it comes to software supply chains, I would even say that some of the larger corporations in the US are much more advanced than many companies in Europe. But it’s the Joe Biden executive order opened the eyes for many people. And then you can also you cannot deny it anymore. It’s there. You cannot say, “Okay, I wasn’t aware of it. I had no idea that we need to do something like it.”

So I think this is now it’s globally started. And it’s also about the vendors, because if you run software, let’s say software as a service, then you don’t know anything about the software bill of materials of the guys who were running the software as a service. So the next step will be, “Hey, they need to disclose the software bill of materials as well. Why should I use a SALSA service where I have no idea if the software behind it leaks data or something like this.” So I think we are just getting started. But it’s a global effort now.

Swapnil Bhartiya: Dennis, thank you so much for taking time out today and talk about not only of course the company, but also the importance of understanding, knowing supply chain and beloved material and the work that you folks are doing there. So thanks for the discussion today. And I look forward to our next conversation. Thank you for joining today.

Dennis Zimmer: But then thank you very much. It was a pleasure.

[/expander_maker]