Guest: Ted Young (LinkedIn, Twitter)
Companies: Lightstep (Twitter)
Show: 2022 Prediction Series

Ted Young, Director of Developer Education at Lightstep, predicts that in 2022, there will be wide mainstream adoption of OpenTelemetry (a project for sharing telemetry). “One of the main reasons people are going to want to do that is because it will allow them to write their instrumentation once, install instrumentation once, and then be able to send that data to any observability tool or provider that they wish to use,” says Young.

He also predicts that open-source software will start to come with native instrumentation. “This also includes things like databases, things like hosted services, all of those things are going to start producing observability data that are hooked into your application traces,” avers Young. Check out the above video to know more.

[expander_maker] 

Swapnil Bhartiya: Hi, this is your host Swapnil Bhartiya and welcome to our 2022 predictions series. And we have with us today, Ted Young, Director of Developer Education at Lightstep. Ted, it is good to have you on the show.

Ted Young: Great. Glad to be here.

Swapnil Bhartiya: Yes. Before I ask you to grab your crystal ball and share your predictions, could you tell us what is Lightstep all about?

Ted Young: Lightstep is an observability company, and what we focus on is integrating all the different kinds of telemetry that your application produces, into something that’s like a coherent work flow. We started with distributed tracing because we see that as actually the backbone that everything else needs to be built on top of, and then have been moving into metrics and other forms of data and providing a workflow that’s augmented with some advanced statistical analysis for finding correlations and basically saving you a lot of time. A lot of the time you spend observing your system is actually hunting around for data, not formulating hypotheses and investigating them. And we try to eliminate that time.

Swapnil Bhartiya: Excellent. Thanks for sharing that. Now it’s time for you to pick up your crystal ball and tell us what predictions you have for 2022?

Ted Young: Yeah, so I actually spend my time working on the OpenTelemetry project. I’m one of the co-founders of the OpenTelemetry project and I’m on the governance committee there. The OpenTelemetry project is a project for sharing telemetry. So having a standardized language that all computer systems are using to describe what they’re doing. It’s built on top of distributed tracing, because you need that to connect all the data between distributed system components, but it emits traces, logs, metrics, resource information about where your programs are running, and generally takes all of that data and combines it into a single data structure.

So that’s my first prediction for 2022, is wide mainstream adoption of OpenTelemetry. One of the main reasons people are going to want to do that is because it will allow them to write their instrumentation once, install instrumentation once, and then be able to send that data to any observability tool or provider that they wish to use. You can even tee it off to send it to multiple providers because all of the different providers are adopting OpenTelemetry, specifically the OTLP protocol that has this combined pipe of tracing logs and metrics and resource information.

So that’s a big problem that people have with their systems right now, is they’re having to just do a lot of work if they want to switch where they’re sending their data, because classically, all of these different observability systems use different data formats, and they only analyze the portion of the data. So you’re going to see a shift in these observability tools to accepting OTLP as a standard pipe, and also spreading out from focusing on just one form of telemetry data like logs or metrics, into providing tooling that’s more like coherent integration of all of that data.

Prediction number two, I would say, is native instrumentation for open-source software. This is the other thing we focus on with OpenTelemetry, is not only providing a standard, but providing that standard with an implementation that’s compartmentalized and lightweight in such a way that open-source software could actually natively instrument itself. This is a problem if you write any kind of shared library that’s going to be run in a bunch of different applications by a bunch of different end users. They all have different choices about where they want to send their data, and if the instrumentation you’re using is dependent upon where the data is going, well, that’s a choice you can’t really make as someone providing a shared library, that’s going to run in lots of different applications.

So by instrumenting with OpenTelemetry, you can take a shared library and install it in many applications, and then the operator of that application can choose where they’re going to send the data. We make sure that OpenTelemetry is very lightweight. It doesn’t cause dependency conflicts or overhead when it’s not in use. So it’s a really good choice for open-source software. I think that’s, in many ways, really a first. I think we’re the first project to really focus on open-source software instrumentation and observability as a high priority. I think that comes with the fact that OpenTelemetry has no backend. There’s no particular system we’re trying to send the data to. The goal of the project is to send the data everywhere.

So that’s my prediction number two, that open-source software will start to come with native instrumentation. This also includes things like databases, things like hosted services, all of those things are going to start producing observability data that are hooked into your application traces.

I would say my third and final prediction for 2022 is this integrated pipe of logging metrics, tracing, and resources is going to drive a leap forward in automated analysis. The reason for that is any form of advanced statistical analysis or machine learning, anything that would fall under the AI category, really require high-quality, well-structured data. The traditional three pillars approach where you had your tracing data over here and your metrics data over here and your logging data over there, all of those data streams were uncorrelated with each other. There was no way to correlate those data streams effectively. So you had to use an ad hoc approach to do any kind of analysis that wanted to analyze things across all of those different data streams.

That’s actually really critical when you’re trying to figure out what’s wrong with your system, because it’s usually a mix of problems. It’s a mix of multiple different transactions interacting the same time with different resources. There is something about the way those transactions are trying to simultaneously use those resources that is creating a problem. Those are most of the hairy problems that operators face in the real world, kind of falls into that category of, it’s not just there aren’t enough resources, it’s not just, there’s a bug in the software and it’s doing something wrong. It’s something subtle about the mix between what the software is trying to do and the way all of those requests are sharing the resources at the same time.

So if you’re looking at an integrated stream that includes statistical data about what those systems are doing through metrics, but then also transactional data like tracing and logs, having that connected into a single graph is the kind of data structure you actually need to do advanced forms of statistical analysis and machine learning.

So since we have that now with OTLP and OpenTelemetry, you’re going to see, I think, a leap forward in the kind of offerings different providers are going to be making on that front. That’s what I got. Those are my three predictions for 2022.

Swapnil Bhartiya: Excellent. Thanks for sharing these predictions. Now, if I ask you what is going to be the focus of the company in 2022?

Ted Young: Yeah. So I would say Lightstep is going to focus on expanding to ingest more for forms of data coming in. There’s lots of different data out there. OpenTelemetry is in the process of integrating new data sources, such as eBPF for low-level network monitoring, for example, and RUM, real user monitoring for being able to have more in-depth client information. You’re going to see LightStep integrating all of that data into a really effective piece of software.

I think the other thing you’re going to see us do is improving the kind of analysis that we offer on the data we already have, and improving the workflows that we have, just to try to make them smoother and more polished. I think the core product that LightStep already has is really, really powerful. So mostly we’re in like a polished phase to this point.

Swapnil Bhartiya: Excellent. Ted, thank you so much for, of course, sharing these insights through these predictions, but also share the focus of the company. I would love to have you back on the show next year so that we can have a scorecard and see how many of your predictions turn out to be true. Then also set up predictions for next year. But thanks for your time today and happy New Year. Thank you.

Ted Young: Absolutely. Thank you for having me.

[/expander_maker]