Guest: Anurag Gupta
Company: Calyptia
Show: Let’s Talk

Calyptia is a first mile data observability platform, created by the folks behind the Fluentd and Fluent Bit projects. Recently the company announced the release of Calyptia Cloud, which is geared specifically for cloud-native observability pipelines to help enterprises optimize data between applications and workflows.

Calyptia has been part of the cloud-native ecosystem for some time now, with both of their original projects reaching critical mass of about two million deployments a day.

According to Anurag Gupta, Co-Founder of Calyptia, “We started to see folks who are going through this large observability journey have trouble with controlling that data, being able to understand how to route that data to send to multiple observability tools at high performance.”

With Kubernetes, users now have the power to spin production environments in a matter of seconds, which means more services, more applications and more load. It’s even more challenging to know and understand what’s going on in one’s environment.

That’s where Calyptia enters the picture to ensure that enterprises have the capability to do that much more efficiently without having to undergo the pains that just come with natural open source management and operations.

Understanding First Mile Observability

“Observability is really about developers and practitioners being able to get insights about their systems, being able to diagnose, troubleshoot, react, and be proactive about all of it,” Gupta explains.

Developers are collecting data from different sources and one of them, for instance, is infrastructure. They are collecting data from the systems themselves, “that’s really where the first mile comes into play,” says Gupta. “It’s the first step where you’re collecting that data. You’re routing it, potentially through, what we call, middle mile markers, with things like Kafka or other data pipelines.” Then from there, it goes into the observability systems such as Splunk, Datadog, Elastic, or New Relic.

The Arrival Of Calyptia Cloud

All of this becomes very complicated very quickly and Calyptia wanted to make it easy for users to consume these open-source tools without any compromises or pains that came with open source on day 2.

“Calyptia Cloud is really an overlay on top of what exists today,” says Gupta. “Instead of trying to displace what already exists in a thriving ecosystem, we wanted to build an overlay that just adds value to anyone using these projects.”

Calyptia has added connectors and the ability to attach your Fluent Bit to Calyptia Cloud within five seconds, and get a wealth of information about how that application is performing.

In first mile observability, this is the data you’re collecting and the data you are routing. “We can tell you things like how much data you’re sending to a particular backend, which sources, which teams might be overloading the system with what they’re collecting. And naturally, that gives you a better ability to troubleshoot and diagnose what might go wrong when your queries or your insights might be a little misguided on the end side,” he adds.

Calyptia Cloud is a fully hosted solution, which is run within the company’s public cloud. It doesn’t matter if you’re running in a public or private cloud, you can connect to Calyptia Cloud via API keys.

The summary of the show is written by Jack Wallen

Here is the rough, unedited transcript of the show…

Swapnil Bhartiya: Hi, this is your host Swapnil Bhartiya. Welcome to TFiR Newsroom. Calyptia is a first-mile data observability platform from the creators and maintainers of Fluentd and Fluent Bit projects. The company recently announced the launch of Calyptia Cloud, which is a cloud native enterprise observability pipeline for optimization of data between applications and workloads. To talk about this announcement, we have with us Anurag Gupta, Co-Founder of Calyptia. Anurag, it’s great to have you on the show.

Anurag Gupta: Awesome. Great to be here.

Swapnil Bhartiya: Let’s start with the basic, tell me about the company. So what problem are you trying to solve with it, that’s why you created it?

Anurag Gupta: For us at Calyptia, we’ve been part of the cloud native ecosystem for a while with the Fluentd and Fluent Bit projects. And those projects have reached a level of a lot of critical mass, right? About two million deployments per day, folks who are using them with all sorts of technologies, all sorts of observability platform, security platforms. And for us, we started to see, as being part of this community, being part of the creators and maintainers, is that folks who are going through this large observability journey have troubles with controlling that data, being able to understand how to route that data to send to multiple observability tools. Folks are having trouble with being able to send that data at high performance. Kubernetes is growing, you have more and more cloud native workloads. The demands that we need to meet from an observability pipeline perspective are just continuing to grow in scale and abundance.

And then the last part is just the management of all of this. We have a server, it’s maybe five or 10, in the corner, they’re all… something managed, you have folks who know them really well, know what they’re named. And with the advent, again, of Kubernetes just growing throughout production environments, folks are starting to say, “Well, now that the loads are increasing, these things will spin up for two to three seconds, how do we make sure that we are creating an environment that we have full understanding of what’s going on?”

So really those three problems are things we saw at Calyptia, and said, “These Open Source projects are really prevalent and they could use some enterprise features and things that we could pour on top to, one, help and improve what the community can do. But, two, enable enterprises to do that much more efficiently without having to undergo the pains that just come with natural Open Source management and operations.

Swapnil Bhartiya: This multi-cloud, hyper-cloud also kind of exaggerate the problem as well, because now it’s not just one cloud, it’s so many clouds from different operators?

Anurag Gupta: Yeah. It’s a great point. We think of folks who are running within multiple public clouds, like AWS, Google, Azure, folks who were running on-prem, folks who are running, say, an air-gapped environment, maybe military installations… Recently folks were running Kubernetes and Kubernetes on the edge at things like jets and all sorts of fun machines, that just continues. And the great part with Fluent Bit and Fluentd is they’re already used by many of the major cloud providers, so the Google Ops Agent, folks in Microsoft with Azure Kubernetes service, EKS, with an AWS, the Elastic Kubernetes Service, they’re already using these Open Source technologies.

And so from the Calyptia side, being able to cater and kind of pour the gasoline on top of these projects that are already present with it, these ecosystems can help strengthen those abilities for those developers that are facing these multi-cloud issues.

Swapnil Bhartiya: I want to talk about these two projects, and before we go there, I want to understand a bit and you are the best person to explain, what exactly is first-mile data observability? How would you define that?

Anurag Gupta: Yeah. Great question. So I think we almost have to take one step above and say, right, we have this observability term, it’s definitely a little overloaded in the industry today where folks will say, “Ah, it’s these three data streams.” It’s all of the things you can encapsulate from a business value. But when we look at that observability journey, it’s really about developers and practitioners being able to get insights about their systems, what’s happening, diagnose, troubleshoot, react and be proactive about all of it. And in that observability journey of, say, the infrastructure, there’s a portion where you’re collecting that data. You’re grabbing that data from the systems, and that’s really where the first-mile comes into play. It’s the first step where you’re collecting that data, you’re routing it. It’s potentially going to, what we call, middle-mile markers, maybe things like Kafka or other data pipelines. And then from there, going into the observability system that we’re so in tune with, with your Splunk’s, or Datadog’s, or Elastic, or New Relics, all of those systems for the practitioners and others are actually doing the interaction.

Swapnil Bhartiya: And now I want to talk a bit about you and your co-founders involvement with these two Open Source project, Fluentd and Fluent Bit. Talk about how you were involved there.

Anurag Gupta: Yeah. So my co-founder Eduardo Silva, he’s actually the creator of the Fluent Bit project. And him and I both met at Treasure Data, so we were both a part of Treasure Data back in 2017, which is the original company that created the Fluentd project. And from there, for us, we were both maintainers, lovers of the Open Source platform. I had, prior to Treasure Data, worked at Microsoft, which had used the Fluentd project as part of its Operations Management Suite agent. And from there, we really said, “Hey, this community, this project is continuing to grow. Here are the features.” And basically, earlier this year, late last year, we said, “Let’s go and really build this community and really strengthen the ability to go cater to these problems that we saw.”

Swapnil Bhartiya: Now, let’s talk about Calyptia Cloud. Where does it fit into all these, the challenges that you are talking about and the solution area that you’re talking about?

Anurag Gupta: Calyptia Cloud is really an overlay on top of what exists today. So we thought about ways to go and say, “Here’s our distribution of Fluent Bit and Fluentd, these Open Source technologies.” But instead of trying to displace what already exists in a thriving ecosystem, we wanted to build an overlay that just adds value to anyone using these projects. So we’ve added connectors and the ability to, within five seconds, attach your Fluent Bit to Calyptia Cloud and get a wealth of information about how that application is performing. Again, in first-mile observability, this is what data you’re collecting and what data are you routing. So we can tell you things like how much data you’re sending to particular backend, which sources, which teams might be overloading the system with what they’re collecting. And naturally, that gives you a better ability to troubleshoot and diagnose what might go wrong when your queries or your insights might be a little misguided on the [inaudible 00:07:23].

Swapnil Bhartiya: And where do you run Calyptia Cloud? Just talk about the cloud component of it.

Anurag Gupta: Yeah. For us, Calyptia Cloud is a fully hosted solution, so it’s something that we run within our own public cloud. And you, as a user, whether you’re running in public cloud or private cloud, you can connect to Calyptia Cloud just via API keys.

Swapnil Bhartiya: And also, it supports the multicloud story as well. It’s not just either on on-prem or public cloud.

Anurag Gupta: Exactly. So, say, you’re running against multiple clouds and you have Calyptia Cloud connected, one of the large problem that also increases is egress charges between the two clouds. So you can quickly monitor that to say, “Hey, am I sending a ton of data with a lot of egress charges across clouds to different locations?” And that’s part of the value that Calyptia Cloud brings.

Swapnil Bhartiya: What kind of symbiotic relationship it has with Fluentd and Fluent Bit projects?

Anurag Gupta: Yeah. For us, a lot of the work we’ve done for Calyptia Cloud in instrumenting, building additional monitoring, we pushed that all into upstream Open Source. And really, Calyptia Cloud is more of a layering on top to say all this data that we’re collecting and bringing, we can give you additional insights on top. So it’s really that transformation layer from the raw data, which is all Open Source, into the insights.

Swapnil Bhartiya: Who would be typical customer of Calyptia Cloud? And if you have already customers, you can or cannot, depending on your NDAs, name them or talk about some use cases?

Anurag Gupta: For Calyptia Cloud, we really focus on observability practitioners, which can encapsulate both developers, operations, this new melding of the two roles. What we’ve had with Calyptia Cloud is both this monitoring ability to visualize your observability pipelines. And we’ve had multiple… about 300 users start to sign up every month, so we have about 1600 users today. And many of them are doing things like understanding how their data’s routing between multiple backends, how their data’s routing in public cloud, in private cloud, within Kubernetes. We have troubleshooting tools that are, again, free to use, so Calyptia Cloud is free to sign up. And this can include things like deciphering regular expressions. Maybe you’re surfacing data from a firewall as your first-mile observability type of a goal, and you need to transcribe what a Barracuda firewall or a Cisco ASA firewall might look like. And we have those tools all readily available within Calyptia Cloud for users just to go out and start using.

Swapnil Bhartiya: I want to quickly talk about observability, which might go out of the scope of this incident, like when we do look at observability, especially with Calyptia Cloud, are we just looking at something is happening somewhere? What about understandability? What about actionability, so that I can also do something about it? So where do you stop, and where other projects takeover?

Anurag Gupta: This is a great question. For observability, we think about understanding the system, taking action, routing it to the proper folks. And there’s a trend, I think, in the general tech sphere where we start to shift the practitioner roles left. And so you might hear in the industry, “Oh, people are shifting security left.” Before it was a cybersecurity team, but slowly the developer gets a lot more power and ability to understand what’s going on. And I feel the same is happening with observability. And when we talk about shifting left, that means that a practitioner or someone who is collecting the data, routing the data, might have that same need to understand insights immediately, do actionability, do that understandably routing.

And these are things where, in the Open Source ecosystem, we’ve started to build the frameworks. So Fluent Bit can talk directly to Slack, and you can run stream processing on top of all the data you’re collecting. And as Calyptia, we want to keep strengthening that natural trend of folks moving from this place of, there’s a separate role, separate siloed organization doing it, to it slowly moves to the left where you might have developers, you might have practitioners that can actually start to do this themselves,

Swapnil Bhartiya: Anurag, thank you so much for taking time out today and talk about Calyptia Cloud. And I am excited to have you again on the show, and look forward to our next conversation. Thank you.

Anurag Gupta: Awesome. Really appreciate the time, Swapnil.