AI/MLCloud Native ComputingOpen Source

Evolution of Hadoop and Hortonworks: An Interview with Alan Gates

Swapnil BhartiyaBy 3468 views
0

Alan Gates is one of the co-founders and architect of Hortonworks. I sat down with Gates at the DataWorks Summit (2018) to talk about a wide range of topics.

Gates touched upon the origins and evolution of Hortonworks, going all the way back to Apache Pig and Apache Hive days.

He reflected on the evolution of Hadoop and said that in the beginning, Hortonworks was trying to help out users consuming technologies like Hadoop, Pig, Hive and Spark with greater ease. Hortonworks built a platform that bundled and tested all of these technologies. But today it’s not just about managing data in a Hadoop cluster; it’s about bringing data in, streaming it, and also processing it on edge.

Now Hortonworks users are using the cloud, and they need the same capabilities. “Even people who aren’t necessarily moving their data off-prem onto the cloud, are seeing that containers are the way to effectively manage their resources, whether they are on-prem or in the cloud,” said Gates.

Hortonworks is dynamically evolving to solve these new problems for its customers.

“What we would once call Hadoop is now really a complete ecosystem. That’s why I think it’s better we call it Big Data. A few years ago we changed the name of this conference from the Hadoop Summit to the DataWorks Summit, and that really distressed that it’s not all about Hadoop anymore. That’s true about Hortonworks as well,” said Gates.

We talked about emerging use-case like IoT and Edge computing. We also talked about how Kubernetes is disrupting Big Data and how Hortonworks is embracing it.

We also discussed how Open Source has democratized software development. “Open Source is the only place where I have seen people solving their own problems instead of running to the vendor every time,” he said.

We concluded the interview by talking about Gate’s hobbies. He does have some interesting hobbies.

Watch the whole interview on YouTube & SoundCloud. Here is the timestamp.

You may also like