AI/MLDevelopersDevOpsFeaturedLet's TalkOpen Source

How Experiment Versioning Is Going To Solve Big Problems Of AI/ML World


Guest: Dmitry Petrov (LinkedIn, Twitter)
Company: (Twitter)
Show: 2022 Prediction Series is building a machine learning (ML) platform on top of a software development stack. Using several open-source tools, they are extending traditional software models such as data versioning control, continuous machine learning (CML), and more. recently announced a new idea called experiment versioning, which Dmitry Petrov, Co-Founder and CEO of, explains as recording all of the changes you make during the modeling process. To this, he says, “When you have a modeling production, for example, you need to know what set of hyperparameters you use for the model, what source code was used, and what version of the data set was used to produce this model.” Petrov adds, “You can always return and change something, retrain the model and have a whole lineage between your data code, hyperparameters and model. That’s the idea.”

But why did the company come up with experiment versioning now and what problem does this solve? To answer these questions, Petrov refers to problems on the modeling and the production side of machine learning. Petrov explains from the modeling side that people working on experiments “cannot understand why they didn’t have the same result and why it’s not reproducible. This breaks this connection between the pieces.” On the production side, they found a lot of their customers saying, “All right, we use some experiment tracking tools, we have models stored in these tools, but they live outside of our code and we have to use a separate set of tools and APIs to store models, to get models.” Those same clients needed a GitHub-based approach, so everything was on the same page.

This new methodology improves the lives of ML engineers and data scientists because it simplifies collaboration between departments. And when your model is built this way, it’s already versioned and on Git. According to Petrov, “You just tell your DevOps, ‘This is my model. This is my Git check sample or Git text.’ It simplifies how you collaborate in the team.” And because of the distributed nature of Git, ML teams will have better control of the experiments so they can run hundreds of experiments per day on a machine (or in the cloud).

The summary of the show is written by Jack Wallen

Read Transcript
Don't miss out great stories, subscribe to our newsletter.

Login/Sign up