ML Experimentation  - Beyond ML flow
Published: 2021-05-04 05:19:38

In this blog, I explain to extend mlflow enabling enhanced tracking Machine Learning Experiments along with its applications in practical ML Ops. Some of these design ideas are sourced from inspirations of how product lifecycle tracking systems are built in the manufacturing sector.

##  What is ML Ops ? 
   ML Ops is the practice of building an ecosystem of services , API, pipelines and dashboards that enable productionizing an ML model. Examples of ML Ops functionality include model monitoring, model explaining, model testing/evaluation to name a few.

If you create a consumer facing ML product (e.g: pricing models/recommendation systems etc)  used by tens of thousands of consumers a day, it stokes several dynamics among stakeholders and consumers alike. There is a need to continually iterate and improve the model and close the feedback loop. This means we keep track of experiments not just on a dashboard visible to a data scientist, but track everything associated with a deployed model's experiment ( hyperparameters, selected features, seasonal adjustment factors, insights etc) and build associations/continual correlations to end user impact metrics across all phases of the model's lifecycle 

###What did we learn from a model that caused a drop in leads that  was deployed 6 months ago ? 

To quantitatively answer this question, we need to systematically collect, store and query  experiment lifecycle data.

### Precedence of experiment/product tracking in other industries

In the manufacturing industry for decades product lifecycle management has been about collecting ,storing and querying  product meta data from conceptualization->Design->Manufacturing->Service -> retirement. This helps an aircraft manufacturer track associations between a design phase assumption and its corresponding tachometer data generated during its 20 year service span. Engineers rely on the tracked associations to close the feedback loop in a future variant /derivative. The analogy applies almost exactly to a machine learning model. An ML engineer should absolutely care about the data generated during consumer interaction of a recommendation system in order to close the feedback loop in a future model training /deployment iteration. (of course, beyond what can be done in real time)

 


## Why Experiment Tracking ? 
ML Flow is an open source product that enables experiment tracking yet has remarkable gaps that need to be addressed. While there is nothing novel about collecting telemetry data in an ML experiment,
systematic ML experiment data collection & storage has profound implications on successful design of ML Ops services.  

## How to collect Experiment Data ?  

![image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/e34js8xysw76576drv7n.png)
 

The above image shows a training pipeline with several steps leading to deployment of an ML model. Each pipeline step performs an "objective task" +   "Incidental /Side Effect" .This causes each pipeline step to "emit" tagged experiment parameters /serialized visualizations to a distributed database.

 
## How does the data we collected look like ? 
![image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3klhp58k6jelu97dta36.png)
 
 The data collected in the above step is sharded into partitions , each partition representing a single Experiment's data graph. 
An Experiment data graph aggregates entities such as hyperparameters, insights and metrics for every model created in an experiment

## We collected experiment Data .Now what ?

## Application I- Inclusive Model Selection
![image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dbimilydf5zpc5qt70yi.png)
 
 
The model selection process must use a combination of metrics along with accuracy - including one that product stake holders care about.

The architecture above can be used to partially automate model selection that uses such a combination of metrics (accuracy, transitions, user interactions). We may have all the compute power to train a few dozen/hundred models for a release. However, systematic data collection enables manageably visualizing & comparing the performance distribution of various models thereby scaling model selection for the complexity. Also, one can notice that serializing visualizations came in handy when model selection dashboards could query them as artifacts (think plotly <-> json ) to display in a browser

Also one can notice that we maintain associations between a deployed model's user behavior metrics and a data scientist's experiment parameters generated while conceptualizing the model. 
Ultimately we hope to understand how the data scientist's actions results in what the user ends up doing. This can also be used in A/B test setting 

## Application II Parameterized ML Ops Pipelines
![image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/glvza3y9ajw71m4wcnmy.png)

The above diagram brings about the pattern of decoupling model generated data from pipelines which represent operations that are applied on the model data

An ML Ops pipeline such as explanation pipeline or monitoring pipeline is parameterized by the model. This means we reuse the same pipeline in preparing data for a dashboard that joins every model's prediction data with cohort/user interaction data. The experiment data model facilitates  lazy evaluation of such joins  

Quite often I have seen data scientists adopt out of the box implementations for experimentation platforms. While there is a need to reuse as much of what's available, we must extend those frameworks ( such as ML Flow) to get the most value needed for our specific data science lifecycle. In my future blogs I will get into detail about ML Ops lifecycles ( Model explanation , Model monitoring and Model deployment)