Repository: evidentlyai/evidently
Issue: Integrating in Prediction pipeline
GitHub labels: enhancement

This package really helps the Data Science Team to find out the data drift . Please let me know is there any way we can integrate into prediction pipeline to generate the metrics.
Is there any way we can generate the json file with this metrics ?

Hi @rambabusure! 

Glad you find the package useful. 

Right now you can generate the reports launching the tool from the terminal. You can schedule a cron job to generate reports on a regular basis (e.g. every time new data arrives) so that the .html file with the data drift report is stored at a chosen location. 

We are working to integrate more tightly with Airflow or MLflow. 

We do not have yet the possibility to send the metrics elsewhere. We are thinking about it as the next direction. Could you share a bit more about how you'd like to use the metrics - do you want them to be displayed externally (e.g. in Grafana), or what would the optimal way to consume them?

If you are open, happy to have a quick chat. You can join our Discord channel https://discord.gg/xZjKRaNp8b or drop a line to hello@evidentlyai.com so that we find the time!

Cheers, 
Elena 

Thank you for reply @elenasamuylova . Right now , json is data exchange format . metrics json can be stored in Database and this data can be consumed by an application like  power BI , Grafana and custom web site through API end points

I have just recently been made aware of this platform so I am still investigating. I wanted to second @rambabusure suggestion. If there was a way to output the metrics so that we could have them saved in a db or send them to our monitoring platform that would be very helpful. Json would be fine for us as well.

Hi @rambabusure, @mcr-ai, thanks for your comments!

We are working on this feature to enable report export as .json.

One open question is whether
1. To make a shorter .json to include only preselected parts of the report. For example, it would include metrics (e.g. p-values in drift, MAE, MAPE in performance reports, etc.) that are easily structured to put in the database. 
2. To make all contents of the report available as .json. In particular, that would mean we include .json descriptions of all plots (e.g. feature distributions). In this case, you might have to select the parts of a larger .json to choose which plots you want to use on your side. 

For now, we look to enable easy export of everything so that you can choose on your side what to ingest, but are curious to learn how you see it best.

Hi @rambabusure, @mcr-ai, 

We implemented a quick solution for option two: a way to output the complete contents of the report as .json. It is already released. You will need to update the package to try it. 

We now have methods ```_json()``` and ```_save_to_json(path)``` for the Dashboard class. 

An example how to generate JSON:
```
drift_report = Dashboard(reference_frame, current_frame, column_mapping = None, tabs=[DriftTab])
drift_report._json()
```
 
An example how to save a JSON file:
```
drift_report = Dashboard(reference_frame, current_frame, column_mapping = None, tabs=[DriftTab])
drift_report._save_to_json('filename.json')
```
 
Also, we added —output_type parameter for the terminal mode, so now you can choose ‘json’ option there as well:
```
python -m evidently analyze --config config.json --reference reference.csv  --current current.csv --output_path save_report_here --output_type json
```
Let us know if you have the chance to test it! We suspect the output might be a bit more than you actually need - feedback is very welcome!

@emeli-dral - This is awesome! I will take a look at this and see how I can use this with our monitoring platform. Ideally I want to be able to use these metrics to report into our automated monitoring platform. My hope is that if we use the same metrics in our monitoring then in the case where there are issues that need to be investigated this with give us a starting point. I appreciate the quick turn around!

@emeli-dral yeah the output file is a lot. I mean I could work with it but it is waaaayyy more than I need and maybe anyone needs. I was really looking for an output file that provide the metrics so I could send them to our monitoring platform. I realize this is the initial version and this is going in the right direction. 

Let me know if I can provide you anything more to help.

> @emeli-dral yeah the output file is a lot. I mean I could work with it but it is waaaayyy more than I need and maybe anyone needs. I was really looking for an output file that provide the metrics so I could send them to our monitoring platform. I realize this is the initial version and this is going in the right direction.
> 
> Let me know if I can provide you anything more to help.

Hundred percent agree :)  We are in the process of implementing nice JSON metrics summary, so that you can use them together with other platforms.

Hi @rambabusure, @mcr-ai! In the latest release, we added the JSON profiles. It generates a summary of metrics and statistical test results. You can easily integrate it into your prediction pipelines or with other visualization tools. Check the release blog here: https://evidentlyai.com/blog/evidently-017-json-profiles and let us know what you think! Would be awesome if you could test it.