Repository: evidentlyai/evidently
Issue: Classification Metric Plots (Distribution) are not included in the report 
Root cause: Unimplemented Feature Gap

Hello, 

According to the documentation, if we want to include information about the visualizations in the snapshot reports, we should use the report.json(include_render=True) command. However, when this is used on the classification metrics, the details about the distribution do not appear in the report. Is there another way to include these details so that these plots can be added to the dashboard in Evidently or another service? I pasted an example of the code, and report below:

import pandas as pd
import numpy as np

from sklearn import datasets

from evidently import ColumnMapping
from evidently.report import Report
from evidently.test_suite import TestSuite
from evidently.metric_preset import DataDriftPreset
from evidently.test_preset import DataDriftTestPreset
from evidently.core import AllDict
from evidently.metrics import ClassificationProbDistribution

data = datasets.fetch_california_housing(as_frame=True)
housing_data = data.frame

housing_data['target'] = np.random.choice([0, 1], size=len(housing_data))
housing_data['prediction'] =  np.random.rand((housing_data.shape[0]))

reference = housing_data.sample(n=5000, replace=False)
current = housing_data.sample(n=5000, replace=False)
report = Report(metrics=[
    ClassificationProbDistribution(), 
])
report.run(reference_data=reference, current_data=current)
report.json()
"""Example of report output:
'{"version": "0.4.31", "metrics": [{"metric": "ClassificationProbDistribution", "result": {}}], "timestamp": "2024-07-24 10:38:27.504064"}'"""
report.json(include_render=False)
"""Example of report output:
'{"version": "0.4.31", "metrics": [{"metric": "ClassificationProbDistribution", "result": {}}], "timestamp": "2024-07-24 10:38:27.504064"}'"""
report.json(include_render=True)
"""Example of report output:
'{"version": "0.4.31", "metrics": [{"metric": "ClassificationProbDistribution", "result": {}}], "timestamp": "2024-07-24 10:38:27.504064"}'"""


I tested the ClassificationClassSeparationPlot() metric and encountered a similar problem. Any assistance in resolving this bug so that I may add these to the dashboard in evidently or in another integrated service would be greatly appreciated 

Full disclosure: I forgot to change the problem from regression to classification. I made this change and there is still an error

Hi @Soundwaveisbestbot, 

Could you share a bit more on what you look to achieve - do you want to:

1. view the resulting Report in Evidently UI (Cloud or self-hosted), 
2. plot a specific value on the Monitoring Panel in the Evidently UI pulling it from multiple reports, or
3. export the render data to some other service.

For option one, when you save a Report as a JSON `snapshot` (which happens automatically if you use the `ws.add_report` https://docs.evidentlyai.com/user-guide/monitoring/snapshots#send-snapshots), the Report is automatically saved with all the render data and will be visible in the the Monitoring UI. 

So in this case you do not need to use `json()` at all - this is a different option in case you want to parse the data and use it elsewhere outside Evidently.

Hi @elenasamuylova,

Thank you for getting back to me so quickly. My goals fall within options 2 and 3.

Option 3: I hope to render the data to either streamlit or PowerBI. After I explore and develop a proof of concept.

Option 2: I had hoped to be able to access the distribution data available in this metric (using the same bins) so that I may plot the changing prediction over time in a monitoring panel using a similar method to:
 p.dashboard.add_panel(
        DashboardPanelDistribution(
            title="Column Distribution: current",
            filter=ReportFilter(metadata_values={}, tag_values=[]),
            value=PanelValue(
                field_path=ColumnDistributionMetric.fields.current,
                metric_id="ColumnDistributionMetric",
                metric_args={"column_name.name": "education"},
            ),
            barmode = HistBarMode.OVERLAY
        )
    )
As well as visualize the different classification metrics in a similar manner

For 2, I believe at the moment you won't be able to parse the results of this specific metric to plot it on a dashboard, but this is something we want to address. 

A possible workaround is to include a `ColumnSummaryMetric(column_name="prediction")` for the column with predicted scores in your Report. It will then be plottable on Evidently Monitoring Dashboard using `DashboardPanelDistribution` https://docs.evidentlyai.com/user-guide/monitoring/design_dashboard#distribution-panel. You can also plot mean, max, min, etc., using `DashboardPanelPlot` https://docs.evidentlyai.com/user-guide/monitoring/design_dashboard#plot, which can be more interpretable. 

For exporting to external tools, you can consider doing the same and checking if the JSON output of this `ColumnSummaryMetric()` or `ColumnDistributionMetric()` works better for you.

That would work great! Thank you!