zhiminy/EvalEng huggingface dataset:
Repository: evidentlyai/evidently
Issue: TargetDriftPreset report fails when using prediction instead of target
Root cause: External Dependency Breakage

Hi.
The documentation for [TargetDriftPresent](https://docs.evidentlyai.com/presets/target-drift) states that you can have `target` and/or `prediction` defined in your dataset.
When you only have `prediction` (no short feedback, for example), it becomes merely a semantic difference - since the report will provide all the amazing report/comparison just for that individual field.

### The problem:
* `TargetDriftPreset(columns=self.most_important_cols)` only works if I specify the `target=` parameter for `ColumnMapping`
* It fails if using 'prediction=` with an error that could not retrieve the key from the dictionary

I __believe__ this is because of the default values set for those parameters. So when I specified `prediction="predictions"`, it assumed the default for `target="target` and tried to generate the full report for both fields - hence failing to retrieve the data since my dataframe does not contain the column **target**.

It took me some trial and error to figure this out.

### Proposed solution
Perhaps the **easiest** option is to update the documentation to make it clear that if you only have:
* a single column: use `target`
* for two columns: use `target` and `prediction`

Hi @marcellovictorino ,
Thank you for letting us know!

Could you please specify as well what version of evidently you have?

The thing is that TargetDriftPreset is supposed to work fine with predictions only; we addressed this problem in the latest release 0.2.6.

Thanks @emeli-dral , my bad!

I am using 0.2.5 - the latest available in conda-forge (Metaflow user here). 

https://anaconda.org/conda-forge/evidently 

Thanks @marcellovictorino ! 

It seems that there is an issue with a conda update frequency (they parse new packages automatically), we'll investigate it now! Thanks for surfacing it.
Repository: evidentlyai/evidently
Issue: General Report chart improvements
Root cause: Unimplemented Feature Gap

I understand this might not be a priority right now, but wanted to provide feedback on the charts generated in the Reports, which I believe would improve their readability.

### 1) For percentages and values varying [0, 1] -> force the y-axis to show [0, 1]
This would make it easier to compare the different charts

Example: TargetDriftPreset, correlation between target and features: 
![image](https://user-images.githubusercontent.com/18636680/223348744-9f2f8302-8a88-4acb-967d-696386f32fd0.png)
![image](https://user-images.githubusercontent.com/18636680/223348790-bd1cd259-1bd5-4fe3-a31e-b57969b8b755.png)

### 2) Some charts contain a green area -> capture the green area meaning in the chart legend
I am still unsure what the green area represents. Capturing it in the documentation or the chart legend itself would be great
![image](https://user-images.githubusercontent.com/18636680/223349439-e61da4f8-f94d-405c-a7b8-50e9f6a31de7.png)

### 3) Some charts have a huge imbalance of data distribution -> would be great to see the percentage instead of the actual count
Similar to some numerical distributions where it is possible to easily apply the `log` distribution, it would be great to easily switch between count and percentage for improved readability
![image](https://user-images.githubusercontent.com/18636680/223350047-6a93bc7d-6c20-4fc8-a601-cf68294d3724.png)

### 4) Capturing multiple reports becomes very long (vertical scrolling) -> automatically place reports into separate tabs, based on their order
Example of multiple tabs in a single container, where we could have `DataDrift | DataQuality | TargetDrift`:
![image](https://user-images.githubusercontent.com/18636680/223352158-85bccf67-658a-4a1f-9aa3-414e19f297f9.png)
Repository: evidentlyai/evidently
Issue: consider compability with pyYAML >=6.0
Root cause: External Dependency Breakage

Dear Evidently Crew,

could you consider compability with pyYAML >=6.0 ?
Because currently we had to upgrade our googleads version to 37.0.0 wich led to a pip dependency conflict making us unable to use evidently any longer :(

#9 344.6 The conflict is caused by:
#9 344.6     googleads 37.0.0 depends on PyYAML<7.0 and >=6.0
#9 344.6     evidently 0.2.6 depends on PyYAML~=5.1

Best Regards
David

Hi @davidbange!

Thank you for bringing this up. Initially, we fixed  PyYAML~=5.1 for compatibility with google colab.
We believe that the issue is already solved. We'll double-check it and update the dependency in the next release.

Hi team - 

I was wondering when the 0.2.7 release, which I believe would include this patch, was scheduled for? My team is getting burned by this PyYAML dependency too (which stems from the `databricks-feature-store` package requiring PyYAML<7.0 and >=6.0). Thanks in advance and appreciate all the work the group has put into this great package.

Cheers,

RS

Hi @rickyschools, @davidbange, 

This fix has been released today: https://github.com/evidentlyai/evidently/releases/tag/v0.2.7
Repository: evidentlyai/evidently
Issue: 0.2.7 - save_json does not work?
Root cause: Validation Gap

I have got an error "TypeError: Object of type Series is not JSON serializable" when trying to save json report for evidently.test_suite.test_suite.TestSuite object

Hi @Mansuete,
thank for reporting this. We had significant Metrics refactoring in the last release, it looks like we left  pd.Series somewhere in MetricResult. 

Could you please share what tests did you use? It will help to locate a problem faster.

Hi @emeli-dral. Thanks for quick reply! Here is an example. Fail in report.save_json(f'report.json') row
```
from evidently.report import Report
from evidently.test_suite import TestSuite
from evidently.test_preset import BinaryClassificationTestPreset, DataStabilityTestPreset, DataQualityTestPreset
from evidently.metric_preset import ClassificationPreset, DataDriftPreset
from sklearn.datasets import load_breast_cancer
import numpy as np
import pandas as pd

dataset = load_breast_cancer(as_frame=True)


ref = pd.DataFrame(dataset['target'])
curr = pd.DataFrame(dataset['target'])


ref['prediction'] = [np.random.random() for i in range(ref.shape[0])]
curr['prediction'] = [np.random.random() for i in range(curr.shape[0])]

report = Report(metrics=[ClassificationPreset(probas_threshold=0.7)])

report.run(reference_data=ref, current_data=curr)

report.save_html(f'report.html')
report.save_json(f'report.json')

```

Thanks a lot! It has helped!

We are testing quick fix in the #537 

@Mansuete fix is merged.

Now you can build evidently from the source code.
Here is a code snapshot to do so:

```
git clone https://github.com/evidentlyai/evidently.git evidently_repo

cd /path/to/evidently_repo

python3 -m venv venv
. venv/bin/activate

pip install -e .
```

Fixed version will be released soon as well.

Released in 0.2.8
Repository: evidentlyai/evidently
Issue: There is a problem with the report style being displayed locally
Root cause: Configuration Error

I follow this guide to generate NLP report. https://colab.research.google.com/drive/15ON-Ub_1QUYkDbdLpyt-XyEx34MD28E1#scrollTo=RJSXCrsvRFCE  

But, if i download the report html to my local, open in browser will have issue:
![image](https://user-images.githubusercontent.com/24414429/227471788-5c4dca61-96a3-4f58-811f-5f689b3e902f.png)


@elenasamuylova you told you have had the same issue.
Could you please also share some details ? Browser version, colab/separate html?
Repository: evidentlyai/evidently
Issue: Assertion Error when trying to import evidently
Root cause: External Dependency Breakage

I have a python code with Evidently library. But when i try to import like this. It causes Assertion error. 

`
import evidently

from evidently import ColumnMapping

from evidently.report import Report

from evidently.metric_preset import DataDriftPreset
`



`Traceback (most recent call last):
  File "/home/app.py", line 4, in <module>
    import evidently_client as evi
  File "/home/evidently_client.py", line 4, in <module>
    from evidently.report import Report
  File "/home/khangnt/anaconda3/envs/scoring/lib/python3.10/site-packages/evidently/report/__init__.py", line 1, in <module>
    from .report import Report
  File "/home/khangnt/anaconda3/envs/scoring/lib/python3.10/site-packages/evidently/report/report.py", line 14, in <module>
    from evidently.metric_preset.metric_preset import MetricPreset
  File "/home/khangnt/anaconda3/envs/scoring/lib/python3.10/site-packages/evidently/metric_preset/__init__.py", line 1, in <module>
    from .classification_performance import ClassificationPreset
  File "/home/khangnt/anaconda3/envs/ss/lib/python3.10/site-packages/evidently/metric_preset/classification_performance.py", line 5, in <module>
    from evidently.calculations.classification_performance import get_prediction_data
  File "/home/khangnt/anaconda3/envs/ss/lib/python3.10/site-packages/evidently/calculations/classification_performance.py", line 18, in <module>
    from evidently.metric_results import Boxes
  File "/home/khangnt/anaconda3/envs/ss/lib/python3.10/site-packages/evidently/metric_results.py", line 28, in <module>
    class ConfusionMatrix(MetricResultField):
  File "/home/khangnt/anaconda3/envs/ss/lib/python3.10/site-packages/pydantic/main.py", line 95, in __new__
    fields[ann_name] = Field.infer(
  File "/home/khangnt/anaconda3/envs/ss/lib/python3.10/site-packages/pydantic/fields.py", line 92, in infer
    return cls(
  File "/home/khangnt/anaconda3/envs/ss/lib/python3.10/site-packages/pydantic/fields.py", line 77, in __init__
    self.prepare()
  File "/home/khangnt/anaconda3/envs/ss/lib/python3.10/site-packages/pydantic/fields.py", line 133, in prepare
    self._populate_sub_fields()
  File "/home/khangnt/anaconda3/envs/ss/lib/python3.10/site-packages/pydantic/fields.py", line 171, in _populate_sub_fields
    assert issubclass(origin, Mapping)
AssertionError
`

Hey @khangt1k25 

the quick solution is to change import from ```from evidently.pipeline.column_mapping import ColumnMapping``` to ```from evidently import ColumnMapping```

Although we'll check what's going on with the initial import 

Hi @emeli-dral , I changed that  but it's still error. I figure out that the error come from two lines of code.
 
from evidently.report import Report
from evidently.metric_preset import DataDriftPreset

Hi @khangt1k25 ! Can you check your pydantic version? Looks like this assertion fails because of old pydantic, try upgrade it. We also should add constraints on it in our requirements

related #568
Repository: evidentlyai/evidently
Issue: Update docs / Add a How-to example for Color Options
GitHub labels: help wanted

**Issue Description:** 

Evidently has an option of changing plot colors. There are two ways to do that: 1. Manually set the colors for specific elements. 2. Use a "color schema" to change all colors simultaneously.     

This functionality is not well-documented. 
- Existing documentation page has not been updated after the API update (replacing Dashboards with Reports). https://docs.evidentlyai.com/user-guide/customization/options-for-color-schema
- There is no "How to use color options" notebook example. 

**We need to add:**

1. Create a "How to use color options" notebook similar to others (https://github.com/evidentlyai/evidently/tree/main/examples/how_to_questions)

It should include the example of changing individual colors and using color schema for both Reports and Test Suites. 

The first part can be directly copied from the sample notebooks (https://github.com/evidentlyai/evidently/blob/main/examples/sample_notebooks/evidently_tests.ipynb and https://github.com/evidentlyai/evidently/blob/main/examples/sample_notebooks/evidently_metrics.ipynb). 

The second should be tested first and written from scratch. 

2. Update the code examples on the documentation page 

https://github.com/evidentlyai/evidently/blob/main/docs/book/customization/options-for-color-schema.md

Update the existing docs page with the example using Reports instead of Dashboards.
Repository: evidentlyai/evidently
Issue: Add a How-to example of using Column Mapping
Root cause: Documentation Deficiency

**Issue Description:** 

Evidently has a column mapping functionality that helps correctly process the input data.

It is documented here https://docs.evidentlyai.com/user-guide/input-data/column-mapping. But there is no "How to use column mapping" notebook example.

**We need to add:**

Create a "How to use column mapping" notebook similar to others (https://github.com/evidentlyai/evidently/tree/main/examples/how_to_questions)

It should include the example of using all available column mapping options for different input data. 

1. Show how to use primary mapping options (target, prediction, ID, datetime*, numerical and categorical features, text features**). 

2. Show how to use additional mapping options (datetime features*, task parameter for target function)

3. Show the different ways to pass labels and predictions for probabilistic multi-class classification. 

To show all 3 we need at least two datasets, for regression and multi-class probabilistic classification tasks. Here is an example notebook that first includes steps to download multiple datasets: https://github.com/evidentlyai/evidently/blob/main/examples/sample_notebooks/evidently_metric_presets.ipynb

*To show how to pass DateTime columns as index and features, we need to include an open example dataset (e.g., openml, creative commons public domain) that contains a DateTime column, or a step to add it artificially. 

**Text features are optional, as there is a separate How-to example for text data. 
https://github.com/evidentlyai/evidently/blob/main/examples/how_to_questions/how_to_run_calculations_over_text_data.ipynb

Closed as completed. Thanks to @AntonisCSt for contributing!
Repository: evidentlyai/evidently
Issue: Request for feature - Performance Report for Clustering Models
Root cause: Unimplemented Feature Gap

Hi!
Evidently provides great reports for the Regression and Classification model performance. What's about Clustering model performance? scikit-learn provides a set of metrics for the clustering models - https://scikit-learn.org/stable/modules/model_evaluation.html#common-cases-predefined-values. What do you think about implementing this type of reports?
Repository: evidentlyai/evidently
Issue: Production data without label column
Root cause: Validation Gap

In case of target feature is missing in production data, gives "'TargetNameWidget' object has no attribute 'wi'"
Which metrics are supported incase the labels are missing for production data

Hi @prarshah ,
In case label data are unavailable there are two reports you still can build:
1. [Data Drift](https://docs.evidentlyai.com/reports/data-drift)
2. If you have model predictions, then you also can build [Numerical Target Drift](https://docs.evidentlyai.com/reports/num-target-drift) or [Categorical Target Drift](https://docs.evidentlyai.com/reports/categorical-target-drift) report (depending on the problem statement you working on). For these reports please set the ```predictions``` column  in the ```column_mapping```.

All other reports are related to the model performance and expect to get labels for quality estimation. But for early assessment both Data Drift  and Prediction Drift can be handy
Repository: evidentlyai/evidently
Issue: Error while using Drift on embeddings
Root cause: Validation Gap

Hi, I'm redoing the notebook on encoders with Drift which was previously univariate detection.
I'm facing the below error when running the embedding drift methods.


![image](https://user-images.githubusercontent.com/35230623/233413981-1a3c307b-0a0f-42bd-8a11-b71d40e2bc50.png)

cc @elenasamuylova 


Hey @SangamSwadiK ,
Thank you for letting us know!

Could you please share with us a link to your colab so that we could see the whole error trace together with the code?

@emeli-dral  

Hi, Sharing the notebooks.
Please let me know in case of any mistakes I've made. Will correct them. Thanks! 

Glove-vectors: https://colab.research.google.com/drive/18pE9pmoQ5TtQQlO3uxmZ_--BPbMwjNl2?usp=sharing

Sentence-transformers:  https://colab.research.google.com/drive/1f0s0MfCMfy9nypYDa2HXHFizVt-n127Q?usp=sharing

Hi @SangamSwadiK !
Thanks for letting us know about this issue! We have fixed the error and the changes will be available in the next release.

By the way, we noticed that you're using the "p_value" parameter in the drift detection methods in your notebook. Just a heads up, we've renamed it to "quantile_probability" in the new release, so you'll need to make that change in your notebook. 
p_value=0.05 -> quantile_probability=0.95

If you have any inquiries, please do not hesitate to ask us!

fixed in https://github.com/evidentlyai/evidently/pull/586
Repository: evidentlyai/evidently
Issue: Grafana example is broken
Root cause: Documentation Deficiency

none of these methods exist: 

> from evidently.model_monitoring import CatTargetDriftMonitor
> from evidently.model_monitoring import ClassificationPerformanceMonitor
> from evidently.model_monitoring import DataDriftMonitor
> from evidently.model_monitoring import DataQualityMonitor
> from evidently.model_monitoring import ModelMonitoring
> from evidently.model_monitoring import NumTargetDriftMonitor
> from evidently.model_monitoring import ProbClassificationPerformanceMonitor
> from evidently.model_monitoring import RegressionPerformanceMonitor

the readme points to several code areas which don't exist. eg. https://github.com/evidentlyai/evidently/blob/main/src/evidently/model_monitoring/monitors/

a search of the api docs and the repo in general didn't surface replacements for these functions. 

seems to work with `evidently==0.2.8` and can be fixed by pinning the version in the [dockerfile](https://github.com/evidentlyai/evidently/blob/main/examples/integrations/grafana_monitoring_service/Dockerfile#L11) 

None of these methods still exist in the most current version (==0.6.5).  Any reason for this?

```
from evidently.model_monitoring import CatTargetDriftMonitor
from evidently.model_monitoring import ClassificationPerformanceMonitor
from evidently.model_monitoring import DataDriftMonitor
from evidently.model_monitoring import DataQualityMonitor
from evidently.model_monitoring import ModelMonitoring
from evidently.model_monitoring import NumTargetDriftMonitor
from evidently.model_monitoring import ProbClassificationPerformanceMonitor
from evidently.model_monitoring import RegressionPerformanceMonitor
```
Repository: evidentlyai/evidently
Issue: Profile
Root cause: Documentation Deficiency

I am using python 3.10 and for some reason even after installing evidently. I still get this error:
ModuleNotFoundError: No module named 'evidently.model_profile'

Hi @midofemi, 

```model_profile``` refers to an old Evidently API. 

Starting from version 0.1.59, there is a new Reports functionality that unified Dashboards and JSON profiles. The old Dashboards API was removed from the code base in 0.3.0. 

We'd suggest migrating to the new API since all the new improvements are available here. 
https://docs.evidentlyai.com/support/migration

As a quick fix to continue using JSON profiles, you can fix the Evidently version to 0.2.8 or earlier.
Repository: evidentlyai/evidently
Issue: Support 'target_names' in Classification Metrics
Root cause: Unimplemented Feature Gap

If ```target_names``` is specified in the ```column_mapping``` it needs to be used in the following metrics:
- ```ClassificationQualityByClass()```
- ```ClassificationClassSeparationPlot()```
- ```ClassificationProbDistribution()```
- ```ClassificationQualityByFeatureTable()```

Screenshots below illustrate where ```target_names``` are supposed to appear on plots:

1. ClassificationQualityByClass()
<img width="1353" alt="ClassificationQualityByClass" src="https://user-images.githubusercontent.com/22995068/233973671-6db24605-122a-460e-8c21-e7d55e410e5b.png">

2. ClassificationClassSeparationPlot()

<img width="1345" alt="ClassificationClassSeparationPlot" src="https://user-images.githubusercontent.com/22995068/233974307-9c8a8ca9-2cd1-4378-899e-69f6b0e6c299.png">

3. ClassificationProbDistribution()

<img width="1354" alt="ClassificationProbDistribution" src="https://user-images.githubusercontent.com/22995068/233973441-9eeffb5d-92b0-4fee-97c9-0824c65091e7.png">

4. ClassificationQualityByFeatureTable()
<img width="1354" alt="ClassificationQualityByFeatureTable" src="https://user-images.githubusercontent.com/22995068/233973714-13f57bfd-177d-442b-a9a8-b10c737c83fc.png">


Hello @emeli-dral , @elenasamuylova , I'm interested in working on this one. 

Is there a minimal example to reproduce the screenshots above, available as a starting point? Also, I have a second question, was this bug always present?

Hi @aanghelidi, thanks for your interest in contributing!

You can reproduce the bug by taking this example notebook: https://github.com/evidentlyai/evidently/blob/main/examples/how_to_questions/how_to_use_column_mapping.ipynb

It shows how to pass `target_names `in column mapping for classification tasks (see 5.4 and below) and generate a Report. It does not include all metrics where the bug is present, but you can easily modify it by passing all the Metrics listed in this issue to the Report.

This bug is present since the initial implementation.

Hi @elenasamuylova, thank you for your detailed answer. As I progress through the PR, I may have other questions, but I'm now ready to work on it.
Repository: evidentlyai/evidently
Issue: How to change the Standard Deviation which is by default 1
Root cause: Unimplemented Feature Gap

How to change the Standard Deviation to like 2 or 3, which is by default 1 in the DataDrift Dashboard?

Hi @nageshsinghc4 

Unfortunately, we do not have this option in the configurations yet. 
At the moment in can be done in the source code by changing the implementation of the https://github.com/evidentlyai/evidently/blob/main/evidently/widgets/data_drift_table_widget.py

To be more precise, this:
```reference_std = np.std(reference_data[feature_name][np.isfinite(reference_data[feature_name])], ddof=1)```
should be updated somewhat like that:
```
n_std = 3
reference_std = n_std*np.std(reference_data[feature_name][np.isfinite(reference_data[feature_name])], ddof=1)
```
And after this update the project should be rebuilt internally.

Do you think this parameter should be presented in the config as well?

Hi @emeli-dral,

Thanks that is quite helpful. 
Indeed I think this should be included because in some use cases we need to consider normal distribution and filter out data points that lie within 3 Standard Deviation along with the mean.
Repository: evidentlyai/evidently
Issue: Handle time series data
Root cause: Unimplemented Feature Gap

Hi, is it possible to use Report and Test with time series data? (eg. using past X values as input for a time series forecasting model that predicts the Y future values) 

Thanks in advance for your reply!


Hi @eldritchhh !
We do not currently have any special time series-focused metrics, but you can use many of the existing metrics and tests with time series data. 

You can pass time series data to Evidently and use all existing presets, tests, and metrics.

- If you already have actuals and want to evaluate true model quality, you can use Regression Performance preset and related tests/metrics to evaluate model errors, etc. 
- If you do not have actual values, you can monitor prediction drift and data drift and use the data quality checks to verify that values stay within ranges, do not contain nulls/constant values, etc.

However, we did not implement any specific methods related to time series (e.g., autocorrelation lag) yet, so you might find some capabilities lacking. You can also implement custom metrics.