zhiminy/EvalEng huggingface dataset:
Repository: evidentlyai/evidently
Issue: data drifts results depends on data size
Root cause: Unimplemented Feature Gap

If I pass let's say 5% of the its show that there is no data drift. But with all data almost all features became drifted. I believe its because of statistic test. I suggest you to give options to choose PSI test as an option 

I encounter the same pattern. The default statistical test is sensitive to sample size.

Drift analysis is based on a statistical test so having different results due to the dataset size is entirely possible. Sample size is one of the underlying parameters that impact the test result.

The result would also vary slightly (e.g. p-values would be slightly different) when e.g. sampling is done several times.

We understand that from a practical standpoint the default tests can be “too sensitive” in case of large datasets.

We plan to address it by adding a different statistical tests (and some preprocessing logic) that will be applied if you flag the dataset as “large”. We will also add the ability to add custom metrics e.g. PSI index.

For now, I’d suggest to use sampling when analysing data drift for large datasets.

Hope that helps, and keep an eye on new releases!
Repository: evidentlyai/evidently
Issue: How do I download data to postgresql?

Hello, everyone!

How can I load the resulting metrics into the postgresql database, in order to further visualize the dashboard in real time with grafana?
I took this repository as a reference, but I am using my own data which is uploaded in real time https://github.com/mzadafiya/grafana_evidently_drift_detection , but nothing works.

The error that pops up is:
```
psycopg2.errors.InvalidTextRepresentation: invalid input syntax for type numeric: ""
LINE 1: ...1'::date,'tech_no','cat','Reference Distribution','',3),('20...
```

I think I figured out what the problem was and how to fix it. I need to use ColumnMapping, but another problem arose. Can you tell me what I did wrong?
```
                    column_mapping = ColumnMapping()
                    column_mapping.categorical_features = df_prev_prod.loc[:, df_prev_prod.dtypes == object].columns
                    column_mapping.numerical_features = list(df_prev_prod.select_dtypes(exclude='object').columns)
                    # data_drift = Dashboard(tabs=[DataDriftTab(verbose_level=False), CatTargetDriftTab(verbose_level=False)])
                    model_profile = Profile(sections=[DataDriftProfileSection()])
                    model_profile.calculate(data_ref[table], df_prev_prod, column_mapping=column_mapping)
                    json_eviden_metrics = json.loads(model_profile.json())
                    evidently_metrics = json_eviden_metrics['data_drift']['data']['metrics']
                    df_metrics = nested_dict_to_dataframe(evidently_metrics)
                    df_metrics.to_sql('data_drift', engine, if_exists='append', index=True)
```

Hi @nfrvnikita, 

we recently released:
- A complete example integration of Evidently with Postgres as a metrics destination: https://github.com/evidentlyai/evidently/tree/main/examples/integrations/postgres_grafana_batch_monitoring
- We also simplified JSON/Python dictionary output and introduced parameters to customize JSON output (e.g. define which metrics to include) https://github.com/evidentlyai/evidently/blob/main/examples/how_to_questions/how_to_customize_json_output.ipynb

Both could be helpful with relation to your question.

Could you share if you still face issues with Postgres integration and Column mapping with the latest Evidently version?
Repository: evidentlyai/evidently
Issue: shorten the time it takes to calculate data drift
Root cause: Unimplemented Feature Gap

currently if i have a big reference data + test data to check drift - it takes few minutes to complete the drift test.
the files sizes are each about 13MB with shape (1400,380)
since the drift check is calculated by comparing histograms of both data sets - why cant the reference data histogram be calculated beforehand (in training phase of ML model for example) , keep the hjistograms , and just calculate the test data histograms when we want to check the drift (in inference time) , i believe it will shorten the time it takes considerably.

Hi @yairVanti ,

Thank you for sharing this idea with us. We also look at this direction. I believe we won’t be able to calculate histograms for reference data during the training because we do not have access to the model training phase. But we might be able to accept calculated histograms for reference as input for drift calculation.

Generally, I agree that it makes sense to calculate reference-based statistics (not only histograms) beforehand and save them to use later for many reasons. At least it looks reasonable for monitoring purposes if the reference dataset is not changing that often, and one can reuse reference-based statistics for many fresh current datasets.

Unfortunately, we do not have the option to save reference-based statistics beforehand yet. 

I also noticed that you have quite a lot of data. Maybe you could consider using sampling to speed up calculations a bit. I mean, sample both reference and current data and use those samples to estimate a drift?

Would you also consider using histograms for the current dataset? I have some monitoring data already aggregated with histograms, and it would be great to use it directly.
If you have thought about how to do it, I am interested in your plans and I may contribute.

Hi @MainRo, 

Apologies for the delay here! We are considering implementing this in the future, by allowing to working with config files - there is no confirmed implementation design yet, but we are exploring this.

Hi @emeli-dral ,

I just started to use this library but I do not understand why the reference dataset statistics are not / cannot be saved by default, because even with small tables, like 50-60MBs it takes a lot of time to get the reports, even for just one column with text data.

Hi @hkristof03 - we plan to implement the ability to separate the reference dataset - However, since it requires a major library-wide change to 100+ metrics and tests, this is not a small task. It will be addressed it in later versions. 

However, specifically for model-based text data drift detection, the reference dataset is always required. In this case, Evidently trains a classifier model, so this metric cannot be computed without the reference dataset.
Repository: evidentlyai/evidently
Issue: Why does TestRocAuc take so long?
Root cause: Algorithmic Error

Hello, I am using `evidently` on developing MLOps base, and facing the issue of slow execution time of `TestRocAuc` (The data inputting for TestRocAuc has about 1 million rows).
Could you please give me some solutions?


What I verified
- I set `TestRocAuc` thresholds, but execution time did not improve [reference](https://github.com/evidentlyai/evidently/blob/main/src/evidently/tests/classification_performance_tests.py#L249-L259).
- There is no parameter in [QualityMetricsOptions](https://github.com/evidentlyai/evidently/blob/main/src/evidently/options/quality_metrics.py#L12) that improves execution time.
- It looks like `scikit-learn.metrics.roc_curve` is used when calculating AUC (Calculations using scikit-learn API should be fast) [reference](https://github.com/evidentlyai/evidently/blob/main/src/evidently/metrics/classification_performance/roc_curve_metric.py#L48-L75).

I faced the same problem and did some investigations.

Calculation of roc-auc invokes 6 calls of the `calculation` method of different classes. One of these classes is `ClassificationQualityMetric` which calls `evidently.calculations.classification_performance.calculate_metrics` function. This function calculates all basic classification metrics. 

First point here that you don't actually need all classification metrics to get `roc-auc` value.

Second, and the most important point is that the code is very inefficient in [these lines](https://github.com/evidentlyai/evidently/blob/main/src/evidently/calculations/classification_performance.py#L395-L402) where for every predicted value some calculations are made using `pandas`.

Hi @feldlime, thank you for taking a look! Would you want to contribute an optimization? 

Hi @elenasamuylova, I can try :)
Repository: evidentlyai/evidently
Issue: evidently.model_profile.Profile.calculate not populating default float values for 'confidence' or 'drift_share'
Root cause: Configuration Error

Hi, when running through the [mlflow integration tutorial](https://github.com/evidentlyai/evidently/blob/main/evidently/tutorials/mlflow_integration.ipynb), the `Profile.calculate` function fails with TypeError exceptions, stating that mathematical operations are not supported between floats and NoneTypes.

The point where this is  failing is in the `dataset_drift_evaluation` function when comparing the calculated drift statistics with the default thresholds, as shown in the attached traceback screenshot:
![conf_level_unsupported_operands](https://user-images.githubusercontent.com/33293631/132728064-313f575d-1b78-4a73-800b-8128c21fbd38.PNG)

If I explicitly specify: 
`column_mapping['drift_conf_level'] = 0.95`
`column_mapping['drift_features_share'] = 0.5`

then everything works as expected.

(evidently version ==  0.1.23.dev0)

Hi @Jay-Down! We just recently implemented the Dataset Drift calculation using column mapping https://docs.evidentlyai.com/reports/data-drift#1-data-drift-summary. So there is no need to write a custom function anymore! We will update this example shortly. Stay tuned, and thanks for reporting an issue!

Hi @Jay-Down ,

Thank you a lot for reporting this! 
Just made a fix, should be working now without explicit specification of the ```drift_conf_level``` and  ```drift_features_share``` in the ```column_mapping``` or config (for CLI running).
Repository: evidentlyai/evidently
Issue: Anderson-Darling with mismatched series lengths
Root cause: Validation Gap

When using two series that have mismatched lengths, the Anderson-Darling test fails:

```python
import pandas as pd
import numpy as np
from scipy import stats
from evidently.calculations.stattests import anderson_darling_test

rng = np.random.default_rng()
var1 = pd.Series(rng.normal(size=50))
var2 = pd.Series(rng.normal(size=30))

anderson_darling_test(var1, var2, "num", 0.1)
```

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/dunchunter9/repos/ds-feature-drift/.venv/lib/python3.8/site-packages/evidently/calculations/stattests/registry.py", line 35, in __call__
    p = self.func(reference_data, current_data, feature_type, actual_threshold)
  File "/home/dunchunter9/repos/ds-feature-drift/.venv/lib/python3.8/site-packages/evidently/calculations/stattests/anderson_darling_stattest.py", line 41, in _anderson_darling
    p_value = anderson_ksamp(np.array([reference_data, current_data]))[2]
ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected shape was (2,) + inhomogeneous part.
```

The inputs are similar [to the example here](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.anderson_ksamp.html#scipy.stats.anderson_ksamp)

If this behaviour is intended, do just close the issue.  If not my suggested change is 

https://github.com/evidentlyai/evidently/blob/main/src/evidently/calculations/stattests/anderson_darling_stattest.py#L41

to
```python
p_value = anderson_ksamp([reference_data.values, current_data.values])[2]
```

Hi @Duncan-Hunter ,
Thank you for reporting the issue.

It is not supposed to be like that. 
You are right, it is ```np.array([var1, var2])``` causing the issue
Repository: evidentlyai/evidently
Issue: pandas pinned version not available with python 3.6
Root cause: Documentation Deficiency

Hi - in CONTRIBUTING.md you mention Python 3.6 is the earliest python version you support.

`Note: 3.6 is the earliest python version we support. `

However when using python 3.6 in a venv, installing pandas==1.3.5 is not possible.

```
~/repos/evidently$ python3 -m pip install -r requirements.txt 
Collecting plotly==5.10.0 (from -r requirements.txt (line 1))
  Downloading https://files.pythonhosted.org/packages/5f/43/6e86a1e1910d434f068a8e032a3cc77ce862f1688776793a6735e5712fd9/plotly-5.10.0-py2.py3-none-any.whl (15.2MB)
    100% |████████████████████████████████| 15.2MB 3.7MB/s 
Collecting statsmodels==0.12.2 (from -r requirements.txt (line 2))
  Downloading https://files.pythonhosted.org/packages/0d/7b/c17815648dc31396af865b9c6627cc3f95705954e30f61106795361c39ee/statsmodels-0.12.2-cp36-cp36m-manylinux1_x86_64.whl (9.5MB)
    100% |████████████████████████████████| 9.5MB 6.1MB/s 
Collecting scikit-learn==0.24.0 (from -r requirements.txt (line 3))
  Downloading https://files.pythonhosted.org/packages/db/e2/9c0bde5f81394b627f623557690536b12017b84988a4a1f98ec826edab9e/scikit-learn-0.24.0.tar.gz (7.4MB)
    100% |████████████████████████████████| 7.4MB 1.1MB/s 
  Installing build dependencies ... done
Collecting pandas==1.3.5 (from -r requirements.txt (line 4))
  Could not find a version that satisfies the requirement pandas==1.3.5 (from -r requirements.txt (line 4)) (from versions: 0.1, 0.2, 0.3.0, 0.4.0, 0.4.1, 0.4.2, 0.4.3, 0.5.0, 0.6.0, 0.6.1, 0.7.0, 0.7.1, 0.7.2, 0.7.3, 0.8.0, 0.8.1, 0.9.0, 0.9.1, 0.10.0, 0.10.1, 0.11.0, 0.12.0, 0.13.0, 0.13.1, 0.14.0, 0.14.1, 0.15.0, 0.15.1, 0.15.2, 0.16.0, 0.16.1, 0.16.2, 0.17.0, 0.17.1, 0.18.0, 0.18.1, 0.19.0, 0.19.1, 0.19.2, 0.20.0, 0.20.1, 0.20.2, 0.20.3, 0.21.0, 0.21.1, 0.22.0, 0.23.0, 0.23.1, 0.23.2, 0.23.3, 0.23.4, 0.24.0, 0.24.1, 0.24.2, 0.25.0, 0.25.1, 0.25.2, 0.25.3, 1.0.0, 1.0.1, 1.0.2, 1.0.3, 1.0.4, 1.0.5, 1.1.0, 1.1.1, 1.1.2, 1.1.3, 1.1.4, 1.1.5)
No matching distribution found for pandas==1.3.5 (from -r requirements.txt (line 4))
```

Either the supported python version needs to go up, or the pandas version go down? Worked well with python 3.8

Hi @Duncan-Hunter ,

You are right. Thank you for raising the issue. 
We ended support for python 3.6 in evidently 0.2.2 release but did not update the contribution guide.

I will do it asap.



CONTRIBUTING.md is updated
Repository: evidentlyai/evidently
Issue: DataQualityAnalyzer breakes when upgrading from 0.2.4 to 0.2.5
Root cause: External Dependency Breakage

Hi!
After upgrading `evidently` from 0.2.4 to 0.2.5 a previously working setup started to crash with `TypeError: Cannot interpret 'string[python]' as a data type`. I suspect that [this change](https://github.com/evidentlyai/evidently/compare/v0.2.4...v0.2.5#diff-7ab34c27a8956aa2454511cda3b8b328822fe9e90f6d9666e2348da6e90c2d06) introduced the error.

ColumnMapping:
```
ColumnMapping(target='target', prediction='prediction', datetime='i_date', id='i_full_visitor_id', numerical_features=['n_product_pages_viewed', 'n_total_hits', 'n_total_pageviews', 'n_total_visits', 'n_total_time_on_site'], categorical_features=['c_device_category', 'c_is_mobile', 'c_operating_system', 'c_browser', 'c_country', 'c_city', 'c_traffic_source', 'c_traffic_medium', 'c_traffic_campaign', 'c_is_first_visit'], datetime_features=None, target_names=None, task='classification', pos_label=1, text_features=None)
```

Stacktrace:
```
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/Users/calychas/.pyenv/versions/3.8.12/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/Users/calychas/.pyenv/versions/3.8.12/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/calychas/Projects/GID/gid-ml-framework/ga4-mlops/src/ga4_mlops/pipelines/monitoring/monitoring.py", line 194, in _loop
    self.model_monitoring.calculate_metrics(self.data_window.get_data())
  File "/Users/calychas/Projects/GID/gid-ml-framework/ga4-mlops/src/ga4_mlops/pipelines/monitoring/monitoring.py", line 149, in calculate_metrics
    self.execute(
  File "/Users/calychas/Projects/GID/gid-ml-framework/ga4-mlops/venv/lib/python3.8/site-packages/evidently/pipeline/pipeline.py", line 52, in execute
    self.analyzers_results[analyzer] = instance.calculate(
  File "/Users/calychas/Projects/GID/gid-ml-framework/ga4-mlops/venv/lib/python3.8/site-packages/evidently/analyzers/data_quality_analyzer.py", line 118, in calculate
    data_definition = create_data_definition(current_data, reference_data, column_mapping)
  File "/Users/calychas/Projects/GID/gid-ml-framework/ga4-mlops/venv/lib/python3.8/site-packages/evidently/utils/data_preprocessing.py", line 216, in create_data_definition
    id_column = _process_column(mapping.id, data)
  File "/Users/calychas/Projects/GID/gid-ml-framework/ga4-mlops/venv/lib/python3.8/site-packages/evidently/utils/data_preprocessing.py", line 158, in _process_column
    column_type = predefined_type if predefined_type is not None else _get_column_type(column_name, data, mapping)
  File "/Users/calychas/Projects/GID/gid-ml-framework/ga4-mlops/venv/lib/python3.8/site-packages/evidently/utils/data_preprocessing.py", line 340, in _get_column_type
    and (ref_type != cur_type and not np.can_cast(cur_type, ref_type) and not np.can_cast(ref_type, cur_type))
  File "<__array_function__ internals>", line 180, in can_cast
TypeError: Cannot interpret 'string[python]' as a data type
```
Repository: evidentlyai/evidently
Issue: Error when a report is run without reference_data even though it is optional
Root cause: Validation Gap

**Issue**:
When generating a report with `DataQualityPreset()` metric, the `reference_data` is optional as described [here](https://docs.evidentlyai.com/presets/data-quality#how-it-works-1). However, calling this method without `reference_data` leads to this error:

```
data_quality_report.run(current_data=current_data,column_mapping=column_mapping)

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
/tmp/ipykernel_4117/1444958639.py in <module>
      1 column_mapping = ColumnMapping(datetime="datetime_end",numerical_features=features[1:])
----> 2 data_quality_report.run(current_data=current_data,column_mapping=column_mapping)

TypeError: run() missing 1 required keyword-only argument: 'reference_data'
```

**Cause**:

The `reference_data` argument is optional when running the `run()` method of `Report` class https://github.com/evidentlyai/evidently/blob/098b95b9343c390ba9910be4bb85755215b7d8b2/src/evidently/report/report.py#L42

But it is not assigned a `None` unlike `column_mapping` here:
https://github.com/evidentlyai/evidently/blob/098b95b9343c390ba9910be4bb85755215b7d8b2/src/evidently/report/report.py#L44

**Solution (suggestion)**:

Initialize the optional `reference_data` as `None` in https://github.com/evidentlyai/evidently/blob/098b95b9343c390ba9910be4bb85755215b7d8b2/src/evidently/report/report.py#L42


I am experiencing the same problem.

Hi @GwenVCX, @abhitoqua, 

You should pass the reference data as None in this scenario.

```
data_quality_report = Report(metrics=[
    DataQualityPreset()
])

data_quality_report.run(current_data=current, reference_data=None)
data_quality_report  
```

That worked. Thank you @elenasamuylova!
Repository: evidentlyai/evidently
Issue: Multiclass classification bug
Root cause: Validation Gap

https://github.com/evidentlyai/evidently/blob/87c38cedaaa8efe4ecd4bfb237ee51bd73725747/src/evidently/metrics/classification_performance/base_classification_metric.py#L54

In the previous line reference if we want to do a multiclass classification you only get classes from prediction features, some classes can happen to not be predicted in a timeframe and if this class is in the target, it will not be added to the confusion matrix get the issue 
because labels in the target are not identified as ac classes because they are not present in the prediction. 

how could I add classes that were not predicted but in target as part of the unique labels for the multiclassification?

ValueError: At least one label specified must be in y_true


Hi @mshislam ,
Thank you for letting us know!

We released the fix in the latest package (v 0.2.6); Fill free to update your local evidently version and let us know if something else needs to be improved :)


Hello Emeli,

Thanks a lot for reaching out and looking forward to using the new version.

Regards,
Mohamed Islam

On 6 Mar 2023, at 12:01 PM, emeli-dral ***@***.***> wrote:

﻿

Hi @mshislam<https://github.com/mshislam> ,
Thank you for letting us know!

We released the fix in the latest package (v 0.2.6); Fill free to update your local evidently version and let us know if something else needs to be improved :)

—
Reply to this email directly, view it on GitHub<https://github.com/evidentlyai/evidently/issues/516#issuecomment-1455827256>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ALJ4AZLUU7SKKUYNAJVGUHTW2WYXDANCNFSM6AAAAAAVJNMJVI>.
You are receiving this because you were mentioned.Message ID: ***@***.***>
Repository: evidentlyai/evidently
Issue: Column mapping not working
Root cause: Validation Gap

Column mapping is not working in the system, I get "TypeError: unsupported operand type(s) for -: 'float' and 'NoneType'" or Zero Division Error when I try to use.
OS : Windows
Python 3.7
Even I tried in Google Colab, still its not working


Hey @raajeshlr ,

Thanks for letting us know! It definitely should not be like this, let's figure out what has happened. 
Could you please share a screenshot with the full error output?

And there are a couple of questions, which also will help to solve the case faster:
- what report you are building?
- how the ```column_mapping``` dict looks like (if you prefer not to share feature names - that’s completely OK)?
- what is the evidently package version (```print(evidently.__version__)```) ?

Hi @emeli-dral 
Thanks for the response, let me share the details.
- I am using Boston dataset only and trying to generate Data Drift and Num Target Drift.
- Evidently package version is 0.1.23.dev0
- Error is **unsupported operand type(s) for -: 'float' and 'NoneType'** when I use column mapping, attached the screenshot.
- My Column mapping is
                                        column_mapping = {}
                                        column_mapping['target'] = 'target'
                                        column_mapping['prediction'] = None
                                        column_mapping['datetime'] = None
                                        column_mapping['numerical_features'] = ['CRIM','ZN','INDUS','NOX','RM','AGE','DIS','TAX',
                                       'PTRATIO','B','LSTAT']
                                       column_mapping['categorical_features'] = ['CHAS','RAD']

- Please connect with me at https://www.linkedin.com/in/raajeshlr/
- I am eager to discuss about this. 
<img width="417" alt="error screenshot" src="https://user-images.githubusercontent.com/37265950/133568077-7a615c51-7318-4a94-861b-687f318c92f0.png">




Hi @raajeshlr ,

Got it, this is the problem we recently fixed in the latest version. 

Could you please try to update the package and run the example again?
```pip install evidently --upgrade```

And than run two following commands in the terminal from the evidently directory:
```jupyter nbextension install --sys-prefix --symlink --overwrite --py evidently```
```jupyter nbextension enable evidently --py --sys-prefix```