zhiminy/EvalEng huggingface dataset:
Repository: evidentlyai/evidently
Issue: Distribution plots of prediction score for binary classifiers
GitHub labels: enhancement

Hi would be nice to extend the binary classifier probabilistic report to include distribution plots (in addition to the scatter plot) of the prediction score grouped by the positive/negative true class. This way it will help identify the right cut-off point.
For the multi-label problem we can do multiple plots using 1-vs-all modeling.

Hi @joaomario-brainly,
We are about to add this visualization to our newest update release. Was wondering if that is something you meant? Let us know your thoughts! 

<img width="786" alt="probability_distr_plot" src="https://user-images.githubusercontent.com/22995068/115554681-6872c900-a2b7-11eb-9864-33fddb53f854.png"> 

That would be awesome yes.
If you can also have a kde interpolation you could measure the overlapping between the two curves which is the area where there are FP and FN.

Thanks. You are doing an amazing work.
Repository: evidentlyai/evidently
Issue: Integration with Dash Plotly 

Hi 
Can you please let me know how to integrate with dash plotly?

Hi @PoweredAutomation, 

I am afraid we did not plan for native integration with Dash. It is potentially possible to add it (since Evidently generates Plotly objects in the code) but that would require a lot of changes to the code. We are open for pull requests and contributions here!

Could you share what is your intended use case? Do you want to host a Dash app to connect to your model logs and acy as a monitoring dashboard?
Repository: evidentlyai/evidently
Issue: Different JSON format when generating json in Docker container
Root cause: Environment Incompatibility

Profile(sections=[DataDriftProfileSection()]) generating different json on windows system and in docker container.
Please see below samples
**Windows JSON:**
"metrics": {
                "n_features": 4,
                "n_drifted_features": 4,
                "share_drifted_features": 1.0,
                "dataset_drift": true,
                "TestFeature": {
                    "current_small_hist": [[], []],
                    "ref_small_hist": [[], []],
                    "feature_type": "num",
                    "p_value": 3.3406466154230965e-07
                },

**Docker JSON:**:
"metrics": 
{"n_features": 4, 
"n_drifted_features": 4, 
"share_drifted_features": 1.0, 
"dataset_drift": true, 
"ViewOfProjectManagementCode":
 {"current_small_hist": [[], []], 
"ref_small_hist": [[], []], 
"feature_type": "num", 
"stattest_name": "K-S p_value", 
"drift_score": 7.954740392478146e-09
},

Please verify p_values replaced by K-S p_values
and drift score added

Hi,  @PoweredAutomation ,

looks like you have different version of library on windows and inside container.
In v0.1.48.dev0 we have changed output of Json profile.

You can check version of library by:
```python
import evidently

print(evidently.__version__)
```
Repository: evidentlyai/evidently
Issue: RegressionPerformanceTab - categorical columns not supported
Root cause: Validation Gap

Documentation [link](https://docs.evidentlyai.com/reports/reg-performance) says it supports categorical columns.
But the following code
```py
import pandas as pd
from evidently import ColumnMapping
from evidently.dashboard import Dashboard
from evidently.dashboard.tabs import (
    RegressionPerformanceTab
)
from sklearn import datasets

iris = datasets.load_iris()
iris_frame = pd.DataFrame(iris.data, columns=iris.feature_names)
iris_frame['target'] = iris.target
iris_frame['prediction'] = iris.target + 0.1
column_mapping = ColumnMapping(target='target', prediction='prediction')
# first column as string
iris_frame['sepal length (cm)'] = iris_frame['sepal length (cm)'].apply(lambda x: f"value_str_{x}")

iris_data_drift_report = Dashboard(tabs=[RegressionPerformanceTab()])
iris_data_drift_report.calculate(iris_frame[:100], iris_frame[100:], column_mapping=column_mapping)
```

raises error

```py
  File "C:\...\lib\site-packages\evidently\analyzers\regression_performance_analyzer.py", line 234, in _error_cat_feature_bias
    majority=float(ref_overal_value),
ValueError: could not convert string to float: 'value_str_value_str_5.0'
```

Hi @iuiu34 ,
thank you for reporting this! Categorical columns are meant to be supported in all Dashboards. We add this fix to the closest release.

Hi @iuiu34  !
Thanks for pointing this out!
The bug is fixed. 👩‍🔧 The update will be published in the next release.
#291
Repository: evidentlyai/evidently
Issue: verbose param for computing time in `.calculate`
GitHub labels: enhancement

Apologies in advance if i'm not looking at the documentation properly.

Evidently dashboards compute several steps. Would be nice to have param `verbose` in `.calculate` function from dashboard classes.
So when `verbose = 1`, then computing time of each step is printed

 Hi @iuiu34 

thank you for sharing this! 
You are right, we do not have this option yet. Added to backlog!

Could you please share as well, do you have some issues with  ```calculate``` execution time? If it is the case, which Tabs/Sections do you use?

I use `DataQualityTab` with all tabs in a data source with >200 columns and >1e6 rows. Probably is due to the correlations plots. But I haven't deep checked.
Repository: evidentlyai/evidently
Issue: Attribute error when trying to pass column mapping
Root cause: Validation Gap

**Here's what I was trying:**
```
data_drift_report = Dashboard(tabs=[DataDriftTab()])
data_drift_report.calculate(reference, production_data, 
column_mapping = {"target": "target", "prediction": None, "datetime": "datetime"})

data_drift_report.show(mode="inline")
```
**The error:**
`AttributeError: 'dict' object has no attribute 'datetime'`

**Traceback**
```
~/evidently/analyzers/utils.py in process_columns(dataset, column_mapping)
     85         # data mapping should not be empy in this step
     86         raise ValueError("column_mapping should be present")
---> 87     date_column = column_mapping.datetime if column_mapping.datetime in dataset else None
     88     # index column name
     89     id_column = column_mapping.id
     
AttributeError: 'dict' object has no attribute 'datetime'
```

**Package versions:**
evidently-0.1.49.dev0
dataclasses-0.6 


Hi @aakash-dusane ,

could you please try to set column_mapping using ColumnMapping object instead of a python dictionary?

The following code should work correctly:

```
from evidently import ColumnMapping

data_drift_report.calculate(reference, production_data, 
column_mapping = ColumnMapping(target="target", prediction=None, datetime="datetime"))
```






Thanks for the reply @emeli-dral. Getting a different error now.

The error:
`KeyError: 1`

**Traceback**
```
~/evidently/dashboard/widgets/data_drift_table_widget.py in _generate_additional_graph_num_feature(name, reference_data, current_data, date_column, data_drift_options, quality_metrics_options, color_options)
    118 
    119     if date_column:
--> 120         x0 = current_data[date_column].sort_values()[1]
    121     else:
    122         x0 = current_data.index.sort_values()[1]
```


If I understand correctly, the index should be reset if date column is passed through `ColumnMapping`. My reference data frame index doesn't range from 0 to len(reference) as it is a spliced frame.

The following code worked.
```
from evidently import ColumnMapping

data_drift_report.calculate(reference.reset_index(drop=True), production_data.reset_index(drop=True), 
column_mapping = ColumnMapping(target="target", prediction=None, datetime="datetime"))
```

Looks like I had to just reset the dataframe indexes.
Repository: evidentlyai/evidently
Issue: Kernel/html crashes when trying to display dashboard after using ColumnMapping
Root cause: Resource Mishandling

**My Code:**
```
import pandas as pd
from evidently.dashboard import Dashboard
from evidently.dashboard.tabs import DataDriftTab, NumTargetDriftTab
from evidently.pipeline.column_mapping import ColumnMapping

column_mapping = ColumnMapping()
column_mapping.target = "target"
column_mapping.datetime = "datetime"

data_drift_report = Dashboard(tabs=[DataDriftTab(), NumTargetDriftTab()])
data_drift_report.calculate(reference, week1, 
                             column_mapping=column_mapping)
                         )

data_drift_report.save("report.html")
data_drift_report.show()
```

After executing `data_drift_report.show()` the kernel crashes. If I try opening report.html, the page crashes with error code: 5

I face no issue if column_mapping is not used.




Looks like crash occurs due to memory limitation as code works fine when limiting data to 10 datapoints.
Repository: evidentlyai/evidently
Issue: Cant load dashboard for Target Drift
Root cause: Resource Mishandling

I've been trying to find out why the html with target drift dashboard always seems to crash. Till now I've figured out that it works only when my data is very less:

This works:
```
data_drift_report = Dashboard(tabs=[NumTargetDriftTab()])
data_drift_report.calculate(reference.head(10), week1.head(10), column_mapping=ColumnMapping(target="target",  datetime="datetime"))
data_drift_report.show()
```

This does not:
```
data_drift_report = Dashboard(tabs=[NumTargetDriftTab()])
data_drift_report.calculate(reference.head(100), week1.head(100), column_mapping=ColumnMapping(target="target",  datetime="datetime"))
data_drift_report.show()
```

Any insights will be helpful.



Hello, 

Looks like it is a problem with specific data or depends on data size. 
Can you describe what happened when the code does not work - and errors, warnings, or maybe just waiting without any result?

And maybe you can add some details about the dataset with `week1.info()`/`reference.info()` and `week1.dtypes`/`reference.dtypes`? 

**week.info()**
```
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 2434 entries, 0 to 2433
Data columns (total 5 columns):
 #   Column                  Non-Null Count  Dtype         
---  ------                  --------------  -----         
 0   datetime         2434 non-null   datetime64[ns]
 1   target              2434 non-null   int64         
 2   feature1          2434 non-null   float64       
 3   feature2         2434 non-null   int64         
 4   feature3         2434 non-null   float64       
dtypes: datetime64[ns](1), float64(2), int64(2)
memory usage: 95.2 KB
```

**reference.info()**
```
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 44466 entries, 0 to 44465
Data columns (total 5 columns):
 #   Column                  Non-Null Count  Dtype         
---  ------                  --------------  -----         
 0   datetime         44466 non-null  datetime64[ns]
 1   target              44466 non-null  int64         
 2   feature1          44466 non-null  float64       
 3   feature2         44466 non-null  int64         
 4   feature3         44466 non-null  float64       
dtypes: datetime64[ns](1), float64(2), int64(2)
memory usage: 1.7 MB
```

**If I save as HTML I get this when I try to open it:**
![image](https://user-images.githubusercontent.com/86416909/167122148-7e6235c0-8794-4a5e-aa42-d36482d0483c.png)

If i try to get the dashboard inline, my kernel crashes.




It looks like the code wants to get a lot of memory and crashes. We will try to investigate how to reduce the memory usage in Evidently.
Repository: evidentlyai/evidently
Issue: stattests not available for categorical features
Root cause: Environment Incompatibility

Hello all,

I'm trying to run the data drift dashboard on my dataset which is composed only by categorical features but I'm getting the following error when selecting one of the following stattests ["psi", "z", "jensenshannon", "kl_div"]:

`StatTestInvalidFeatureTypeError: Stattest psi isn't applicable to feature of type cat. Available feature types: ['num']`

This is weird, because I was following the [adult dataset](https://github.com/evidentlyai/evidently/blob/main/examples/how_to_questions/stat_test_specification_for_data_drift_adult.ipynb) example and they seem to use those stattests in their categorical features. Are they not available anymore?


Hi @BrunoRoth95 ,
Can you check what version of evidently package are you using?
PSI support for categorical features was introduced in 0.1.49.dev0.

Hi @Liraim ,
Thank you for your message. Indeed I was using an older version. Now it is working correctly :)
Repository: evidentlyai/evidently
Issue: How to visualize large data?
Root cause: Architectural Constraint

Hi Everyone,

We are very new to community and have a large dataset. When we try to load the data for visualization, browser can not load this html.

Is is possible to figure out this problem or how to deal with large datasets in evidently
![evidently](https://user-images.githubusercontent.com/57112729/168219111-6cc6f161-bfe1-4903-bc8d-4ca4026d94a3.PNG)
?

Hi @yerol89! 
The report is large because the tool stores all the data necessary to generate interactive plots directly inside the HTML. We plan to fix it when we create a service version of the tool (where we decouple the data storage and the browser-based web service).

For now, there are two workarounds:

1. Use some sampling strategy for your dataset, for instance, random sapling. For the Jupyter notebook, that can be done directly with pandas. For the command-line interface, we have a configuration - you can choose random sampling or pick the n-th row. 

2. Use JSON profile. This way Evidently calculates the metrics and statistical tests and returns a JSON summary instead of a visual report. The results can be logged or displayed elsewhere. We have an example for MLflow and Airflow.

We understand this limits how you can use the tool now and are working hard to get to the more feature-full version!
Repository: evidentlyai/evidently
Issue: Multiple comparisons problem
Root cause: Unimplemented Feature Gap

Hi!

I wanted to know whether you think it would be interesting to introduce some method for correcting the [multiple testing problem](https://en.wikipedia.org/wiki/Multiple_comparisons_problem), taking into account the number of tests performed for the evaluation of data drift. I realise that reporting a drift in the dataset when at least 50% of the features have a drift somewhat quells this problem, but perhaps it would be useful to have a threshold that is a function of the number of tests (e.g. Bonferroni correction to select the significance level, without using 0.05 by default).

Let me know what  do you think, in case I can try to implement it

Hi @mfumanelli, 

Thanks for your question and willingness to contribute!

Bonferroni correction is indeed one of the possible approaches! Personally, I find it a bit tricky to interpret. Introducing a custom drift function over the results of the individual tests (or only running drift detection for the top features) gives you a bit more control. 

However, we know some users might prefer it! 

We do not plan to add Bonferroni correction as s a default option, but instead, what we are working on now is to make it easier to add custom metrics / tests / functions to the library so that users can set them as custom options. Right now there are a few statistical tests that are already implemented as options, and we are happy to see more: https://docs.evidentlyai.com/user-guide/customization/options-for-statistical-tests#available-stattest-functions

We are open for contributions! I'd only suggest to wait a couple more weeks. Soon we'll have an updated release with a change in the API, and it would be good moment to revisit the "metrics/tests library" composition.

Perfect, thank you very much for your answer 🌈
Repository: evidentlyai/evidently
Issue: probability classification report - does not show the entire size of data
Root cause: Validation Gap

Hello,
have been investigating your great library. However, after updating the tool and running a classification report I am noticing that the count of rows in the report is inconsistent with the data used for the calculation.
I am not in the position of sharing the error but I hope is a quick thing to check on your side?
I tried all the possible changes and debugging. I know for sure that the reference data has 2500 rows but the report only shows 480 records. Really not sure what else to check and any helps would be really appreciated.

Hi @Guidosalimbeni ,
thank for sharing, we will try to figure it out.

I have quick questions:
- Do I got it right, that this bug appeared in the latest version, and in the older one everything worked correctly? Or you  built the dashboard in the latest version only? This will allow us to understand a little faster what the problem might come from.
- For some reports we filter out rows with nan values, it might be the reason of the problem here. Could you please check the amount of rows with at least one nan value: ```df.isna().any(axis=1).sum()```  ?  May it be, that there are 2020 rows with nan values? 


Hi @Guidosalimbeni, 


In case it helps, these might be related: #241 and #242

Do the data include columns that have nans or `np.inf` values (even if they are not the target/prediction columns)? 
Does the count change if used with a dataset with only the  target and/or prediction columns? 



Thanks, @emeli-dral and @danieljmv01 
1) I have noticed it only in the new version but it might be that I did not notice the error in the previous version. Apologies I am not a great help here.
2) I feel the point on Null values might likely be the issue. Let me test on Monday and I will let you know.
Repository: evidentlyai/evidently
Issue: Not used columns should not affect calculating Categorical Target Drift
Root cause: Validation Gap

`CatTargetDriftAnalyzer` only uses  the target or the prediction columns, but it drops rows that contains NaNs in any column, even if that column is not used in this analyzer. 

Also if any used column is empty after dropping invalid values, the exception  raised is not informative at all. 

I added a test to replicate the problem, here is the pytest output:

```
______________________ test_structure_no_drift_with_nulls ______________________

analyzer = <evidently.analyzers.cat_target_drift_analyzer.CatTargetDriftAnalyzer object at 0x7f88ccf6af98>

    def test_drift_with_null_colums(analyzer: CatTargetDriftAnalyzer) -> None:
        """Test drift with columns with nulls.
    
        Test that not used columns with nulls does not change
        target drift.
        """
        data = {
            "target": ["a"] * 10 + ["b"] * 10,
            "foo": [1]*10 + [np.nan] * 10,
            "bar": [np.nan] * 10 + [1] * 10,
        }
        df1 = DataFrame(data)
        df2 = DataFrame(data)
    
>       result = analyzer.calculate(df1, df2, ColumnMapping())

tests/analyzers/test_categorical_target_drift_analyzer.py:217: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
src/evidently/analyzers/cat_target_drift_analyzer.py:154: in calculate
    reference_data, current_data, feature_type, target_column, target_test, threshold
src/evidently/analyzers/cat_target_drift_analyzer.py:38: in _compute_statistic
    return stattest(reference_data[column_name], current_data[column_name], feature_type, threshold)
src/evidently/analyzers/stattests/registry.py:28: in __call__
    self.default_threshold if threshold is None else threshold)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

reference_data = Series([], Name: target, dtype: object)
current_data = Series([], Name: target, dtype: object), feature_type = 'cat'
threshold = 0.05

    def _z_stat_test(
            reference_data: pd.Series,
            current_data: pd.Series,
            feature_type: str,
            threshold: float) -> Tuple[float, bool]:
        #  TODO: simplify ignoring NaN values here, in chi_stat_test and data_drift_analyzer
        if (reference_data.nunique() == 1
                and current_data.nunique() == 1
                and reference_data.unique()[0] == current_data.unique()[0]):
            p_value = 1
        else:
            keys = set(list(reference_data.unique()) + list(current_data.unique())) - {np.nan}
            ordered_keys = sorted(list(keys))
            p_value = proportions_diff_z_test(
                proportions_diff_z_stat_ind(
>                   reference_data.apply(lambda x, key=ordered_keys[0]: 0 if x == key else 1),
                    current_data.apply(lambda x, key=ordered_keys[0]: 0 if x == key else 1)
                )
            )
E           IndexError: list index out of range
```
Repository: evidentlyai/evidently
Issue: Conda version way behind Pip
Root cause: Unimplemented Feature Gap

Current version on anaconda is: 0.1.23.dev0 way behind 0.1.50

Hi @seanyboi ,
As far as I'm aware of we do not upload version to Anaconda on regular basis.

I'll check what version is there (maybe it was one of tests of upload).

Does your work require for library be available in Anaconda or Pypi is enough?

PyPi is definitely enough, but I know a lot of data scientists / machine learning engineer who use anaconda especially with the new MacBook M1's so if you can it would be nice! :) 

Added to the backlog.
Conda support will start from the stable release.
Repository: evidentlyai/evidently
Issue: How does it work?

Do you have some explanations how drift is detected? For example what statistical test is used for which data type?  I can't find it in the docs. Thanks!

Hi @pplonski! Here is the relevant section of the docs: https://docs.evidentlyai.com/reports/data-drift#how-it-works. There is a default logic that picks a statistical test based on feature type, the number of unique values, and the number of observations. You can also modify it by choosing a different test or adding your own.

Thank you!
Repository: evidentlyai/evidently
Issue: Update sklearn and numpy?
Root cause: Environment Incompatibility

Any plans to update numpy and sklearn to newest versions?

Hi @pplonski,
Library does not have upper bound for versions for those packages, so you can use any version above.
But we can't lock only to newest version because not all users can use it (for variety of reasons).

Why do you think we need to raise minimal version of those dependencies?

Thanks @Liraim, my bad, I've looked at requrements.txt instead setup.py. Sorry!

I'm trying to build Evidently dashboard and host on free Heroku dyno and having hard time with numpy installation.

Anyway, Evidently rocks!

@elenasamuylova @Liraim I've managed to move some of example notebooks to the web application. I'm using [`Mercury`](https://github.com/mljar/mercury) for converting notebooks to web apps. The web app is hosted on free Heroku dyno https://data-drift-detection.herokuapp.com/ and all code is here https://github.com/pplonski/data-drift-detection

Happy to hear any feedback from you! I'm writing an article how to create dashboard for data drift monitoring in Python - will let you know when ready.
Repository: evidentlyai/evidently
Issue: The ChiSquareTest  function parameters are in the wrong order
Root cause: Algorithmic Error

evidently.analyzers.stattests.chisquare_stattest
![image](https://user-images.githubusercontent.com/14029570/176342293-718305e8-ced2-41b4-b0cc-c17b8708a87b.png)
![image](https://user-images.githubusercontent.com/14029570/176342324-f793c617-40c5-42c7-bdc9-80eb075e4236.png)

result = sum((obs-exp)^2/exp);

In ChiSquared distribution, different order result in different results。

Hi @Coder-HHN !
Thanks for reporting!
The bug is fixed. 👩‍🔧 The update will be published in the next release.
#289 

你好，来信已收到，将尽快回复。谢谢。