zhiminy/EvalEng huggingface dataset:
Repository: evidentlyai/evidently
Issue: ValueError in ProbClassificationPerformanceTab
Root cause: Validation Gap

Hi, 

I'm testing evidently with  light gbm binary classifier and getting the following error:

ValueError: Found array with 0 sample(s) (shape=(0, 2)) while a minimum of 1 is required

on calculate quality metrics for ProbClassificationPerformanceTab, though DataDriftTab works fine.

Can you help please? is it possible to add new quality metric e.g. gini  to the dashboard?

Thanks for your amazing work

Hi @sanram905 ,
first of all we do apologise for the late response.
Apparently, we  missed your question 😔 

That's very interesting, especially tacking into account that DataDriftReport  works.
We know, that we have a problem with the N/A values. Particularly, we try to filter out them from the dataset and sometimes it leads to the unforeseen consequences. Do you think that might be connected to the error you got?  Were there any missing values among the targets or projections?

Hi Emeli,
Thanks for getting back on it. Yes you are the right. The issue is due to missing values in the dataset. It works if all missing values are dropped or imputed. I have another question: is it possible to add another dimension to data distribution plots?
Repository: evidentlyai/evidently
Issue: Custom stattest Profile (DataDriftProfileSection(), CatTargetDriftProfileSection())
Root cause: Documentation Deficiency

Hi,

I would like to request some help in changing the statistical tests used in DataDriftProfileSection(), CatTargetDriftProfileSection() on a numerical/categorical level.

Here is what I have so far:

```
# Initialization 
column_mapping = ColumnMapping()
column_mapping.task = "classification"
column_mapping.target = "target"
column_mapping.prediction = None
column.mapping.id = "id"
column_mapping.datetime = "date"

column_mapping.numerical_features = ["N1", "N2", "N3"]
column_mapping.categorical_features = ["C1", "C2", "C3"]

# Calculation
data_drift_profile = Profile(sections=[DataDriftProfileSection(), CatDriftProfileSection()])
data_drift_profile.calculate(historical, last_quarter, column_mapping=column_mapping)
```
How can I apply a specific metric inside each Profile() section for numerical and categorical variables?

I viewed the following "Options for Statistical Tests" but am unable to incorporate this into the Profile() section:

https://github.com/evidentlyai/evidently/blob/main/docs/book/customization/options-for-statistical-tests.md

Hi!

Thank you for the question.

If I understand your question correctly, you want to apply different stattests for num and cat features with JSON profile.

You can do it with `options` parameter for `Profile` like:

```
drift_options = DataDriftOptions(
        cat_features_stattest="kl_div",
        num_features_stattest="ks"
  )
my_profile = Profile(
      sections=[DataDriftProfileSection(), CatTargetDriftProfileSection(), NumTargetDriftProfileSection()],
      options=[drift_options]
 )

my_profile.calculate(
      reference_data=reference_data, current_data=current_data, column_mapping=data_mapping
)
```

Let me know If I've missed something in the question and your case!

That was exactly what I was missing thank you!
Repository: evidentlyai/evidently
Issue: String categorical features causes error in "Error Bias Predicted vs Actual" visualization
Root cause: Validation Gap

I have a dataset with some string categorical features. When generating regression performance report, I'm encountering the following error with the "Predicted vs Actual" scatter plot:

```
ValueError:
    Invalid value of type 'builtins.str' received for the 'cmax' property of scatter.marker       
        Received value: 'mycategoricalvalue'

    The 'cmax' property is a number and may be specified as:
      - An int or float
```

Root of the error is in the creation of the scatter plot `segment_fig` for categorical features, at site-`packages\evidently\dashboard\widgets\reg_underperform_segments_table_widget.py", line 211, in calculate`, which tries to create a color bar, which could make sense if categorical variable is ordinal, but not for this case.

A potential fix creating the figure for string feature types as follows:

```python
segment_fig = px.scatter(
    merged_data,
    x=target_name,
    y=prediction_name,
    color=feature_name,
    facet_col='dataset',
)
```

`merged_data` is already created (from error bias histogram plot) and contains `dataset` column to make distinction of 'reference' / 'current' dataset.

This modification would result in:

![image](https://user-images.githubusercontent.com/44030048/177965234-eacac33e-0415-4327-b6fa-83cce1731eb9.png)



Hi @frtendero !
Thanks for reporting!
The bug is fixed. 👩‍🔧 The update will be published in the next release.
#291
Repository: evidentlyai/evidently
Issue: Report summary error in success_tests and failed_tests
Root cause: Algorithmic Error

## The issue
As a result of a TestSuite I get this report summary:
```
{'all_passed': False,
 'total_tests': 26,
 'success_tests': 26,
 'failed_tests': 26,
 'by_status': Counter({'SUCCESS': 25, 'FAIL': 1})}
```
I believe that `success_tests` and `failed_tests` are wrong, and should rather be `25` and `1` respectively...? Or do I interpret their meaning incorrectly?

## How to reproduce

* python 3.10.4
* pip install pandas==1.4.3 evidently==0.1.52.dev0
* get [data.pkl.zip](https://github.com/evidentlyai/evidently/files/9091359/data.pkl.zip)
* then run

```python
data = pd.read_pickle("data.pkl")
ref = data.head(2)
cur = data.tail(1)
K = 0.33

feature_importance_test_suite = TestSuite(
    tests=[
        tests.TestShareOfOutRangeValues(
            column_name=col,
            left=ref[col].min() - K * ref[col].mean(),
            right=ref[col].max() + K * ref[col].mean(),
        )
        for col in data.columns
    ]
)

feature_importance_test_suite.run(reference_data=ref, current_data=cur)
d = feature_importance_test_suite.as_dict()
print(d["summary"])
```

Hello!

Thank you a lot for the issue report.

I've made a fix for it. 

https://github.com/evidentlyai/evidently/pull/273

Thanks @Tapot ! I took the liberty to have a quick look and make a small suggestion :)
Repository: evidentlyai/evidently
Issue: Incorrect AUC ROC score and log loss.
Root cause: Algorithmic Error

The probability classification report gives wrong values for the above mentioned metrics. Also when I looked into this code - prob_class_ref_quality_metrics_widget.py.
the lines where auc and log_loss were calculated are mentioned in comments as problem

Hi @vikrant-sahu! 
You got it right from the comments! We are aware of it and will fix it really soon :)
Could you share your use case when you got a wrong result? We will add it to the testing now.

If you want the data. Then sorry I can't share it as it a confidential dataset and can't be shared outside our network.

Hi @vikrant-sahu 
Could share something of the below:
- the number of classes in the dataset
- the rough number of observations (are we talking thousands, or tens of thousands, or more)
- whether the metric was incorrect or did not compute at all. 

That would help! 

We understand it can be sensitive - appreciate the response in any case.

Cool..

- the number of classes in the dataset - **2 classes**
- the rough number of observations (are we talking thousands, or tens of thousands, or more) -  **thousands, within 10K**
- whether the metric was incorrect or did not compute at all. **It was incorrect.  I used sklearn and it gave different metric. Even AUC curve was incorrect. Also please check other curves too.**

Hi @vikrant-sahu,

Apologies for the late reply! (🦠)

Thanks for your answers! We are still struggling to reproduce the bug. Here are a couple ideas we have on why the calculation is wrong.

1. Could you please check if your target (positive) class is the first in the column-mapping ['prediction'] list? 

In other tools including sklearn, in order to calculate the ROC AUC metric, you have to explicitly pass the probabilities for the target class. Evidently expects the class columns to follow the order. If it is reversed, the metrics will be calculated incorrectly. 

2. Do you pass the target function as is (categorical label) or there is some post-processing (e.g. you pass the class label like "0" or "1")?

If you pass a categorical label to Evidently, it should automatically binarize it. However, the problem might have happened at this stage, and we'd look further!

Overall, we believe that if you:
- pass a binarized target label to the Evidently (1 for target class and 0 for other); and
- make sure that positive class is the first in the column-mapping ['prediction'] list
the calculation should be correct. 

If it is not, please let us know!

Hi @vikrant-sahu! 

We found a bug and fixed it in #122
Repository: evidentlyai/evidently
Issue: Numpy aliases deprecation warning
Root cause: External Dependency Breakage

I installed the dev version and when running the tests I am getting 3 deprecation warnings from numpy
`numpy_encoder.py`:
```
src/evidently/utils/numpy_encoder.py:17: DeprecationWarning: `np.bool` is a deprecated alias for the builtin `bool`. To silence this warning, use `bool` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.bool_` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
((np.bool, np.bool_), bool)
```
and then similar ones for `np.float` and once more for `np.bool` in `test_utils.py` module.
```
tests/test_utils.py:38: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
[_type(0.5) for _type in (np.float, np.float_, np.float16, np.float32, np.float64)]
```

```
tests/test_utils.py:41: DeprecationWarning: `np.bool` is a deprecated alias for the builtin `bool`. To silence this warning, use `bool` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.bool_` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
([(_type(1), _type(0)) for _type in (np.bool, np.bool_)], "[[true, false], [true, false]]")
```
It could probably be easily gotten rid of by removing `np.bool` and `np.float` from the code.
Repository: evidentlyai/evidently
Issue: Target and prediction are not excluded from the data drift report
Root cause: Documentation Deficiency

According to the [evidently documentation](https://docs.evidentlyai.com/features/dashboards/column_mapping#column-mapping), if target and prediction are specified, they should be excluded from the data drift report. This is fine with evidently v0.1.49.dev0. However, when I upgraded evidently to v0.1.53.dev0, target and prediction are shown in the data drift report. 

I created a notebook to reproduce this bug: https://github.com/yumoL/evidently_bug_reproduction 

Hi @yumoL ,
you are right. 
We have change the behaviour of DataDriftTab()in the 0.1.51 release: https://github.com/evidentlyai/evidently/releases/tag/v0.1.51.dev0

The idea behind that was that some users wanted to obtain drift score for target and predictions together with all the rest features, but did not want to add an additional tab to the report like ```tabs=[DataDriftTab(), NumTargetDriftTab()]``` 

What are you thoughts on the topic? Do you think that target and prediction should not be there?

Sorry for the mess with the docs, we will update it shortly.






Hi @emeli-dral , thanks for your quick reply. Should have read the release notes more carefully... I don't have a strong opinion on whether the target and prediction should be included in the data drift report or not, so it's fine to me that the target and prediction are included in the latest version :)


@yumoL thank you!
Fixed in the docs.
Repository: evidentlyai/evidently
Issue: KS test on discrete variables?
Root cause: Documentation Deficiency

KS test is recommended to be applied for continuous data but for discrete is KS advisable over chisquare?
The following evidently example provided in tutorials is applying ks test on discrete data.

https://colab.research.google.com/drive/1xjAGInfh_LDenTxxTflazsKJp_YKmUiD#scrollTo=4R8X06cuf6zC
Repository: evidentlyai/evidently
Issue: Classification Quality by Feature missing in ClassificationPerformanceProfile json
Root cause: Unimplemented Feature Gap

https://discord.com/channels/815991896916361276/1009001638071119933

Hello, I just started to work with evidently. It's a great tool! I want to create the dashboards for Classification Performance in my application using apexcharts library. So I used profiling to get json profile. However, I couldn't figure how to get Classification Quality by Feature part's data. I get the profile using this: 

```
classification_performance_profile = Profile(sections=[ClassificationPerformanceProfileSection()])
# I don't use current data
classification_performance_profile.calculate(df, None, column_mapping=column_mapping)

There are no json data related to Classification Quality by Feature widgets, because there are no calculated statistics yet.
However, we have histograms with feature distribution by classes and we potentially can add these data to report in a json format.
Repository: evidentlyai/evidently
Issue: How to interpret this graph?
GitHub labels: question

<img width="1407" alt="Screenshot 2020-12-15 at 2 51 19 AM" src="https://user-images.githubusercontent.com/27916695/102140055-87cc9600-3e80-11eb-8539-759dc0ca9d29.png">


Hi Muddassar

The plot shows the individual data points for the chosen feature from the “Production” dataset. Since we do not have a time dimension, the data is organised by index number (ID). 

The green line is the mean, as seen in the reference dataset (the baseline data you are comparing against). The green area covers one standard deviation from the mean. 

You can use this plot to explore feature behavior - e.g. visually understand when the feature started drifting, zoom in on specific data points etc.
Repository: evidentlyai/evidently
Issue: Change titles in HTML reports

I was wondering how to change the titles in the HTML report created. For instance, when I do **RegressionPerformanceTab** comparing two models, the report writes current and reference but I would like to change these names to the  model names used as current and reference. Do you know if that is possible? or is a possible new feature?
Thanks

I actually found the solution. This code should make it:

`dashboard = Dashboard(tabs=[RegressionPerformanceTab(include_widgets=[                                      (RegPredActualTimeWidget("YOUR TITLE",  dataset="current"))], verbose_level=1)])`
Repository: evidentlyai/evidently
Issue: Update DataDriftMonitor to show statistical test and generalize metric
Root cause: Unimplemented Feature Gap

Currently, the `DataDriftMonitorMetrics` class defines a `p_value` variable with a hardcoded `p_value` tag. This doesn't generalize to other statistical tests like the Wasserstein distance or the Jensen-Shannon distance since they don't return p-values (and it's especially confusing for the Wasserstein distance since the metric is not bounded by 1 like Jensen-Shannon). 
Moreover, currently the `metrics` method of the `DataDriftMonitor` class doesn't yield the statistical test's name to `DataDriftMonitorMetrics`, which I think would be immensely useful in distinguishing between metrics that come from chi-square and KS vs Jensen-Shannon for example. 

So I propose replacing
`p_value = ModelMonitoringMetric(f"{_tag}:p_value", ["feature", "feature_type"])`
with
`value = ModelMonitoringMetric(f"{_tag}:value", ["feature", "feature_type", "stat_test"])` 

and
 
`feature_metric.p_value, dict(feature=feature_name, feature_type=feature_metric.feature_type)`
with
`feature_metric.p_value, dict(feature=feature_name, feature_type=feature_metric.feature_type, stat_test=feature_metric.stattest_name)`

I have a local build with those changes and believe that's all that's required to change in the source code, however some unit tests will break. 

I will note that the `DataDriftAnalyzer` class does make use of hardcoded "p_value" variables which is technically also inaccurate since it works for more than just chi-square and KS, but that doesn't impact functionality. 

Let me know if you'd like me to introduce the change in a PR!
Thanks

Hello, thank you for the report.

I think it will be great to get a stat test name in the metric, the PR will be very useful!

Hello @Tapot , I just created the PR! I followed the contribution guidelines and tested locally, all good on my end. Let me know if you have any questions!


Hello, thank you a lot, I've merged the PR!
Repository: evidentlyai/evidently
Issue: ModuleNotfoundError: No module named 'evidently.report'
Root cause: Documentation Deficiency

Hello! 

I was trying the example notebook 'metrics_presets.ipynb' and can not load report module. Is it specific to some version of evidently?
I have already been using evidently for classification performance, data drift and data quality dashboards, which look great.  Now I need the report for getting some performance metrics for custom visualisation, but it wont load evidently.report module.

thanks for your awesome work and would appreciate your help on this issue


Hi, 
This is WIP reports, we haven't released it yet. It works on version from sources only for now.

Thanks much for prompt response. I tried it with source and it works perfectly well for my use case where I need to run performance reports over subsets of data (group by some category) and get quality metric to further calculate some  custom metrics per category.
Now I put a table (where cells are color coded based on variable value) and barchart in plotly horizontally in one figure. the code works fine. But when I define it in a widget, the table doesn't get displayed in my dashboard, but the barchart does.  I haven't seen the use of go.Table in Evidently source code, is it not supported?
Thanks again

Yes, we don't have support for plotly tables, but you can use simple version of tables from Evidently: 

```python
BaseWidgetInfo(
    title="",
    type="table",
    params={"header": ["value", "count"], "data":[['a', 1], ['b', 2]]},
    size=2,
)```

Does the Evidently table support color code based on cell value? And how about plotly subplots or multiple yaxis ?? are they supported?
Repository: evidentlyai/evidently
Issue: DataDriftProfileSection Filtered Results
Root cause: Unimplemented Feature Gap

There should be an option to choose what we want as data drift profile result. At present, we get the following fields for each feature - 

- "current_small_hist": feature_metrics.current_small_hist,
- "ref_small_hist": feature_metrics.ref_small_hist,
- "feature_type": feature_metrics.feature_type,
- "stattest_name": feature_metrics.stattest_name,
- "drift_score": feature_metrics.p_value,
- "drift_detected": feature_metrics.drift_detected,
  
Suppose I don't want `current_small_hist` and `ref_small_hist` in my output. I should be having an option to skip the calculatios for these fields as it takes longer to generate the data drift output . Consider the following code snippet - 

https://github.com/evidentlyai/evidently/blob/0f279d3d908b20d6df47b88bff8800bbdf3d516e/src/evidently/calculations/data_drift.py#L281

```
ref_counts = feature_ref_data.value_counts(sort=False)
cur_counts = feature_cur_data.value_counts(sort=False)
keys = set(ref_counts.keys()).union(set(cur_counts.keys()))
for key in keys:
    if key not in ref_counts:
        ref_counts.loc[key] = 0
    if key not in cur_counts:
        cur_counts.loc[key] = 0
 ```
 For a high cardinality categorical feature having thousands of categories (e.g. zipcode, ip address etc.), this loop takes longer to get the count of all keys for reference and current data which in turn delays the process. It is just used to calculate `current_small_hist` and `ref_small_hist` in data drift profile output. Therefore we should have an option to skip this calculation.

@emeli-dral
Repository: evidentlyai/evidently
Issue: Font in Text Widget (markdown rendered as html) doesn't match other tabs in dashboard
GitHub labels: enhancement

I used the TextWidget as an example, I can't get the react component to take html in the text so I can set the font, but ideally I'd like the font of the rendered markdown to match the font in the other dashboards please.

Thank for another valid feature request @darenr! 
Agree we should implement it. As mentioned, UI customization is not a priority due to ongoing refactoring but we hope to address this later, and happy to accept contributions.
Repository: evidentlyai/evidently
Issue: feature request: allow html in react-markdown and/or support github markdown (or remark-gfm)
GitHub labels: enhancement

the text component is very limiting as-is, it would be super useful it you added plugins to the markdown react component so it would allow html and the github markdown syntax, or at least gfm so I can use tables.

I'd adding my own dashboard reports using this markdown format.


Thanks for reporting @darenr!

Do I understand correctly that you have already tried using the existing options (headers, lists), as shown in the example https://github.com/evidentlyai/evidently/blob/main/examples/data_stories/nyc_taxi_data_drift_dashboard_customization.ipynb?

Otherwise, indeed Evidently currently does not support additional markdown inside the text field.

Unfortunately, we cannot focus on the UI updates right now (though there is a huge backlog of known issues, and it does require a significant rework). We are going through an API update and focus first on the core calculation components. But we expect to get there eventually!  

Also happy to take contributions here!


@emeli-dral the link no longer seems to work
Just raising for awareness

Hi @marcellovictorino, this feature has now been reimplemented in the new Evidently API https://docs.evidentlyai.com/user-guide/customization/text-comments!