zhiminy/EvalEng huggingface dataset:
Repository: evidentlyai/evidently
Issue: Update Grafana example dashboard visuals

In the [new Prefect + Evidently + Grafana example](https://github.com/evidentlyai/evidently/tree/main/examples/integrations/postgres_grafana_batch_monitoring) we have a visual Grafana dashboard that shows distribution drift.

The dashboard title says "Drift score (p-value)", however, we use Wasserstein distance metric. 

**Suggested change:** either delete mention of p-value, or replace it for "distance".
Repository: evidentlyai/evidently
Issue: latest version (new API) gives error
Root cause: Validation Gap

i am using version 0.3.1
running the following code :

from evidently import ColumnMapping

    from evidently.report import Report
    from evidently.metrics.base_metric import generate_column_metrics
    from evidently.metric_preset import DataDriftPreset, TargetDriftPreset
    from evidently.metrics import *

    column_mapping = ColumnMapping()
    column_mapping.numerical_features = numeric_features
    column_mapping.categorical_features = categorical_features

    data_drift_report = Report(metrics=[DataDriftPreset()])

    data_drift_report.run(reference, predictions, column_mapping=column_mapping)
    json_report = json.loads(data_drift_report)

gives the following error :

    from src.monitoring.monitor_data_drift_evidently import detect_dataset_drift
../../../../src/monitoring/monitor_data_drift_evidently.py:6: in <module>
    from evidently.report import Report
../../../../lean_env/lib/python3.9/site-packages/evidently/report/__init__.py:1: in <module>
    from .report import Report
../../../../lean_env/lib/python3.9/site-packages/evidently/report/report.py:15: in <module>
    from evidently.metric_preset.metric_preset import MetricPreset
../../../../lean_env/lib/python3.9/site-packages/evidently/metric_preset/__init__.py:10: in <module>
    c.__name__,
E   NameError: name 'c' is not defined

what can be the reason for this failure ?
Repository: evidentlyai/evidently
Issue: The calculation for DataQualityPreset()['DatasetMissingValuesMetric()'] is taking a significant amount of time(approx. 90% of the calculation time))
Root cause: Algorithmic Error

 The profiling of the **DataQualityPreset()[DatasetMissingValuesMetric]** preset reveals that this calculation becomes significantly slower for any datasets, causing it to take up approximately 90% of the time among all presets in the process

I can work on addressing this issue and submit a pull request

See below profiling of DatasetMissingValuesMetric preset



<img width="949" alt="data_quality_missing" src="https://user-images.githubusercontent.com/69073137/236301001-98d632ff-6e72-419d-811c-b907ea90187e.PNG">


I *think* I'm seeing the same error with evidently `0.4.15`.
Repository: evidentlyai/evidently
Issue: Allow passing kwargs to ColumnMetrics that are passed to underlying numpy functions
Root cause: Unimplemented Feature Gap

I'm interested in using `ColumnDistributionMetric` with numeric columns but the only option i can give to this class initialization is column name. Is it possible to include some kwargs in the initialization that get passed to the underlying functions?

I looked at the implementation and it appears that `calculate` calls `get_distribtion_for_column` which dispatches to either `get_distribution_for_numerical_column` or `get_distribtuion_for_category_column`. Can you please make adjustments so that we can send in extra kwargs which can be passed to the underlying functions? So for instance, i wish to pass `density=True` to the underlying `np.histogram` function for numeric columns.

I further believe that this makes sense as many of the popular libraries, for e.g., pandas and dask, already do this quite extensively. For instance, looking at the function signature of `pandas.read_parquet` we see that the last arg is `**kwargs` which gets passed to the underlying engine.

I will be happy to create a branch and do a pull request if there is any interest in this. However, i wanted to ask before doing this to learn if this has already been done in some other branch[es].
Repository: evidentlyai/evidently
Issue: Custom binning showing error
Root cause: Documentation Deficiency

By default custom binning value(nbinsx) is 10. I tried to change the default value by passing below values to column mapping. 

column_mapping['nbinsx'] = 15

It is showing error. I am also attaching the error images. 

Is there any other way to give the custom binning value. 

![image](https://user-images.githubusercontent.com/89089880/135033894-b2f8504f-66c7-4281-ba57-5f557b22b5d3.png)
![image](https://user-images.githubusercontent.com/89089880/135033960-3ad8fb57-b475-400e-ba5f-af07258e8f17.png)



Hi @NuthulaTarun 

Thank you for asking that!
The thing is that currently we do not support this scenario of the binning customisation. 
Instead we offer to set the custom number of bins for each feature separately, for instance like this: ```column_mapping['nbinsx'] = {'TAX': 3, 'PTRATIO': 5}```
You do not have to list in this ```dict``` all features you have in the dataset, only those where you want to have non-default number of bins (default=10).

Here are some more details available in our docs: https://docs.evidentlyai.com/reports/data-drift#histogram-plots-customization.

Do you think that setting a custom ```nbisx``` for the whole dataset will be useful as well?

Thanks emeli-dral for the reply and solution. 
The above mentioned custom bin solution is working.
Thanks once again for the Great Tool and Support
Repository: evidentlyai/evidently
Issue: Internal Server Error 
Root cause: Configuration Error

Hi!!!! I am getting Internal server error when trying to run the whole setup of evidently with my dataset. 
I am attaching the screenshot of it. can you please help with this?
![image](https://github.com/evidentlyai/evidently/assets/100350083/5803ab16-84be-4c8e-a985-69e5f5ff3556)


Hey hii Jyothsna , Even I'm also getting that same error after setup all the environment and run the run_example.py file with my data.
If you find any solution let me know. 

Thank you

Hi guys , 

What is your system configuration and data size, And can please let me know the steps to reproduce it. 

Thanks

Hi @ketangangal !!!
Thank you for the response. We have done the setup using (evidently/examples/integrations/grafana_monitoring_service at main · evidentlyai/evidently · GitHub). We have fixed that issue. It was because of the config file. 
Can you please help us on how to get the classification and regression performance drift using evidently. Can you please share us if there are any documentations available? 

Hello @JyothsnaPendyala , 

There is something called as 
1. RegressionPreset
2. ClassificationPreset
 
 Can also tell me when you say performance drift, where exactly you are focusing with metrics. 
 
(https://docs.evidentlyai.com/reference/all-metrics)

FYI @Vamsi-Cloudangles 

hi @ketangangal, As I have mentioned earlier, I am following the evidently and grafana integration git code from the evidently docs. I am trying to use that code for my example but data is not getting passed to some dashboards. Regression performance dashboard is showing no data
Repository: evidentlyai/evidently
Issue: Can this be used for LLMs?


Hi @kf-rahman 

If you run LLM models in production, you can use Evidently to evaluate and monitor input text data quality, text data drift, and embedding drift.

We do not have any LLM-specific evaluation metrics yet, but depending on the task, you can also use Evidently to evaluate the model quality (e.g., use Classification Presets if you work on text classification tasks).

Which use case do you have in mind?

Hello @kf-rahman , 

Its very handy to use for LLMs all you need to do is Generate embeddings with you llm model and use evidently for monitoring. 
You can also refer : https://docs.evidentlyai.com/user-guide/customization/embeddings-drift-parameters

FYI @elenasamuylova
Repository: evidentlyai/evidently
Issue: Calculate Function throws unpacking error
Root cause: Interface Contract Mismatch

Hi Team,
I ran the example notebook provided on git on my local machine but getting the following error when I call the "calculate" function on the Dashboard or Profile object. 
![image](https://user-images.githubusercontent.com/8390850/136078117-75174570-12b8-406d-8050-2422d2bdcc5c.png)

Can someone help me to understand why this is happening?
Thanks!

Hi @nilamshete3001 , 
thanks for reporting, that's an issue on our side.
Working on a bug fix!

Hi @nilamshete3001 
We updated the package (version 0.1.26.dev0), hopefully now it should be fine.
Could you please upgrade the package version on your side and check if the function works?

Hi @emeli-dral,
Yes, it's working on my end as well 👍 
Thank you for your prompt response and resolution. 
Looking forward to exploring the library and include in our model monitoring framework.
Thanks again!
Repository: evidentlyai/evidently
Issue: TypeError: Sequential.forward() got an unexpected keyword argument 'threshold'
Root cause: Interface Contract Mismatch

Hello,

When running example notebook `how_to_run_drift_report_for_text_data.ipynb` on Colab or locally, I run into the following issue:
```
report = Report(metrics = [
    EmbeddingsDriftMetric('small_subset', 
                          drift_method = model(
                              threshold = 0.55,
                              bootstrap = None,
                              quantile_probability = 0.95,
                              pca_components = None,
                          )
                         )
])

report.run(reference_data = ref_df[:50], current_data = cur_df[:50], 
           column_mapping = column_mapping)
report
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
[<ipython-input-30-b6292193e8aa>](https://localhost:8080/#) in <cell line: 1>()
      1 report = Report(metrics = [
      2     EmbeddingsDriftMetric('small_subset', 
----> 3                           drift_method = model(
      4                               threshold = 0.55,
      5                               bootstrap = None,

[/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py](https://localhost:8080/#) in _call_impl(self, *args, **kwargs)
   1499                 or _global_backward_pre_hooks or _global_backward_hooks
   1500                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1501             return forward_call(*args, **kwargs)
   1502         # Do not call functions when jit is used
   1503         full_backward_hooks, non_full_backward_hooks = [], []

TypeError: Sequential.forward() got an unexpected keyword argument 'threshold'
```

Evidently version is 0.3.2, torch version is 2.0.1+cu118. Python 3.10.11

Hi @edloginova, apologies for the delay - this should have been fixed in Evidently 0.3.3.
Repository: evidentlyai/evidently
Issue: what does 1m, 0.5m or 1B, 2B means in axes ticks labels of data/target drift dashboard

see of eg. image below.

<img width="988" alt="target" src="https://github.com/evidentlyai/evidently/assets/124593281/c1901322-90ef-41c0-9ae3-0fbed5af164b">


Hi @Rajendra-mckesson , 

The x-axis splits all values of a numerical feature into bins. The y-axis (height of the bar) represents the number of objects in the dataset that fall into that range.

The values you refer to on the x-axis show the corresponding bin value. (If you hover over the left age of any histogram bin, it will show the left edge of a given bin). 

When the number is large, the number is shortened. "K" stays for a thousand, "M" stays for million, "B" for billion.  

On you image, it appears you are dealing with a numerical feature that contains large numbers (it takes values from 0 to 5 Billion). So the annotation is shortened to 1B (1 billion),  2B and so on.
Repository: evidentlyai/evidently
Issue: KeyError when using TestShareOfDriftedColumns suite with dataframes with named index
Root cause: Validation Gap

https://github.com/evidentlyai/evidently/blob/3aeccf186b0f85bf3a09748797d81bdff6b3e7ef/src/evidently/utils/visualizations.py#L969-L994

There is a problem with this function in `utils/visualization.py` (referenced above). 
If a data frame with a named index was passed to the function, the line (991)   
`plot_df = df[column_name].reset_index().sort_values("index")`   
will throw a KeyError exception because there will be no "index" column in the data frame after the index reset. Instead of this line, there should be a line  
`plot_df = df[column_name].reset_index().sort_values(index_name)` 

Hi @khavlaren , apologies for the delay - this should have been fixed in Evidently 0.3.3.
Repository: evidentlyai/evidently
Issue: How to ensure directionality of drift?

Suppose we only want to detect the drift in one direction (positive or negative). Does the Evidently AI provide any such facility? 
Example: If we are checking drift in RMSE, we only want to detect drift in a positive direction because RMSE is lower the better kind of metric. Similarly, for r2_score, we only want to detect drift in a negative direction as r2_score is higher the better kind of metric.

@emeli-dral Please guide me on this.

Hi @MokshaVora,

If you are referring specifically to detecting statistical distribution drift ("data drift" metric in Evidently), these tests either measure the distance between distributions or return a P-value of a statistical test. The nature of these checks is to test if distributions are different, not in which "direction" the numerical value changes. Moreover, most of our drift detection methods are normalized, so that you can assess the "size" of the drift, rather than the "direction".  You can read more about different tests here: https://www.evidentlyai.com/blog/data-drift-detection-large-datasets 

If you referring to the ability to detect whether the RMSE has increased compared to some expectations, you can set a rule-based expectation on the RMSE threshold. For example:
- TestValueRMSE(lte=15)  - this will alert if the RMSE is equal or < 15 (pick the threshold that works for your use case)
In this case, you pass a dataset with model predictions and actuals to the tool. So you are comparing a single RMSE value against the threshold.

What I would do, if I got you right, is to calculate the root squared error for each row of the dataset and create a column with these errors. Basically it will be just a numerical column in the dataset
Then you can pass this column as a numerical column to Evidently and run a rule-based check like:
- TestColumnValueDrift
- TestColumnValueMeanInSigmas (I personally believe this is the best one, and I like the visualisation. I think it is pretty helpful for the understanding of drift direction)
- TestColumnValueMax(column_name='RMSE-column', lte=15) - this will alert if any of the values are above 15.

Let me know if that answers your question!
Repository: evidentlyai/evidently
Issue: Save and load reference datasets profile
Root cause: Unimplemented Feature Gap

Hi,
Is there possibility to save reference dateset profile and later load from file to do the data drift calculation in production environment?

Hi @marisabele!

Technically you can save a JSON profile (the summary of metrics/statistical tests results) as a JSON file anywhere.

For example, you can save a JSON of the regression model performance report as a "performance reference". It would contain metrics such as MAE. Then, during production you can build a workflow that would compare the newly calculated production metrics against the baseline values from JSON and follow some rule-based trigger. (Right now such workflow is external to Evidently since the tool only provides the metrics). 

However, if you want to calculate data and prediction drift in production, the tool needs a complete baseline dataset (or a sample of it). Since it uses statistical tests under the hood, it won't work if only a summary of statistical properties (e.g. mean feature values, etc.) is available. In this case, it is not possible to just save some dataset "reference expectations".  

We plan to later add a feature profiling report which can be more suitable in this case. 

I hope we understood your intention right :) Otherwise please let us know a bit more about your ideal workflow!

Hi @emeli-dral, sorry to comment on this closed issue, but has this been aded to Evidently? I am trying to find documentation related to it, but can't find any.

Hi @fferegrino, 

Apologies for the delay here! No, this feature has not been added to Evidently yet.

You can save the dataset profile as JSON (it includes summary of metrics, averages, etc.), but you cannot load it as "reference" for Reports or Test Suites. 

Right now, you either need to pass the reference dataset, or manually specify the conditions (applicable for tests related to data quality, model quality, etc. - exlcuding data drift). 

We are currently reviewing the possibility to add extrenal config, and might add this in the future.
Repository: evidentlyai/evidently
Issue: Is ztstat calculation correct?
Root cause: Algorithmic Error

The Zstat calculation shown here does not appear to match any of the common zstat calculations. Where is it from?

https://github.com/evidentlyai/evidently/blob/fe8be9bbb7fa077a176305d7f87a7f18e8c60919/src/evidently/calculations/stattests/z_stattest.py#L36-L45

Some documentation on comparitive z-test:

- https://www.cliffsnotes.com/study-guides/statistics/univariate-inferential-tests/two-sample-z-test-for-comparing-two-means
- https://en.wikipedia.org/wiki/Student%27s_t-test#Independent_two-sample_t-test

I see this is actually a z-test for proportions:

https://mverbakel.github.io/2021-02-13/two-sample-proportions

Z-test is specifically for binary case
Repository: evidentlyai/evidently
Issue: Is Anderson Darling test built-in function?

Hello,

I got information from the following link about supporting Anderson Darling test. And then, I applied it to detect drift on my datasets. https://docs.evidentlyai.com/user-guide/customization/options-for-statistical-tests
I got some results and after reviewing it, I want to keep using it. By the way, I have questions about built-in method. 
Is this method for two sample test? (The Anderson Darling test is known as a method for comparison with the normal population distribution.) And I couldn't fine any arguments or functions having name like 'Anderson Darling' or 'anderson'. I found the default functions only. (z-score, ks, etc.) Would you tell me about description page or papers on the build-in method?



I found it. Please close this issue.
Repository: evidentlyai/evidently
Issue: TestF1Score by class
Root cause: Documentation Deficiency

Hello dear evidently Team!

I would like to request the possibility to add TestF1Score by class option, if it possible. This is done for Precision and Recall already, but not for F1-Score. Ideally would be nice to have option to chose which f1-score strategy to use ('micro-average','macro-average' etc. Reference: https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html) 
Many thanks.

Hi @danilyef, thanks for the feature request!

We currently have:
* `TestF1ByClass(label='classN') `
* `TestF1Score()`

So it is already possible to test F1 by class.

To double-check, you also would like to have a micro/macro parameter for the `TestF1Score()`, correct?
Repository: evidentlyai/evidently
Issue: Trace 0 on charts of RegressionPerformanceTab
Root cause: Unimplemented Feature Gap

Hi, 

I am having problems trying to understand if trace 0 on graphs has any meaning or if it just a trace on y = 0.

Is it possible to remove this trace? My graph outputs doesnt scale well with it.

Thank you!

Hi @Pakard1 ,

do you mean zero traces like the green one in the picture below?
<img width="474" alt="Screenshot 2021-10-19 at 18 30 19" src="https://user-images.githubusercontent.com/22995068/137943286-06546dfe-6fa6-4d6a-bf21-c63cc960d025.png">

In this case it is just a line indicting the zero error (ideal model). We added it to help quickly understand how far is the existing error form the ideal one. Do you want to get rid of it completely or it will be good enough to have a checkbox with zero trace (and option to include/exclude it like other traces)?



Hi, yes i was talking abou that. I would prefer if it was possible to
remove that trace (be optional) or set it to a certain value

The problem is not on that graph tho. It in the predict vs reference and
predict vs current graphs

![image](https://user-images.githubusercontent.com/57632070/138063863-24bc5e01-5f4a-4320-9660-03e3bb4d28f9.png)

I am referencing this part of code in reg_pred_and_actual_in_time_widget.py

If you change showlegend to True you cant filter out, the other 2 traces, Actual/Current and Predicted legend are show by default

Won't be fixed, we decided to keep the zero trace in the default configurations.
However, the tool allows to create a custom widgets: https://github.com/evidentlyai/evidently/tree/main/examples/how_to_questions/custom_widget_and_tab_example 
The zero line can be entirely removed there.
Repository: evidentlyai/evidently
Issue: RMSE metric in RegressionQualityMetric is wrong
Root cause: Algorithmic Error

Hi,

The RegressionQualityMetric contains a metric value for RMSE. However, the calculation of the metric is wrong and is actually computing the MSE, not the RMSE.

The calculation is using scikit-learn mean_squared_error function, and the default for this function is to output the MSE. To output the RMSE, we need to pass the parameter `squared=False`.

Using version 0.3.3.

Hi @tbaptista, thanks for reporting! 

Could you try building Evidently from main and see if the issue persists?  

This bug should have been fixed recently but is not yet released in the package.

Hi @elenasamuylova,

Thank you for the quick reply. Yes, I can confirm that the issue is solved in the most recent code in main. Waiting for the release then.

Thanks for checking @tbaptista! This will be published next week together with the next release.
Repository: evidentlyai/evidently
Issue: No module named 'evidently.dashboard'
Root cause: External Dependency Breakage

why not installed on my notebook instance 

Hi @jilan804, 

Most likely you are using old Evidently API with a newer Evidently version that no longer has a `dashboard` object.

You can fix the Evidently version to 0.2.8 or earlier, or migrate to the new `Report `object (recommended).

This should help: https://docs.evidentlyai.com/support/migration
Repository: evidentlyai/evidently
Issue: Does not work for cyrilic. Filters all other non-latin letters, and therefore throws error Division by Zero, when the whole text consists of only cyrilic letters
Root cause: Validation Gap

https://github.com/evidentlyai/evidently/blob/eee97cb856c72f713ebe5cc22487852f2046a4cc/src/evidently/features/OOV_words_percentage_feature.py#L34