zhiminy/EvalEng huggingface dataset:
Repository: evidentlyai/evidently
Issue: [Feature Request] Verbosity level in Reports + API Docs!
GitHub labels: enhancement

Hi there!

I have been playing around with the new `Report` objects (I know they haven't been "officially" released yet, but wanted to get a taste!). Most of the stuff I was doing before with the `Dashboard` objects works just fine, but I couldn't find a way of changing the verbosity level of the metrics/widgets as I could before. Hopefully this is in your roadmap for the Report class, but I thought I'd mention it just in case :)

Which brings me to my second request. Trying to figure out how to do some stuff I frequently have to resort to going to the source code files. Your documentation does a great job at showing examples, but it is not easy to discover what all the available options or parameters for functions and classes are. It would be very helpful if the docs contained an "API Reference Guide" where one could explore each class/method/function and see its docstring, parameters, and default values!

Hi @jlopezpena, sorry for the delay here!

You are spot on in both cases. We are already working on it 😉  

For verbosity level: we plan to address it slightly differently and do it the same way it works for Tests. There you can create a custom test suite by listing individual tests to include https://docs.evidentlyai.com/user-guide/tests-and-reports/run-tests#custom-test-suite

For Reports, you will be able to create a Report by listing individual Metrics you want it to calculate and/or show. We believe this will make it even simpler to customize to your needs. What do you think? 

API docs is a known pain😳 It is in the immediate plans. 

Expect both to happen within 1 month.

Hi @jlopezpena! It's been already some time, just wanted to flag that we released:
- Metrics as individual components. You can now create a report by combining individual metrics and control "verbosity" this way https://docs.evidentlyai.com/tutorial#5.-customize-the-report
- API docs draft https://docs.evidentlyai.com/reference/api-reference 

Hi @emeli-dral and @elenasamuylova, thanks for your replies and for the additions to Evidently!
I am already finding the new presets and customizations very useful and using them to generate content for my team's model cards which get created every time we make a new model. The API docs have also been really helpful for that!

I think we can close this issue as resolved, thanks again for your help and keep up the good work!

Thanks, @jlopezpena! Please let us know if face any other issues!
Repository: evidentlyai/evidently
Issue: Drifted Tabular Data Generation

Hi there,

I've been exploring data drift detection and have been wanting to test how good evidently is at determining how much a given dataset has drifted. However, my main concern right now is wondering how to generate drifted data in the first place, and how much to skew them, so that I can get evidently to detect how much drift was applied to them.

So let's say I have a tabular dataframe like this, where I want to drift just the feature of Age.

![adult df](https://user-images.githubusercontent.com/61004414/191708948-f89bd6a0-c0a0-4faf-8bc4-5f9a5de464b4.PNG)

**What are the types of ways to artificially create a drifted dataset from a given dataset?** 

What I've been doing is splitting it into 2 extreme ranges (e.g. one set of <50 Age and one set of >=50 Age), and then mixing the two datasets more and more to create "less" drift. But supposedly for tabular data would something simpler do the trick, such as applying a uniform difference to all the Ages of one dataset work? Applying a random noise to all of the Ages, the noise following some normal distribution? What other standard techniques could be used to apply drift in this manner, and of a degree that can be varied?

Thank you!

Hi @LifeBoey, you might find this blog useful https://www.evidentlyai.com/blog/data-drift-detection-large-datasets 

There, we generate artificial drift and then explore how each statistical test reacts to it. 

There is also a notebook with all the code, including the code where we created artificial drift: https://colab.research.google.com/drive/1EFFcs0wDzToxSR6nw1umXDgPyeoP_Uk6
Repository: evidentlyai/evidently
Issue: Exception with string and int mixed category features in DataDrift
Root cause: Validation Gap

If a category feature has mixed string and int-like values, DataDrift report raises an error

`TypeError: '<' not supported between instances of 'str' and 'int'`
in method `get_overall_data_drift` on line 
`ref_small_hist = list(reversed(list(map(list, zip(*sorted(ref_counts.items(), key=lambda x: x[0]))))))`

Code for reproducing:
```
import pandas as pd
from evidently import ColumnMapping

from evidently.dashboard import Dashboard
from evidently.dashboard.tabs import DataDriftTab

data = pd.DataFrame({
    "cat": ["a", "b", "c", 123],
    "target": [1, 2, 3, 4]
})

data_mapping = ColumnMapping(
    categorical_features=["cat"],
)

report = Dashboard(tabs=[DataDriftTab()])
report.calculate(data[:100], data[100:101], column_mapping=data_mapping)
```

Discord discussion with real-data dataset https://discord.com/channels/815991896916361276/1020337385202012183
Repository: evidentlyai/evidently
Issue: Add parameters for under- and overestimation metrics
GitHub labels: enhancement

Metrics Group: Regression Performance 

Feature request: to add the ability to change the ranges for over, under, or majority in error bias per segment (like from 90% to 80%). 

Source: Discord conversation https://discord.com/channels/815991896916361276/1022847316614459392 

Fixed in Evidently 0.2
Repository: evidentlyai/evidently
Issue: Some errors in quick start docs
Root cause: Documentation Deficiency

https://docs.evidentlyai.com/quick-start#generating-dashboards-and-html-reports

```
iris_data_and_target_drift_report = Dashboard(tabs=[DataDriftTab, CatTargetDriftTab])
iris_data_and_target_drift_report.calculate(iris_frame[:75], iris_frame[:75], 
    column_mapping=None)
iris_data_drift_report.save("reports/my_report_with_2_tabs.html")
```
you got same reference and current data here, and exported the same report without target drift

I guess it might be:
```
iris_data_and_target_drift_report = Dashboard(tabs=[DataDriftTab, CatTargetDriftTab])
iris_data_and_target_drift_report.calculate(iris_frame[:75], iris_frame[75:], 
    column_mapping=None)
iris_data_and_target_drift_report.save("reports/my_report_with_2_tabs.html")
```


Hi @chengchengbai 

Thanks so much for pointing this out!
We have just updated the docs.
Repository: evidentlyai/evidently
Issue: Cramér-von Mises (CVM) stattest for Drift Analysis
GitHub labels: enhancement

good first issue

hacktoberfest

Cramér-von Mises Criterion: https://en.wikipedia.org/wiki/Cram%C3%A9r%E2%80%93von_Mises_criterion
How to contribute stattest for Drift Analysis: https://github.com/evidentlyai/evidently/wiki/Hacktoberfest

- [x] test implementation 
- [x] unittest
- [x] docs update 
- [x] example update

Hi @emeli-dral, I would like to take this!

I had a look at the steps to implement the stat test at https://github.com/evidentlyai/evidently/wiki/Hacktoberfest.

A also had a look at the template/example code at https://github.com/evidentlyai/evidently/blob/main/src/evidently/calculations/stattests/ks_stattest.py.

Could you point me to someplace I can obtain sample inputs that the function expects? I can start from there. 

Hi @ReubenThomasJohn, thanks for your interest to contribute!

Regarding the inputs that the function expects, you can understand it from unit tests: https://github.com/evidentlyai/evidently/blob/main/tests/stattests/test_stattests.py

The function takes 2 pandas Series objects (current and reference data), feature type and threshold for drift detection.  
```
reference_data: pd.Series, 
current_data: pd.Series, 
feature_type: str, 
threshold: float
```

Example:
```
reference = pd.Series([38.7, 41.5, 43.8, 44.5, 45.5, 46.0, 47.7, 58.0])
current = pd.Series([39.2, 39.3, 39.7, 41.4, 41.8, 42.9, 43.3, 45.8])
feature_type = "num"
threshold = 0.1
```

It returns 2 numbers, test score (for example, p_value or distance) and whether the drift has been detected (p_value <= threshold or p_value > threshold for distance metrics).

Oh, I think I need to mention that there is already a PR on this specific test by @SangamSwadik waiting for minor fixes. 
But you can also choose a different test to implement, including a custom one if you use something different in your practice!

Ah okay! Will see if there are any other stattests that need to be implemented. Thank you! @emeli-dral 

We recently added an [example](https://github.com/evidentlyai/evidently/blob/main/examples/how_to_questions/how_to_specify_stattest_for_a_testsuite.ipynb) of setting stattests in TestSuites. We are happy to receive PRs to update it with new drift detection methods!
Repository: evidentlyai/evidently
Issue: Anderson–Darling stat test for Drift Analysis
GitHub labels: enhancement

good first issue

hacktoberfest

Anderson–Darling test: https://en.wikipedia.org/wiki/Anderson%E2%80%93Darling_test 
How to contribute stattest for Drift Analysis: https://github.com/evidentlyai/evidently/wiki/Hacktoberfest

- [x] test implementation 
- [x] unittest
- [x] docs update 
- [x] example update

Hi, Is it possible to be assigned for this enhancement? I am new to contributing to open source (I also joined hackoctoberfest).

**Update/Edit:** I actually saw from other repos that: you do not formally "reserve" issues. So this is just an interest message :) and not asking for reservations! 

We recently added an [example](https://github.com/evidentlyai/evidently/blob/main/examples/how_to_questions/how_to_specify_stattest_for_a_testsuite.ipynb) of setting stattests in TestSuites. We are happy to receive PRs to update it with new drift detection methods!

Hey @SangamSwadiK , 
Thanks for you contribution!
Repository: evidentlyai/evidently
Issue: Fisher’s Exact test for Drift Analysis
GitHub labels: enhancement

good first issue

hacktoberfest

Fisher’s Exact test: https://en.wikipedia.org/wiki/Fisher%27s_exact_test 
How to contribute stat test for Drift Analysis: https://github.com/evidentlyai/evidently/wiki/Hacktoberfest

- [ ] test implementation 
- [ ] unittest
- [ ] docs update 
- [ ] example update

I would like to assign this issue to myself. I want to contribute evidently! 🤗

Hi @mertbozkir! Thanks for your interest! 
Feel free to start working on the issue, we will review pull requests as they come!


We recently added an [example](https://github.com/evidentlyai/evidently/blob/main/examples/how_to_questions/how_to_specify_stattest_for_a_testsuite.ipynb) of setting stattests in TestSuites. We are happy to receive PRs to update it with new drift detection methods!


@SangamSwadiK asked below question in the Discord server 

> Hi, for the  fisher's exact test in the issues, would fisher for 2x2 contingency table suffice or should it be able to work for m x n contingency table ?
Repository: evidentlyai/evidently
Issue: 2 sample t-test for means comparison
GitHub labels: enhancement

good first issue

hacktoberfest

t-test: https://en.wikipedia.org/wiki/Student%27s_t-test 
How to contribute stat test for Drift Analysis: https://github.com/evidentlyai/evidently/wiki/Hacktoberfest 

- [ ] test implementation 
- [ ] unittest
- [ ] docs update 
- [ ] example update

Hi @emeli-dral I am interested in working on this issue. 

Hi @inderpreetsingh01, feel free to take it! 
We will review contributions on first come basis.


Sure will raise pr soon

We recently added an [example](https://github.com/evidentlyai/evidently/blob/main/examples/how_to_questions/how_to_specify_stattest_for_a_testsuite.ipynb) of setting stattests in TestSuites. We are happy to receive PRs to update it with new drift detection methods!
Repository: evidentlyai/evidently
Issue: Hellinger distance for Drift Analysis
GitHub labels: enhancement

good first issue

hacktoberfest

Hellinger distance: https://en.wikipedia.org/wiki/Hellinger_distance
How to contribute stattest for Drift Analysis: https://github.com/evidentlyai/evidently/wiki/Hacktoberfest

- [x] test implementation 
- [x] unittest
- [x] docs update 
- [x] example update

Hi, @emeli-dral I would like to take this, 

But I am new to evidently, could you please assign this to me and help me during the progress?

Are there any time constraints you have, do I need to complete it within a short time from assigning it to me?

Hi @Mathanraj-Sharma I already started working on this issue yesterday, forgot to mention it here, @emeli-dral please review this pr, will add tests and example shortly.

Hi @Mathanraj-Sharma!

Thanks for your interest to contribute! We do not have a specific timeline here and will not formally "reserve" issues - though we encourage everyone to post when they start working on the issue so that others can see it (and maybe pick a different test). We will review contributions as they come, on the first come - first reviewed basis.

Note that you can also choose and implement a different statistical test of your choice! 


We recently added an [example](https://github.com/evidentlyai/evidently/blob/main/examples/how_to_questions/how_to_specify_stattest_for_a_testsuite.ipynb) of setting stattests in TestSuites. We are happy to receive PRs to update it with new drift detection methods!
Repository: evidentlyai/evidently
Issue: Parallel backend support
GitHub labels: enhancement

Hi there

Thank you for a great package. I would like to request whether a parallel backend could be supported when running drift calculations across large numbers of features.

Regards

Hi @niklasvm , and thanks for opening up the issue!

It is in the roadmap. Unfortunately, not immediate one due some resource constraints, but we will definitely add this.

Amazing, thank you!

Instead of the parallel backend we will support Spark. This should speed up calculations
Repository: evidentlyai/evidently
Issue: loading but not showing any image
Root cause: Environment Incompatibility

When making this example ,  there is not any image showing


![image](https://user-images.githubusercontent.com/46805838/193979525-36ac3ad9-0fa7-493e-96da-791f6d7bbcb1.png)

I'm using WSL with anaconda 



Hi @ccfbeltran, 
Unfortunately we have problems to load visualisations in jupyter notebook in Windows.

Lets try a  couple of things, it should be better with WSL.

1. If you did not enable nbextention before running the example, could you please try to that? 
To enable it, run the two following commands in the terminal from the project directory.
```
jupyter nbextension install --sys-prefix --symlink --overwrite --py evidently
jupyter nbextension enable evidently --py --sys-prefix
```

2. If you did enable nbextention and it did not help, could you try to specify mode parameter in show method, and call it like this:
```bcancer_target_drift.show(mode='inline')``` ?
Repository: evidentlyai/evidently
Issue: AttributeError when trying to save report
Root cause: Validation Gap

When saving a report using 

data_drift_report.save("/dbfs/FileStore/drift/drift_report.html")

I get 

AttributeError: 'DataDriftTableWidget' object has no attribute 'wi'

caused by 

/databricks/python/lib/python3.8/site-packages/evidently/widgets/data_drift_table_widget.py in get_info(self)
     25 
     26     def get_info(self) -> BaseWidgetInfo:
---> 27         if self.wi:
     28             return self.wi
     29         raise ValueError("no widget info provided")

I'm using azure databricks, and can save the json profile, and get the same error if I use data_drift_report.show(), though I never installed nbextension.   

I can save the iris report, and am using numerical and categorical feature mappings.

I had the same issue when I forgot to include the `prediction` key in the `column_mapping` dictionary.

@Chien10 thanks for helping out!

Including prediction didn't help, and I was only running a data drift report.  

Hi @lrumanes, could you share 1) a snippet of how your pandas dataframe looks for reference and current datasets (e.g. a header and 1 line of data) 2) a line of code where you run the calculate function?

This appears to have been caused by encoding the categorical data incorrectly.  Is there any way to pass a map of the original values of the data in so column values display in the graphs in admit_data_drift_report.save and admit_data_drift_report.show?

Hi @lrumanes, apologies for taking the time to get back to you here!

We tried a bunch of combinations but where unable to reproduce the error. Our best guess for now is that there is some error on the dataframe side or e.g. in column mapping.

Could you share the snippet of your code where you specify the column mapping and call the "calculate" command? If it is not sensitive, ideally with a preview of your dataframe (just 2-5 lines would do). You can also send it to email hello@evidentlyai.com or privately in our Discord channel https://discord.gg/xZjKRaNp8b

> This appears to have been caused by encoding the categorical data incorrectly. Is there any way to pass a map of the original values of the data in so column values display in the graphs in admit_data_drift_report.save and admit_data_drift_report.show?

yes, this is still an open issue. Unfortunately, at the moment there are no ways to pass a map for categorical data encoding. 
However, we already working on this issue in PR: https://github.com/evidentlyai/evidently/pull/113
Hopefully, this will be fixed soon enough and all Dashboards will work correctly with categorical data without encoding.
Repository: evidentlyai/evidently
Issue: chisquare error for Boston Dataset
Root cause: External Dependency Breakage

Thanks for your work.

I'm going through Target and Data Drift Dashboard for Boston Dataset Notebook and getting this error:

```
/lib/python3.8/site-packages/evidently/pipeline/pipeline.py in execute(self, reference_data, current_data, column_mapping)
     16                 column_mapping: dict = None):
     17         for analyzer in self.get_analyzers():
---> 18             self.analyzers_results[analyzer] = analyzer().calculate(reference_data, current_data, column_mapping)

/lib/python3.8/site-packages/evidently/analyzers/data_drift_analyzer.py in calculate(self, reference_data, current_data, column_mapping)
     75 
     76             # CHI2 to be implemented for cases with different categories
---> 77             p_value = chisquare(f_exp, f_obs)[1]
     78 
     79             result['metrics'][feature_name] = dict(

/lib/python3.8/site-packages/scipy/stats/stats.py in chisquare(f_obs, f_exp, ddof, axis)
   6850 
   6851     """
-> 6852     return power_divergence(f_obs, f_exp=f_exp, ddof=ddof, axis=axis,
   6853                             lambda_="pearson")
   6854 

/lib/python3.8/site-packages/scipy/stats/stats.py in power_divergence(f_obs, f_exp, ddof, axis, lambda_)
   6692                    f"of {rtol}, but the percent differences are:\n"
   6693                    f"{relative_diff}")
-> 6694             raise ValueError(msg)
   6695 
   6696     else:

ValueError: For each axis slice, the sum of the observed frequencies must agree with the sum of the expected frequencies to a relative tolerance of 1e-08, but the percent differences are:
0.53
```

I have `scipy=1.7.0`. Can you please have a look. Thanks

I have downgraded scipy to 1.5.0 and the error is gone. 

Hi @asadabbas09, glad it was solved!
Repository: evidentlyai/evidently
Issue: ImportError: cannot import name 'ColumnDriftMetric' from 'evidently.metrics
Root cause: Environment Incompatibility

Hi I'm testing out the evidently_metrics.ipynb, all metrics notebook.
 I've come across an issue importing anything on the following import:

from evidently.metrics import ColumnDriftMetric, DataDriftTable, DatasetDriftMetric, ColumnRegExpMetric

Any suggestions as to what the problem is?

Thanks in advance

Linking to a parallel Discord discussion on the same topic https://discord.com/channels/815991896916361276/815992391076806657/threads/1029052234438291458 (right now, one needs to run Evidently from source to use Metrics). 

Hi - I came across the same error. @elenasamuylova - this might be noob query - but could you please let know how to run Evidently from source ? 

Appreciate your time.

Hi @sadashiv19, you can follow the steps described in the Contribution Guide to run the version from the source locally.
https://github.com/evidentlyai/evidently/blob/main/CONTRIBUTING.md

In step 4, you need to make a change (since you do not need the dev version)

pip install -e .[dev] -> pip install -e .

After this, make sure to install and enable nbextension. 

jupyter nbextension install --sys-prefix --symlink --overwrite --py evidently

jupyter nbextension enable evidently --py --sys-prefix
 

You can also wait a couple of weeks, as it will be added to the package in the next release. In the meantime, you can still use the Dashboards API (described here https://docs.evidentlyai.com/features/dashboards/generate_dashboards, examples for drift here https://docs.evidentlyai.com/examples#sample-notebooks)

Hi @ManunEbo, just dropping a note that the Metrics have been released recently in a package https://github.com/evidentlyai/evidently/releases/tag/v0.1.59.dev0

You can work with the "All metrics" example notebook to explore  https://github.com/evidentlyai/evidently/blob/main/examples/sample_notebooks/evidently_metrics.ipynb

Thanks Elena,

Perfect timing.

I'll have a look later on in the week.

Regards

Elvy
Repository: evidentlyai/evidently
Issue: Can you add a save option for TestSuite it takes a very long time to load
Root cause: Documentation Deficiency

I'm following the examples provided: 

#dataset-level tests
data_integrity_dataset_tests = TestSuite(tests=[
    TestNumberOfColumns(),
    TestNumberOfRows(),
    TestNumberOfNulls(),
    TestShareOfNulls(),
    TestNumberOfColumnsWithNulls(),
    TestNumberOfRowsWithNulls(),
    TestShareOfColumnsWithNulls(),
    TestShareOfRowsWithNulls(),
    TestNumberOfDifferentNulls(),
    TestNumberOfConstantColumns(),
    TestNumberOfEmptyRows(),
    TestNumberOfEmptyColumns(),
    TestNumberOfDuplicatedRows(),
    TestNumberOfDuplicatedColumns(),
    TestColumnsType(),
    
])

data_integrity_dataset_tests.run(reference_data=adult_ref, current_data=adult_cur)
data_integrity_dataset_tests

This is taking forever to load, if a save to html option was available, It would be really useful.
I imagine, if I tried this on a much larger dataset, it would be problematic
Is there currently, a way to save TestSuite() output to html?

Regards

Elvy

Hi @ManunEbo, here is the complete documentation for tests: https://docs.evidentlyai.com/user-guide/tests-and-reports/run-tests#output-formats

You can indeed save an HTML, a JSON or export the results as a Python dictionary.  

To export HTML as a separate file:
`data_integrity_dataset_tests.save_html(“file.html”)`

Thanks,
I looked at the documents earlier and didn't see the save_html
I was expecting a save("filename.html") which would have been consistent with the reports.

I digress ...

Thank you

@ManunEbo, it will work the same way for Reports as it does for Test Suites - we are currently in the process of migrating reports to the new API: instead of Profile/Dashboard object, there is now a single Report object https://docs.evidentlyai.com/user-guide/tests-and-reports/get-reports. You can still use the old Dashboards/Profiles API in the meantime, but it will be depreciated after the migration is completed.