zhiminy/EvalEng huggingface dataset:
Repository: evidentlyai/evidently
Issue: Add a new data drift detection method to Evidently
GitHub labels: enhancement

good first issue

hacktoberfest

This is an open issue. We invite you add new data drift detection tests or metrics to the Evidently library. 

Here are general instruction on how to add a new test:
https://github.com/evidentlyai/evidently/wiki/Hacktoberfest 

We invite you contribute new drift detection methods that you consider useful. If you want to add a new method, you can create a separate issue to let us know that you are working on it. You can also directly send the pull-request. 

Please do share some details about the new method and when it can be useful (or when you are using it!) 

Each contribution should contain:
- Test implementation
- Unittest
- Docs update
- Example update

Our goal is not to implement as many statistical tests as possible (as the choice can be confusing to the users!) but to add tests that have different properties and might be chosen over already existing ones for a specific reason. 

We highly welcome additions that come from your practice: e.g., if you, as a data scientist, already use a different drift detection method in your work. 

Here are some ideas for methods you can explore:
- Bhattacharyya distance https://en.wikipedia.org/wiki/Bhattacharyya_distance 
- Energy distance ​https://en.wikipedia.org/wiki/Energy_distance ​ 
- Total variation https://en.wikipedia.org/wiki/Total_variation 
- Matrix or vector norm https://numpy.org/doc/stable/reference/generated/numpy.linalg.norm.html 
 
You can also study implementations that exist in research papers and propose ones that could be a valuable addition. Please cite the paper if you implement any of the approaches proposed by the authors so that we reflect it in the documentation.

**Mahalanobis distance** is also another candidate. 

Here is a paper that discusses about Mahalanobis distance and its applications in Outlier detection
[https://core.ac.uk/download/pdf/233075917.pdf](https://core.ac.uk/download/pdf/233075917.pdf)
Repository: evidentlyai/evidently
Issue: The dashboard showing Loading... on local Jupyter Notebook
Root cause: Documentation Deficiency

**Steps taken**
I installed nbextension using
`jupyter nbextension install --sys-prefix --symlink --overwrite --py evidently`

Enabled nbextension evidently
`jupyter nbextension enable evidently --py --sys-prefix`

**Preview of Error**
![image](https://user-images.githubusercontent.com/1046939/195439597-642d5946-6ec9-4de2-8c7f-47a85be6c6b2.png)

**Issue**
Still I get the above error, Loading... 

Hi @aswinvk28 ,

what OS do you use?

Could you please try to specify ```mode``` parameter in ```show``` method, and call it like this:
```bcancer_target_drift.show(mode='inline')``` ?

mode='inline' works!  @emeli-dral 

> Hi @aswinvk28 ,
> 
> what OS do you use?
> 
> Could you please try to specify `mode` parameter in `show` method, and call it like this: `bcancer_target_drift.show(mode='inline')` ?

Yes, thank you @emeli-dral mode='inline' works
Repository: evidentlyai/evidently
Issue: [Feature Request] Include more simulation datasets to capture the data drift
Root cause: Unimplemented Feature Gap

**Description**
The simulation datasets make the use case for synthetic data generation as well as involve algorithms that use the dataset without a machine learning model. 

[https://github.com/czagoni/darkgreybox/blob/master/docs/tutorials/data/demo_data.csv](https://github.com/czagoni/darkgreybox/blob/master/docs/tutorials/data/demo_data.csv)

For example, [https://github.com/czagoni/darkgreybox](https://github.com/czagoni/darkgreybox), uses Statistical models such as lmfit to estimate the parameters. 

We should include more examples of these then, the ML models usage can be avoided for scoring.
Repository: evidentlyai/evidently
Issue: Invalid JSON being generated by `save_json` methods
Root cause: Validation Gap

The JSON encoder used can output NaN and infinite values (see the `most_common_value` label in the example below).

This is valid javascript, but not valid JSON, and can create problems down the pipeline when another service tries to deserialize it.

The standard `json.dumps` method has an `allow_nan` parameter that can be used to control this behaviour, and it would be good to have access to it (with the default being left as it is right now for compatibility).

Generally speaking, it would be good to have access to all the parameters used by `json.dumps`, perhaps by adding some catch-all `**kwargs` to the `.json()` method in evidently and passing that along to the `json.dumps` call.

Example:
```json
{
    "feature_type": "num",
    "count": 2492,
    "infinite_count": 0,
    "infinite_percentage": 0.0,
    "missing_count": 190,
    "missing_percentage": 7.08,
    "unique_count": 2482,
    "unique_percentage": 92.54,
    "percentile_25": -0.11,
    "percentile_50": 0.35,
    "percentile_75": 1.41,
    "max": 241887.34,
    "min": -1.0,
    "mean": 138.11,
    "most_common_value": NaN,
    "most_common_value_percentage": 7.08,
    "std": 4949.03,
    "most_common_not_null_value": 1.098466433173,
    "most_common_not_null_value_percentage": 0.07,
    "new_in_current_values_count": null,
    "unused_in_current_values_count": null
}
```



Hi @jlopezpena, thanks for your suggestion, it is a great point. We were unable to fit in the current release as we focused on major refactoring. But we took it into work and will make the changes in the next release in November.
Repository: evidentlyai/evidently
Issue: Predicted Vs Actual is not setup to work ?
Root cause: Unimplemented Feature Gap

Hi there,

I am trying to set up a regression performance dashboard that includes the predicted vs actual metric, just like what is shown in the docs [here](https://docs.evidentlyai.com/reports/reg-performance#2.-predicted-vs-actual). However, looking through the codebase it seems the predicted vs actual metric is not fully implemented? There is only a metric file for it and not a calculation, monitor or analyzer? 

For context, I have integrated evidentlyai with Prometheus and Grafana, so have [this ](https://github.com/evidentlyai/evidently/blob/main/examples/integrations/grafana_monitoring_service/dashboards/regression_performance.json)dashboard json file pointing at the regression performance analyzer.

Hi @Joshwlks, you caught us in the middle of refactoring as we move to the new API (where you can get a Report comprised of individual Metrics). All the metrics will be implemented there very soon. 

You can still get the regression performance dashboard, including this plot, as shown in the docs, using the old API (that uses Analyzers) and calling the `RegressionPerformanceTab` in Dashboards https://docs.evidentlyai.com/features/dashboards/generate_dashboards or JSON profile with `RegressionPerformanceProfileSection `https://docs.evidentlyai.com/features/profiling/generate_profiles

Thanks for the reply @elenasamuylova ! Which version of the repo do you recommend using to achieve what I am looking for?

Hi @Joshwlks!

Apologies, as I missed you specifically referring to the Grafana example. 

I am afraid there is no easy way to pass this plot to the Grafana in the existing example. The thing is that Predicted VS Actual is not technically a "metric" (there is nothing additional that Evidently computes). Both "target" and "prediction" are passed to Evidently as inputs in the first place. 

Evidently generates the Predicted VS Actual visualization in the Evidently HTML reports output, but it does not produce any specific JSON output that can be useful in real time. 

If you use Grafana for visualization, you'd need to write some custom code to implement and pass this data (predictions and target) directly to Prometheus from your service, bypassing the Evidently service.
Repository: evidentlyai/evidently
Issue: Contribution Guidelines 

Hi there,

I think it is really a good package, and I want to ask about how contributions can be made to this project?

Thanks.

Hi @DRMALEK! 

Awesome that you like the package! We plan to add contribution guidelines in the fall. 

Right now, it is a more manual process and review. If you want to contribute, let me know what you think to add - we can give you some pointers. 

We can also chat in our Discord: https://discord.com/invite/xZjKRaNp8b

Hi,
here is a contributing guide: https://github.com/evidentlyai/evidently/blob/main/CONTRIBUTING.md 

We are happy to get contributions to any part of the library, from documentation to the core modules and new dashboards :)
Repository: evidentlyai/evidently
Issue: Bhattacharya Distance between Two datasets
Root cause: Unimplemented Feature Gap

**Steps taken**
----------------

**Bhattacharya Kernel**
--------------------------

![image](https://user-images.githubusercontent.com/1046939/197400299-b645d567-69c7-4896-968f-2ef8fb5208cb.png)

The above image is that of the Bhattacharya Distance calculation when no kernel trick is applied. Here the formula shows how to calculate the Bhattacharya Kernel without performing any normalization steps. 

**Bhattacharya Kernel with Kernel Trick**
---------------------------------------------

![image](https://user-images.githubusercontent.com/1046939/197400483-b70ff1d8-ae23-494f-ac1a-1ce58173cffb.png)

The above image is that of the Bhattacharya Distance calculation applied with kernel trick. 

[http://math.uchicago.edu/~may/REU2013/REUPapers/Choe.pdf](http://math.uchicago.edu/~may/REU2013/REUPapers/Choe.pdf)
Repository: evidentlyai/evidently
Issue: DataDriftProfileSection for categorical features is not working with scipy version >= 1.7.1
Root cause: External Dependency Breakage

I've encountered an issue when trying to generate DataDriftProfileSection of categorical features column with Scipy version >= 1.7.1 after downgrade Scipy to 1.6.3 everything is working fine.


Hi @nadcharin 
Thanks for reporting! That will be fixed in the near term.

Hi there,

it is fixed, one more thanks for the reporting.
Repository: evidentlyai/evidently
Issue: Cannot import and use TestHighlyCorrelatedColumns()
Root cause: Documentation Deficiency

Hello, very simple issue that I've just found. When I try to import TestHighlyCorrelatedColumns I got this:


```
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
Input In [34], in <cell line: 1>()
----> 1 from evidently.tests.data_quality_tests import TestHighlyCorrelatedColumns

ImportError: cannot import name 'TestHighlyCorrelatedColumns' from 'evidently.tests.data_quality_tests' (/Users/samico/Desktop/Codes/CoachMe/API-Dev/carrot_latest/lib/python3.9/site-packages/evidently/tests/data_quality_tests.py)
```


```
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Input In [31], in <cell line: 3>()
      1 #from evidently.tests import *
      3 data_quality = TestSuite(tests=[
----> 4     TestHighlyCorrelatedColumns()
      5 ])
      7 data_quality.run(reference_data=ref,current_data=curr,column_mapping=schema)
      8 data_quality.as_dict()

NameError: name 'TestHighlyCorrelatedColumns' is not defined
```

My Code:
```
from evidently.tests import *

data_quality = TestSuite(tests=[
    TestHighlyCorrelatedColumns()
])

data_quality.run(reference_data=ref,current_data=curr,column_mapping=schema)
data_quality.as_dict()
```



Evidently Version = '0.1.58.dev0'
Python Version = 3.8

Hi @samuelamico, there is an upcoming update containing the renamed tests (it has just been merged https://github.com/evidentlyai/evidently/pull/419 and should be in the package in a day).

You can now use the old name `TestHighlyCorrelatedFeatures` or wait until the update, which should happen very soon.

Thanks @elenasamuylova , could you update the Docs later ? Thanks so much

Hi @samuelamico, you can see the complete list of renamed tests in the linked PR https://github.com/evidentlyai/evidently/pull/419 

The docs are one day ahead of the published package right now, but it will be synced very soon :)  

Hi @samuelamico, there is a new release https://github.com/evidentlyai/evidently/releases/tag/v0.1.59.dev2 that should solve the naming issues, and now the docs and release are aligned. There should be no further changes of this type. Apologies if the workflow was disrupted! Hope it solves the issues now.
Repository: evidentlyai/evidently
Issue: library issue : DataDrift,DataQuality
Root cause: Documentation Deficiency

Hello community,

when I try to import libraries
1. from evidently.metric_preset import DataDrift, NumTargetDrift
2. from evidently.test_preset import DataQuality, DataStability

I am getting following error 
1. cannot import name 'DataDrift' from 'evidently.metric_preset'
2. cannot import name 'DataQuality' from 'evidently.test_preset'

this was just example notebook from evidently.

Any suggestions are highly appreciated.

thanks,

Hi @IITGoaPyVidya! There was an update to the names of the presets last week, as we now have both Metric presets and Test presets. https://github.com/evidentlyai/evidently/releases/tag/v0.1.59.dev0 

Here is the Getting Started tutorial (with updated API): https://github.com/evidentlyai/evidently/blob/main/examples/sample_notebooks/getting_started_tutorial.ipynb

Here are all test presets:
https://github.com/evidentlyai/evidently/blob/main/examples/sample_notebooks/evidently_test_presets.ipynb

Here are all metric presets
https://github.com/evidentlyai/evidently/blob/main/examples/sample_notebooks/evidently_metric_presets.ipynb

In your case, you need to import for Metric Presets:
from evidently.metric_preset import DataDriftPreset
from evidently.metric_preset import TargetDriftPreset

For Test Presets:
from evidently.test_preset import DataStabilityTestPreset
from evidently.test_preset import DataQualityTestPreset




We should have updated all the examples in the documentation but could have overlooked something. Could you point to the example notebook you refer to? We'll update it as well!

@elenasamuylova Thanks for timely response! 

I renamed those import statements now its working fine. I was referring [getting started tutorial](https://colab.research.google.com/drive/1j0Wh4LM0mgMuDY7LQciLaUV4G1khB-zb). that colab notebook is yet to be updated as per new changes.

However this is all for structured data, I want to analyse drift for unstructured data, is there any guidebook on this?

Hi @IITGoaPyVidya, thanks - we will update that one soon. 

Unfortunately, right now, Evidently only natively supports tabular data inputs. As a workaround, some users feed embeddings in the tools, but it can be hard to interpret.

 

Do you have text or image data? 

I have text data, I will be working around NLP

Thanks for clarifying! We plan to add support for the text data next, but right now, there is no dedicated implementation yet.

@IITGoaPyVidya, you can also have a look at this example notebook https://github.com/evidentlyai/evidently/pull/427 (work in progress, I believe) by @SangamSwadiK where he demonstrates exactly the idea of how you can feed text data in Evidently right now.
Repository: evidentlyai/evidently
Issue: Profile misses out data in Data Drift json
Root cause: Unimplemented Feature Gap

When using 'Profile' to generate data_drift json, it does not generate feature specific detailed report like 'Share' graph from categorical columns and 'Data Drift' & 'Data distribution' graphs are missing which is there in data_drift Dashboard, when we click on any particular feature dropdown.
Sharing the sample code from one of the shared sample notebook file to generate the Data drift json:
`iris_data_drift_profile = Profile(sections=[DataDriftProfileSection()])
iris_data_drift_profile.calculate(iris_frame[:75], iris_frame[75:], column_mapping=None)
iris_data_drift_profile.json()`
![image](https://user-images.githubusercontent.com/42404233/199448708-e36f6861-4eb5-4ea0-b438-0205fb7c9359.png)
![image](https://user-images.githubusercontent.com/42404233/199449280-36a9653a-d35f-40f5-99a7-40756f1b553d.png)


Hi @shaktiman101! 

Thanks for the question! 

Btw, we recently changed the API. Instead of a JSON profile as a separate object, we now have a Report object that you can choose to display as a visual report or export as a JSON or Python dictionary. 

You can follow the steps in this example notebook: https://github.com/evidentlyai/evidently/blob/main/examples/sample_notebooks/evidently_metric_presets.ipynb

Here is how you get the JSON for the DataDriftPreset:

```
data_drift_report = Report(metrics=[
    DataDriftPreset(),
])

data_drift_report.run(reference_data=adult_ref, current_data=adult_cur)
data_drift_report.json()
```

Originally we did not include all the data for all the plots to make JSON more lightweight. We might change this in the future.

Could you share what your intended use case is? Where do you plan to use the JSON output - is it a particular visualization tool you want to send it to?


Hi @elenasamuylova,

Thanks for the reply.

Regarding the new Report object for data drift, I think I looked into this also but this also doesn't have all the details rather I see histogram data has also been removed, I guess this has to do with making the json object lightweight.
Could we have some params or some conf file passed to the new Report class & let the user decide what & all data they want in the output json response?

And as for your question regarding the use case, actually we have a tool witch which we want to integrate some of the functionalities of evidently. Having evidently send the json response & visualizing the response within the tools looks more reasonable as it will give seamless user experience rather than integrating the Dashboard/tabs. 
This is what we think would be the right approach but open for any suggestion, feedback you might have, would be happy to include it.

Hi @shaktiman101, thanks for the extra details! 

We actually plan to update the JSON output (both for Tests and Reports) in the next release. We will review how to best implement it literally over the next few days. Cc @emeli-dral to share more once we define it.
Repository: evidentlyai/evidently
Issue: Decimal type is not JSON serializable
Root cause: Validation Gap

Testing some report creation and saving in html that I have in my repo against latest evidently build, I get this errors:

```
  File "/workspaces/score/.venv/lib/python3.10/site-packages/evidently/suite/base_suite.py", line 136, in save_html
    out_file.write(self._render(determine_template("inline"), template_params))
  File "/workspaces/score/.venv/lib/python3.10/site-packages/evidently/suite/base_suite.py", line 165, in _render
    return temple_func(params=template_params)
  File "/workspaces/score/.venv/lib/python3.10/site-packages/evidently/dashboard/dashboard.py", line 105, in file_html_template
    var {params.dashboard_id} = {_dashboard_info_to_json(params.dashboard_info)};
  File "/workspaces/score/.venv/lib/python3.10/site-packages/evidently/dashboard/dashboard.py", line 48, in _dashboard_info_to_json
    return json.dumps(asdict_result, cls=NumpyEncoder)
  File "/usr/lib/python3.10/json/__init__.py", line 238, in dumps
    **kw).encode(obj)
  File "/usr/lib/python3.10/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python3.10/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/workspaces/score/.venv/lib/python3.10/site-packages/evidently/utils/numpy_encoder.py", line 53, in default
    return json.JSONEncoder.default(self, o)
  File "/usr/lib/python3.10/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type Decimal is not JSON serializable
```

this was working in the previous version. All I changed in my code was replacing `CatTargetDrift` and `NumTargetDrift` by `TargetDriftPreset` and `DataDrift` by `DataDriftPreset`

The solution should be as simple as adding `Decimal` to `_TYPES_MAPPING` in `evidently/dashboard/dashboard.py`

Upon further inspection, I found out our issue was from a change in our database connector that was now returning some numbers as decimals instead of floats and not `evidently`, sorry for the noise!

That notwithstanding, it might still be useful to add a rule for serialization of Decimal to `NumpyEncoder`!

Thanks for flagging it @jlopezpena and glad it was solved! 

@jlopezpena do you by chance recall how you were able to resolve the issue? running into the same error on my end
Repository: evidentlyai/evidently
Issue: Produce a Jupyter Notebook Visualizing the Cosine Similarity of any Dataset
Root cause: Unimplemented Feature Gap

**Description**
Using the custom implementation of Cosine Similarity, 

1. Choose a Dataset of choice: such as Breast cancer dataset or any tabular data
2. Create Visualizations for:
3. Similarity measure between dataset, on the whole
4. Similarity measure for each column, individually, based on specifications
Repository: evidentlyai/evidently
Issue: Request for feature - Transformer models/ Clustering/ Pandas Data-frame descriptions 
Root cause: Unimplemented Feature Gap

Do we have the flexibility to create dashboard for NLP related tasks  from `Hugging face` libraries https://huggingface.co/.

**Example1**: Comparing two classification models using NLP models like `BERT` and `Roberta`.  Compare metrics and show them in dashboards. Also show their loss in the dashboard.

**Example2**: Comparing two Clustering  models using on text data.  Compare metrics such as Silhouette score and show them in dashboards. Also plot the clustering result for `3D `and `2D `by using `Tsne`


Thanks for opening the issue @pratikchhapolika! 

Adding features for text-based data is in our mid-term roadmap. However, we want to expand the functionality with tabular data first. 

Adding the possibility to add custom plots/metrics to existing reports or create custom reports from pre-built plots is in the near-term roadmap. But we will again focus on tabular data first.

Do you have any updates regarding the topic?
Maybe "how to" examples with Hugging Face?
Repository: evidentlyai/evidently
Issue: About the style of  legend

hello team,
   I put the report html in an iframe, but sometimes there is a problem with the translation value of the legend. Can you help me see how to deal with it?
![image](https://user-images.githubusercontent.com/89903340/201302355-63eccfb8-43c3-4720-9cda-b4f8dcbb227d.png)
Repository: evidentlyai/evidently
Issue: Target names from column mapping not showing in reports
Root cause: Configuration Error

Even after setting column_mapping.target_names, class names are not used in the legend for Reports with TargetDriftPreset() and for ClassificationPreset() (class representation and confusion matrix)
See example in Colab Notebook https://colab.research.google.com/drive/14WNUC36p86T4kV4KABtWA5CZCi-2Gl9G?usp=sharing 

Hi @NataliaTarasovaNatoshir,

Thank you for raising the issue.
This is finally fixed in the latest release (0.2.5).

Feel free to raise an issue if anything else does not work!
Repository: evidentlyai/evidently
Issue: Predicted labels are treated as a numerical column for multi-class classification
Root cause: Validation Gap

For a classification task with >5 classes predicted labels are treated as a numerical feature in Reports. column_mapping.task = 'classification' is specified but it only affects the target treatment, predicted labels are still treated as a numerical feature.
See example: https://colab.research.google.com/drive/1XnYj1fAeqn9D6cKlxmaJI8nSNvIKyel1?usp=sharing

Hi @NataliaTarasovaNatoshir ,
thank you for bringing this up!

fixed in #458