zhiminy/EvalEng huggingface dataset:
Repository: evidentlyai/evidently
Issue: Model monitoring without modifying existing code

Hi Team.

We have an ML model deployed as a REST API endpoint in GCP CloudRun service. We want to monitor the endpoint and effectively validate drift detection. May I know if this is possible without modifying my code. I am looking for a plug-n-play option if available. Also, let me know if EvidentlyAI has any enterprise offering that covers my requirement.

Hi @iamavi9, 

Since Evidently is open-source, it requires you to manage the integration on your own. You can either:
- Run scheduled monitoring jobs over the model monitoring logs (e.g., predictions stored in the data warehouse)
- Host the Evidently collector service on your own and send predictions from your ML service 

You can also run ad hoc checks e.g. by performing analysis (generating Evidently Reports) in Jupyter notebook.

We are working on the Evidently Cloud to simplify the integrations and provide additional enterprise features. It is not available publicly, but you can sign up for the waitlist here https://www.evidentlyai.com/cloud-waitlist.
Repository: evidentlyai/evidently
Issue: Index of dataframe must be called "index"
Root cause: Validation Gap

Hi, 

This is my first issue submission. Sorry by advance if does not meet the requirements.

I've been using evidently for the first time and I got an error while providing dataframes with an index named differently than 'index'.

It can be witnessed in [this notebook](https://nbviewer.org/github/JulienfLeBoucher/OC_MLOps_credit_scoring/blob/main/data_drift/data_drift.ipynb) in the 4th section called 'data drift on main features'.

I have let the error on purpose so you can see the traceback there.

The problem comes from evidently/utils/visualizations.py line 1225 assuming that after reset_index() the index would be named 'index' which was False in my case.

I do not know if fix suggestions are welcome or not, but if so, may be one could extend the Column_mapping() with the index name and use it in replacement of 'index' in the problematic line, or preprocess dataframe ensuring the index name is 'index'.

Thanks for reading me.

Hi @JulienfLeBoucher ,
Thank you for rising the issue and for fix suggestion! It is very welcomed :) 

Could you please clarify the version of evidently you used? 

Great!

pip freeze gives : 
evidently @ git+https://github.com/evidentlyai/evidently.git@0f9496e1aa817addaa4d86cb22515bd0f54e0ee6

@0lgaF could you please help with that? 

Hi @JulienfLeBoucher , we merged the fix recently! If you use evidently from the source code it is already available in the main branch. The fix will be included in the next release as well.

Ty!
Repository: evidentlyai/evidently
Issue: Support for tags and metadata in the UI
Root cause: Unimplemented Feature Gap

Users can specify tags and metadata for snapshots. See the [docs](https://docs.evidentlyai.com/user-guide/monitoring/snapshots#add-tags) for more details.

This data then goes to the snapshot json and is accessible via API in the UI service.
The task is to let users see those tags and metadata in the UI and filter & order snapshots by tags and metadata. This functionality should be supported in Reports and Test Suites tabs in the UI.

Here is a support picture to illustrate the idea:


![Image](https://github.com/evidentlyai/evidently/assets/22995068/90cc38f8-8c00-4707-88f0-1effae2ce5e8)



Blocked by #786 

- [X]  Tags (finished by #798 )
- [ ] Metadata


How we should display metadata?

* All collapsed by default? (no on screenshot below)
* What about colors? (see screenshot)
 

![Image](https://github.com/evidentlyai/evidently/assets/32310771/38b27ae4-fc7b-4f94-8f9e-2797c7db91a9)

@emeli-dral ^^

This color theme better i believe



![Image](https://github.com/evidentlyai/evidently/assets/32310771/f109b9f8-11be-472e-b891-8568f74d3658)



we will see deploy soon i believe

With @mou we deployed it here
https://github.com/evidentlyai/cloud_service/pull/175

But there is no projects and i don't know why

@mike0sv @emeli-dral FYI

Wait for proper deploy

- [X]  Tags (finished by #798 )
- [X] Metadata (basics) #873
Repository: evidentlyai/evidently
Issue: Update the Bike Demand Forecasting notebook
Root cause: Documentation Deficiency

**Description:**

Update the Bike Demand Forecasting notebook example adding the `"render": {"raw_data": True} option` to all the Reports.

**Context:**

A blog tutorial https://www.evidentlyai.com/blog/tutorial-1-model-analytics-in-production shows an example of using an Evidently Regression Performance report. 

Here is the accompanying code example:
https://github.com/evidentlyai/evidently/blob/main/examples/data_stories/bicycle_demand_monitoring.ipynb 

Since creating this blog, we updated the Evidently Reports, introducing raw data aggregation that is now applicable by default. Because of this, the visuals you get when running the example do not match the ones in the blog. To get the visuals that match the ones in the blog, users need to turn the `raw_data` option. 

**Implementation:**

Edit the notebook, adding the `raw_data` option https://github.com/evidentlyai/evidently/blob/main/examples/data_stories/bicycle_demand_monitoring.ipynb to all generated Reports.

Follow the documentation on using the raw data option https://docs.evidentlyai.com/user-guide/customization/report-data-aggregation.
Repository: evidentlyai/evidently
Issue: Update the sample "Metric Presets" Notebook
Root cause: Documentation Deficiency

**Description:**

Update the sample notebook showing how to generate `DataDriftPreset` with `"render": {"raw_data": True}` option.

**Context:**

Evidently documentation includes a sample notebook that shows all Metric Presets for tabular data.
https://github.com/evidentlyai/evidently/blob/main/examples/sample_notebooks/evidently_metric_presets.ipynb

By default, all Evidently Reports show aggregated visuals. However, there is an option to include raw data in the Reports, generating different (non-aggregated) visualizations. 

We want to add an example of using this functionality for the Data Drift report.

**Implementation**:

Edit the notebook, adding an example using the `raw_data` option for Data Drift. https://github.com/evidentlyai/evidently/blob/main/examples/data_stories/bicycle_demand_monitoring.ipynb.

Add a new header under "How to get a Python object with Report's main data?" - for example, "How to generate a Report with raw data visualizations?"

Follow the documentation on using the raw data option https://docs.evidentlyai.com/user-guide/customization/report-data-aggregation.

Hi @anirudhsudhir, we do not assign the issues directly - feel free to send the PR, and we'll review them as they come. 

I am afraid though there is already a submitted PR for this issue.
Repository: evidentlyai/evidently
Issue: Update the sample "Metrics" notebook
Root cause: Documentation Deficiency

**Description:**

Update the sample notebook showing how to use Metrics with` "render": {"raw_data": True}` option.

**Context:**

Evidently documentation includes a sample notebook that shows all Metrics for tabular data.
https://github.com/evidentlyai/evidently/blob/main/examples/sample_notebooks/evidently_metrics.ipynb

By default, all Evidently Reports show aggregated visuals. However, there is an option to include raw data in the Reports, generating different (non-aggregated) visualizations.

We want to add an example of using this functionality to the example notebook.

**Implementation:**

Edit the notebook, adding an example using the `raw_data` option for a couple of Metrics - for example, create a simple Report with  `ColumnDriftMetric` and  `ColumnValuePlot` Metrics. https://github.com/evidentlyai/evidently/blob/main/examples/sample_notebooks/evidently_metrics.ipynb 

Add a new header under "How to set metric parameters?" - for example, "How to use Metrics with raw data?"

Follow the documentation on using the raw data option https://docs.evidentlyai.com/user-guide/customization/report-data-aggregation.
Repository: evidentlyai/evidently
Issue: Change the display name for DatasetCorrelationsMetric()
GitHub labels: good first issue

hacktoberfest

**Description:**

Change the the display name for` DatasetCorrelationsMetric()`. Instead of "Dataset Correlations Metric", it should simply have "Dataset Correlations" in the title.

<img width="1283" alt="Screenshot 2023-09-29 at 03 52 04" src="https://github.com/evidentlyai/evidently/assets/67064421/8c2826e3-02cc-499c-ad03-b048a9aa14f0">


@elenasamuylova  Can I get assigned to this issue? As this is for the Hacktoberfest, I will submit the PR as soon as it starts. I have completed the task. Here is a screenshot of the updated file.

<img width="1438" alt="Screenshot 2023-09-29 at 1 46 25 PM" src="https://github.com/evidentlyai/evidently/assets/62090715/0bc489ef-e19f-41a0-96b5-5d4c477d4b2e">


If this is not the change you had expected, please notify me and I will update with the new change.

Thank you,
Dev.

Hey @elenasamuylova, Just asking if I can be assigned or if should I create a PR.

Hi @Dv04, we do not assign issues directly - but feel free to leave a comment here that you are working on it and send the PR; we will review them on a first-come basis!  

The screenshot you shared looks good!

Hey @elenasamuylova, Thanks. Yes I will be working on this issue and submit it as soon as possible.

Hey @elenasamuylova, in which branch do I have to merge the issue, as it is blocking me from doing it in the base branch.

Hi, I’d love to work on this—can I take it
Repository: evidentlyai/evidently
Issue: Fix docs and example on classification column mapping
Root cause: Documentation Deficiency

**Description:**

There is an error on the docs page regarding column mapping for binary probabilistic classification https://docs.evidentlyai.com/user-guide/input-data/column-mapping#binary-probabilistic-classification-option-1

There is the same error for all 3 options. For example, instead of:

```
column_mapping = ColumnMapping()
column_mapping.target = 'target'
column_mapping.prediction = 'prediction'
pos_label = 1
column_mapping.target_names = ['churn', 'not_churn']
```

We should specify that `pos_label` is passed inside `column_mapping`:

```
column_mapping = ColumnMapping()
column_mapping.target = 'target'
column_mapping.prediction = 'prediction'
column_mapping.pos_label = 1
column_mapping.target_names = ['churn', 'not_churn']
```

This error duplicates in the how-to example notebook: 
https://github.com/elenasamuylova/evidently/blob/main/examples/how_to_questions/how_to_use_column_mapping.ipynb

Related: https://github.com/evidentlyai/evidently/issues/799

**Implementation**

Fix the error:
- Update the docs markdown page https://github.com/evidentlyai/evidently/blob/main/docs/book/input-data/column-mapping.md
- Update the how-to example notebook https://github.com/elenasamuylova/evidently/blob/main/examples/how_to_questions/how_to_use_column_mapping.ipynb

Hey @elenasamuylova 
I would like to work on this issue.
Could you please assign?

Hi @ujwalkumar1995, since it is a small issue - we do not directly assign them, but you are free to submit the PR. We'll review them on a first-come basis.

> Hi @ujwalkumar1995, since it is a small issue - we do not directly assign them, but you are free to submit the PR. We'll review them on a first-come basis.

Hey @elenasamuylova 
Sure. Have submitted a PR. Kindly review and let me know in case of any feedback.
Repository: evidentlyai/evidently
Issue: Add example integration with Hamilton 
Root cause: Unimplemented Feature Gap

**Description:**

Document how to use Evidently together with Hamilton, and implement Evidently Test Suites / Reports as part of the DAG.

**Implementation**
- Add a documentation page similar to this one https://docs.evidentlyai.com/integrations/evidently-and-metaflow documenting the example usage. Specify that this is a community-contributed integration.  

**Note**: Requires the implementation of the materializer first. Related issue in Hamilton repo: https://github.com/DAGWorks-Inc/hamilton/issues/439
Repository: evidentlyai/evidently
Issue: iframing evidenlty Ui

Hi Team,
Good Day!

we tried to iframe the evidently ui in our application. but we are getting content blocked. could you pelase advice, is there any flags we need to check before iframing.

Hi @nagasaipureti-infosys

There are no flags for that

I suppose in your application you do something like:
```html
<iframe src="https://your-evidently-ui-instance.com">
```

First of all you have to check [`X-Frame-Options`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Frame-Options) response header from your `https://your-evidently-ui-instance.com` deployment

I don't know details about your infrastructure behind `https://your-evidently-ui-instance.com`, but it is highly recommended to use web servers (like nginx and etc.) before your application server (evidently UI)

```
https://your-evidently-ui-instance.com --->  Web Server like (better to manage headers here) ---> Application server (evidently UI)
```

And then you should set (or remove) `X-Frame-Options` header (and other headers) on web server side
Repository: evidentlyai/evidently
Issue: Integrate pre-commit to improve UX and repository style guide maintenance
GitHub labels: question

There was an attempt to integrate pre-commit into the repository, but it was unsuccessful. But in the future, such integration could save the community a lot of time by preventing style-guide violations and lint check fails to be committed. This will save time on rerunning tests after style and lint fixes.

Previous attempt #527 .
Repository: evidentlyai/evidently
Issue: Pillow Vulnerabilities Pre 10.0.1
Root cause: External Dependency Breakage

Good Morning,

We are using Evidently as our monitoring tool and Dependabot is alerting us to a potential [vulnerability](https://github.com/advisories/GHSA-hhrh-69hc-fgg7) in the currently pinned version of `pillow` which is 9.5.0. 

Is there any appetite to upgrade `pillow` to a later version?

Thanks!
Seán
Repository: evidentlyai/evidently
Issue: Documentation addition: DVC integration
GitHub labels: hacktoberfest

It would be nice to demonstrate how DVC can integrate with Evidently, similarly to [what is shown for MLFLow](https://docs.evidentlyai.com/integrations/evidently-and-mlflow).

Inspired by that, I have already added [one page on the DVC documentation](https://dvc.org/doc/user-guide/integrations/evidently), following the same example as for MLFlow.
I thought that it would be nice to show the same also on the evidently side.

I would be happy to contribute to that, if that sounds interesting to you.

Hi @francesco086, thanks for the willingness to contribute - we'd appreciate it!

Could you specify two things when making the contribution:
- That this is an example that shows how to use Evidently Reports with [DVCLive](https://dvc.org/doc/dvclive) - and name it **Evidently and DVCLive**. Since there are other ways to integrate the two tools (for example, with DVC pipelines like shown [here](https://www.youtube.com/watch?v=qFnwZ653Aks), this might be a bit confusing otherwise.
- Could you add a note that it is a community-contributed integration like in [this ](example) example with Metaflow?

Thank you!

Hi @elenasamuylova !

I did what you asked and a little more, added the notebook in the examples folder 🤗

You can preview the result here: https://francesco.gitbook.io/docs.evidentlyai.com/

By the way, if it is not asking for too much, I would appreciate if you could add the [hacktoberfest](https://hacktoberfest.com/) tag to this issue, as I am participating. I see that you are also involved in this already...

Hi @francesco086 - thanks for the contribution, we will take a look soon!

I added the `hacktoberfest` tag to the issue just in case. However, I believe it should work even without it - the requirement is to have a “hacktoberfest” topic on the project level (which Evidently does).
Repository: evidentlyai/evidently
Issue: Fix visualization in Data Drift metrics
Root cause: Algorithmic Error

**Description**
There is an incorrect visualization of the feature values over time in `ColumnDriftMetric` and inside the `DataDriftPreset` when:
- Datetime is used as index 
- The period of observations is less than 24 hours
- "Raw data" option is off (default).

Here is an example of the bug. The data has 60K observations for 23 hours in October, but they do not render, and the index has an incorrect date.
<img width="1025" alt="Screenshot 2023-10-26 at 11 42 39" src="https://github.com/evidentlyai/evidently/assets/67064421/5ea3d5f5-9b94-4d39-bdac-20bff2b6f5ee">

If the period is longer than 24 hours, the data visualization and index are correct:
 
<img width="919" alt="Screenshot 2023-10-26 at 11 44 04" src="https://github.com/evidentlyai/evidently/assets/67064421/a6818e3b-e3cf-4213-b341-306d382ad3b5">

The data also renders correctly if `raw_data` option is on.

**Reproducible example**:
https://colab.research.google.com/drive/13KLlLRYCxOLn_MKYgrgv5khPsU4ya_hV?usp=sharing

**Implementation**:
Fix the bug for all related metrics that use the same aggregation mechanism (`DataDriftPreset`, `ColumnDriftMetric`, .. possibly `ColumnValuePlot`).


Completed in #845
Repository: evidentlyai/evidently
Issue: Fix issue with datetime_features column mapping
Root cause: Validation Gap

**Description**

An error occurs under the following conditions:
- you specify `datetime_features` in column mapping
- the datatime feature name starts with "prediction"
- you run the Target Drift preset

It fails with an error `Cannot calculate drift metric for column 'prediction' with type ColumnType.Datetime`. It appears to incorrectly parse the datatime feature as the prediction column. 

**Example mapping:**
```
column_mapping_1 = ColumnMapping()

column_mapping_1.prediction="prediction"
column_mapping_1.datetime_features="prediction_timestamp_utc"
```

**Example report (fails):**
```
target_drift_report = Report(metrics=[
    TargetDriftPreset(),
])

target_drift_report.run(reference_data=ref, current_data=cur, column_mapping=column_mapping_1)
target_drift_report
```

However, it works correctly if:
* the datetime feature column does not have "prediction" in its name (for example, it is called `timestamp_utc`)
* the datatime column is passed as `datetime` and not `datetime_features`

**Reproducible example**
https://colab.research.google.com/drive/13hdgU0mSPTiwjq5pAEAgQZApUcM5v3nV

**Implementation**:
Fix the bug + it seems worth checking how `datetime_features` are treated in other reports.

Figured this one out. It is simply because the expression `"prediction" in "prediction_timestamp_utc"` evaluates to `True`. Here is where that is happening in `data_preprocessing.py`:

<img width="698" alt="Screenshot 2024-10-09 at 11 02 54 PM" src="https://github.com/user-attachments/assets/afcb53a9-c9df-4ed6-adf0-5d5c7ac6a292">

This is because the `ColumnMapping` object expects a List[str] type and so if the example code were like this:
```python
column_mapping_1.datetime_features=["prediction_timestamp_utc"]
```

It would work out ok. 

2 possible solutions:

1. Enforce that these column names are passed as lists only
2. Update the type expectation to be Union[str, List[str]] and then update the condition to check that it is either in the list if it is a list or equal to the string if it is a string. 

Hope this helps!
Repository: evidentlyai/evidently
Issue:  Dashboards and Profiles customization
Root cause: Unimplemented Feature Gap

To make Dashboards and Profiles customisable we perform the following refactoring:
- add  an ```Options```  entity to set some changeable parameters in Tabs/Sections and Widgets
- create ```verbose``` parameter for Tabs/Sections  in order to specify the amount of the details in the reports
- create ```include_widgets``` parameter for Tabs/Sections   in order to specify the list of the widgets to show in report
- add ```verbose``` , ```include_widgets``` parameters and ```Options``` support for CLI interface
- add ```verbose``` , ```include_widgets``` parameters and ```Options``` support for monitoring_service

Additionally, to demonstrate how ```Options`` works we need to specify ```drift_conf_level```,  ```drift_features_share```,  ```xbins```,  ```nbinsx``` options for all data_drift reports (Tab, Section, Monitor)
Repository: evidentlyai/evidently
Issue: Reports can only be saved to a file path instead of a handle.
Root cause: Unimplemented Feature Gap

If calling `save_html` (see https://github.com/evidentlyai/evidently/blob/main/src/evidently/suite/base_suite.py#L182C25-L182C33), it only allows to provide a file name as parameter.

However, for saving it to S3 or writing the stream somewhere else, it would be nice to obtain a handle that supports `write`.

Hi @thomasniebler . You can try `report.get_html()`. You can just pass the result to your `stream.write()` method

Still, would you mind adding a PR to allow passing a stream instead of a file handle?

Hey @thomasniebler , we just supported s3 for ```save_html``` method. It is not released in the package yet, but you can check it out in the main branch.
We would love to get your feedback. 

Hi @emeli-dral, the inclusion of an IO type seems like a clever way to solve this issue, thank you! :)
Repository: evidentlyai/evidently
Issue: ValueError: badly formed hexadecimal UUID string
Root cause: Validation Gap

I follow this tutorial (Part 2) 
https://docs.evidentlyai.com/get-started/tutorial-monitoring

Got an error when opening project dashboard

`ValueError: badly formed hexadecimal UUID string`

![image](https://github.com/evidentlyai/evidently/assets/46805454/e3c4dd83-e6c2-4522-9abb-446c2d9a3dfe)
![image](https://github.com/evidentlyai/evidently/assets/46805454/1f876c45-d9da-4ede-b021-9f5d04f10739)
Repository: evidentlyai/evidently
Issue: TypeError: analyzers() missing 1 required positional argument: 'self'
Root cause: Validation Gap

I am interested in the historical drift visualization tutorial. However, after downloading the notebook and running all cells, I encountered an error at cell 200: "TypeError: analyzers() missing 1 required positional argument: 'self'"

Source: https://github.com/evidentlyai/evidently/blob/main/evidently/tutorials/historical_drift_visualization.ipynb

Hi @TawabG ,
thank you for pointing this out!

Here is a fix: https://github.com/evidentlyai/evidently/blob/main/evidently/tutorials/historical_drift_visualization.ipynb

Hey @emeli-dral, thanks for the quick reply! It's fixed! Thank you😃
Repository: evidentlyai/evidently
Issue: How to set custom target feature name ?
Root cause: Documentation Deficiency

It's pretty annoying to rename my target to "target" each time .

Anyway to pass custom target feature name ?

My snippet : 

``
from evidently.test_suite import TestSuite
from evidently.test_preset import BinaryClassificationTestPreset
from evidently.tests import *

rename_dict = {
    "class" : "target"
}

reference_data_dp.rename(columns=rename_dict, inplace=True)
prod_data_dp.rename(columns=rename_dict, inplace=True)

model_performance_suite = TestSuite(tests=[BinaryClassificationTestPreset()])
``

Hi @snassimr, you can use column mapping as shown here: https://docs.evidentlyai.com/user-guide/input-data/column-mapping.



Hi ,

I am trying to use 

model_performance_suite.run(reference_data=reference_data_dp, current_data=prod_data_dp, column_mapping=column_mapping)

But model_performance_suite returned is None

@snassimr Could you share how you define the column mapping object?

You can use this reference example with different ways to map your columns for classification: https://github.com/elenasamuylova/evidently/blob/main/examples/how_to_questions/how_to_use_column_mapping.ipynb

Yep . The second link helped to understand that the error on my side