zhiminy/EvalEng huggingface dataset:
Repository: evidentlyai/evidently
Issue: pydantic version upper bound
Root cause: External Dependency Breakage

Hello, 

is there a reason for capping _pydantic_ at <2? 
I'd like to use some of _pydantic_ 2.x features such as  `computed_field` or `model_validator` which is not currently possible in conjunction with Evidently.

Thank you.

Pydantic v2 is almost full re-write, so there are many incompatible stuff. We put this restriction before it's release in case something breaks. I will investigate if we can loosen the restriction without breaking anything (but I highly doubt it). At some point we will migrate to pydantic v2 though

Thanks, I'm aware that currently Pydantic v2 isn't quite compatible with Evidently. 
Hope the changes would not be extensive and the adoption as seamless as possible. 

I just tried updating to v2 and its definitely will not be a walk in the park :( 

It seems to me that after #792 it now should be possible to remove the `pydantic<2` capping, so evidently can be used together with other libraries that do use it
Repository: evidentlyai/evidently
Issue: Unable to view Evidently drifting test graphs on ReviewNB on Github.
Root cause: Documentation Deficiency


Hello Evidently AI team. I was able to use your package and generate drifting test and graphs for my data. I was able to see the interactive plots on my Jupyter notebook. But when I push my notebook to github and share it with my team, the graphs don't show up on ReviewNB. We use ReviewNB as a tool in conjunction with github for reviewing notebooks and discussions on pull requests within our team. I was curious what packages do we need to install so that the Evidently report graphs are displayed on ReviewNB on Github. Thank you for your help.

Hi @ananda-duetto, I am afraid this is not possible. Evidently generates dynamic HTML visuals, so you cannot render them directly on GitHub. As an option, you generate Reports as HTML files and link to them.

Thank you for answering my query @elenasamuylova and providing an alternate solution. It was helpful. Thanks!!
Repository: evidentlyai/evidently
Issue: Test suite not visible in monitoring dashboard
Root cause: Validation Gap

When creating a test suite for a text column using a text descriptor function, the test suite is not displayed in the monitoring dashboard.

Here is my code:
```
def create_test_suite(train, test):
    column_mapping = ColumnMapping(text_features=['text'])

    data_drift_test_suite = TestSuite(tests=[TestColumnDrift(column_name=WordCount().for_column("text"))])
    data_drift_test_suite.run(reference_data=train, current_data=test, column_mapping=column_mapping)
    return data_drift_test_suite

def create_project(workspace: WorkspaceBase):
    project = workspace.create_project(YOUR_PROJECT_NAME)
    project.save()
    return project

def create_demo_project(train, test, workspace: str):
    ws = Workspace.create(workspace)
    project = create_project(ws)

    test_suite = create_test_suite(train, test)
    ws.add_test_suite(project.id, test_suite)


df = sklearn.datasets.fetch_20newsgroups()
df = pd.DataFrame(df.data, columns=["text"])
df_train = df.iloc[0:500,:]
df_test = df.iloc[500:1000,:]

create_demo_project(df_train, df_test, "workspace")
subprocess.run(["evidently", "ui"])

```

I'm using evidently version 0.4.1


It is however displayed when I call the test suite in a notebook. It is also displayed if I replace the function ```TestColumnDrift``` by ```TestValueRange```

This is how the test suite looks like with ```TestColumnDrift```:
![broken](https://github.com/evidentlyai/evidently/assets/39080117/516fa7ee-8f4c-45de-8da8-785d2fa28c92)


I tried different test suite function:
```TestColumnDrift```  --> test suite is not displayed
```TestNumberOfOutRangeValues```   --> test suite is not displayed
```TestShareOfOutRangeValues```  --> test suite is not displayed
```TestColumnValueMin```  --> test suite is not displayed
```TestColumnValueMax```   --> test suite is not displayed
```TestColumnValueMedian```  --> test suite is not displayed
```TestColumnValueMean```  --> test suite is not displayed

```TestValueRange```  --> test suite is displayed

In all the functions above, the test suite was properly displayed in a notebook.






Hey @max-lutz ,
Thank you for reporting this!

We released evidently 0.4.3, the bug should be fixed there.
Repository: evidentlyai/evidently
Issue: Normalize volume of drift reports
Root cause: Documentation Deficiency

As volume of current and reference data is always gonna be different, need to customize the preset such that, plot uses different y-scale for reference and current data. 
Tried creating a custom metric for drift reports, but there is not much information regarding how to implement plotly in the code.
Also why evidently is only using bar graphs, i would like to plot histogram
![image](https://github.com/evidentlyai/evidently/assets/124362800/8564ec72-fcdd-4d11-a9f1-29dc117d7363)
trying to create plots like this.

Hi @VaishnaviMendhe, 

You can use the "percentage" switch on the plot to switch from absolute values to % - this will help align the distributions when the reference and current datasets have different sizes. Basically, you can switch from:

Absolute values:
![](https://user-images.githubusercontent.com/67064421/259807679-6f7045cf-fcb1-4b32-aa7d-eaa3e4f402dc.png)

To percentage values:
![](https://user-images.githubusercontent.com/67064421/259807773-fc67dd34-dd48-4059-94db-ead494dfedd4.png)

A bar chart represents a distribution, so it works like a histogram but is faster. 

If you do not see this switch "button" on your plots, make sure you are using the latest Evidently version.

Does this solve an issue for you?


Thats awesome, Thanks @elenasamuylova 

Updating evidently version will resolve my problem, hope this won't break anything else.

Hi @elenasamuylova , could I set the data distribution default to "percentage"? Thanks a lot!

Hi @dehiker - there is no such parameter, I am afraid. Both visualizations are always available, and you can manually switch between them when viewing the plots.
Repository: evidentlyai/evidently
Issue: TestColumnCorrelation(column_name="XYZ") Enhancement
GitHub labels: enhancement

As part of my reporting on model performance I've been using `ColumnCorrelationsMetric(column_name='model_score')` to keep an eye on changes between how the inputs are coming in and the `model_score` coming out. I was hoping to use a `TestColumnCorrelation` style call in my TestSuite to go along with the report form of the call. I can use `TestCorrelationChanges(is_critical=False)` broadly for the dataset, but I'd like to have a much more limited test to specifically are there changes in the corellations between this column I specifically care about and all of the other columns provided in the reference/current datasets. 

Is there any plan to add Tests based on the already existing metric looking at changes between the reference and current datasets? Would be quite useful for highlighting changes between high value columns (like the model output) and the rest of the dataset instead of getting warnings for all correlation changes.

Thank you @ZWMiller - this indeed makes sense! We'll try to add this functionality in one of the next releases. 

Hi @ZWMiller, we added the `column_name` parameter to the `TestCorrelationChanges` test - available in Evidently 0.4.5. Kudos to @0lgaF for the implementation!

If you pass this parameter, only the correlations between the selected column and other columns in the dataset will be considered.

To check correlation changes for several chosen columns, you should create several Tests in a suite with different `column_name`.

Let us know how it'll work for you!

Hey @elenasamuylova . I tested this today and it works as I expected, thanks! Only request - if there is a `column_name` provided, is it possible to update the title of the test outcome to say "Change in Correlation **with `column_name`**" . Right now, the title implies there's no change in any correlations. 

Thanks!

Thanks for testing it out @ZWMiller!

cc @0lgaF - could you take at a look at how we update the wording in the test outcome?
Repository: evidentlyai/evidently
Issue: I have a evidently UI hosted in ECS fargate. Now how can I add my reports to the UI running in cloud server...?
Root cause: Documentation Deficiency


Hi @Annamalaisaravanan, 

You must send the snapshots to the same sever where the UI runs. You can compute them locally (or anywhere where you have the Evidently library installed), and then push them to the remote server.

Use the remote workspace API to create and manage projects https://docs.evidentlyai.com/user-guide/monitoring/workspace_project#remote-workspace
Repository: evidentlyai/evidently
Issue: Different bins for reference and current distribution and minor ideas 
Root cause: Unimplemented Feature Gap

hey I saw your presentation at pydata global and really appreciate your work and played around with the data-drift report a little.

1) My main problem is, that different bins are used for reference an current distribiton, so i find the diagrams somewhat misleading
![image](https://user-images.githubusercontent.com/69851777/141085546-900eba8e-0887-4b81-ba4e-7b23a803fada.PNG)

![image](https://user-images.githubusercontent.com/69851777/141085802-2fbc67ef-09b6-45ec-b6a6-07c8f1bc33f2.png)

2) is there a way to disable the data drift plot at all? because in my case with a lot of data points with high deviation this plot takes a lot of memory/performance but is rarly useful

3) what i would find useful is a quick overview of the means and standard deviations in the overview of all features, maybe this could be added as a feature :)

And again thank you for your work. This really helps me a lot since i only tracked shifts in the means in the past.

Hi @davidbange 
Thank you for sharing your thoughts! We really appreciate! 
Let me try to comment a bit.
 
1. Yep, there is a problem indeed. Now we calculate bins for reference and current datasets separately, and we are considering different approaches.
What do you think will be the best option? Do you think we should reuse bins from reference dataset for current data also? Or may be something else?

2. Currently there is no easy way to include or exclude specific widgets in the reports for the moment. Now the only way is to make it at the code level: explicitly remove the instructions for distributions plotting from ```class DataDriftTableWidget(Widget)``` in https://github.com/evidentlyai/evidently/blob/main/evidently/widgets/data_drift_table_widget.py. And then rebuild the project to use it locally. 
Also, we are releasing soon the ability to parametrise Tabs and Dashboards, and an option to switch on/switch off widgets will be there 😎 

3. Indeed! It will be included in our next report: we designing somewhat like dataset overview report! Good to know that it seems interesting 🙂

1.  Won't be fixed (although it is partially updated, the order of the bins if fixed now)
2. The option to enable/disable the data drift plots will we implemented for TestSuites
3. The overview of the means and standard deviations of all features is implemented in the DataQualityTab() of Dashboards
here is the example: https://github.com/evidentlyai/evidently/blob/main/examples/sample_notebooks/data_quality_bike_sharing_demand.ipynb
Repository: evidentlyai/evidently
Issue: Typer dependency conflict 
Root cause: External Dependency Breakage

Reported in Discord. Applies to version 4.0 and above:

```
There are incompatible versions in the resolved dependencies:
  typer>=0.9 (from evidently==0.4.1->-r requirements/../requirements.in (line 18))
  typer<0.8.0,>=0.3.0 (from spacy==3.4.4->en-core-web-sm==3.4.0->-r requirements/../lib/requirements.txt (line 6))
```

Can we check if could accept an earlier typer version?
Repository: evidentlyai/evidently
Issue: UI demo-project cli error
Root cause: Algorithmic Error

Hi, I got the following error after running the `evidently ui --demo-project` 

KeyError: `Value based partial slicing on non-monotonic DatetimeIndexes with non-existing keys is not allowed.`

The root cause is that the raw_data DatetimeIndex is not sort in a monotonic order

suggested fix:
add  `raw_data = raw_data.sort_index()`  here:  

https://github.com/evidentlyai/evidently/blob/d05b20629fa7f55ab6d4cf01450cc92c39b84a12/src/evidently/ui/demo_project.py#L46



same error here! 

same error here

Thanks for reporting @KiralyOr, @jbauer12, @wouwou4444 , the fix will be out soon!
Repository: evidentlyai/evidently
Issue: Data Quality Repoting Limitations
Root cause: Documentation Deficiency

**Data Quality** report currently displays only the count of missing categories or new categories. 

A suggestion is to also include the names of the categories missing in each particular variable for better context and analysis.

Hi @VaishnaviMendhe, 

You can create a custom Report and include a `ColumnValueListMetric()` for all relevant columns (or use a [column metric generator](https://docs.evidentlyai.com/user-guide/tests-and-reports/test-metric-generator) to generate this Metric for all categorical columns from the very beginning). 

The Metric will show both missing categories and new categories (top-10) as compared to the reference or to a defined value list.

<img width="749" alt="Screenshot 2023-09-06 at 13 51 33" src="https://github.com/evidentlyai/evidently/assets/67064421/64e81579-148f-44a1-9565-2c627c8468e7">

Hope this will solve an issue for you!
Repository: evidentlyai/evidently
Issue: Test suit Limitation
Root cause: Unimplemented Feature Gap

If a test fails in a Test Suit, for example, for **duplicated columns**, the column name won't be displayed. 

A suggested improvement is to display the column names, making investigation easier.

![image](https://github.com/evidentlyai/evidently/assets/124362800/43abcff6-8ac9-4c33-8ee6-4e1b2c6c5b4b)


Hi @VaishnaviMendhe, 

Thanks for the feature request! We did not implement it initially since the number of duplicated columns/rows potentially can be in hundreds or millions, which might return an impractically long list of entries.

However, we plan to review this in the near future (for this and similar metrics) and potentially add a parameter to include, e.g., top-10 results, return primary keys/identifiers, etc.
Repository: evidentlyai/evidently
Issue: Feature to check variable bounds, not just min max but also multi bounds
Root cause: Unimplemented Feature Gap

Evidently does not provide a feature to check and compare the multiple bounds of numeric variables between the current data and reference data. for example test should be flagged true if the values of variable are within the bound [[5,10],[50,100]] and for values between 11 to 49 it should be flagged false.

It would be great f evidently can create a json itself for variables bounds in the data set.

Also ,If a column is entirely null in any dataset, Evidently fails to generate the report. 
Instead, it could ignore the column and issue a warning message if it's not present in any dataset.

Hi @VaishnaviMendhe, 

Thanks for sharing your thoughts!

**1. Complex conditions for value range**

There is indeed no special metric to set double conditions for the value range. As a workaround, you can test similar complex conditions using regular expressions. You can use a `TestColumnRegExp` test that accepts a regular expression via `reg_exp` parameter.

**2. Creating a JSON with test conditions**

If I understood you right, you are referring to the ability to:
* export the test conditions that are automatically generated by Evidently (such as derived value ranges) as a separate JSON, and 
* for tests to accept the conditions expressed as a JSON (instead of passing the reference dataset / specifying the conditions as parameters)

This is a great feature indeed, and we plan to add it in the future. However, this is a significant change that requires reworking all metrics and tests, so it will take some time to implement.

**3. Empty columns in the dataset**

We did consider this - it might indeed be more convenient for ad hoc analysis when the results of Reports are reviewed manually. However, we believe an explicit error is a better universal outcome in this scenario - especially when Reports or Test Suites are generated as part of an automated pipeline, and passing an empty column might be unintentional.

**Empty columns in the dataset:**
So if I wish to create a report for live data for some time intervals and let's say for that specific interval if a col is empty, evidently totally fails generating the reports.
this can be passed as a parameter, to include_empty_cols =True , and default this to False

Thanks @VaishnaviMendhe - that's a good point, specifically for near real-time monitoring at short intervals. Cc @emeli-dral to review how we can address this better.
Repository: evidentlyai/evidently
Issue: Issue with importing Dashboard after latest release 0.1.28.dev0 ?
Root cause: External Dependency Breakage

I am exploring evidently and I am new to the tool. I was trying to use it on google colab. It worked fine until today, but when I tried using it today the following import statement started throwing error. 

'from evidently.dashboard import Dashboard'

The error is : 
![image](https://user-images.githubusercontent.com/13672259/141222833-5b5910da-d96a-4567-9465-d6bb84c761d0.png)
 
When I revert to the older version using "pip install evidently==0.1.27.dev0" , the import works well again. 
Has the new version stopped supporting any particular python version?

Hi @Neel-Renavikar 

thank you for reporting this! 
We do some experiments with testing design, and we lost tests for python < 3.8 by mistake. 

Fixing the 🐞!


fixed in ```0.1.29.dev``` version
Repository: evidentlyai/evidently
Issue: grafana_monitoring_service example compatibility with v4.0
Root cause: Documentation Deficiency

Hello all,
I have been following the [prometheus-grafana integration example](https://github.com/evidentlyai/evidently/tree/v0.4.2/examples/integrations/grafana_monitoring_service) few months ago.
I was on v1.59.dev3. Now I check that evidently has been updated a lot since then.
So I wish to update and want to know if the mentioned example work for evidently v4.0 and above?
If not then what changes will I have to implement for same (the [code logic](https://github.com/evidentlyai/evidently/tree/v0.4.2/examples/integrations/grafana_monitoring_service/metrics_app) for calculating metrics) ?
Please guide and let me know if that example needs to be updated so I can contribute to it simultaneously.

Hi @that-rahul-guy! 

This is indeed an old example, with each is not currently maintained. We made several major changes to the library since - specifically, completely rewrote how metrics are computed, replacing `analyzers` for `metrics`. So, updating the example will require significant work.

If you are looking for an example of near real-time integration where you can send the predictions directly from the ML monitoring service, you can check out this new example with a collector service.
Docs: https://docs.evidentlyai.com/user-guide/monitoring/collector_service
Example code: https://github.com/evidentlyai/evidently/tree/main/examples/integrations/collector_service

![image](https://github.com/evidentlyai/evidently/assets/67064421/14a1584f-1a9f-4d6f-99d6-074aa39ab81e)

In this scenario, you self-host the Evidently monitoring UI instead of Grafana - to visualize the metrics over time. 
Quickstart for ML monitoring is here: https://docs.evidentlyai.com/get-started/tutorial-monitoring 

We’d recommend it as a suggested course of action since as a team, we are currently focused on this monitoring functionality - it gets all the attention and support :) 

If you are keen on using Grafana, you can look for an up-to-date example here: https://github.com/evidentlyai/evidently/tree/main/examples/integrations/postgres_grafana_batch_monitoring

However, it follows a different architecture compared to the old one: you run batch monitoring jobs, compute metrics using Evidently, log to Postgres, and visualize on a Grafana dashboard.
Repository: evidentlyai/evidently
Issue: What design pattern have you followed when starting with this repo?

Hi, I would love to know what kind of design pattern(s) are being followed, or were taken into account when starting with this repository. 
Thanks 

Hi @omerihtizaz, if you are looking to start with Evidently, here are the two QuickStart tutorials:

- Get Started with Tests and Reports https://docs.evidentlyai.com/get-started/tutorial
- Get Started with ML monitoring https://docs.evidentlyai.com/get-started/tutorial-monitoring

If you are looking for deployment patterns, you can browse the integrations folder: https://github.com/evidentlyai/evidently/tree/main/examples/integrations

Hello @elenasamuylova , 

I think he is asking more specific to SOILD design principles: https://www.freecodecamp.org/news/solid-design-principles-in-software-development/
 

Hello @elenasamuylova. 
I think @ketangangal is correct, I am looking more specifically to ask around which SOLID design principles did you guys follow when starting with the development of Evidently. 
Thanks.
Repository: evidentlyai/evidently
Issue: Issue with ipynb_checkpoints on JupyterLab
Root cause: Validation Gap

Reported on Discord - minor issue to look at.

**Description:**

When I open `snapshot.json` inside `snapshots/` on JupyterLab, it creates `.ipynb_checkpoints/` and it raises this error and freezes at loading when accessing Evidently ui "ValueError: badly formed hexadecimal UUID string". Nothing crucial, remove .ipynb_checkpoints, refresh, and it works fine.
Repository: evidentlyai/evidently
Issue: Data drift volume issues. Any workaround ?
Root cause: Algorithmic Error

Hello,

I'm currently using Evidently to perform a data drift analysis on my dataset. The dataset has a total shape of (356,251 rows, 797 columns) for both reference and current data.

When I execute the Run() function in Evidently, it seems to run indefinitely. To give you an idea of the issue:

When I analyze 50 columns, it takes approximately 3 minutes.
However, when I increase the number of columns to 100, the process takes about 23 minutes.
I'm wondering if there's a workaround for this situation. One idea I had is to break down the analysis into smaller chunks, perhaps 50 columns at a time, and then merge the results into a single comprehensive report.

Additionally, having a progress bar feature would be extremely helpful to monitor the analysis's progress, especially in cases where it takes a significant amount of time.

Any guidance or suggestions would be greatly appreciated.

Thank you 

True when try to do column level metric test, it takes lot of time if you have more columns .
  Try : 
  1. feature selection 
  2. PCA


@YassineR Datadrift test time depends a lot on
- which test you're using
- type of column (numerical or categorical)
- structure of data, and particularly number of unique values for cat. columns

So it's possible that your columns are just different.
To check this you can run test separately for each column and measure the time.
Repository: evidentlyai/evidently
Issue: Reports and Testsuite should show , distance metric name instead of just distance. 
Root cause: Documentation Deficiency

Hello @emeli-dral 

I have used multiple metrics or Tests to check drift , i am getting distance as a metric instead of the method being used in as_pandas method.

As can been seen 
![image](https://github.com/evidentlyai/evidently/assets/40850370/a2a03fc3-ec20-4ec1-a329-f3e7d8935a64)

**expected would to get distance metric name.**

FYI @mike0sv 


also ref and current data columns not required in the generated report !