zhiminy/EvalEng huggingface dataset:
Repository: evidentlyai/evidently
Issue: TypeError: multiple bases have instance lay-out conflict
Root cause: Interface Contract Mismatch

When I try to import something from `evidently.future` I got the error:

```bash
Traceback (most recent call last):
  File "/home/.venv/lib/python3.13/site-packages/evidently/__init__.py", line 3, in <module>
    from . import _registry
  File "/home/.venv/lib/python3.13/site-packages/evidently/_registry.py", line 3, in <module>
    from .future import _registry as future_registry
  File "/home/.venv/lib/python3.13/site-packages/evidently/future/__init__.py", line 1, in <module>
    from . import _registry
  File "/home/.venv/lib/python3.13/site-packages/evidently/future/_registry.py", line 3, in <module>
    from evidently.future.metric_types import BoundTest
  File "/home/.venv/lib/python3.13/site-packages/evidently/future/metric_types.py", line 40, in <module>
    from evidently.tests.base_test import TestStatus
  File "/home/.venv/lib/python3.13/site-packages/evidently/tests/__init__.py", line 6, in <module>
    from .classification_performance_tests import TestAccuracyScore
  File "/home/.venv/lib/python3.13/site-packages/evidently/tests/classification_performance_tests.py", line 12, in <module>
    from evidently.metrics.classification_performance.classification_dummy_metric import ClassificationDummyMetric
  File "/home/.venv/lib/python3.13/site-packages/evidently/metrics/__init__.py", line 24, in <module>
    from .data_drift.data_drift_table import DataDriftTable
  File "/home/.venv/lib/python3.13/site-packages/evidently/metrics/data_drift/data_drift_table.py", line 17, in <module>
    from evidently.metrics.data_drift.base import WithDriftOptions
  File "/home/.venv/lib/python3.13/site-packages/evidently/metrics/data_drift/base.py", line 31, in <module>
    class WithDriftOptions(WithDriftOptionsFields, Metric[T], Generic[T], abc.ABC):
    ...<4 lines>...
            return self._drift_options
  File "/home/.venv/lib/python3.13/site-packages/evidently/pydantic_utils.py", line 71, in __new__
    res = super().__new__(mcs, name, bases, namespace, **kwargs)
  File "/home/.venv/lib/python3.13/site-packages/pydantic/v1/main.py", line 282, in __new__
    cls = super().__new__(mcs, name, bases, new_namespace, **kwargs)
  File "<frozen abc>", line 106, in __new__
TypeError: multiple bases have instance lay-out conflict
```

My versions:

Pydantic: 2.10.6
Evidently: 0.6.5

Anything that I'm doing wrong?

I have faced the same error when trying to use evidently with Python 3.13. Using an older Python version (in my case 3.10) solved the issue


Same problem with Pydantic: 2.10.6, Evidently: 0.6 and Python 3.13. Working with python 3.10 as mentioned above. 

I'll try. Thanks
Repository: evidentlyai/evidently
Issue: Integrating Evidently's LLM Judge Template for Copilot Agent based on azure ai search
Root cause: Documentation Deficiency

Hi team,

I'm exploring ways to evaluate our LLM-generated answers using Evidently LLM Judge Template. Currently, we rely on Azure Cognitive Search/ai search as our source of ground truth, but I'm interested in adding an evaluation layer. 

Could you provide guidance on the following?

1) What is the best way to integrate Evidently's LLM Judge Template with our current setup for post-hoc evaluation of LLM responses?
2) How can we align the evaluation outputs from Evidently with our ground truth data obtained from Azure Cognitive Search?

Hi @KishoreKunalPraval,  

If you’re using Evidently as a Python library, you can run evaluations as Python scripts:  

### Evaluation  
- Import the dataset you want to evaluate as a pandas DataFrame.  
- Invoke the chosen LLM judge evaluator over your dataset using `descriptors`.  
- Summarize the results using the `Report`.  

Check the basic tutorial here: [Quickstart LLM](https://docs.evidentlyai.com/quickstart_llm)  

### Comparing evaluations against ground truth  
If by "aligning," you mean evaluating the quality of LLM-generated labels against yours, you can:  
- Import the dataset that contains your ground truth labels.  
- Generate the LLM judgments as shown above.  
- Treat the LLM judgments as "predictions" and the true labels as "targets," then evaluate the classification quality.  

Here’s a tutorial on how to create a custom LLM judge and evaluate it against your labels as a classification problem: [LLM Judge Tutorial](https://docs.evidentlyai.com/examples/LLM_judge)  

Let me know if you have more specific questions. You can also ask in our [Discord community](https://discord.gg/xZjKRaNp8b).
Repository: evidentlyai/evidently
Issue: `DriftDriftPreset()` takes 8 hours for data with 3M rows and 1k columns
Root cause: Algorithmic Error

Originally my reference data have 10M rows and current data have 1M rows, and I am computing data drift using evidently - but the process gets killed (possibly due to OOM).
I then downsample my reference data to 30% - so reference data have 3M rows and current data have 0.3M rows - but the process takes 8 hours to run.
Note: There are 1k columns.
My code:
```python
from evidently.metric_preset import DataDriftPreset
from evidently.report import Report
 
report = Report(metrics=[DataDriftPreset()])
report.run(reference_data=reference_data, current_data=current_data)

drift_report = report.as_dict()
is_data_drift_present = drift_report["metrics"][0]["result"]["dataset_drift"]

drifted_features = [
            feature_name
            for feature_name, drift_info in drift_report["metrics"][1]["result"][
                "drift_by_columns"
            ].items()
            if drift_info["drift_detected"]
        ]
```


Is there a way to fasten data drift computation to reduce runtime from 8hrs to 2hrs? 

Would it be worthwhile to look at parallel execution at column level? Or can I just specify additional in evidently function call to fasten it?

Hi @iamharshit,

For data drift detection, you can take even a smaller sample. This is often effective because you typically don’t need to analyze all records to identify distribution shifts.

Another option is to generate per-column checks separately using column drift detection. You can then extract individual drift scores or test results for each column in Python. This allows you to split the process into multiple smaller steps instead of running everything at once.

It looks like you’re using the older Evidently API, where the metric is called `ColumnDriftMetric()`.
In the newer version, the equivalent is `ValueDrift()`:
👉 https://docs.evidentlyai.com/metrics/all_metrics#data-drift
Repository: evidentlyai/evidently
Issue: Reference Data Not Accessible to Metric Inside GroupBy
Root cause: Configuration Error


## Description
When using the `GroupBy` component, metrics enclosed within it cannot access the reference dataset, causing an error.

## Current Behavior
Currently, the following works perfectly:

```python
report = Report(
    metrics=[
        ValueDrift(column="rating"),
        group_by.GroupBy(
            MaxValue(column="rating"), 
            column_name="month",
        )
    ],
)
eval_result = report.run(current_dataset, reference_dataset)
```

When examining the report from this example, it's evident that no reference data is being used for the metrics inside the GroupBy.

But when trying to use a drift metric inside `GroupBy`, it fails:

```python
report = Report(
    metrics=[
        ValueDrift(column="rating"),
        group_by.GroupBy(
            ValueDrift(column="rating"), 
            column_name="month",
        )
    ],
)
eval_result = report.run(current_dataset, reference_dataset)
```

## Error Message
```
ValueError: An empty column 'rating' was provided for drift calculation in the reference dataset.
```

## Expected Behavior
The reference dataset should be accessible to all metrics, including those inside a `GroupBy`, allowing for grouped drift calculations.

## Environment
- Evidently version: 0.6.6

Hi @jeandre-db, 

You should be able to use the drift calculation by group as the following.

```
report = Report([
    GroupBy(ValueDrift(column="Salary"), "Department"),
])
snapshot = report.run(data, data)
snapshot.dict()
```

this will compute separate value drift checks for Salary column by Department labels.
Repository: evidentlyai/evidently
Issue: Fix Scipy Deprecation Warning
Root cause: External Dependency Breakage

In our organisation we want to start using Evidently. However, we have strict tests on Deprecation warnings not being allowed. Currently, there is a Deprecation warning within the Evidently package around SciPy. It would be much appreciated if this could be tackled. 

## Error message
<img width="939" alt="Image" src="https://github.com/user-attachments/assets/d7a88e1e-69d2-40c7-a1ec-87dbeb82ed51" />
Repository: evidentlyai/evidently
Issue: Issue in handling real-time monitoring
Root cause: Unimplemented Feature Gap

Hi Team,

In Evidently previous version (0.4.4), Collector service was provided to handle real time Monitoring. But in new version of Evidently (0.6.6), Collector service is not offered. Is there any other way you are offering to handle Realtime monitoring?

Thanks,
Priya Saha

@Priya3Saha Hi! 
Here is the answer from [discord channel](https://discord.com/channels/815991896916361276/1430270689516191897) of the project (I am not a maintaner, just had the same question). 

> Collector was deprecated last year and we don't plan to bring it back at the moment. 
> 
> However, effectively it was executing batch Evidently Report calculation upon specified frequency keeping the data in memory - you can create a similar script yourself.
> 
> If you are using Evidently Cloud, you can use tracing to collect raw data and use built-in task scheduler to run report computation in the web service.
Repository: evidentlyai/evidently
Issue: Retrieving Reports from Evidently UI
Root cause: Documentation Deficiency

Dear Evidently AI team,

I have the self hosted Evidently UI service running on a remote server and I'm trying to retrieve reports from it. Until now, I could retrieve the project by ID using the following code:
```
workspace = RemoteWorkspace(SERVER_URI)
loaded_project = workspace.get_project(PROJECT_ID)
```
In addition, I could list all reports in that project (using the REST API though, couldn't figure out how to do it with python):
```
curl -X GET "http://SERVER_URI/api/projects/PROJECT_ID/reports/"
```
However, I couldn't find out how to retrieve a report by its ID. When sending the request `curl -X GET "http://SERVER_URI/api/projects/PROJECT_ID/reports/REPORT_ID"` I get a 404 Not Found error. And when sending the request `curl -X GET "http://SERVER_URI/api/projects/PROJECT_ID/REPORT_ID"` I get a 405 Method Not Allowed error.

So my question to you is: How can I retrieve the report data from the server? (Preferably using REST API). Unfortunately I couldn't find this out anywhere in the documentation.

Thanks a lot!

Hi @mahdyshabeeb for now we provide 2 options to download report: as html or as json


You can try it on our [demo](https://demo.evidentlyai.com/) deployment (replace vars values for your case)

```bash
SERVICE_URL="https://demo.evidentlyai.com"
PROJECT_ID="c172570b-0111-4d57-a9fd-580b37ff5901"
REPORT_ID="b30d8202-feb4-427f-8747-e2bb6f1ba57d"
FORMAT_JSON="json"
FORMAT_HTML="html"

# get JSON
curl -X 'GET' \
  "$SERVICE_URL/api/projects/$PROJECT_ID/$REPORT_ID/download?report_format=$FORMAT_JSON"

# get HTML
curl -X 'GET' \
  "$SERVICE_URL/api/projects/$PROJECT_ID/$REPORT_ID/download?report_format=$FORMAT_HTML"

```

P.S. you can see full service api in [swagger](https://demo.evidentlyai.com/schema/swagger)
Link for your deployed service api located at `https://SERVER_URI/schema/swagger`

Thanks a lot @DimaAmega ! This is what I was looking for and it's working perfectly.
However, I couldn't see this request in swagger. I would expect to see something like /api/projects/{project_id}/{report_id}/download
Am I missing something here?

I guess you miss, yes. Follow this [link](https://demo.evidentlyai.com/schema/swagger#/default/ApiProjectsProjectIdSnapshotIdDownloadGetSnapshotDownload)

And for your deployment url is: `https://SERVER_URI/schema/swagger#/default/ApiProjectsProjectIdSnapshotIdDownloadGetSnapshotDownload`



Got it. Thank you very much for your help!
Repository: evidentlyai/evidently
Issue: ModuleNotFoundError: No module named 'evidently.test_suite'
Root cause: Documentation Deficiency

In the new version 0.7.0 when trying to import test_suite I run into the error:

from evidently.test_suite import TestSuite
ModuleNotFoundError: No module named 'evidently.test_suite'

Is this module removed in this new version or is a bug?

Hi @jwilliamn,

Yes this module is no longer available starting from version 0.7.0 - you can use the new Evidently API https://docs.evidentlyai.com/faq/migration

If you want to use older API, fix the version to 0.6.7 or lower 

Thank you @elenasamuylova 
I had been wondering what the migration method was.

@vibhansh-mdlz Do you need any pointers - which Test Suite are you looking to replicate?

@elenasamuylova 
I am trying to replicate what in previous version was possible: 
suite.as_dict()['tests'], which gave the results in a dictionary

Now is supposed to be:
report.dict() or report.json
but it doesn't work because those attributes are not present in the Report class.

How can I convert the report to dict?

Specifically, I am dealing with TestColumnDrift (in previous version) and now I am trying with ValueDrift

Following through on this, I am running drift and stability reports. I see what looks like a fix by calling deprecated 'legacy' modules, e.g.
from evidently.legacy.test_suite import TestSuite
from evidently.legacy.test_preset import DataStabilityTestPreset

So, is this the preferred path or do you recommend installing 0.6.7 or older? Do I lose out on new features if I do the latter? Thanks!

Hi @ccfarah, for now these will remain equivalent options. However, please note that no new features will be added to the legacy modules, and they will eventually be removed from the codebase in a future release. To ensure stability, pinning a version is the safer choice.

Thanks @elenasamuylova. Makes sense. I think I will install the current version and import the legacy modules, so I can still access any of the newer features.
Repository: evidentlyai/evidently
Issue: Numpy 2.x support?
Root cause: External Dependency Breakage

Numpy 2.x has been out for a while now (since June 2024, I think) and they have a really useful migration guide [here](https://numpy.org/doc/stable/numpy_2_0_migration_guide.html). Any plans for evidently also to upgrade? We've found the strict requirements with Evidently make it hard to install things together esp since many other packages have upgraded to 2.x.

It seems you use it quite extensively in the repo, but we (ZenML) made the upgrade and found it not such a huge issue given the really nice upgrade document.
Repository: evidentlyai/evidently
Issue: ValueError: Cannot use ClassificationPreset without a classification configration
Root cause: Documentation Deficiency

Hi

When doing ClassificationPreset() with version 0.7.3 I am getting error `ValueError: Cannot use ClassificationPreset without a classification configration`. 

Here is my code:
```
report = Report(
        [ClassificationPreset()],
        include_tests="True")
my_eval = report.run(cur, ref)
```
and dataset sample:

probability | prediction
-- | --
0.396355 | 0
0.978325 | 1
0.129338 | 0
0.733420 | 0

Hi @jankogasic, 

This code does not show how you create `ref` and `cur` Dataset objects in Evidently - could you share this part of the code?



Here is the relevant part of the documentation 

How to create the Dataset
https://docs.evidentlyai.com/docs/library/data_definition#basic-flow

How to create Data Definition for classification
https://docs.evidentlyai.com/docs/library/data_definition#classification

I was using `pd.Dataframe` instead of `Dataset`, which was the case for older versions. 

Maybe you can add _prerequisites_ section [here](https://docs.evidentlyai.com/metrics/preset_classification), similar to what you did for some of the other pages in the documentation, which I find very useful.

Thank you
Repository: evidentlyai/evidently
Issue: [1404] EvidentlyError: Request Entity Too Large
Root cause: Configuration Error

> I am facing this same issue in version 0.7.3
> 
> ```
> ---------------------------------------------------------------------------
> EvidentlyError                            Traceback (most recent call last)
> Cell In[83], [line 1](vscode-notebook-cell:?execution_count=83&line=1)
> ----> [1](vscode-notebook-cell:?execution_count=83&line=1) workspace.add_run("0196a49d-bfa5-7de2-be6d-378aba5c2a6e", rep, include_data=False)
> 
> EvidentlyError: Request Entity Too Large
> ```
> 
> Also from above comments I came across , like `request_max_body_size = None` may solve the issue, but how to pass this ?
> A details steps on this would be helpful.
>  

 _Originally posted by @abhishek-pattern in [#1404](https://github.com/evidentlyai/evidently/issues/1404#issuecomment-2854061108)_

hello @tanay-pattern 

we fixed it in version 0.7.5

Thanks @DimaAmega
Repository: evidentlyai/evidently
Issue: MAE tests are missing in report
Root cause: Validation Gap

# Overview

Some of the regression metrics (`MAE` and `MeanError` at least) are not being executed when running a report.

# Example code

```python
import pandas as pd
from evidently.future.datasets import Dataset, DataDefinition, Regression
from evidently.future.metrics import MAE, RMSE
from evidently.future.report import Report
from evidently.future import tests

data = pd.DataFrame({
    "id_col": [1, 2, 3],
    "actual": [10.0, 20.0, 30.0],
    "predicted": [11.0, 19.0, 29.0]
})
regression_ = [Regression(target="actual", prediction="predicted")]
data_definition = DataDefinition(id_column="id_col", regression=regression_)
dataset = Dataset.from_pandas(data, data_definition=data_definition)
report = Report(
    [
        # 3 tests for RMSE, 1 test for MAE -> Expected: 4 tests in total
        RMSE(tests=[tests.lt(100), tests.gt(0), tests.not_eq(0)]),
        MAE(tests=[tests.lt(100)]),
    ],
)
snapshot = report.run(dataset).dict()
assert len(snapshot["tests"]) == 4, "Expected 4 tests, but got {}".format(len(snapshot["tests"]))

>>> assert len(snapshot["tests"]) == 4, "Expected 4 tests, but got {}".format(len(snapshot["tests"]))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
>>> AssertionError: Expected 4 tests, but got 3
```

# Versions

Evidently: 0.7.6
Python: 3.12.7


Hi @emacha, 

MAE metric returns two different values (mean error and std), so when you create a test you'd need to use a different parameter depending in the value you are testing:  `mean_tests` or `std_tests`. 

```python
# Updated imports

import pandas as pd

from evidently import Dataset, DataDefinition, Report, Regression
from evidently.metrics import MAE, RMSE
from evidently.tests import gt, not_eq, lt

# Data prep 
data = pd.DataFrame({
    "id_col": [1, 2, 3],
    "actual": [10.0, 20.0, 30.0],
    "predicted": [11.0, 19.0, 29.0]
})

regression_ = [Regression(target="actual", prediction="predicted")]
data_definition = DataDefinition(id_column="id_col", regression=regression_)

dataset = Dataset.from_pandas(data, data_definition=data_definition)

# Report

report = Report(
    [
        RMSE(tests=[lt(100), gt(0), not_eq(0)]),
        MAE(mean_tests=[lt(100)]), # use mean_tests or std_tests here
    ],
)
snapshot = report.run(dataset)

# Get output as dict
snapshot.dict()
``` 

@elenasamuylova Yeah that worked, thanks!

Do you think you could raise an exception in this case? It's a pretty large footgun.

Hi @elenasamuylova!
I’ve created a PR that raises an explicit exception when tests= is passed to MAE instead of mean_tests or std_tests. Happy to adjust for any suggested changes. Let me know your thoughts!
Repository: evidentlyai/evidently
Issue: Provider "gemini" gives unexpected error related to "vertex credentials"
Root cause: Documentation Deficiency

I was running through the latest AI Builders course (great job everyone!) and was having token errors with my OpenAI and Anthropic API keys, so I tried Gemini next. It gives a weird error, I don't know if this is a bug or not, but I think at the very least the documentation here could be improved. 

Here is my baseline, calling Gemini 1.5 using the `requests` module, it works as expected with my token:

<img width="858" alt="Image" src="https://github.com/user-attachments/assets/628f320f-5277-4c4e-920c-3086370e8b10" />

Then with the LiteLLM package:

<img width="1007" alt="Image" src="https://github.com/user-attachments/assets/c194b326-f6b6-490f-919c-8e70cea98535" />

Then I try using the "gemini" provider listing the model name. Based on what I could find in the repo and evidently docs this is the right way to do it, but get this error (notice the "vertex" credential error description):

<img width="1177" alt="Image" src="https://github.com/user-attachments/assets/b149098e-2dcc-493c-a88b-847ab8f46a5f" />

When I raised the error during the course, I was directed to this test notebook:

https://github.com/evidentlyai/evidently/blob/main/examples/future_examples/llm_providers.ipynb

So I tried using the `GeminiOptions` based on that notebook, but got the same error:

<img width="1337" alt="Image" src="https://github.com/user-attachments/assets/c81cdd57-6c43-407b-a35b-10026397865c" />

**Further comments:**

1. From the code/docs it seems `gemini` is a separate provider to `vertex_ai`, also `GeminiOptions` is not a `LiteLLM`wrapper, so its strange to me that the error is showing vertex and litellm issues. 
2. The error it gives is huge, I can provide that full print out if needed, but its mostly what I'm showing here repeated many times.
3. Seems like there are two different approaches to `gemini` provider in the docs vs the example notebooks, adding some more clarity here would be really helpful!





Hi @jonbown, this (`GeminiOptions`) should be available in the version 0.7.7 - could you test if everything works correctly with the latest?

Thanks for the clarification @elenasamuylova I'm now trying with 0.7.9 and its working as expected!

@jonbown awesome, thanks for confirming!
Repository: evidentlyai/evidently
Issue: Limit the number of runs by timestamp
Root cause: Unimplemented Feature Gap

Hello,

I'm working with evidently 0.7.7.  I would like to remove runs older than the last seven days from a project, when I try to list the runs associated with the project with the code below I get an error message saying "NotImplementedError  # todo: snapshot api". Is there another way I can delete runs?

`workspace.list_runs(self.project.id) `