zhiminy/EvalEng huggingface dataset:
Repository: evidentlyai/evidently
Issue: BinaryClassification with name != "default" crashes
Root cause: Validation Gap

I don't know if this is a bug or not supposed to be implemented. But I'll ask the question here.

The premise is that I have a model that trains in more than one target. So I want to pass more than one `BinaryClassification` class to my data_definition.
Given the signature is `classification: List[Classification] | None`, I'd assume this is intended as a feature.

If I don't define a unique `name` for `BinaryClassification`, it crashes on `report.run` because it's not unique. 

```python
>>>   raise ValueError("More than one classification with id {}".format(classification_id))
>>>ValueError: More than one classification with id default
```

However, if I _do_ define a name, it doesn't work like my example below.

```python
import pandas as pd
from evidently import Dataset, DataDefinition, Report, BinaryClassification
from evidently.metrics import Accuracy

data = pd.DataFrame({
    "id_col": [1, 2, 3],
    "actual_1": [10.0, 20.0, 30.0],
    "predicted_1": [11.0, 19.0, 29.0],
    "actual_2": [10.0, 20.0, 30.0],
    "predicted_2": [11.0, 19.0, 29.0],
})
classification_ = [BinaryClassification(target="actual_1", prediction_labels="predicted_1", name="regression_1"), BinaryClassification(target="actual_2", prediction_labels="predicted_2", name="regression_2")]
data_definition = DataDefinition(id_column="id_col", classification=classification_)
dataset = Dataset.from_pandas(data, data_definition=data_definition)
report = Report(metrics=[Accuracy(target="actual_1", prediction="predicted_1"),])
report.run(dataset)

>>>     raise KeyError(key) from err
>>> KeyError: None
```

I think I traced it to this function `_default_input_data_generator`, that hardcodes the name to `default`. So when running the report it tries to slice a DataFrame with none.


Is there "BinaryClassification" library from evidently? i didn't saw any library matrices like this in last version!
Repository: evidentlyai/evidently
Issue: Installation of Extras
Root cause: Documentation Deficiency

The `contributing.md` guidelines suggest to install the dev extras before running tests, but the tests also depend on the `spark` and `llm` extras. I ran the tests in a virtual environment with only the `dev` extras:
```
ERROR tests/test_torch_numpy.py
ERROR tests/spark/metrics/test_data_drift.py
```

Created a PR with suggested amendments https://github.com/evidentlyai/evidently/pull/1674
Repository: evidentlyai/evidently
Issue: Support current data None by default and good error message if reference data is None
Root cause: Unimplemented Feature Gap

Make a case with `dashboard.calculate(some_data)` works fine (calculate results if all analyzers are OK with one dataset)

Add tests for the case.

For case `dashboard.calculate()` or `dashboard.calculate(None)` return a value error. And add tests for the cases also.
Repository: evidentlyai/evidently
Issue: Dependency Issue: 'sklearn'
Root cause: Environment Incompatibility

Hi, Evidently Team! Thanks for the great package! Very very useful.

I had an issue installing `evidently==0.1.14.dev0`:

```python
Traceback (most recent call last):
  File "main.py", line 8, in <module>
    from evidently.dashboard import Dashboard
  File "/usr/local/lib/python3.6/site-packages/evidently/dashboard/__init__.py", line 4, in <module>
    from .dashboard import Dashboard
  File "/usr/local/lib/python3.6/site-packages/evidently/dashboard/dashboard.py", line 17, in <module>
    from evidently.tabs.base_tab import Tab
  File "/usr/local/lib/python3.6/site-packages/evidently/tabs/__init__.py", line 8, in <module>
    from .classification_performance_tab import ClassificationPerformanceTab
  File "/usr/local/lib/python3.6/site-packages/evidently/tabs/classification_performance_tab.py", line 8, in <module>
    from evidently.widgets.class_ref_quality_metrics_widget import ClassRefQualityMetricsWidget
  File "/usr/local/lib/python3.6/site-packages/evidently/widgets/class_ref_quality_metrics_widget.py", line 11, in <module>
    from sklearn import metrics
ModuleNotFoundError: No module named 'sklearn'
```

I've already solved this by installing `scikit-learn==0.23.2`.

So, I don't know if this is a real bug and scikit needs to be a dependency of evidently or not. It's up to you!

:smile: 

Hi @lfpelison , thank you for pointing this out! 
This is definitely a bug, will be fixed in the next package.

fixed in ```evidently==0.1.15.dev0```
Repository: evidentlyai/evidently
Issue: Pandas extra feature `[parquet]` is only available as of version v2
Root cause: Configuration Error

Hi Evidently AI team,

I found an issue in the definition of the dependencies for `evidently>=0.4.11`. In #885 by @mike0sv, the definition of `pandas>=1.3.5` was updated to `pandas[parquet]>=1.3.5` so that `pyarrow` was not a separate dependency. However, this extra feature was only added to `pandas` as of version 2.0.0. As such, when using `pandas<2`, this leads to an error when checking the installed dependencies:

<img width="945" height="183" alt="Image" src="https://github.com/user-attachments/assets/16bfbabc-f5c8-4ddf-b954-04909fa7b256" />

As you can see here in version 1.5.3 (last 1.x release), the extra is not yet there: https://github.com/pandas-dev/pandas/blob/2e218d10984e9919f0296931d92ea851c6a6faf5/setup.cfg#L53C1-L57C23. In version 2.0.0, it is there: https://github.com/pandas-dev/pandas/blob/v2.0.0/pyproject.toml#L66.

As such, would it be possible to either revert this change or to handle this more smartly, that either `pandas[parquet]>=2.0.0` or `pandas>=1.3.5,<2 && pyarrow` are required?

Thanks! :)
Repository: evidentlyai/evidently
Issue: GroupBy(ValueDrift) display_name not shown properly
Root cause: Algorithmic Error

Hi,

When I use

```
report = Report([
    GroupBy(MaxValue(column="column1"), "column2"),
    DataDriftPreset(),
    RegressionPreset()
],
include_tests=True)
```
the resulting report displays properly, "Maximum value of 'column1' group by 'column2' for label: 'abc'"

On the other hand, when I use 

```
report = Report([
    GroupBy(ValueDrift(column="column1"), "column2"),
    DataDriftPreset(),
    RegressionPreset()
],
include_tests=True)
```


the resulting report displays just "Drift in column 'column1'".

Could you fix this issue? Thank you for developing such a great framework.
Repository: evidentlyai/evidently
Issue: Grafana dashboard detect data drift even though both reference and production data are same
Root cause: Documentation Deficiency

I was experimenting with Evidently's Real-time ML monitoring with Grafana and getting data drift detection even though when both reference and production datasets are exactly same (production data is just the copy of reference data).

**Please find the below example:**
![image](https://user-images.githubusercontent.com/24756965/155931962-c088b5f7-5961-4f67-9518-302431af98d4.png)

Here is the preview of **Reference data**
![image](https://user-images.githubusercontent.com/24756965/155932292-2c1db483-3fca-4133-abb5-5863ddb0e1a2.png)


Here is the preview of **Production data**
![image](https://user-images.githubusercontent.com/24756965/155932150-dbdb63a7-ad23-4b68-aaa8-04bef79030d2.png)



Hello, thank you for the question.

Looks like the drift depends on data window size and season changes in Bike dataset.

I'm going to check the case with the same reference and production/current datasets and will come back with more details and explanations.

Hello @Tapot,
Thanks for the reply 👍
Actually, everything is working fine when I am using the default Bike Dataset. But when I replace this data with other datasets (for example this [Weather Data](https://www.kaggle.com/budincsevity/szeged-weather)), I am facing the data drift issue when both reference and production/current datasets are same.

Hi @alokrajg! 

I think that we really met the drift here. Weather data have strong seasonality, so the piece of the dataset can differ significantly from the whole data.

<img width="200" alt="image" src="https://user-images.githubusercontent.com/36808731/159704402-00bab01b-00d7-4944-be44-da831f03c26f.png">

<img width="220" alt="image" src="https://user-images.githubusercontent.com/36808731/159704647-e7adf0ef-8624-4bf6-8dac-48c8a60c22f0.png">

<img width="677" alt="image" src="https://user-images.githubusercontent.com/36808731/159704953-260aa1d5-bbc9-4f23-b46c-00022de65e8f.png">



Hi @0lgaF,
Thanks for the nice explanation :blush:
Now I got the reason for the data drift.
Repository: evidentlyai/evidently
Issue: Evidently latest: How to create a custom metric with value specified at creation time
Root cause: Documentation Deficiency

WHAT:
I'm trying to create a custom metric following the example in the documentation MaxMetric .
https://docs.evidentlyai.com/metrics/customize_metric

We want the current value of the metric to be specified as a constructor field. That is, rather than being a value derived from the dataset, the value is specified when the metric is created.

When we add the metric to the project the server UI show each point as a separate point with a different label name.
Locally it works fine, and show the plot as a line. But not in the server.

How to fix the code below, so the server show a single line like the MaxMetric example.
Or alternatively, how to use an existing Metric to do the same.
Is there any link to example you can share showing how to pass values to the metric to render on server?
Is there any repository/folder with community contributed custom Metrics?!

Thanks!

```
class CustomMetric(SingleValueMetric):
    column: str
    value: float

    def _default_tests(self) -> List[BoundTest]:
        return [eq(0).bind_single(self.get_fingerprint())]

    def _default_tests_with_reference(self) -> List[BoundTest]:
        return [eq(Reference(relative=0.1)).bind_single(self.get_fingerprint())]


class CustomMetricImplementation(SingleValueCalculation[CustomMetric]):
    values = []
    
    def calculate(self, context: Context, current_data: Dataset, reference_data: Optional[Dataset]) -> SingleValue:
        value = float(self.metric.value)
        self.values.append(value)
        result = self.result(value=value)
        figure = line(self.values)
        result.widget = [plotly_figure(title=self.display_name(), figure=figure)] 
        return result

    def display_name(self) -> str:
        return f"{self.metric.column}"
```

Hi, @jsimao71!

Thanks for your question!
From what I can see, your sample code looks mostly correct, with just a small detail to clarify:
- The custom metric takes column and value as parameters.
- In the implementation, the metric simply returns value — which means it will always produce that fixed result.
- The metric also customizes the widget to show a line graph with a single point. That part might not work as expected.

It seems the issue may be more about how the widget is displayed in the report rather than the metric itself.
Could you tell me a bit more about what you’d like the metric to show in the report?
That will help in suggesting the right approach.
Repository: evidentlyai/evidently
Issue: ModuleNotFoundError: No module named 'evidently.dashboard' with 0.7.14 - python 3.10.4
Root cause: Unimplemented Feature Gap

Dear,
I`ve installed the 0.7.14 version and I see issues in the modules present in the installation.
In particular I don`t see **'evidently.dashboard'**. 

Typing dir(evidently), I got:

['BinaryClassification', 'ColumnType', 'DataDefinition', 'Dataset', 'LLMClassification', 'MulticlassClassification', 'Recsys', 'Regression', 'Report', 'Run', '__all__', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', '__version__', '_jupyter_nbextension_paths', '_pydantic_compat', '_registry', '_version', 'compare', 'core', 'errors', 'legacy', 'llm', 'metrics', 'nbextension', 'presets', 'pydantic_utils', 'sdk', 'tests', 'ui', 'utils', 'version_info']

Could someone help me with this issue?
Thanks in advance.
Regards.
Repository: evidentlyai/evidently
Issue: grafana_monitoring_service does not work
Root cause: Configuration Error

Hi Evidenty team, I tried to use the example for real time monitoring and I went through your explanation. I had all three files that you mentioned in step 4, but  there was no pre-built Grafana dashboard, and Prometheus was empty.  I waited for a while, but nothing happened. Would you please let me know what the problem is? Thanks.

Hi @Amirshirazi-tech,
Thank you for opening the issue.

We believe the reason might be in the order of the services starting. We performed some test internally and figured out that sometimes Prometheus and Grafana starts before Evidently service. In this scenario service often cannot connect to Prometheus and send data there.  We now work to create a script that will simplify the example run (and fix the order of services launching).

Currently, I can suggest launching services in the right order manually. I mean, first launch evidently service (docker compose up evidently_service), then launch Prometheus and finally Grafana the same way. Hopefully, it should help.


Hi @emeli-dral,
Thank you  for your response. I did, but nothing happened. Even I started from beginning and went step by step, but I had the same issue. The Evidently service works, but Prometheus and Grafana not.

Hi @Amirshirazi-tech ,
I just wanted to mention that we reworked the example of the grafana integration heavily and automated a lot of launching steps.
I believe it worth try now (if it is still relevant of course :) ), hopefully the example should work straight away.
Repository: evidentlyai/evidently
Issue: Bug: Valid report&run metadata leads to `Report` tab being bricked with Server error 500 on evidently server/dashboard
Root cause: Interface Contract Mismatch

# Issue

When I attach meta data to `Report` objects and/or the `Report`'s `run` method (so the resulting `Snapshot` inherits the merged metadata), it breaks the dashboard's ability to parse those reports on the server, giving:

<img width="2503" height="410" alt="Image" src="https://github.com/user-attachments/assets/5f129c0d-b05c-4ecd-a063-d7d51b5aafd0" />

This seems to happen whenever i use[ non-string type values in the metadata that are officially supported as per here](https://github.com/evidentlyai/evidently/blob/d0506f6e63353b5d899fd93d8d04697ccb35045b/src/evidently/legacy/suite/base_suite.py#L475), i.e. this will not work:

```
metadata={'invalid-list':['test'],'invalid-dict':{'test-key':'test-value'}}
```

This works

```
metadata={'valid-string':'test'}
```

Oddly, this issue does NOT apply to datasets. The dashboard/server CAN resolve dict and list type values in dataset metadata and displays those correctly (if somewhat oddly with additional indices for each list entry):

<img width="2503" height="410" alt="Image" src="https://github.com/user-attachments/assets/7d6f066c-4c4b-4f66-9fed-911344f3a572" />

# System

Python 3.10.18
Evidently (client & server): 0.7.16

@SebastianScherer88 are you or anyone working on this issue?

Datasetmetadata contains the metadata: Dict[str, MetadataValueType] = {}
https://github.com/evidentlyai/evidently/blob/15b69bfc3419bde6499bfc5c0f7f908aa2cecb6f/src/evidently/ui/service/datasets/metadata.py#L64

where as SnapshotMetadataModel contains the   metadata: Dict[str, str]
https://github.com/evidentlyai/evidently/blob/15b69bfc3419bde6499bfc5c0f7f908aa2cecb6f/src/evidently/sdk/models.py#L73

we have to change the SnapshotMetadataModel.metadata to Dict[str, MetadataValueType] also.
Repository: evidentlyai/evidently
Issue: Mini chart in monitoring dashboard mixes up ref and current for categorical vars
Root cause: Algorithmic Error

```
import pandas as pd
from evidently.dashboard import Dashboard
from evidently.dashboard.tabs import DataDriftTab
from evidently.pipeline.column_mapping import ColumnMapping

ref = pd.DataFrame({'a':[1,2,3,4], 'b':['A','A','B','A']})
cur = pd.DataFrame({'a':[4,1,3,1], 'b':['B','A','B','A']})

report = Dashboard(tabs=[DataDriftTab()])
column_mapping = ColumnMapping()
column_mapping.numerical_features = ['a']
column_mapping.categorical_features = ['b']

report.calculate(
    ref,
    cur,
    column_mapping=column_mapping,
)

report.save('monitoring.html')
```

![2022-03-08_14-06-57](https://user-images.githubusercontent.com/36078019/157308706-e6981104-0a13-45ee-ba23-eb5ebee90796.png)

For the categorical vars, in the mini  view charts, the reference distribution is mixed up with the current.

**Version 0.1.45.dev0**


Hi @FelixDent, thanks for reporting this!

fixed in 543711483baa8e27dfbc81a6575d553ab57cc897
Repository: evidentlyai/evidently
Issue: IndexError in infer_column_type when column contains only null values
Root cause: Validation Gap

# `IndexError` in `infer_column_type` when column contains only null values

## Description

When creating a `Dataset` from a pandas DataFrame that contains a column with all null/NaN values, Evidently raises an `IndexError: single positional indexer is out-of-bounds`. This occurs because `infer_column_type` attempts to access `.iloc[0]` on an empty series after dropping NA values.

## Steps to Reproduce

```python
import pandas as pd
from evidently import Dataset

# Create DataFrame with an all-null column
df = pd.DataFrame({
    "valid_column": [1, 2, 3, 4, 5],
    "all_null_column": [None, None, None, None, None]
})

# This raises IndexError
dataset = Dataset.from_pandas(df)
```

## Expected Behavior

Evidently should gracefully handle columns that are entirely null, either by:
- Skipping them during type inference
- Assigning a default column type
- Raising a descriptive error message indicating which column is problematic

## Actual Behavior

```
IndexError: single positional indexer is out-of-bounds
```

Full traceback points to `evidently/core/datasets.py` line 653:

```python
def infer_column_type(column_data):
    # ...
    if column_data.dtype.name == "object":
        without_na = column_data.dropna()
        if isinstance(without_na.iloc[0], str) and isinstance(without_na.iloc[-1], str):  # <- crashes here
```

## Root Cause

In `infer_column_type`, when the column dtype is `object`, the function drops NA values and then immediately tries to access `iloc[0]` and `iloc[-1]` without checking if the resulting series is empty.

## Suggested Fix

Add an empty check before indexing:

```python
if column_data.dtype.name == "object":
    without_na = column_data.dropna()
    if len(without_na) == 0:
        return ColumnType.Unknown  # or handle appropriately
    if isinstance(without_na.iloc[0], str) and isinstance(without_na.iloc[-1], str):
        # ...
```

## Workaround

For anyone encountering this issue, you can filter out all-null columns before passing to Evidently:

```python
df_clean = df.dropna(axis=1, how='all')
dataset = Dataset.from_pandas(df_clean)
```

### Tested on current `main` branch

I tested this issue on the current `main` branch (`v0.7.18`) using the exact reproduction steps provided.

```python
import pandas as pd
from evidently import Dataset

df = pd.DataFrame({
    "valid_column": [1, 2, 3, 4, 5],
    "all_null_column": [None, None, None, None, None],
})

Dataset.from_pandas(df)
```
Based on this test, the issue does not reproduce on the current `main` branch and appears to be already fixed upstream.
Repository: evidentlyai/evidently
Issue: embedding drift with domain classifier
Root cause: Documentation Deficiency

Hi Hi, 

I would like to know if the embedding drift can still use domain classifier to detect? I saw the thread [here](https://github.com/evidentlyai/evidently/issues/706)  but the link is not available anymore. I tried to 
- shape the embedding data with one element of the embedding one column as showed [here](https://docs.evidentlyai.com/docs/library/overview#embeddings); 
- currently I was using `embedding_report = Report([DataDriftPreset(num_method="wasserstein", num_threshold=0.1)])` to test out but of course when I try to push the report to our self-host evidently instance, it gave me error `evidently.errors.EvidentlyError: Request Entity Too Large` probably due to the number of columns from the embeddings

Question:
1. after the migration, where can I find the docs as shown in the issue link?
2.  is there a classification method implemented for detecting embedding drift(embeddings, not raw text column like described [here](https://docs.evidentlyai.com/metrics/explainer_drift#text-data-drift)) (I did saw [there]"(https://github.com/evidentlyai/evidently/blob/main/src/evidently/presets/drift.py#L27) is embedding_drift_method parameter but it is not clear how we use that)?
3. is there other ways to detect embedding drift?

thank you
Repository: evidentlyai/evidently
Issue: Regression Performance Tab has ZeroDvisionError when target values have zero values
Root cause: Validation Gap

I have a dataset with some zeros values in target and this line is throwing error the Zero division zero
```
abs_perc_err = list(map(lambda x : 100*abs(x[0] - x[1])/x[0], zip(reference_data[target_column], reference_data[prediction_column])))
ZeroDivisionError: float division by zero
```
I was thinking since the datasets are already in DataFrame can't we do something like this
```
abs_perc_err = ((reference_data[target_column]-reference_data[prediction_column])/reference_data[target_column]*100).abs()
```
and the zero target values will now be returned with -inf instead

Hi @yanhong-zhao-ef, thank you for pointing this out, this is a bug!
This is an old piece of code, it will be replaced by vectorised version in the next package.

fixed in ```evidently==0.1.15.dev0```
Repository: evidentlyai/evidently
Issue: grafana monitoring service shows no data for regression performance
Root cause: Documentation Deficiency

I have installed and run grafana using below instructions
`https://github.com/evidentlyai/evidently/tree/main/examples/integrations/grafana_monitoring_service`
I used bike sharing data, but I see data drift dashboard only, 
How can I enable regression performance dashboard on grafana.

![image](https://user-images.githubusercontent.com/84674037/159204083-e806cfc2-f071-4ec8-ace6-db62ecd1e221.png)

![image](https://user-images.githubusercontent.com/84674037/159206183-c034c57b-94cc-4096-ae91-534f6e85c838.png)

![image](https://user-images.githubusercontent.com/84674037/159206237-52f1a7f6-62fd-470d-b3b0-acae9ec263c2.png)


its working fine now
Repository: evidentlyai/evidently
Issue: How to monitor multiple models using grafana

Hi, 
I have multiple models in my project and want to monitor all on grafana.
but I am unable to do so. 
The below link doesn't tell how to setup multiple model monitoring.

> https://github.com/evidentlyai/evidently/tree/main/examples/integrations/grafana_monitoring_service

Is there any other document/guide I can make use of

Hi!

The current example implements a separate application logic for sending data to Prometheus and then on to Grafana for one model.

We have not a guide for this case yet(.

If you want to monitor multiple models, you can change `MonitoringService` and add separate labels for each model and filters for Grafana dashboards by the labels.

Can you describe what case do you want to implement  - different Grafana dashboards for each model, compare models both other? If we better understand the issue, maybe we can add a guide for it or update the current example.

@Tapot , Thanks for responding!
Let me tell you in a simpler way
Currently you are providing 3 dashboards for grafana- regression, classification and data drift.

let's say I have 2 regression models to monitor. how can I create 2 regression dashboards. How Evidently will send metrics to prometheus for these 2 models separately?
I am not able to find a way to create multiple dashbords.

Hi @debimishra89!

I know it's been a while, but we recently released an example that does exactly what you describe. You can try out the updated example here: https://github.com/evidentlyai/evidently/tree/main/examples/integrations/grafana_monitoring_service