zhiminy/EvalEng huggingface dataset:
Repository: evidentlyai/evidently
Issue: Evaluating the classification quality of model
Root cause: Unimplemented Feature Gap

Mathematically, the classification quality of the model is known as Matthews Correlation Coefficient.

The metric is a true judge for the classification problems having an imbalanced set of classes.

https://en.wikipedia.org/wiki/Matthews_correlation_coefficient

### Model Accuracy
![model_accuracy](https://user-images.githubusercontent.com/74548577/113571851-bee3d480-9634-11eb-9d02-3667b9c548a5.png)

### Classification quality of model
![model_classification_quality](https://user-images.githubusercontent.com/74548577/113571881-c905d300-9634-11eb-9b25-e8d97185e188.png)

If we observe, the accuracy is not giving us a clear picture of the classification capability of the model.

Can this feature be added?

Hi @xsansha ,

It can indeed be useful for problems with imbalanced classes!

However, we try to minimise the number of plots for now. The goal is to provide maximum information in a single report without duplication. One of the concerns is not inflate the report in size too much (since all visualizations for now are stored in .html file). We had to vote out a few plots in a heated debate :)

The reason we excluded Matthews Correlation is that 1) it is mostly useful for imbalanced classification (especially binary), whereas we have a single report that works for all binary and multi-class alike 2) the metric itself is harder to interpret. We suggest to refer to Quality Metrics by Class to understand the variations in model performance per class.

That said, we understand some might prefer different plots for different use cases, and it is impossible to build universal "defaults". We are working on a service version of a tool where all the plots can be configured, and expect to include the full selection of options there!

Thanks @emeli-dral 

Hi @xsansha ,

I just wanted to mention that now you can design custom Metrics, Widgets and Tabs. Maybe you will find convenient to customize or create your ClassificationPerformanceTab, where you can remove all irrelevant visuals and add your own ones :)
Here is the blog where we introduced the customization functionality:
https://evidentlyai.com/blog/evidently-custom-tabs-widgets-and-metrics

And here you can see how to create a custom widget with a different statistical test
https://docs.evidentlyai.com/step-by-step-guides/report-customization/add-a-custom-widget-or-tab

If you need any help with specific code or implementation, please join our Discord https://discord.com/invite/xZjKRaNp8b support channel!
Repository: evidentlyai/evidently
Issue: n_bins Argument Not Used in _psi Function
Root cause: Validation Gap

The `n_bins` argument is defined in the `_psi` stat test function signature but is not utilized in the function's logic. This issue creates confusion for developers expecting the n_bins parameter to influence the calculation of PSI (Population Stability Index).

The `get_binned_data` function is called within _psi, and while it presumably requires the number of bins for binning the data, the `n_bins` argument is not used by it [(check here)](https://github.com/evidentlyai/evidently/blob/main/src/evidently/calculations/stattests/utils.py#L14-L39)

Suggested change in `get_binned_data` :

- Pass the n argument directly to `np.histogram_bin_edges` as the number of bins:
So [line 30 ](https://github.com/evidentlyai/evidently/blob/main/src/evidently/calculations/stattests/utils.py#L30) changes to:

`bins = np.histogram_bin_edges(pd.concat([reference_data, current_data], axis=0).values, bins=n)
`
Repository: evidentlyai/evidently
Issue: TargetByFeaturesTable does not save with JSON
Root cause: Configuration Error

When using `TargetDriftPreset`, the `Target (Prediction) Behavior By Feature` metric does not appear when saving report as html or json.

```
report = Report(metrics=[TargetDriftPreset()])
report.run(current_data=cur, reference_data=ref, column_mapping=column_mapping)
report.save_html('report.html')
report_json = report.json()
```
It seems to be the `Aggregated visuals in plots` feature that removes it. I'm aware the feature is suppose to reduce the size of the raw data, but is it suppose to remove the metric completely? When passing, `raw_data=True`, the html report shows the metric, but the JSON is still returning an empty dictionary. I also tried setting `include_render=True` for the save json method.

```
from evidently.options.agg_data import RenderOptions 
render = RenderOptions(raw_data=True)
report = Report(metrics=[TargetDriftPreset()], options=[render])
report.run(current_data=cur, reference_data=ref, column_mapping=column_mapping)
report_json = json.loads(report.json(include_render=True))
print(report_json['metrics'][4])
```
OUT:

> {'metric': 'TargetByFeaturesTable', 'result': {}}

**EDIT**: Passing the `TargetByFeaturesTable` as an `include` parameter as follows seemed to work:
```
report_json = json.loads(report.json(include={'TargetByFeaturesTable': {'reference', 'current'}}))
```

It would still be nice to have the metric show up on its own instead of having to manually include it.

Evidently:  0.4.40
Python: 3.12.7
Repository: evidentlyai/evidently
Issue: Request Entity Too Large in v0.5.1
Root cause: Configuration Error

In version v0.5.1 starts a problem with `Request Entity Too Large` on save snapshot.
```

File "/home/airflow/.local/lib/python3.11/site-packages/evidently/ui/managers/projects.py", line 159, in add_snapshot
    await self.project_metadata.add_snapshot(project_id, snapshot, blob)
  File "/home/airflow/.local/lib/python3.11/site-packages/evidently/ui/workspace/remote.py", line 206, in add_snapshot
    return self._request(f"/api/projects/{project_id}/snapshots", "POST", body=snapshot.dict())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.11/site-packages/evidently/ui/workspace/remote.py", line 137, in _request
    raise EvidentlyServiceError(details)
evidently.ui.errors.EvidentlyServiceError: Request Entity Too Large
```

Seems it correspond to new version of litestar whitch has this new features and restrict it by default to 10Mb:
<img width="927" alt="image" src="https://github.com/user-attachments/assets/87657a03-5fd4-4a06-8e0d-f341def8737b" />

Workaround: need add a way to specify increase max request size (snapshots can be much more then 10mb).



Hi @oleg-savko

Nice catch! Thanks for it!

Would you like to contribute 😊?

Just add  `request_max_body_size = None,` to this [file](https://github.com/evidentlyai/evidently/blob/main/src/evidently/ui/components/base.py#L50) before `**self.kwargs`

I guess this is nice to have no limits by default in application layer (and good to have limits in other layers like proxy (nginx etc.))



I am facing this same issue in version 0.7.3

```
---------------------------------------------------------------------------
EvidentlyError                            Traceback (most recent call last)
Cell In[83], [line 1](vscode-notebook-cell:?execution_count=83&line=1)
----> [1](vscode-notebook-cell:?execution_count=83&line=1) workspace.add_run("0196a49d-bfa5-7de2-be6d-378aba5c2a6e", rep, include_data=False)

EvidentlyError: Request Entity Too Large
```

Also from above comments I came across , like `request_max_body_size = None` may solve the issue, but how to pass this ?
A details steps on this would be helpful.
Repository: evidentlyai/evidently
Issue: Update scikit-learn version requirement to support v1.6.0
Root cause: External Dependency Breakage

Hello!

I’ve encountered a compatibility issue with the current `install_requires` setting in the `setup.py` file:
```
install_requires=[
    ...
    "scikit-learn>=1.0.1,<1.6.0",
]
```
The upper limit (`<1.6.0`) prevents the use of the latest stable version of `scikit-learn` (`1.6.0`). This creates conflicts in projects where `scikit-learn` is already updated to the latest version.

---
Note:
Previously, the function `sklearn.metrics.mean_squared_error` allowed the RMSE calculation with the argument `squared=True`. However, in `scikit-learn` `1.6.0`, the `squared` argument has been removed. Instead, a dedicated function, `sklearn.metrics.root_mean_squared_error`, has been introduced.

Admittedly, such a breaking change in a minor version release (`1.6.0`) was unexpected, as this would typically be reserved for a major version update. Regardless, it impacts the compatibility between `evidently` and `scikit-learn` in its latest release.

Hi @marta-guimaraes yes, thanks for this note

I started PR about it #1451 should be done soon

fixed in v 0.6.1
Repository: evidentlyai/evidently
Issue: Update evidently hashlib usage for FIPS-Compliant Systems and Security Best Practices
Root cause: Validation Gap

From python 3.9 `hashlib` introduced the `usedforsecurity` argument:
> Changed in version 3.9: All hashlib constructors take a keyword-only argument usedforsecurity with default value True. A false value allows the use of insecure and blocked hashing algorithms in restricted environments. False indicates that the hashing algorithm is not used in a security context, e.g. as a non-cryptographic one-way compression function.

`evidently` use hashing where the purpose is indeed _not for security_ purposes. This should be specifed in the code. My guess is that pretty much all federal systems in the world would have this issue.
> Federal Information Processing Standards (FIPS) 140-2 is a mandatory standard for the protection of sensitive or valuable data within Federal systems. - https://www.wolfssl.com/license/fips/

Without this fix organisations following certain security standard (FIPS compliance could be an example) are not able to use this package. 

The same fix has been applied to a range of key repositories:
- https://github.com/streamlit/streamlit/issues/7120
- https://github.com/mlflow/mlflow/issues/10106
- https://github.com/huggingface/transformers/issues/27034
- https://github.com/chroma-core/chroma/issues/1672

# Proposed solution

- Explicitly set `usedforsecurity=False` when calling `hashlib` constructors where the hashing is **not security-related**.
- Since `usedforsecurity` is only available from Python 3.9 onwards and `evidently` supports Python 3.8+, implement logic to handle this difference:
  - For Python 3.8: Call `hashlib` without `usedforsecurity`.  
  - For Python 3.9+: Specify `usedforsecurity=False`.

I will submit a PR implementing this change. Let me know if there are other concerns or suggestions for handling this!
Repository: evidentlyai/evidently
Issue: Error when trying to create collector config in self-hosted environment
Root cause: Configuration Error

Hello everyone! 

I'm facing an issue when trying to set up evidently self-hosted, basically what I want to do is set the collector config using the api and get data data displayed on the ui, but when I try to create the config via API, evidently complains about not having a key in their LockStorage. 

I'll share a sample project with my configurations:

The following zip contains:
- Dockerfile for evidently ui
- Dockerfile for evidently collector
- Docker compose for running both of them together
- Script used for creating config via API
- requirements.txt to install dependencies

[sample-project.zip](https://github.com/user-attachments/files/18378778/sample-project.zip)

The steps for running the project is
- docker compose up -d (wait for containers to go up)
- uv venv --seed
- uv pip install
- uv pip install -r requirements.txt
- uv run create-collector-config-api.py

Here's the docker logs that contain the error I'm facing
<img width="1143" alt="image" src="https://github.com/user-attachments/assets/9fa6d3b4-9f05-476c-8f45-2804cef732ba" />
Repository: evidentlyai/evidently
Issue: Need more examples about how to use user own statistical tests function.
Root cause: Documentation Deficiency

Hi.

I want to use own statistical tests function with evidently dashboard.
I added my function to stattest_func argument in DataDriftOptions.
I got an error but solving is not easy for me. I don't know the correct usage.
So, I tried to find detail docs or example but I couldn't.

Can I have some guide?

Hi @KevinRyu,
Can you provide the error you got there?

Statistical test function should meet several requirements:
- signature `(reference_data: pd.Series, current_data: pd.Series) -> float`
- return value in range from 0.0 to 1.0
Here an example with custom stattest function [examples/how_to_questions/drift_dashboard_with_options_california_housing.ipynb ](https://github.com/evidentlyai/evidently/blob/main/examples/how_to_questions/drift_dashboard_with_options_california_housing.ipynb)

Hi.

Thanks a lot. I got off work today.
I will check and give you feedback.

2022년 2월 8일 (화) 오후 6:58, Vyacheslav Morov ***@***.***>님이 작성:

> Hi @KevinRyu <https://github.com/KevinRyu>,
> Can you provide the error you got there?
>
> Statistical test function should meet several requirements:
>
>    - signature (reference_data: pd.Series, current_data: pd.Series) ->
>    float
>    - return value in range from 0.0 to 1.0
>    Here an example with custom stattest function examples/how_to_questions/drift_dashboard_with_options_california_housing.ipynb
>
>    <https://github.com/evidentlyai/evidently/blob/main/examples/how_to_questions/drift_dashboard_with_options_california_housing.ipynb>
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/evidentlyai/evidently/issues/143#issuecomment-1032421433>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AG43MBJALUL72PNZ2IFZR43U2DSLZANCNFSM5NZ5KM6Q>
> .
> Triage notifications on the go with GitHub Mobile for iOS
> <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
> or Android
> <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
>
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>


We implemented many statistical tests, here is a full list of available tests: https://docs.evidentlyai.com/user-guide/customization/options-for-statistical-tests May be on of them will suit straight away.

And here is an example of the custom stat test usage: https://github.com/evidentlyai/evidently/blob/main/examples/how_to_questions/drift_dashboard_with_options_california_housing.ipynb
Repository: evidentlyai/evidently
Issue: Whole report crashes when column is empty
Root cause: Validation Gap

I'm encountering an issue when running the following code:
```

num_target_drift_report = Report(metrics=[DataDriftPreset(), DataQualityPreset()])
num_target_drift_report.run(
    reference_data=reference, current_data=current, column_mapping=column_mapping
)
num_target_drift_report.save_html(path)
```

If my DataFrame contains an empty column, the process crashes with the following error:

ValueError: An empty column '''' was provided for drift calculation in the reference dataset.

This behavior is quite unfavorable in my case because I still want the report to be generated for the other columns. Ideally, I would just receive a warning that one column couldn't be processed, rather than having the entire process fail. Additionally, I still want the Data Quality Report to indicate that the column is empty.

Is this the intended behavior of the function? If so, is there an option to add and argument that prevent the crash and allow the report to be generated despite the empty column?

Hi @seb2704, this is expected behavior https://docs.evidentlyai.com/metrics/preset_data_drift#data-requirements - the columns passed for drift detection should be non-empty.

You can pass a list of columns to the preset as parameter: in this case only the passed columns will be tested for drift.
Repository: evidentlyai/evidently
Issue: Error during data drift report
Root cause: Validation Gap

Hey I'm getting the following Error due a data drift report:

```
File "C:\Users\UE93540\repos\ki-krankenhausautomat-code\.venv\lib\site-packages\evidently\suite\base_suite.py", line 276, in save_html
    render = self._render(file_html_template, template_params)
  File "C:\Users\UE93540\repos\ki-krankenhausautomat-code\.venv\lib\site-packages\evidently\suite\base_suite.py", line 362, in _render
    return temple_func(params=template_params)
  File "C:\Users\UE93540\repos\ki-krankenhausautomat-code\.venv\lib\site-packages\evidently\utils\dashboard.py", line 98, in file_html_template
    var {params.dashboard_id} = {dashboard_info_to_json(params.dashboard_info)};
  File "C:\Users\UE93540\repos\ki-krankenhausautomat-code\.venv\lib\site-packages\evidently\utils\dashboard.py", line 90, in dashboard_info_to_json     
    return json.dumps(asdict_result, cls=NumpyEncoder)
  File "C:\Users\UE93540\.pyenv\pyenv-win\versions\3.9.13\lib\json\__init__.py", line 234, in dumps
    return cls(
  File "C:\Users\UE93540\.pyenv\pyenv-win\versions\3.9.13\lib\json\encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "C:\Users\UE93540\.pyenv\pyenv-win\versions\3.9.13\lib\json\encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "C:\Users\UE93540\repos\ki-krankenhausautomat-code\.venv\lib\site-packages\evidently\utils\numpy_encoder.py", line 61, in default
    return json.JSONEncoder.default(self, o)
  File "C:\Users\UE93540\.pyenv\pyenv-win\versions\3.9.13\lib\json\encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type time is not JSON serializable
```
any idea why?

Convert the time and datetimes to string first. This is JSON encoder error.

Maybe remove some column from your pandas DF that has this data type to confirm?
Repository: evidentlyai/evidently
Issue: AttributeError: 'NoneType' object has no attribute 'columns'
Root cause: Unimplemented Feature Gap

When trying to build a report with `DataQualityPreset` for spark dataframe, there is an error: `AttributeError: 'NoneType' object has no attribute 'columns'
`

here is the code: 
```
from evidently.report import Report
from evidently.spark.engine import SparkEngine
from evidently.metric_preset import DataQualityPreset


data_quality_report_spark = Report(metrics=[
    DataQualityPreset(), 
])

data_quality_report_spark.run(current_data=df_spark, reference_data=None, engine=SparkEngine)
```

Hi @amalibayev, 

We have only limited support for some of the metrics on Spark in the old Evidently API. You can check the list here https://docs-old.evidentlyai.com/user-guide/tests-and-reports/spark

Data quality is not supported.

@elenasamuylova  thank you for quick reply. 

May I ask if there are plans for support Data Quality with spark ? 


Hi @amalibayev, not in the immediate roadmap - we are not currently actively developing the open-source spark module.
Repository: evidentlyai/evidently
Issue: Typo in 0.6 report

`mertics` instead of `metrics`

https://github.com/evidentlyai/evidently/blob/d7e2c2d379c032eef43d4a0e7ebb5173e0b33f2b/ui/packages/evidently-ui-lib/src/components/WidgetsContent.tsx#L69

Hi @emacha 

Would you like to contribute?

Hey, I guess. I reported this from my work laptop as it was the lazy route. It would take longer to clone the repo and open a PR than the actual change.

It's a trivial fix, so I can do it later when off the clock

> It would take longer to clone the repo and open a PR than the actual change

Actually you can use github's UI for it. You can click on edit button when you explore some file like:

![Image](https://github.com/user-attachments/assets/d3acfe9e-0f77-49f4-8789-f8066073fa23)

This is nice moment to try this feature @emacha 

It's alright, opened a [PR](https://github.com/evidentlyai/evidently/pull/1503).

Using the UI feels wrong :)
Repository: evidentlyai/evidently
Issue: "share" is an unintuitive word, whereas "proportion" or "percent" would be instantly understandable

`share_tests` <- see this variable, think: "what does share mean - who / what are these tests _shared with_?"

Search in the docs to try and find out what "share" means, see:

> If you want to test share instead of count, use share_tests instead of tests.

OK, but what does "share" mean!?

> For example, the check on missing values will validate if the current share of missing values is within +/-10% of the reference.

Share of missing values? Huh!? Why not: "will validate if the current proportion of missing values in within +/-10% of the reference"?

Find an explanation in the docs that links share to percentage:

> Testing count vs. share. Some Metrics like MissingValueCount or CategoryCount return both absolute counts and percentage. The default tests parameter lets you set condition against the absolute value. To test the relative value, use share_tests parameter.

But why not just use proportion/percentage all the time, instead of the word "share", which at first glance implies something being "shared"?

---

Side note; what about having `lte_proportion`, `lte_count`, etc. instead of `tests` and `share_tests`:

```
report = Report([
    MissingValueCount(column="Age", tests=[lte_proportion(0.1)]),
])
```

Or

```
report = Report([
    MissingValueCount(column="Age", tests=[lte_count(100)]),
])
```
Repository: evidentlyai/evidently
Issue: Real-time ML monitoring with Evidently and Grafana
Root cause: External Dependency Breakage

> None of these methods still exist in the most current version (==0.6.5).  Any reason for this?
> src: https://github.com/evidentlyai/evidently/tree/main/examples/integrations/grafana_monitoring_service
>
> ```
> from evidently.model_monitoring import CatTargetDriftMonitor
> from evidently.model_monitoring import ClassificationPerformanceMonitor
> from evidently.model_monitoring import DataDriftMonitor
> from evidently.model_monitoring import DataQualityMonitor
> from evidently.model_monitoring import ModelMonitoring
> from evidently.model_monitoring import NumTargetDriftMonitor
> from evidently.model_monitoring import ProbClassificationPerformanceMonitor
> from evidently.model_monitoring import RegressionPerformanceMonitor
> ```
> 
>  

 _Originally posted by @wanderson42 in [#573](https://github.com/evidentlyai/evidently/issues/573#issuecomment-2669486687)_
Repository: evidentlyai/evidently
Issue: ValueError: The truth value of a DataFrame is ambiguous.
Root cause: Validation Gap

https://github.com/evidentlyai/evidently/blame/645639c1cd3d930083640cccea649cd32778d15b/src/evidently/future/report.py#L411

This was introduced couple of weeks ago, whole dataframe can't be truthy/falsy and `run()` method fails because of that.