zhiminy/EvalEng huggingface dataset:
Repository: evidentlyai/evidently
Issue: help to upgrade to new api
Root cause: Documentation Deficiency

I have a code with old api, but if i use old "evidently" it downgrade other dependencies, so i need use new api.
i take a look on documentation but i don´t know how to change... my code:


```
        from evidently.dashboard import Dashboard
        from evidently.pipeline.column_mapping import ColumnMapping
        from evidently.tabs import CatTargetDriftTab, DataDriftTab

        column_mapping = ColumnMapping()
        column_mapping.target = target
        column_mapping.prediction = None
        column_mapping.datetime = None
        column_mapping.numerical_features = numeric_features
        column_mapping.categorical_features = categorical_features
        column_mapping.datetime_features = date_features

        if target not in reference_data.columns or target not in current_data.columns:
            raise ValueError(
                f"Both dataset must contain a label column {target} "
                "in order to create a drift report."
            )

        dashboard = Dashboard(tabs=[DataDriftTab(), CatTargetDriftTab()])
        dashboard.calculate(reference_data, current_data, column_mapping=column_mapping)
        filename = (
            filename or f"{self.exp_name_log}_{int(time.time())}_Drift_Report.html"
        )
        dashboard.save(filename)
        return filename
```

upgraded to:


```
        from evidently import ColumnMapping
        from evidently.metric_preset import DataDriftPreset, TargetDriftPreset
        from evidently.report import Report

        column_mapping = ColumnMapping()
        column_mapping.target = target
        column_mapping.prediction = None
        column_mapping.datetime = None
        column_mapping.numerical_features = numeric_features
        column_mapping.categorical_features = categorical_features
        column_mapping.datetime_features = date_features

        report = Report(metrics=[DataDriftPreset(), TargetDriftPreset()])

        # Generate the report filename
        filename = f"{self.exp_name_log}_{int(time.time())}_Drift_Report.html"

        # Save the report as an HTML file
        report.save(filename)

        # Return the filename
        return filename

```

It its correctly upgraded? Any suggestions?
Repository: evidentlyai/evidently
Issue: Evidently Collector in v0.4.16
Root cause: Validation Gap

When I try to run `evidently collector` from version 0.4.16, I am getting the following error:

```
ImproperlyConfiguredException: 500: Kwarg resolution ambiguity detected for the following keys: i, d. Make sure to use distinct keys for your dependencies, path parameters and
aliased parameters.
```

thanks for reporting! let us take a look at this

@mike0sv 
I fix it by del query in type Annotation, i can make PR

@c0t0ber yes, go ahead. Also you'll need to annotate some routes's return types
Repository: evidentlyai/evidently
Issue: Can we still customize histogram plots in the latest version?
Root cause: Unimplemented Feature Gap

According to the [v0.1.57 user guide](https://docs.evidentlyai.com/v/v0.1.57/user-guide/customization/options-for-data-target-drift#id-3.-customize-the-histogram-plots), we can define `DataDriftOptions` to customize histogram plot and define `QualityMetricsOptions` to select chunk of data we interest.

However, `DataDriftOptions` and `QualityMetricsOptions` are no longer compatible in the latest release since we have to pass `options` as a subclass of `Options`. `DataDriftOptions` is inherited from `pydantic.BaseModel`, while `QualityMetricsOptions` is inherited from object. 

By looking at [latest user guide](https://docs.evidentlyai.com/user-guide/customization/options-for-color-schema), we can only see customization on color schema and raw data. `DataDriftOptions` and `QualityMetricsOptions` are still there in the API Reference, so I'm pretty not sure whether we still can customize our report via `DataDriftOptions` and `QualityMetricsOptions` or there is a workaround for this.



Hi @naveen-pit, 

There are currently no parameters to specify e.g. number of bins in the histograms. 

SInce the API update, we divided the customizations into two groups:
- Report-level `Options` (like changing the color schema or opting to see raw data).
- Metric-level `parameters` (that you can specify for a specific Metric inside the Report and for some Presets). For example, the definition of classification threshold for Classification quality metrics now works through `probas_threshold` parameter. You can see the complete list of Metric-level parameters here https://docs.evidentlyai.com/reference/all-metrics 

However, there are currently now parameters to visually customize the bins. 

We'd like to add them, but it is not on the immediate roadmap. 

Thank you for your reply. will keep checking the updates.
Repository: evidentlyai/evidently
Issue: Setting S3 as back store for remote server

Hey, I was wondering if its possible to configure a remote evidently server to use say an s3 bucket for all its metadata and snapshot storage. In [this link](https://docs.evidentlyai.com/user-guide/monitoring/workspace_project#remote-snapshot-storage), it mentioned that the UI server can read snapshots from a remote FS like AWS but it does not mention if we can also write to the FS.

My final solution would look like running the UI server in an EC2 instance connected to an EBS volume. I want my UI server to use this EBS storage (or s3 too) to read and write snapshots. Is this possible?

Yes it is possible. If you setup s3 as a storage backend, UI server will upload new snapshots there as well
Repository: evidentlyai/evidently
Issue: Feature Request: Allow array of confidence scores to be passed in to match features.
Root cause: Unimplemented Feature Gap

Current design only allows one confidence score as an option. Would be nice to allow an array of confidence scores so each feature could be evaluated differently. 
Thanks for any consideration on this.

Hi @yodanater ,

We are planning to include such feature in next release.

Hi again,

We added this in new release 0.1.39.dev0. You can found example of that in https://github.com/evidentlyai/evidently/blob/main/evidently/examples/california_housing_drift_with_options.ipynb
Repository: evidentlyai/evidently
Issue: Weird labelling for y-axis in some error plots
Root cause: Validation Gap

In a regression test report (using default preset) I noticed some weird labelling in the error plots for the negative part of the axis (see screenshot), with ticks labelled eg `"â^'0.3"` instead of `-0.3`

![image](https://github.com/evidentlyai/evidently/assets/3518950/ef66ffa7-5dd9-4534-a0d2-9433a98c520f)


Hi @emeli-dral and @elenasamuylova,

I have created a fix for this in https://github.com/evidentlyai/evidently/pull/1183 with the following with the following result:

**Currrent State**
<img width="476" alt="current_state" src="https://github.com/evidentlyai/evidently/assets/54825049/25df40a8-432f-4e88-8d99-dfeaee69403f">

**After Fix**
<img width="475" alt="after_fix" src="https://github.com/evidentlyai/evidently/assets/54825049/3c393d9f-b756-4a97-b9ad-21c9682f6e4c">

Please review and merge if it's ok.

Cheers, 

Gwen

Hi @jlopezpena,

I am experiencing the same problem and have proposed a fix in https://github.com/evidentlyai/evidently/pull/1183. In the meantime, if you want a quick fix, you can add the following code around the output code:

**Before the given code**
`
    <!DOCTYPE html>
    <html lang="en">
      <head>
        <meta charset="utf8">
      </head>
      <body>
`

**After the given code**
`</html>`
Repository: evidentlyai/evidently
Issue: Collector service cannot process np.nan, np.inf, np.NINF values passed to endpoints
Root cause: External Dependency Breakage

```python
import uuid

import numpy as np
import pandas as pd

from evidently.collector.client import CollectorClient
from evidently.collector.config import RowsCountTrigger, CollectorConfig, ReportConfig
from evidently.metrics import DatasetMissingValuesMetric
from evidently.report import Report


missing_vals = (np.nan, np.inf, np.NINF)

for bug_val in missing_vals:
    rep = Report(
            metrics=[
                DatasetMissingValuesMetric(missing_values=[bug_val]),
            ],
        )
    rep.run(reference_data=None, current_data=pd.DataFrame())
    conf = CollectorConfig(
        trigger=RowsCountTrigger(),
        report_config=ReportConfig.from_report(rep),
        project_id=str(uuid.uuid4()),
    )
    collector_id = str(uuid.uuid4())
    collector_client = CollectorClient("http://localhost:8001")
    try:
        collector_client.create_collector(collector_id, conf)
    except Exception as e:
        print(e)
```

Client error: 400 Client Error: Bad Request for url: http://localhost:8001/7ff10857-346c-474f-b557-1fa1040d53fa
Collector error: 
```
INFO:     127.0.0.1:61679 - "POST /7ff10857-346c-474f-b557-1fa1040d53fa HTTP/1.1" 400 Bad Request
ERROR - 2024-03-18 21:21:40,458 - litestar - config - exception raised on http connection to route /7ff10857-346c-474f-b557-1fa1040d53fa

Traceback (most recent call last):
  File "evidently\venv38\lib\site-packages\litestar\serialization\msgspec_hooks.py", line 186, in decode_json
    return _msgspec_json_decoder.decode(value)
msgspec.DecodeError: JSON is malformed: invalid character (byte 306)

litestar.exceptions.base_exceptions.SerializationException: JSON is malformed: invalid character (byte 306)
```


The issue is related to litestar using msgspec, which does not support np.nan, np.inf, np.NINF. Adding support for these values without substituting them, for example, with strings, is not feasible.

Possible solutions:

- Discontinue support for np.inf, np.NINF values and use the pyarrow backend for pandas, which will convert nan to None and subsequently be processed normally.
- Change the transport protocol, as nan and inf are not standards and are not supported by most libraries. As an alternative, msgpack can be considered.
- A very controversial solution since it depends on the framework implementation, use fastapi, as it utilizes Python's built-in json encoder, which supports nan and inf.
Repository: evidentlyai/evidently
Issue: Multi-output regression metrics
Root cause: Unimplemented Feature Gap

Such as in regression models like [Multi-task Lasso](https://scikit-learn.org/stable/modules/linear_model.html#multi-task-lasso), where problems are solved jointly, would it be possible/is it planned to evaluate regression metrics jointly? i.e. We can see the effect of (single or multiple) x on multiple dimensions of y. This would mean allowing multiple target columns and multiple prediction columns for regressions.

Would be very much appreciated! A use-case would be a forecasting model in which many weeks are forecasted and you'd want to monitor short-term forecasting and long-term forecasting.
Repository: evidentlyai/evidently
Issue: UI 500 when opening a report or tests if one of the metrics or tests ends with an error
Root cause: Validation Gap

```
import logging

import numpy as np
import pandas as pd

from evidently.metrics import ColumnDriftMetric
from evidently.report import Report
from evidently.ui.base import Project
from evidently.ui.workspace import Workspace

logging.basicConfig(level=logging.DEBUG)

def create_evidently_project(workspace_name: str, project_name: str) -> Project:
    ws = Workspace.create(workspace_name)
    if projs := ws.search_project(project_name):
        proj = projs[0]
    else:
        proj = ws.create_project(project_name)

    return proj


proj = create_evidently_project("test_error_bug", "test_error_bug")

data = pd.DataFrame(
    {"vals": [np.nan] * 100}
)
ref_data = pd.DataFrame(
    {"vals": [1] * 1000},
)

ref_data.to_parquet("ref_data.parquet", index=False)
used_features = ["vals"]
rep = Report(
    metrics=[
        *[ColumnDriftMetric(column_name=column) for column in used_features],
    ],
)
rep.run(reference_data=ref_data, current_data=data)

try:
    rep._inner_suite.raise_for_error()
except Exception as e:
    print(e)

rep.save(f"test_error_bug/{str(proj.id)}/snapshots/{str(rep.id)}.json")
```

If u open generated report you get 500 and 

```
  File "evidently\src\evidently\report\report.py", line 212, in _build_dashboard_info
    html_info = renderer.render_html(test)
  File "evidently\src\evidently\metrics\data_drift\column_drift_metric.py", line 322, in render_html
    if result.drift_detected:
AttributeError: 'MetricResult' object has no attribute 'drift_detected'
```
Repository: evidentlyai/evidently
Issue: Slow test execution and metric calculation
Root cause: Algorithmic Error

When using real data with a size of 100k rows and a large number of columns, metrics, and tests (around 1000), their calculation can take up to 20 minutes. Additionally, computer resources are not fully utilized, with a powerful processor not exceeding 20% of one core's capacity. Consequently, with many tests and metrics and a high RPS of new data, Evidently may not be able to process them in time.

I checked the insides of ClassificationPreset and DataDriftPreset. I've seen a lot of data copying, which is far from ideal. Also pandas is used here, while in some cases faster alternatives could be utilized.
But imo the most inefficient part is embedding actual data into html reports

I wonder if metric calculation can be done at least in several processes

@nick-konovalchuk 

The best solution, I believe, would be to use parallel execution, but we need to explore the feasibility of its application. Trying to optimize individual sections is of little use because in my case, we are calculating ~2000 different tests and metrics. 

I don't see any problems with generating HTML since you're only using HTML when necessary.


Also using polars with lazy calculations instead of pandas can be a good solution if we are talking about calc optimization

@c0t0ber 
Personally I've never used polars, but I think I remember it using all the cores of a CPU. In such setting multiprocessing would be harmful. 

@c0t0ber 
I don't see a problem with generating HTML. I wish they also had an option of generating actual plotly objects, that I can display using streamlit, for instance. I still can display HTML in streamlit.
The problem is embedding redundant data in HTML. Do you really need all data point to draw a histogram given that you can't change the bin size after the report is generated? Because as far as I understand they embed ALL data points for ClassificationProbDistribution and charts.

Also I'm less sure about it, but the data points in HTML may be duplicated in context of several metrics/tests

Idk if this is correct and/or possible, but the following would be cool
1. Generate actual Plotly objects
2. Extract HTML from them when the report is rendered. I think this HTML won't have redundant data embedded
Repository: evidentlyai/evidently
Issue: Report import error in evidently.report package in databricks notebook
Root cause: Environment Incompatibility

`/databricks/python/lib/python3.9/site-packages/sklearn/__check_build/../../scikit_learn.libs/libgomp-d22c30c5.so.1.0.0: cannot allocate memory in static TLS block`

Error goes away on copying code to a new notebook and reappears after some time.

It is some weird error arising because import sequence. Seems like sklearn needs to be imported first. Closing issue
Repository: evidentlyai/evidently
Issue: UI dashboard doesn't automatically update
Root cause: Documentation Deficiency

Hello, I recently started using your library and found it to be very impressive. However, I encountered an issue when running a test with the RemoteWorkSpace function. Specifically, when I add new data, the dashboard UI does not refresh to display this new data. It only refreshes the reports and test suites sections. I would really appreciate your assistance with this. Thank you.

Hello @Binh24399 ! 
I just tested it and it actually works, just refresh the dashboard page (or you can just click on the bread crumb)

Please, provide more info about your setup: how you launch `evidently ui` and so on
Repository: evidentlyai/evidently
Issue:  Support for using embeded images instead of Plotly interactive component
Root cause: Unimplemented Feature Gap

Out reports contain multiple large datasets. In general we can do sampling but for a few of them we want to keep all data. The size of the final report html is around 100MB which cause the browser crashed. So I wonder if it's possible to have an option to generate reports with static images instead of Plotly figures. 

Hi @2qU24Tlb, 

Which Evidently version are you using? (Pre-0.3.2, all visualizations retained raw data. Currently, they are aggregated by default).

It is not possible to swap Plotly, but you can get the Report output as JSON and optionally include the render data https://docs.evidentlyai.com/user-guide/customization/json-dict-output#include-render, and then visualize it elsewhere using a different tool.

If you are using a Preset (e.g. ClassificationPreset), you can also experiment with creating a custom Report instead and including only some of the metrics – if not all of them are required. 

Hi @elenasamuylova ,
We are using latest version of Evidently. 
I will try your suggestions. Thank you very much.
Repository: evidentlyai/evidently
Issue: DataDriftPreset : drift_share threshehold does not work with spark engine
Root cause: Algorithmic Error

Hello, I recently started using your library and found it to be very impressive. However, I encountered an issue when running a DataDriftPreset using the spark engine. The Dataset Drift is always detected event when i set the drift_share as 1 (or 100). I'm forced to covert my spark df to Pandas to use the standard engine
Repository: evidentlyai/evidently
Issue: If current df doesn't have an expected column, running tests raises assertion error
Root cause: Validation Gap

Hey there! Discovered this stepping away from OSS Tutorial

```py
from sklearn.datasets import fetch_california_housing

from evidently.test_suite import TestSuite
from evidently.test_preset import DataStabilityTestPreset
from evidently.tests import *

data = fetch_california_housing(as_frame=True)
housing_data = data.frame
housing_data.head(1)

reference = housing_data.sample(n=5000, replace=False)
current = housing_data.sample(n=5000, replace=False).drop(["AveOccup"], axis=1)

suite = TestSuite(tests=[
    DataStabilityTestPreset(),
])

suite.run(reference_data=reference, current_data=current)
suite
```

tb:
```py
{
	"name": "AssertionError",
	"message": "",
	"stack": "---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
File ~/Git/kc-mlops/testing_monitoring/.venv/lib/python3.12/site-packages/IPython/core/formatters.py:347, in BaseFormatter.__call__(self, obj)
    345     method = get_real_method(obj, self.print_method)
    346     if method is not None:
--> 347         return method()
    348     return None
    349 else:

File ~/Git/kc-mlops/testing_monitoring/.venv/lib/python3.12/site-packages/evidently/suite/base_suite.py:167, in Display._repr_html_(self)
    166 def _repr_html_(self):
--> 167     dashboard_id, dashboard_info, graphs = self._build_dashboard_info()
    168     template_params = TemplateParams(
    169         dashboard_id=dashboard_id,
    170         dashboard_info=dashboard_info,
    171         additional_graphs=graphs,
    172     )
    173     return self._render(determine_template(\"auto\"), template_params)

File ~/Git/kc-mlops/testing_monitoring/.venv/lib/python3.12/site-packages/evidently/test_suite/test_suite.py:212, in TestSuite._build_dashboard_info(self)
    210 renderer.color_options = color_options
    211 by_status[test_result.status] = by_status.get(test_result.status, 0) + 1
--> 212 html = renderer.render_html(test)
    213 replace_test_widget_ids(html, generator)
    214 test_results.append(html)

File ~/Git/kc-mlops/testing_monitoring/.venv/lib/python3.12/site-packages/evidently/tests/data_integrity_tests.py:993, in TestColumnsTypeRenderer.render_html(self, obj)
    990 info = super().render_html(obj)
    992 parameters = obj.get_result().parameters
--> 993 assert isinstance(parameters, ColumnTypesParameter)
    994 info.details = [
    995     DetailsInfo(
    996         title=\"\",
   (...)
   1006     ),
   1007 ]
   1008 return info

AssertionError: "
}
```

evidently==0.4.18
Repository: evidentlyai/evidently
Issue: Failed to load module script: Expected a JavaScript module script but the server responded with a MIME type of "text/html". Strict MIME type checking is enforced for module scripts per HTML spec.
Root cause: Environment Incompatibility

When running evidently ui locally, I'm getting blank page and the following error message: "Failed to load module script: Expected a JavaScript module script but the server responded with a MIME type of "text/html". Strict MIME type checking is enforced for module scripts per HTML spec."
![2024-04-03_17h59_29](https://github.com/evidentlyai/evidently/assets/76261338/b11ef0b0-0592-46fb-a042-b279655020f2)

Running it using separate conda environment with Python 3.11 installed.
![2024-04-03_18h15_44](https://github.com/evidentlyai/evidently/assets/76261338/4391e001-efc9-4ce5-bfac-2f60a47afaa4)

Does anyone know what's causing this?

The folders have the following structure
![2024-04-03_18h17_47](https://github.com/evidentlyai/evidently/assets/76261338/18e55ca5-3661-4ff4-bda5-8f76cb2968ad)

Found solution: https://github.com/golang/go/issues/32350#issuecomment-635859542
The problem is specific to Windows 10
Repository: evidentlyai/evidently
Issue: New pydantic version crashes import resolving
Root cause: External Dependency Breakage

### Description

Yesterday, a compatibility fix was released in Pydantic version 1, which added the "v1" submodule in versions 1.., causing the try/except feature to stop working. 

Pydantic release: https://github.com/pydantic/pydantic/pull/9157

Evidently code https://github.com/evidentlyai/evidently/blob/v0.4.18/src/evidently/_pydantic_compat.py#L7

```python
try:
    from pydantic import v1  # noqa

    v = 2
except ImportError:
    v = 1
```

> This bug can be reproduced ONLY if you have other packages installed before evidently with an unfixed Pydantic version, such as "<2" or "1.10", etc.

### Steps to reproduce:
1. Install pydantic==1.10.15.
2. Install evidently.
3. Try to build a classification report, for example, or perform an import:
```python
classification_performance_report = Report(
        metrics=[
            ClassificationPreset(),
        ]
    )
classification_performance_report.run(reference_data=None, current_data=df)
html = classification_performance_report.get_html()
```
Repository: evidentlyai/evidently
Issue: What are you guys using for dependency management?

I can see that you have wisely separated your dependencies into `requirements.txt` and `dev-requirements.txt`. Which tool are you using to separate those dependencies? Is it `poetry`?

Hi @bhattbhuwan13 ,
For now we are doing this manually, because adding new dependencies isn't happen often.
And we are looking for better alternative (`poetry` + `pip compile` is one of them).