zhiminy/EvalEng huggingface dataset:
Repository: evidentlyai/evidently
Issue: Save report to file cause "unclosed file" ResourceWarning
Root cause: Resource Mishandling

## save report with code:
```
drift_report = Report(metrics=[
DataDriftPreset(), TargetDriftPreset()
])
drift_report.run(reference_data=reference, current_data=current, column_mapping=column_mapping)
drift_html_file = "./drift_report.html"
drift_report.save_html(drift_html_file)
drift_json_file = "./drift_report.json"
drift_report.save_json(drift_json_file)
```
## Unclose file Warning log with:
```
D:\ProgramData\Anaconda3\envs\models\lib\site-packages\evidently\utils\dashboard.py:231: ResourceWarning:
unclosed file <_io.TextIOWrapper name='D:\ProgramData\Anaconda3\envs\models\lib\site-packages\evidently\nbextension\static\index.js' mode='r' encoding='utf-8'>
D:\ProgramData\Anaconda3\envs\models\lib\site-packages\evidently\utils\dashboard.py:236: ResourceWarning:
unclosed file <_io.BufferedReader name='D:\ProgramData\Anaconda3\envs\models\lib\site-packages\evidently\nbextension\static\material-ui-icons.woff2'>
```
Repository: evidentlyai/evidently
Issue: Custom StatTest Failing in Python Due to missing Python Engine
Root cause: External Dependency Breakage

When attempting to implement a custom StatTest in python, the run function is failing with the following error:


- **Test failed with exceptions: 'mann-whitney-u' is not implemented for <class 'evidently.calculation_engine.python_engine.PythonEngine'>**




See below for the code to reproduce the error.
- The code was taken from this link: https://github.com/evidentlyai/evidently/blob/main/examples/how_to_questions/how_to_specify_stattest_for_a_testsuite.ipynb






import pandas as pd
import numpy as np

from scipy.stats import mannwhitneyu
from sklearn import datasets

from evidently.calculations.stattests import StatTest
from evidently.test_suite import TestSuite
from evidently.tests import *

#Dataset for Data Quality and Integrity
adult_data = datasets.fetch_openml(name='adult', version=2, as_frame='auto')
adult = adult_data.frame

adult_ref = adult[~adult.education.isin(['Some-college', 'HS-grad', 'Bachelors'])]
adult_cur = adult[adult.education.isin(['Some-college', 'HS-grad', 'Bachelors'])]

adult_cur.iloc[:2000, 3:5] = np.nan

def _mann_whitney_u(reference_data: pd.Series, current_data: pd.Series, _feature_type: str, threshold: float):
    p_value = mannwhitneyu(np.array(reference_data), np.array(current_data))[1]
    return p_value, p_value < threshold

mann_whitney_stat_test = StatTest(
    name="mann-whitney-u",
    display_name="mann-whitney-u test",
    func=_mann_whitney_u,
    allowed_feature_types=["num"]
)

data_drift_dataset_tests = TestSuite(tests=[
    TestShareOfDriftedColumns(num_stattest=mann_whitney_stat_test),
])

data_drift_dataset_tests.run(reference_data=adult_ref, current_data=adult_cur)
data_drift_dataset_tests

I tried with 0.4.2 and it worked. On 0.4.7 there were some other issues, and 0.4.0 was missing some tests. They definitely stuffed something up with Python engine , and since they made func a property
Repository: evidentlyai/evidently
Issue: Feedback and bugs (mostly about Cloud)

Here's feedback and number of bugs I encountered while trying out evidently:

https://aguschin.notion.site/evidently-feedback-f1e932a60b6e46c1947f2df084f31e14?pvs=4. 

It's in russian, try translating the page if necessary. Ping me if something is unclear.

Added another comment there. Please feel free to strikethrough issues/left comment there if that's easier for you to use it this way.
Repository: evidentlyai/evidently
Issue: Missing "Per feature"-metrics in Regression
Root cause: Documentation Deficiency

Somehow I can't replicatete full regression performance report as it is shown on: https://docs.evidentlyai.com/presets/reg-performance

All the "per-feature" analyses plots seem to be missing (components 10-12 in the above link)

I ran this notebook (as a norml python-script): https://github.com/evidentlyai/community-examples/blob/main/tutorials/bicycle_demand_monitoring_setup.ipynb 

And still all the "per-feature" components seem to be missing. I don't see any warnings or errors occurring and I updated evidently to several versions, including the latest (0.4.19) , so currently I am a bit in the dark on what's going on. 

Thanks in advance!

Edit: by debugging/digging in the evidently code I found that the problem was that the option "render.raw_data" needed to be set to True. I've got the plots I want, but it might be a good idea to document this somewhere or explain in the tutorials section why this is necessary.

Hi @DonnyWhoLovedBowling,

You found the correct solution, this visualization requires passing the raw data option. 

```
report = Report(
    metrics=[
      ClassificationQualityByFeatureTable(),
    ],
    options={"render": {"raw_data": True}}
  )
report.run(reference_data=ref, current_data=cur)
report
```

It is in the backlog to re-implement this metric to support the non-aggregated visualization. (Either this or the documentation update is coming soon :) You are also welcome to submit a PR to the docs to add a note about this!

Hi @elenasamuylova,

thanks for your quick reply! I just noticed that if you search your website on "raw data", you actually find the right documentation (https://docs.evidentlyai.com/user-guide/customization/report-data-aggregation#non-aggregated-plots-for-reports)

Perhaps it might be an easy solution just to place a link to this page at places where it is relevant. For example: https://docs.evidentlyai.com/presets/reg-performance 
Or it might be nice for completeness/overview to add a mention in this table for each metric where it is relevant? https://docs.evidentlyai.com/reference/all-metrics

For now I'm closing this issue, thanks again.
Repository: evidentlyai/evidently
Issue: Question: Targetdrift doesn't detect drift when trend is reversed.
Root cause: Algorithmic Error

Hello,
I am using two pandas dataframe:
data1 = {
    'Sqft': [600, 700, 800, 900, 1000],
    'Price': [235000, 255000, 293350, 321200, 350000]
}
df1 = pd.DataFrame(data1)

data2 = {
    'Sqft': [600, 700, 800, 900, 1000],
    'Price': [350000, 321200, 293350, 255000, 235000]
}
df2 = pd.DataFrame(data2)

The target here is price. The second dataframe has exact opposite trend as compared to the first. Yet, when I try to use TargetDriftPreset, drift score remains 1 and drift is not detected. Am I doing something wrong?
Repository: evidentlyai/evidently
Issue: How to set a different threshold for classification probability tab that is not 0.5?
Root cause: Unimplemented Feature Gap

I am having issues to understand how to use  ProbClassificationPerformanceTab() as it seems the position of the item in the list column_mapping.prediction is very important to the output of the recall precision curve and area under the curve.  

On the other hand, the classification report that comes in the same dashboard seems to use a default 0,5 threshold to cut the probability predictions. Is there a way to change the threshold?

also is there a way to set what to consider as the positive class in the precision recall curve?

thanks


Hi @Guidosalimbeni 

Yes, you are right, for this report class order matters. In particular, column order in Probabilistic Binary Classification. The tool expects that the target (so-called positive) class is the first in the column_mapping['prediction'] list.
Unfortunately, now we do not have an option to set a positive class explicitly, but it is definitely needed. Thank you for pointing it out.

At the moment there is no way to set a custom threshold for the probability predictions, but it will be added in the next release. We are adding ```QualityMetricOptions``` and the threshold will be there.

Hi @Guidosalimbeni 

We added this functionality in #122 
You can get more details here https://docs.evidentlyai.com/user-guide/customization/options-for-quality-metrics
Repository: evidentlyai/evidently
Issue: Data quality test suite saved as HTML is much bigger than data quality preset metric report (300MB vs. 3MB)
Root cause: Documentation Deficiency

The two files in the screenshot are generated with the code below:
```python
print("Generating data quality report...")
data_quality_report = Report(metrics=[
    DataQualityPreset(),
])
data_quality_report.run(reference_data=df, current_data=df, column_mapping=data_column_mapping)
data_quality_report.save_html(
    os.path.join(data_profile_dir, "data_quality.html")
)
print("Data quality report generated successfully!")
print("Running data quality test suite...")
data_quality_test_suite = TestSuite(tests=[
    DataDriftTestPreset(),
    DataQualityTestPreset(),
    DataStabilityTestPreset(),
])
data_quality_test_suite.run(reference_data=df, current_data=df, column_mapping=data_column_mapping)
data_quality_test_suite.save_html(
    os.path.join(data_profile_dir, "data_quality_test.html")
)
print("Data quality test suite generated successfully!")
```
<img width="714" alt="Screenshot 2024-05-02 at 17 15 05" src="https://github.com/evidentlyai/evidently/assets/30638982/68e6827f-10c0-455f-a1de-07399d87565d">

What can I do to shrink the size of the HTML output from the test suite?




Hi @billlyzhaoyh, 

In the second instance (when you combine multiple Test Presets), you generate a very large number of column-level tests, compared to the first instance (where `DataQualityPreset()` generates summaries for all columns only once). 

Many of these individual Tests have a visual render (e.g., distribution of each column), increasing the resulting HTML's size. 

The solution is to create a custom Test Suite that includes the individual Tests you'd like to see, instead of combining Test Presets.
https://docs.evidentlyai.com/user-guide/tests-and-reports/custom-test-suite

Thank you for this @elenasamuylova  I was trying to look up but is there any way that we can disable visual render functionality in favour of a smaller HTML?

Hi @billlyzhaoyh, I am afraid there is no such feature currently. However, you can export the results as a JSON or Python dictionary instead: https://docs.evidentlyai.com/user-guide/tests-and-reports/run-tests#output-formats
Repository: evidentlyai/evidently
Issue: ModuleNotFoundError: No module named 'evidently.metrics'; 'evidently' is not a package
Root cause: Environment Incompatibility

Hello, 

python version : **3.10.11**

this what I did :
in terminal
`pip install evidently==0.4.19`
in python script : 
```import datetime

from sklearn import datasets

from evidently.metrics import ColumnDriftMetric
from evidently.metrics import ColumnSummaryMetric```
Repository: evidentlyai/evidently
Issue: Keyword Error
Root cause: Unimplemented Feature Gap

facing object 'dashboard' is not J-Son serializable 

Hi! Yes, the Dashboard object is not JSON serialisable by default and we did not implement specific method to dump it is as a JSON yet. But we are considering this direction though!

Are you trying to do something like:
```
import json
dashboard_json = json.dumps(dashboard)
```
?
Could you please tell us more about the end problem you are truing to solve? We might be able to suggest something.

Hey @achandras1597!
Despite the fact that we do not have a specific method to dump dashboard as a nice JSON file, we implemented a quick solution for you.

We implemented methods ‘_json()’ and ‘_save_to_json(path)’ for Dashboard class. Current implementation dumps the whole Dashboard data into the JSON, which is probably way too much! :) But these methods are already available (you will need to update the package), so you can give it a try.

This is an example of the JSON generation:
```
drift_report = Dashboard(reference_frame, current_frame, column_mapping = None, tabs=[DriftTab])
drift_report._json()
``` 
And  an example of the saving JSON file:
```
drift_report = Dashboard(reference_frame, current_frame, column_mapping = None, tabs=[DriftTab])
drift_report._save_to_json('filename.json')
``` 
Also, we added —output_type parameter for terminal mode, so now you can choose ‘json’ option there as well:
```
python -m evidently analyze --config config.json --reference reference.csv  --current current.csv --output_path save_report_here --output_type json
```
Hope, this will help!
Repository: evidentlyai/evidently
Issue: Passing Bearer token when connecting to remote workspace
Root cause: Documentation Deficiency

I am trying to setup a self hosted ML monitoring with Evidently.

I have Evidently running inside a container on a remote host. The host requires an Authorization header with a Bearer token. 

According to the documentation, passing a secret is possible via: 

However, it looks like Evidently passes the secret value as `evidently-secret` header instead of an `Authorization` header. (see [here](https://github.com/evidentlyai/evidently/blob/main/src/evidently/ui/storage/common.py#L26)  and [here](https://github.com/evidentlyai/evidently/blob/main/src/evidently/ui/workspace/remote.py#L122)) 

Would it be possible to allow users to specify the name of the header the secret should be attached to?

Hi

We are using "evidently-secret" header in order to protect write api. You should never use this functionality for authorization.

But, of course, you can have your own proxy authorization layer just before running "evidently ui" (i think modifying evidently source code is bad idea for your purposes). This layer just check token, if ok - proxy request to running "evidently ui" server

Is it helpful for you?

Thanks for the reply @DimaAmega, perhaps the Documentation on this could be clarified a little bit. Right now it simply states:
<img width="629" alt="image" src="https://github.com/evidentlyai/evidently/assets/37832014/0d40efd0-e199-426f-ab15-3b755a5105d5">

It is a bit ambiguous and I initially assumed this was basically the auth token required to access the remote server. MLFlow has done something similar (see https://mlflow.org/docs/latest/auth/index.html#using-environment-variables).

I understand this is unfortunately not possible with Evidently. We can work around it and secure the service in a different way, but more clarity in the documentation would be appreciated.
Repository: evidentlyai/evidently
Issue: SSL_Verify Disbale for Connecting with the Evidently Remote Client (Within VPN)
Root cause: Configuration Error

Right now, I'm running the Evidently remote server in a Kubernetes cluster within a Organisation VPN space. When I try to reach out to the evidently Server URL with evidently Client, I'm getting an SSL_verify error. I wanted to know how we can disable the ssl authentication to connect with the same client Or can you avail a mathod to disable the ssl verify method.

**ERROR:**
`SSLError: HTTPSConnectionPool(host='evidentlyui.gpu.<domain-name>.com', port=443): Max retries exceeded with url: /api/version (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1006)'))).`

I am also running into this issue. Here's what a colleague of mine and I found:

I went into evidently.ui.workspace.remote::RemoteBase._request and altered line 132: 
 
FROM: response = s.send(request.prepare())
TO: response = s.send(request.prepare(), verify=os.environ["SSL_CERT_FILE"])

It works with that change.

We think the issue is that Evidently creates a Session.  From that session, it calls session.send().  send() does NOT use self.request to send a thing...   Why do you care? -  It's the request function within the Session which calls the self.merge_environment_settings to pull ssl bundle through REQUESTS_CA_BUNDLE env var...
 
Without that call, I don't see the ssl certs injected/used.

how to repackage the library with new changes??
Repository: evidentlyai/evidently
Issue: API documentation in repo
GitHub labels: enhancement

In addition to the existing documentation, we need to document the API.

Since such documentation is generated automatically, and the codebase is not entirely covered with comments, we shall do it step-by-step. As a first step, I suggest we:
- [ ] upload the first version of the automatically generated documentation 
- [ ] upload the instructions for building the docs from the repository

This will allow us to:
1. see how the API documentation looks now and iteratively improve the content and visuals
2. give the users the opportunity to build it from the repository according to the instructions

Any thoughts on what you want to use? shpinx? readthedocs?
Repository: evidentlyai/evidently
Issue: ModuleNotFoundError: No module named 'evidently.profile', 'evidently.model_profile'
Root cause: External Dependency Breakage

Hi Team,

I am getting ModuleNotFoundError while importing 'evidently.profile' & 'evidently.model_profile'.

Installed Python version: 3.8.19 (Also found the same error with version : 3.10.12)

Installed evidently version : 0.4.28

After checking all the folders & files, I found there are no .py files available for these modules. Can you please suggest how can I use these modules with the above configuration.


Thank You.

Hi @abhinandansamal , check this docs section: https://docs.evidentlyai.com/support/migration

> Hi [@abhinandansamal](https://github.com/abhinandansamal) , check this docs section: https://docs.evidentlyai.com/support/migration

The doc returns 404. @abhinandansamal, were you able to find a solution for this?
Repository: evidentlyai/evidently
Issue: Repeated pandas in requirements.txt
Root cause: Documentation Deficiency

https://github.com/evidentlyai/evidently/blob/fe4f58446b1d7a29ee61404acb9ef4b14c7606f2/examples/integrations/streamlit_dashboard/requirements.txt#L12

Hi @emrynHofmannElephant, 

Would you mind sending a PR to fix this?
Repository: evidentlyai/evidently
Issue: Not able to Integrate Evidently Output HTML report with Power BI Desktop

When integrating Evidently's HTML reports into Power BI Desktop,  i am  facing  issues as Power BI does not natively support displaying HTML content.

Hi @ssget2sumit, to integrate Evidently with external BI platforms, you would need to generate the Reports or Test Suites results as JSON or Python dictionary, then parse the values you need to a database and plot the metrics in the chosen BI tool. You can check this example with Grafana for inspiration: https://www.evidentlyai.com/blog/batch-ml-monitoring-architecture

@elenasamuylova Can this be done with SQL Server?
Repository: evidentlyai/evidently
Issue: labextension support for Pylab Dashbord visualization
Root cause: Unimplemented Feature Gap

We have nbextension mode to show the UI using nbextension in Jupyter notebooks. It does not work for Pylab. Pylab has the  labextention instead. Let's figure out whether we should support it to display UI inside of Pylab.  Currently, ```dashboard.show(mode='inline')``` works for Pylab as well, but labextension  can be more efficient.

Now Report visualisation works in different environments with ```show(mode='inline')```
That is why we won't implement anything specific for Pylab.
Repository: evidentlyai/evidently
Issue: TopKMetrics not showable on evidently ui dashboard
Root cause: Interface Contract Mismatch

In my project, I'm trying to set up a dashboard to display metrics for a search engine (ranking).

Here's a minimal example:

```python
from evidently.ui.dashboards import DashboardPanelPlot, ReportFilter, PanelValue, PlotType, DashboardPanelCounter, CounterAgg
from evidently.renderers.html_widgets import WidgetSize
from evidently import ColumnMapping
from evidently import metrics
from evidently.ui.workspace import Workspace

from evidently.report import Report

#Create the workspace and project
ws = Workspace.create(path="./workspace_evidently")
project = ws.create_project("My test project")
project.save()

dataset = pd.DataFrame([
{
    "score": 0.5,
    "query": "my_query",
    "document_id": "docuemtn_id_1",
    "relevance": 0
},
{
    "score": 0.4,
    "query": "my_query",
    "document_id": "docuemtn_id_2",
    "relevance": 1
},
{
    "score": 0.3,
    "query": "my_query",
    "document_id": "docuemtn_id_5",
    "relevance": 0
},
{
    "score": 0.2,
    "query": "my_query",
    "document_id": "docuemtn_id_4",
    "relevance": 0
},
{
    "score": 0.1,
    "query": "my_query",
    "document_id": "docuemtn_id_3",
    "relevance": 1
}
])

report = Report(
    metrics=[
        metrics.MAPKMetric(4),
    ]
)

column_mapping = ColumnMapping(
    recommendations_type="score", target="relevance", prediction="score", item_id="document_id", user_id="query"
)
column_mapping.task = "recsys"
report.run(
    reference_data=None,
    current_data=dataset,
    column_mapping=column_mapping,
)
ws.add_report(project.id, report)
```

Let's add a dashboard like this:

```python
project.dashboard.add_panel(
        DashboardPanelCounter(
            title="MAPKMetric",
            filter=ReportFilter(metadata_values={}, tag_values=[]),
            value=PanelValue(
                	metric_id="MAPKMetric",
                	field_path=metrics.MAPKMetric.fields.current,
                	legend="MAPKMetric",
                ),
            size=WidgetSize.HALF,
            agg=CounterAgg.SUM
        )
)
project.save()
```

If I now run `evidently ui --workspace “./workspace_evidently”` and go to the “Dashboard” page, I get the following error:

```python
pydantic.v1.error_wrappers.ValidationError: 1 validation error for ParsingModel[float]
__root__
  value is not a valid float (type=type_error.float)
```

From what I understand, this is due to the fact that the TopKMetrics data are panda series showing the evolution of the metric with k ranging from 1 to 10 (useful for showing the curve). But it seems that, when building the Panel, there's no access to fetch the metric at the k I specified when creating my report ` metrics.MAPKMetric(4),`. So, no float available when I do `field_path=metrics.MAPKMetric.fields....` only a Series, which seems to broke the dashboard ui
Repository: evidentlyai/evidently
Issue: RecSys not working
Root cause: Validation Gap

data = {
    'target': [[1, 1],[1,1],[1,1]],
    'prediction': [[0,0],[0,0],[0,0]],
    'user_id':[1,2,3],
    'item_id':[1,2,3]
}

df = pd.DataFrame(data)

data_drift_report = Report(metrics=[RecallTopKMetric(k=1)])

column_mapping = ColumnMapping(recommendations_type='rank', item_id='item_id', user_id='user_id')

report.run(current_data=df, reference_data=df,column_mapping=column_mapping)

report.show() #error  

is there a way for me to achieve the above, I have tried and have not found a way to make it work.



It appears that the `target` and `prediction` columns in your dataset are lists, which is causing the Report generation to fail. The `RecallTopKMetric` expects these columns to contain plain numerical values. Here is your modified example that works:

```
data = {
    'target': [1, 1, 1],
    'prediction': [0, 0, 0],
    'user_id':[1, 2, 3],
    'item_id':[1, 2, 3]
}

df = pd.DataFrame(data)
report = Report(metrics=[RecallTopKMetric(k=1)])
column_mapping = ColumnMapping(recommendations_type='rank', item_id='item_id', user_id='user_id')
report.run(current_data=df, reference_data=df, column_mapping=column_mapping)
report.show()
```