zhiminy/EvalEng huggingface dataset:
Repository: evidentlyai/evidently
Issue: QUESTION : Test with Custom Metric
Root cause: Documentation Deficiency

Hi , 

I want to calculate a weighted Drift Score that encounter some feature weights

I want to get Drift scores (i.e. PSI that more or less doesn't depend on feature type (categorical/numerical) and weight them according feature weights , get weighted Drift score and test it to be less than 0.1. Does it sound right statistically ?

Later I might use it in report also. Any DOC can help me on it ?

I see this [notebook](https://github.com/evidentlyai/evidently/blob/main/examples/how_to_questions/how_to_make_custom_metric_and_test.ipynb) 

But it uses input data as basis for calculations and my calculation above requires using already existing metrics (Individual Drift score) . I can imagine there no need for custom metric test here and best approach exists.
Repository: evidentlyai/evidently
Issue: Feature request: support array data inputs  
Root cause: Unimplemented Feature Gap

**Description:**

Sometimes users have array or vector-type columns that have a sequence of numbers, e.g. [1, 3, 4, 5, 6] 
Currently, we can only perform drift detection with such inputs if a user splits it into separate columns before passing the data to Evidently.

**Possible implementation:**
- We can support a new column array/vector type in column mapping, perform the split on the background, and apply embedding drift detection methods to the generated columns.
- Worth considering how to implement it for sequences of varying lengths.
Repository: evidentlyai/evidently
Issue: ModuleNotFoundError: No module named 'evidently.dashboard'
Root cause: External Dependency Breakage

I was able to install evidently using pip install. However, when I add
from evidently.dashboard import Dashboard
from evidently.dashboard.tabs import DataDriftTab, CatTargetDriftTab
it is giving me the error in the subject  line.

Hi @sumesh1812, the `dashboard` refers to the old API in Evidently version 0.2.8 and earlier. The latest current Evidently version is 0.4.10.

You must either install an older Evidently version or use a more up-to-date API (recommended). Here is a migration guide if you have some old code you want to upgrade: https://docs.evidentlyai.com/support/migration

If you are new to Evidently, it is best to start with the Get Started tutorial https://docs.evidentlyai.com/get-started/tutorial and refer to the official documentation with latest examples.

Hello, 
I`ve just installed the 0.7.15 and Dashboard is still not there
Repository: evidentlyai/evidently
Issue: Question: For images related monitoring

Hi
 
  i wish to know is there any example that i can reference how to use evident ai for image related tasks such as object detection and classification? thanks

Hi @lchunleo, 

Evidently currently does not work with raw image data, so there are some limitations:
- For model quality (classification), you can use the usual Classification Quality Reports / Test Suites. 
- If you have tabular metadata about your images, you can use tabular drift detection methods.
- If you have embeddings, you can use embedding drift detection methods (e.g Euclidean distance, model-based drift detection, etc.)

However, there are no specific presets or metrics focused on computer vision at the moment.
Repository: evidentlyai/evidently
Issue: Improve test presets to handle primary key columns
Root cause: Unimplemented Feature Gap

Thanks to the test presets, Ex: `DataQualityTestPreset` one can get up and running with `evidently` immediately. During my early exploration of the package I noticed some failing tests with my dataset and want to capture it as potential idea for improvement.

| Current dataset | Reference dataset |
| --- | --- |
| <img width="1047" alt="Screenshot 2023-12-14 at 08 08 04" src="https://github.com/evidentlyai/evidently/assets/2899501/bbf531ad-3438-4818-a649-a3685a5d7f1a"> | <img width="1047" alt="Screenshot 2023-12-14 at 08 07 53" src="https://github.com/evidentlyai/evidently/assets/2899501/977c2c05-663e-426f-80f8-834c9b0339d2"> | 

Like we see in the screenshots above, the dataset has an `ID` column which is the unique primary key column on which tests have run and the result has come out as a test failure. Since, every value of `ID` occours only once we see that both the current and reference datasets have `count` in the Top 10 as `1`.  In such a scenario, is this test failure relevant?

Maybe the recommended solution here is to pass all the columns in the DataFrame, except for the `id` column while calling `DataQualityTestPreset`. That does make sense. But, should the preset itself be intelligent to handle the `id` column? 🤔 

---

_NOTE: Please feel free to close this if it's not actionable for now._
Repository: evidentlyai/evidently
Issue: Export Evidently Reports and Tests to flat tables/dataframes
Root cause: Unimplemented Feature Gap

Hello,

We plan to integrate Evidently to our data platform (pandas, spark, data lake, ...), and we'd appreciate to export data from reports/tests as flat tables (e.g., dataframes) instead of nested structures (e.g., dicts).

For instance, we want to export data from `metrics.DataDriftTable` as follow:

```python
evidently_report = Report(
    metrics=[metrics.DataDriftTable()],
)
evidently_report.run(
    current_data=current,
    reference_data=reference,
    column_mapping=evidently_column_mapping,
)
# result
records = evidently_report.as_dict()
result = records["metrics"][0]["result"]
drifts = result["drift_by_columns"]
# evaluation
evaluation = pd.DataFrame(drifts.values())
evaluation = evaluation.drop(columns=["current", "reference"])
```

We would love to export the results by simply using `report.as_dataframe()` (to be agnostic of pandas, spark, or other frameworks like polars).

You may also consider the upcoming Dataframe API: https://data-apis.org/dataframe-api/draft/index.html.

Hi @mederic-hurier-partner ,
Thank you so much for raising this up!

We previously had an attempt to implement this and it looks like the time has come :) 

There are two issues with transforming the report into a flat table:
1. There is no universally good way to understand which dataframe is desired for each metric. Additionally, different users may need dataframes of different structures.
2. The second problem is that it is unclear how to merge dataframes from different metrics in one report into a single dataframes. On the one hand, metrics can be completely different with various fields, and on the other hand, they can be identical (for example, from generators - many identical metrics calculated for different columns), and in this case, merging is desired.

So, tanking this into account we propose the following approach:

1. Firstly, we will implement a not-so-elegant but universal way to create a dataframe by simply flattening the metric into a series with complex keys.
2. We will also implement a mechanism to override this logic and redefine the transformation methods into a dataframe for several important metrics. In particular, we will use what you have extended for the DataDriftTable as a reference.

After this, we will be able to accept PRs for redefining the dataframe format for metrics that you use :)
How does it sound?

Hi @mederic-hurier-partner!

We polishing up ``as_pandas`` implementation here #966 
We already have a nice pandas renderer for ```DataDriftTable``` metric, feel free to check it out in feature/as_dataframe branch. We would love to know you opinion :)
Repository: evidentlyai/evidently
Issue: Differences in metrics rmse / rmse_default and what is me_default_sigma in the Report json output
Root cause: Documentation Deficiency

I am using your library to work out when to send an alert. My colleagues, wants me to send an alert when the RMSE value is high or the z-score or the sigma gets out of the 2 standard deviations. I want to use the Report json output for these values, however, the metrics are unclear. Could you please clarify the following?

1) What is the difference between rmse and rmse_default and which one should I use?
2) What is me_default_sigma? Can I use that if I am using rmse?

Thank you.

Hi @GwenVCX ,
Thank you for your interest!

1. What is the difference between rmse and rmse_default and which one should I use?
The rmse is a root mean squared error, which is calculated on top passed target and prediction data. This is a standard metric. If you something like ```sklearn.mean_squared_error(target, prediction, squared=False)``` for your data, you will get this rmse value.
However, together with this standard rmse we also calculate a root mean squared error for a "dummy model", so that one can compare the current rmse with a baseline. In our case dummy model is the model, which always predicts a mean target value for each object. So rmse_default is a error which you will get if you calculate an average target value on you dataset, and then use it as a model prediction for all objects.

2. What is me_default_sigma? Can I use that if I am using rmse?
me_default_sigma is a standard deviation around prediction error. This is somewhat like ```(target - prediction).std()``` . I would not go for me_default_sigma instead of rmse because me_default_sigma does not give us any information about mean error itself. It gives us only information about how much individual errors are deviated from the mean error, however this mean error can possibly be high or low.

I hope it has helped to clarify this things!
Repository: evidentlyai/evidently
Issue: Spark dataframes integration?
GitHub labels: enhancement

Thank you for this nice project. 
I was wondering if there is going to be any integration with spark data frames or big data with your work. 
Really sorry if that integration exists and I couldn't find it. 

Hi @echarso,
Thank you for this question. 

Currently, the tool works only with pandas DataFrames or CSV files (CLI version). This means that you can either transform Spark DataFrame to Pandas DataFrame and then run evidently. In this scenario, having a smaller data sample will make sense. 

We are adapting tool to larger amounts of data; this will be addressed in the next releases. 


Aloha @emeli-dral ,

I was wondering if there has been any updates regarding any plans or further discussion regarding integration with Spark DataFrames? Extremely excited to see the library continue to grow :D 

Cheers

Hi @echarso, @lowballedintern, we are now starting to work on the beta for Spark integration. I was wondering if any of you'd be open to chatting about how you want to see that implemented?   

If yes, feel free to stop by Discord https://discord.com/invite/xZjKRaNp8b, drop a line to hello@evidentlyai.com, or maybe describe here how you'd see the ideal solution?

Hi, may I know if is there any update on the spark integration? is there any timeline for this? Thank you!

Is it possible to use Evidently with Spark Dataframe now? I have huge amount of data in spark dataframe and converting it in pandas dataframe would be time taking. What are other ways to integrate it? Let me know if I can utilize beta version of the feature.

Hi @prity-k, 

Spark support is currently in development. If you want to test it pre-release, here are the instructions (currently works from several data drift metrics): https://github.com/evidentlyai/evidently/pull/806 

I see that Spark DataFrames has been moved to a legacy module. Could somebody elaborate on this decision and the future plans for supporting Spark DataFrames?
Repository: evidentlyai/evidently
Issue: After the Linux server deployment, the Test_suite display failed, but the Report was fine
Root cause: Environment Incompatibility

Evidently is deployed to a personal Linux server, and Evidently's Test_suite visualization function is accessed based on ip and port number. Clicking the "view" button on the page cannot be displayed, and "Somthing went wrong" is displayed.
My environment is an internal environment that cannot connect to the Internet

Hi @msyJY,
Thank you for reporting this!

Could you please specify evidently package version?
Do you install evidently from pypi or dockerhub?

The Evidently package version is 0.4.10 and the installation method is pypi
Repository: evidentlyai/evidently
Issue: Datastructure for classification report
Root cause: Validation Gap

hello,

I would like to use evidently to analyze my model results. I use YoloV8 for object detection. During detection, there are of course true positives, false positives and false negatives.

I enter my results in two columns of the pandas-dataframe. In the column "target" the ground-truth and in the column "pred" the symbol class predicted at this bounding box. There is a value for false positives, but none for false negatives. This results in an error message in the ClassificationConfusionMatrixRenderer class:
`curr_matrix.labels = [target_names[x] for x in curr_matrix.labels]`
`KeyError: ''`

'' is the value in the pred column for false negatives.

With which value do I have to enter false negatives in the pred column so that the error does not occur?

Many thanks for the answer.

Kind regards

Robb

Hi @RobbStarkAustria, 

Are you using column mapping to specify the name of the prediction column?

If not, check out these docs https://docs.evidentlyai.com/user-guide/input-data/column-mapping#prediction-column-s-in-classification or an example notebook for different options on how to map your input data: https://github.com/elenasamuylova/evidently/blob/main/examples/how_to_questions/how_to_use_column_mapping.ipynb

If the issue persists, could you share a small reproducible code example? 

Hi @elenasamuylova,

Thank you for your quick answer.

I have read the suggested pages. I am not sure if I have understood everything correctly or if I am using the wrong data structure.

So I'll send you a csv file of my dataframe and the code I used. Maybe you can recognize my misunderstanding.

        data = {"target": target, "pred": pred}
        evidently_df = pd.DataFrame(data)
        column_mapping = ColumnMapping()
        column_mapping.target = 'target'
        column_mapping.prediction = 'pred'
        column_mapping.target_names = cl_list
        column_mapping.task = 'classification'

        classification_report = Report(metrics=[
            ClassificationConfusionMatrix(),
            ClassificationQualityByClass(),
            ClassificationQualityMetric(),
        ])

        classification_report.run(reference_data=None, current_data=evidently_df, column_mapping=column_mapping)
        classification_report.save_html("report.html")

Index 0-225: true positives
Index 226-228: false positives
Index 229-230: false negatives
[evidently_issue.csv](https://github.com/evidentlyai/evidently/files/13797173/evidently_issue.csv)
[cl_list.csv](https://github.com/evidentlyai/evidently/files/13797192/class_list.csv)
Repository: evidentlyai/evidently
Issue: Does the Evidently local deployment Report feature display only preset metrics_presets?
Root cause: Documentation Deficiency

Hi～
I am Evidently deployed locally, and I think that the default "metric_presets" have too many features for me to use, and I would like to show only a few of them.
However, I find that the parameter filled in does not present results for other separate functions except Preset, which I guess may be unsuitable for a separate function.
In the following figure, one is the code of my Report function, I have separated the preset functions and only retained the functions I need; the other is the Report function directory displayed after my deployment, which does not show the functions I retain. I would like to ask whether it is not supported at present.
![7171704872095_ pic](https://github.com/evidentlyai/evidently/assets/95612623/5a203bee-6aaf-4e46-bd98-7ff5c0a4dbc2)
![7161704871958_ pic](https://github.com/evidentlyai/evidently/assets/95612623/918b6f16-2b6b-4dfe-829c-6fc03e84e37e)
The question I expressed may be a little abstract, because my English is not very good, I hope you can understand.
Many thanks for the answer.
Kind regards

Hi @msyJY, if the Report includes the Preset, it is automatically passed as metadata. In other scenarios, you can assign custom tags/metadata. For example, you can add the tag "model quality" or "data drift" (or any other custom tag you want) to a snapshot. After you add the tag or metadata, it will be visible in the interface, and you will be able to filter the Reports in the UI.

Here is the documentation: https://docs.evidentlyai.com/user-guide/monitoring/snapshots#add-tags

> 嗨，如果报告包含预设，它将自动作为元数据传递。在其他情况下，您可以分配自定义标记/元数据。例如，您可以将标记“模型质量”或“数据偏移”（或所需的任何其他自定义标记）添加到快照中。添加标记或元数据后，它将在界面中可见，并且您将能够在 UI 中过滤报告。
> 
> 以下是文档：https://docs.evidentlyai.com/user-guide/monitoring/snapshots#add-tags

Thank you very much～
Repository: evidentlyai/evidently
Issue: Ranking metric coverage under PopularityBias doesn't use k parameter
Root cause: Algorithmic Error

Hi!
Recently I decided to adopt Evidently to automate reporting around a recommender model, focusing on ranking metrics first.

I believe there might the a problem with the implementation of coverage under `PopularityBias` metric. The formula present in the [docs](https://docs.evidentlyai.com/reference/all-metrics/ranking-metrics#id-2.-coverage) includes the K parameter:
Coverage = Number of unique items recommended@K / Total number of unique items

However, the value of the coverage returned by the `Report` does not change upon varying k (5, 10, etc.). I also checked the source and [the calculation does not include k parameter](https://github.com/evidentlyai/evidently/blob/ccade89689a6b9f9afbe46fbd8589ecc083477d3/src/evidently/metrics/recsys/popularity_bias.py#L121).
 
This looks like a bug to me, but would like some confirmation whether this is the intended behaviour or not. If it is indeed a bug I would be happy to contribute with a fix.

Thank you for the amazing project! 

Hi @soogui , thanks for reporting this - it does indeed look like an omission. Would you like to send a pull request with a fix?

Sure, I would be happy to contribute!

Hey @soogui , are you still working on this? Let me know, if not, I can take this up.

@sky-2002 Thanks for checking in! I plan to work on this in the next few days.
Repository: evidentlyai/evidently
Issue: Question on non-encoded categorical feature
Root cause: Unimplemented Feature Gap

Hi, 

I noticed that drift reports ignore the columns that are categorical and not encoded. For example, look at the native-country, race, sex, etc. columns in the following dataset:

![image](https://user-images.githubusercontent.com/54699234/145296887-05182180-54a1-4d8a-afa8-3f98ca7f4251.png)

How should they be treated? At the moment evidently will generate a report but it will ignore all the categorical features that are not encoded in the report even though they appear correctly in `ColumnMapping`. See below:

![image](https://user-images.githubusercontent.com/54699234/145298453-3fdafe9a-c5d1-45b6-a88c-a91b07b99f4c.png)

Report:
![image](https://user-images.githubusercontent.com/54699234/145298999-e9b65403-ea31-4f08-bb38-0f1c3921cc53.png)


Can you share some piece of code? That would be an excellent test case!

Hi @moezali1 ,

Try new version (v0.1.35.dev0). It contains several fixes in determining and filtering of categorical feature and should fix your problem.

Hi @moezali1,
hope those fixes have helped. 

To be honest, not encoded categorical features are not supported in all the Dashboards, but this will be fixed soon.
We will support the usage of not encoded categorical features in all relevant reports. This work is in progress: https://github.com/evidentlyai/evidently/pull/113
Repository: evidentlyai/evidently
Issue: DataQualityPreset graph for numerical feature does not look right.
Root cause: Algorithmic Error

In a report with DataQualityPreset(), the graph for the passenger_count numerical feature does not look right. 

The most common value is 1 but from the graph you can see value 0 has the most row count.

![image](https://github.com/evidentlyai/evidently/assets/12087142/e2a73ef7-086b-41a3-86b4-0a9d10debf9a)

The python code for the sample that generated report is
[data_quality_report.txt](https://github.com/evidentlyai/evidently/files/13985621/data_quality_report.txt)

The input csv is 7.5MB. If it is ok, I can attach it as well.

Evidently version is the latest 0.4.14

Hi @venkat1m ,
Thank you for raising this up! 

As far as I can get from the screenshot the scale grade in the picture is pretty large. Could you please try to zoom in a picture? I'm trying to figure out what are X coordinates of the bin in the histogram. Or alternatively if the dataset is public I'd love to get a link to the data and do it myself.

Hi @emeli-dral 

I have copied the csv, generated report html to my github page.

The dataset is a subset of nyc trip fare which is publicly available.
https://github.com/venkat1m/samples/tree/main/evidently

Thanks



Hello, @emeli-dral 
I am also interested in this question - So the report OP posted, looks like it has some outliers where mean is 1.68, but max is 200 so the binning logic put all data in one bin.
My question - is there a way to deal with this within the library, by defining the number of bins or is dataset cleaning expected to be done outside of the library before generating report?
I've looked around and it seems like there used to be some customization possibility in earlier versions but now I don't see a way to pass such options
Thanks!
Repository: evidentlyai/evidently
Issue: χ2 test usage in border cases
Root cause: Validation Gap

Beware that there are some border cases when evidently computes a statistic on categorial data columns:

Now consider these two cases:

```
def test_current_data_contains_one_class_less(self):
        reference = DataFrame({
            'column_name': ['a'] * 5 + ['b'] * 5 + ['c'] * 5
        })
        current = DataFrame({
            'column_name': ['a'] * 8 + ['b'] * 3
        })
        result = chisquare_stattest.chi_stat_test(reference['column_name'], current['column_name'])

def test_reference_data_contains_one_class_less(self):
    reference = DataFrame({
        'column_name': ['a'] * 5 + ['b'] * 5
    })
    current = DataFrame({
        'column_name': ['a'] * 8 + ['b'] * 3 + ['c'] * 5
    })
    result = chisquare_stattest.chi_stat_test(reference['column_name'], current['column_name'])
```

In the fist case the `p_value` will be 0 because (when checking the code):
```
f_exp = [5, 5, 5]
f_obs = [8, 3, 0]
chisquare(f_exp, f_obs)
Out: Power_divergenceResult(statistic=inf, pvalue=0.0)
```
In the second case the statistic is actually fine
```
f_exp = [5, 5, 0]
f_obs = [8, 3, 5]
chisquare(f_exp, f_obs)
Power_divergenceResult(statistic=7.458333333333333, pvalue=0.024012838199733478)
```

That confuses me somehow. I find it alarming when `current_data` does have more classes than the `reference_data`. It means, is see new labels in new data that I haven't considered in reference data!

Any thoughts?