zhiminy/EvalEng huggingface dataset:
Repository: evidentlyai/evidently
Issue: inconsistent dataset_drift value in the data drift report
Root cause: Configuration Error

Hi, inside data_drift_report, there are two metrics (`DatasetDriftMetric` and `DataDriftTable`) reporting `dataset_drift`. Contrary to the `DatasetDriftMetric`, One cannot set the `drift_share` value for `DataDriftTable`. There are situations that `dataset_drift` is **true** for `DatasetDriftMetric` while **false** for `DataDriftTable`. It caused some confusion to some data scientists in my project. 
<img width="1460" alt="Screenshot 2023-07-14 at 09 43 35" src="https://github.com/evidentlyai/evidently/assets/7295145/2e9b9f9a-93c5-4f09-8cf7-dc0db1b7ae6b">

I attached a screen shot above. Inside the given google colab example, I set the `drift_share = 0.4`, and the `dataset_drift` is inconsistent. I also briefly checked the source code, it seems for `DataDriftTable` function https://github.com/evidentlyai/evidently/blob/174bde94d7ba682e125395bfe100b4d4b99f74e2/src/evidently/metrics/data_drift/data_drift_table.py#L43, it does not pass `drift_share` value anywhere while it outputs the `dataset_drift` value. Would it be better to remove `dataset_drift` for `DataDriftTable` function? 

Thanks for sharing @userkkw - it would indeed make sense to pass the `drift_share` parameter to the `DataDriftTable`. We will address this in one of the following releases.
Repository: evidentlyai/evidently
Issue: per_column_stattest_threshold would reset the value of stattest_threshold to default 0.05
Root cause: Algorithmic Error

When I am testing the threshold priority among `per_column_stattest`, `stattest`, and `num_stattest` in `TargetDriftPreset`, I might find a potential bug. 
If I set metric parameters as following: 
```python
  num_stattest="mannw",
  num_stattest_threshold=0.5,
  # per_column_stattest = None,
  # per_column_stattest_threshold = None,
  stattest="ks",
  stattest_threshold=0.4,
```
The `stattest_threshold` in the json report would be **0.5**, and the `stattest` is **Mann-Whitney U-rank test**, which is fine. 
<img width="1453" alt="Screenshot 2023-07-17 at 16 43 11" src="https://github.com/evidentlyai/evidently/assets/7295145/fe8697c8-d23f-4c1b-ba41-b8e42d80731d">


However, if I add `per_column_stattest_threshold` for my feature columns e.g. `AvgRooms`, the `stattest_threshold` would be reset to the **default 0.05**, see the following screenshot:
<img width="1452" alt="Screenshot 2023-07-17 at 16 42 38" src="https://github.com/evidentlyai/evidently/assets/7295145/fbcb7fa7-f379-42db-a3b1-9eedc1b9acdb">

Same thing also happens if I just pass an empty Dict to `per_column_stattest_threshold`, see the screenshot: 
<img width="1456" alt="Screenshot 2023-07-17 at 16 37 01" src="https://github.com/evidentlyai/evidently/assets/7295145/d36a6c68-25df-464a-9ddd-0d125557650d">

-------
I am using the latest Evidently AI version and the google colab examples from the official documentation. 


Thanks for reporting @userkkw, we will take a look!
Repository: evidentlyai/evidently
Issue: Data Drift by Feature standard deviation from mean
GitHub labels: enhancement

Hi,

I was using the Data Drift Tab by Feature.

Is it possible to change the green area that covers one standard deviation from the mean? I wanted it to be 2 std deviation making it less sensible to data drift

Thank you

Hi @Pakard1 , 

unfortunately there are no such parameter in the config yet.

Currently it can be done only at the code level. 
For doing this you need to clone the repo and update Data Drift Table Widget: https://github.com/evidentlyai/evidently/blob/main/evidently/widgets/data_drift_table_widget.py. 
There we have the instruction to calculate the green area width: ```reference_std = np.std(reference_data[feature_name][np.isfinite(reference_data[feature_name])], ddof=1)```,  you might want to multiply it by 2 or 3. 

After all code updates you need to build evidently locally:  
1. run command ```pin install -e .``` in the project folder 
2. install nbextention for visualizations ```jupyter nbextension install --sys-prefix --symlink --overwrite --py evidently```
3. enable it for evidently ```jupyter nbextension enable evidently --py --sys-prefix```

Hi, thank you for you reply. I have changed it but was having problems
deploying it in production on a container.

I Will try you suggestion and see if it works

A terça, 19/10/2021, 20:34, emeli-dral ***@***.***> escreveu:

> Hi @Pakard1 <https://github.com/Pakard1> ,
>
> unfortunately there are no such parameter in the config yet.
>
> Currently it can be done only at the code level.
> For doing this you need to clone the repo and update Data Drift Table
> Widget:
> https://github.com/evidentlyai/evidently/blob/main/evidently/widgets/data_drift_table_widget.py
> .
> There we have the instruction to calculate the green area width: reference_std
> =
> np.std(reference_data[feature_name][np.isfinite(reference_data[feature_name])],
> ddof=1), you might want to multiply it by 2 or 3.
>
> After all code updates you need to build evidently locally:
>
>    1. run command pin install -e . in the project folder
>    2. install nbextention for visualizations jupyter nbextension install
>    --sys-prefix --symlink --overwrite --py evidently
>    3. enable it for evidently jupyter nbextension enable evidently --py
>    --sys-prefix
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/evidentlyai/evidently/issues/68#issuecomment-947043483>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ANXWKRRL6YGYHOM4LMFOZP3UHXB2PANCNFSM5GB7P5ZQ>
> .
> Triage notifications on the go with GitHub Mobile for iOS
> <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
> or Android
> <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
>
>


Hi @Pakard1 

We added this functionality in #122 
You can get more details here https://docs.evidentlyai.com/user-guide/customization/options-for-quality-metrics
Repository: evidentlyai/evidently
Issue: New scikit-learn version 1.3.0 breaks EmbeddingsDriftMetric model drift analysis
Root cause: External Dependency Breakage

Due to a new release of scikit-learn, version 1.3.0, I am having the following error using the `EmbeddingsDriftMetric `:
```
InvalidParameterError: The 'loss' parameter of SGDClassifier must be a str among {'modified_huber', 'squared_hinge', 'huber', 'squared_epsilon_insensitive', 'epsilon_insensitive', 'hinge', 'squared_error', 'perceptron', 'log_loss'}. Got 'log' instead.
```
In the Evidently `model` method from `from evidently.metrics.data_drift.embedding_drift_methods` the `SGDClassifier instance` is using the argument loss as `log`. That was removed from scikit-learn. Right now is just available the list is shown on the error stack.
Repository: evidentlyai/evidently
Issue: Diffrent sizes of refrence dataset and current dataset
Root cause: Unimplemented Feature Gap

@emeli-dral
In case the reference dataset which is usually the historic dataset is much bigger in size as compared to the current dataset, in that case, should we rescale the frequencies so that we can compare distributions of similar size?

ref:
https://commons.apache.org/proper/commons-math/javadocs/api-3.3/org/apache/commons/math3/stat/inference/ChiSquareTest.html

Hi @prarshah ,

thank you for bringing this up!

Currently we do not account for the dataset sizes in this test, it makes sense to rescale it.

To add to it, this test does not work properly when the observed or expected frequencies in each category are too small (each less than 5). For this case it is better to use something like Barnard’s test, which we are adding soon as well.

We have added several drift detection metrics (like Wassershtein distance), which are less sensitive to the dataset sizes.
Repository: evidentlyai/evidently
Issue: ColumnCorrelationsMetric - extension for categorial targets

`ColumnCorrelationsMetric` as far as I know only compares a numeric column with other numeric columns, or a categorical column with other categorical columns. There can be however cases, where a categorical target is highly dependent on some numeric feature (e.g. an outcome 'A' tied to a value 0). 

This relation goes unnoticed in the current metric. Do you plan to incorporate some sort of ANOVA test to make the metric more powerful? Or at least enable a binary/ordinal outcome to be treated as a numeric one to enable correlations?
Repository: evidentlyai/evidently
Issue: Modify scales of plots generated in report
Root cause: Unimplemented Feature Gap

I have been looking at some of the options for data drift report and I haven't seen an option to generate the plots inside the report with the Y-axis modified for example to percentages or to have a different scale than the default. 
Within your documentation the distribution graphs are shown with the Y-axis as share by default however within my tests the results always come out with bin counts

Are you considering to have a parameter to modify those axis related options inside the plots generated for reports?

Hi @andrescardenas-abinbev, 

Thanks for the feature request! It is not currently possible - but we'll implement it in one of the following releases. 

Hi @andrescardenas-abinbev, 

Wanted to update you that we released this functionality recently. 

You can use the "percentage" switch on the plot to switch from absolute values to % - this will help align the distributions when the reference and current datasets have different sizes. Basically, you can switch from:

Absolute values:

![](https://user-images.githubusercontent.com/67064421/259807679-6f7045cf-fcb1-4b32-aa7d-eaa3e4f402dc.png)

To percentage values:

![](https://user-images.githubusercontent.com/67064421/259807773-fc67dd34-dd48-4059-94db-ead494dfedd4.png)

Hope this solves the issue for you!
Repository: evidentlyai/evidently
Issue: Add density plot as an option to DataDriftPreset Report
Root cause: Unimplemented Feature Gap

I am interested in using the `DataDriftPreset` to create a drift `Report`. It would be beneficial for the resulting report to show density plots opposed to the histogram count plots. The purpose being that often times I am trying to compare a `current` dataframe with a much smaller sample size, to a `reference` dataframe that was used in training and contains much more data compared the current dataframe. 

the full code I am using to generate the report looks like this 
```
from evidently.report import Report
from evidently.metric_preset import DataDriftPreset

data_drift_report = Report(
    metrics=[
        DataDriftPreset(
            num_stattest=z_stat_test,
            cat_stattest='psi',
            num_stattest_threshold=0.2,
            cat_stattest_threshold=0.2
        )],
)

data_drift_report.run(reference_data=df_reference, current_data=df_target, column_mapping=None)
data_drift_report.show(mode='inline')
```

Hi @elutins, 

Thanks for sharing the feature request!

We will add the option to switch from the absolute number of observations to the percentage scale on a histogram. (Both options will be available on the plot). This way, if you have a significant difference in sample size, you can switch to the % view to visually compare the distributions on a histogram. 

This will be included in an upcoming release in August.

Hopefully, this will solve the visibility issue!


Thanks @elenasamuylova! looking forward to it 

Hi @elutins, to update - the ability switch between absolute and % view on distrbution plots have been added in Evidently 0.4.1. Hope it will make the distribution analysis more convenient. 

 https://github.com/evidentlyai/evidently/releases/tag/v0.4.1 

looked through the code and still wondering how can this parameter can be specified in a `Report`? eg I am using `ColumnSummaryMetric` to produce a `Report`
```
metrics = [ColumnSummaryMetric(column_name=col) for col in self.live_failed_features]
data_drift_report = Report(metrics=metrics)  # DataDriftPreset(num_stattest=z_stat_test)
data_drift_report.run(reference_data=df_ref, current_data=df_current)
```

It is available directly on the visual plot as a button - you can use it to switch between absolute and % view. So both types of visualizations are always generated and there is no parameter.

Absolute:
<img width="855" alt="Screenshot 2023-08-10 at 17 19 37" src="https://github.com/evidentlyai/evidently/assets/67064421/6f7045cf-fcb1-4b32-aa7d-eaa3e4f402dc">

Percentage:

<img width="844" alt="Screenshot 2023-08-10 at 17 20 02" src="https://github.com/evidentlyai/evidently/assets/67064421/fc67dd34-dd48-4059-94db-ead494dfedd4">





@elenasamuylova thanks for clarifying! great feature addition 💯
Repository: evidentlyai/evidently
Issue: Tabular Data Drift By Binary Classifier
Root cause: Unimplemented Feature Gap

Hi, 

I saw that for text & embedding there is binary classification method:
"Evidently trains a binary classification model to discriminate between data from reference and current distribution"

I want to use same method for tabular data but didn't see it. Is it supported for tabular? If not how can I implement one?

and general question about data drift, can it be run as test in test suite or just report?

Thanks!

Hi @Miriam2040, 

**Classifier drift detection method**

The classifier method is available for embeddings (docs here https://docs.evidentlyai.com/user-guide/customization/embeddings-drift-parameters) and for text data (docs here https://docs.evidentlyai.com/user-guide/customization/options-for-statistical-tests). For text data, it includes text-specific pre-processing.

It is not implemented for tabular data. 

If you want to pass a custom drift detection method, here is the explanation of how to pass a custom function: https://docs.evidentlyai.com/user-guide/customization/add-custom-drift-method

**Data drift test**

Yes, you can use data drift detection as a Test Suite. There is a test preset (`DataDriftTestPreset()`), and separate tests you can choose from: `TestNumberOfDriftedColumns()`, `TestShareOfDriftedColumns()` for the dataset and `TestColumnDrift(column_name='name')` for individual columns.

You can see them in the example notebooks linked here
https://docs.evidentlyai.com/examples 

Or in the all tests list
https://docs.evidentlyai.com/reference/all-tests#data-drift

Great, thanks!
Repository: evidentlyai/evidently
Issue: Dataset size limitations?

Hi, do I understand correctly that evidently uses Pandas under the hood? If so, does this constrain the size of the datasets that it can effectively work with? What's the largest dataset size you're aware of that's been successfully analyzed with evidently in a production context?
Thanks a lot

Hi @EgorKraevTransferwise, 

Yes - currently, Evidently uses pandas. We know of users running Evidently on 100s thousand or low million rows per batch. Since the computation happens in memory, the exact limitation and computation time depend on your infrastructure and the exact Evidently metric(s) being used. 

We are currently working on Spark support - this will be more suited for larger datasets. However, it requires re-implementing the underlying metrics, so we will likely release Spark support only for some metrics initially. 

If this is relevant to your use case and you are considering using Evidently with Spark - let us know which metrics are most important (e.g., data quality, data drift, etc.)
Repository: evidentlyai/evidently
Issue: DaTa Drift | Dataset volume

Hi Team,
Good Day!

We are trying to implement Data Drift detection in our project.
In the Latest documentation of evidently, we found that, for large datasets ,we have sample them before  passing it to evidently.
Please specify , an approximate size of dataset, at what threshold size we have to turn sampling before passing it to evidently.


Hi @nagasaipureti, 

I am afraid we cannot give a precise answer here.

The performance varies based on your infrastructure, the number of rows/columns, and the exact metrics used (e.g., some drift detection methods are faster than others).

Also, the need for sampling may vary based on whether you run reports ad hoc in your notebook (when waiting too long for a report to appear might be inconvenient) or run an automated pipeline (when it's more acceptable that the computation will take some time).

I'd suggest running a few tests on your sample datasets to develop your heuristics here.
Repository: evidentlyai/evidently
Issue: Create parameter to set warning columns
Root cause: Unimplemented Feature Gap

I am building out a test suite with the following code

```
drift_suite = TestSuite(
            tests=[
                generate_column_tests(TestMeanInNSigmas, columns="num"),
                generate_column_tests(TestColumnShareOfMissingValues, columns="all"),
                generate_column_tests(TestShareOfOutListValues, columns="cat"),
               ]
)
drift_suite.run(
            reference_data=self.df_live_reference[live_features],
            current_data=self.df_live[live_features],
            column_mapping=cmap,
)
```

however certain column failures are more concerning than others. Is it possible to pass in a column parameter, where if a test fails for that particular column it appears as a `warning` instead of a `fail`? this would be useful to know if/when I need to take immediate action.

Hi @elutins, 

This is a great feature request, and it is on the roadmap. (The design of the Test Suite output already has a spot for the "warnings"  in addition to "failed" tests). 

Unfortunately, I cannot confirm the exact timeline of when it will be released yet. 

In the meantime, the best way to address this is to create two or several different Test Suites grouping the "critical" or "less critical" checks in different suites and manage the alert/warning workflow externally. 

Hi @elutins, 

We did release this update in v0.4.2.

By default, all Tests will return a `Fail` if the Test condition is not fulfilled. If you want to get a `Warning` instead, use the `is_critical` parameter and set it to `False`. Example: 

```python
data_integrity_column_tests = TestSuite(tests=[
    TestColumnAllConstantValues(column_name='education', is_critical=False),
    TestColumnAllUniqueValues(column_name='education', is_critical=False),
])
```

Let me know how it works for you!
Repository: evidentlyai/evidently
Issue: Bug: Similar Test & Metric with different parameter support
Root cause: Algorithmic Error

Hello @emeli-dral , 

As can be seen from the image  **TestEmbeddingsDrift**  with different parameters does not support **TestSuite** and we end up having result of only first written Test. 

![image](https://github.com/evidentlyai/evidently/assets/40850370/748607ea-f69e-4de3-8193-42eb0fdb87ce)


```
TestSuite(
    tests: Optional[List[Union[evidently.tests.base_test.Test, evidently.test_preset.test_preset.TestPreset, evidently.utils.generators.BaseGenerator]]],
    options: Union[evidently.options.base.Options, dict, List[evidently.options.option.Option], NoneType] = None,
    timestamp: Optional[datetime.datetime] = None,
    id: Optional[uuid.UUID] = None,
    metadata: Dict[str, Union[str, Dict[str, str], List[str]]] = None,
    tags: List[str] = None,
)
```

**Usecase:** 
1. Lets say we want to create one testsuit with same metric but with different parameters being used and based on those we want to drive the retraining etc. Although we can create different TestSuits but again we want to combine results in same response.



Hello @emeli-dral is this possible ?


In general this should be possible, but in this case our de-duplication algorithm thinks it's the same metric, even with different parameters. I will take a look

Hi @ketangangal , we merged the fix #721 
It is available in main now and will be added to the nearest release

Thanks !
Repository: evidentlyai/evidently
Issue: Allow custom label ordering in plot for categorical data
Root cause: Unimplemented Feature Gap

_Using Evidently 0.4.0 on Databricks runtime 10.4_
_Looking at outputs for ColumnSummaryMetric and ColumnDriftMetric_

When generating results plots for categorical data, the labels on the x-axis are ordered by associated value.
I would like to order the results by defined category order.

I have tried assigning category order to the Pandas series that is being evaluated, but that ordering is not carried through to the plot. Would it be possible to include a flag that lets Evidently know to use the Pandas categorical ordering in the output of the plot?

Thanks for the feature request @AvanBerne! 

I am afraid we won't be able to address this in the short term. The reason is that this is a type of parameter can work for all metrics and tests; and library-wide implementation for all checks and output formats will require significant work. 

Hopefully we'll be able to address it in the future.
Repository: evidentlyai/evidently
Issue: Discord link does not work

It seems that https://discord.gg/xZjKRaNp8b has expired. Are you still maintain a discord community?

Hi @burkovae ! Yes, the community is indeed active. I am testing the link and it works on my side and is set to never expire. While we drill down to the core, can you try this temporary link as an alternative? https://discord.gg/tCRgAbny

That's totally strange!

![image](https://user-images.githubusercontent.com/15078122/140448607-2e9ddd26-a99d-45e4-bb9c-83601cd397b6.png)

It tells me, that I have no permission to join.

And now I tested it from another (private) PC and it worked.

Okay, sorry, it was some strange cache invalidation on my part.

@burkovae great that it is solved!
Repository: evidentlyai/evidently
Issue: Evidently 0.3 is compatible with DVC 3, but Evidently 0.4 is not
Root cause: External Dependency Breakage

Output from trying to solve the dependencies (using `poetry`):

```
Because no versions of evidently match >0.4.0,<0.4.1 || >0.4.1,<0.5.0
 and evidently (0.4.0) depends on iterative-telemetry (0.0.5), evidently (>=0.4.0,<0.4.1 || >0.4.1,<0.5.0) requires iterative-telemetry (0.0.5).
    And because evidently (0.4.1) depends on iterative-telemetry (0.0.5), evidently (>=0.4.0,<0.5.0) requires iterative-telemetry (0.0.5).
    And because dvc (>=3,<4) requires iterative-telemetry (>=0.0.7) (1), dvc (>=3,<4) is incompatible with evidently (>=0.4.0,<0.5.0)
```

Could be solved by relaxing the dependency on `iterative-telemetry` on Evidently's side. 
Happy to open a PR if you ok with the change!

After doing some reading, the core issue seems to be that `iterative-telemetry` version 0.0.6 dropped support for python 3.7, which Evidently still offers. Since DVC 3 does not support python 3.7, setting up a dynamic dependency on `iterative-telemetry` based on python version (0.0.5 for 3.7, anything >= 0.0.6 for python 3.8) should allow to solve this without removing python 3.7 support 

That is exactly what I have found out! You can actually just change it to `>=0.0.5`, in this case in py37 env 0.0.5 will be installed and in other envs latest will be used. Can you create a PR with this?