zhiminy/EvalEng huggingface dataset:
Repository: evidentlyai/evidently
Issue: get_binned_data and Reference Normalization Causes Statistical Tests to Become Asymmetric
Root cause: Algorithmic Error

Hello!
Firstly, I would like to express my sincere appreciation for the exceptional work you've done in creating and maintaining this tool.

I'm using your tool for data drift detection and have recently encountered some edge cases. When a distribution (either reference or current) has few values or consists solely of zeros, the statistical tests lose their characteristic symmetry. This is due to the fact that the get_binned_data function only processes the histogram if there are more than 20 unique values in the reference distribution:

```python
n_vals = reference_data.nunique()
if feature_type == ColumnType.Numerical and n_vals > 20:
```

Is there a reason we would want to break this symmetry using the reference distribution?

Additionally, the same issue occurs when using the normalization for the Wasserstein distance by the reference distribution:

```python
norm = max(np.std(reference_data), 0.001)
```

Using only the normalization by the reference also causes an asymmetry in the result.

Given that all these statistical tests have symmetry as a property (PSI, Jensen-Shannon, Wasserstein Distance, etc):

```python
stattest(a, b) == stattest(b, a)
```

by normalizing or making these cases of limitation with n_bins > 20 from reference, we cause this assymetry:
```python
stattest(a, b) != stattest(b, a)
```

I understand that in the case of the Wasserstein distance, the normalization is done to scale the distance according to the maximum variability of the reference, but does it really make such a difference to only compare with the reference instead of taking the minimum or maximum value between the np.std of the two distributions?

```python
norm = max(np.std(reference_data), 0.001) # currently
norm = max(np.std(reference_data), np.std(current_data)) # this could be a solution to solve this issue
```

Is this intentional, or is it a side effect of some more general implementation? Would it be a good idea to make these tests symmetric again by removing the n_vals > 20?
Repository: evidentlyai/evidently
Issue: Monitoring of NLP embeddings
Root cause: Documentation Deficiency

Hello,

**Briefly:**
I'm interested in a guide detailing the calculation of drift in embeddings for NLP-specific tasks. Specifically, I'm dealing with Word2Vec, BERT, OpenAI embeddings, etc.

**Additional context:**
I've explored [this documentation](https://docs.evidentlyai.com/user-guide/customization/embeddings-drift-parameters) and gone through [this tutorial](https://github.com/evidentlyai/evidently/blob/main/examples/how_to_questions/how_to_calculate_embeddings_drift.ipynb). However, the tutorial seems to be centered around image datasets with fewer dimensions compared to NLP embeddings. This raises uncertainty about whether and how I can utilize Evidently for calculating drift in NLP embeddings.

Best regards,
Trine

Hi @TrineEngelund, 

You might find this blog + code example useful. We explain the use of different embedding drift detection methods on text datasets using two different embedding models.
https://www.evidentlyai.com/blog/embedding-drift-detection 

There are also examples of text drift detection with raw text data, e.g. using descriptors (here https://www.evidentlyai.com/blog/unstructured-data-monitoring) and using raw text drift detection (https://www.evidentlyai.com/blog/tutorial-detecting-drift-in-text-data). 

Hi @elenasamuylova,

The blog post is great. Thanks for the quick response! Sorry if I'm missing something, but where can I find the code example for this blog post?

There are 6 links under the "Experiment code" subheader in the blog. Note that it does not show how to visualize the results with Evidently - this is more about the research we did before implementing different embedding drift detection methods in Evidently (it plots the drift score given different artificial shifts introduced). 

For Evidently example, the links you shared in the first message are the correct ones - they show how to apply any of the methods described in the blog for your data.

Thank you so much!
Repository: evidentlyai/evidently
Issue: Classification Report: Handle time series data??
Root cause: Documentation Deficiency

Hi, i hope you're doing well :)!

I wanted to know, there's any feature/preset that could help me to make a plot like this but grouping by timestamp??
<img width="1243" alt="image" src="https://github.com/evidentlyai/evidently/assets/81534737/c123cef3-5ed2-4df4-8786-3e5ad2b558ab">


In order to see if i should re train the model or not, i would really like to see if the accuracy decreases through time.

Thanks in advance!

Hi @benja20029, a Report shows the performance for a particular batch of data. If you want to monitor the model quality over time, you can consider generating such Reports on a cadence (e.g. daily, weekly, etc) and then launching the ML monitoring dashboard that allows building time series plots from individual snapshots. Here is the quickstart for self-hosting: https://docs.evidentlyai.com/get-started/tutorial-monitoring

You can also consider implementing a custom metric with its render for the Report to add a plot showing the chosen quality metric (inside the Report period) over time. However, this requires familiarity with Plotly. Here is the guide for the custom metric: https://github.com/evidentlyai/evidently/blob/main/examples/how_to_questions/how_to_make_custom_metric_and_test.ipynb

I will try the first option then, thanks!

There's any way of generating the dashboard as files that doesn't need to be running in the terminal and instead works with a lambda function or a cron job (for example)??

@benja20029 to create the Reports, you can run a cron job (or orchestrate it however you like - e.g. using a tool like Airflow etc.) This will result in a JSON snapshot (that describes the model performance for a specific period in your case) in a chosen directory. But the ML monitoring dashboard is a service (that reads JSONs from the directory), so you need to run it from the Terminal to launch. 

There is also a hosted commercial version (Evidently Cloud, which is currently in private beta https://docs.evidentlyai.com/get-started/tutorial-cloud) if you do not want to self-host the ML monitoring dashboard.

I understand, thanks!
Repository: evidentlyai/evidently
Issue: To set fixed time interval in dashboard pannel
Root cause: Documentation Deficiency


How to set fixed time interval in dashboard panel for real time monitoring.
I used timestamp () method to set time interval but whenever the no. of report increased the time interval get auto adjusted.


Hi @Priya3Saha, have you tried passing the `date_from: ` and `date_to: ` parameters to the Project?

https://docs.evidentlyai.com/user-guide/monitoring/workspace_project#project-parameters 

> Hi @Priya3Saha, have you tried passing the `date_from: ` and `date_to: ` parameters to the Project?
> 
> https://docs.evidentlyai.com/user-guide/monitoring/workspace_project#project-parameters

Hi,

Thank you, but I have tried this option, but no changes noticed on Dashboard. Actually, finding way to show only past 30 mins report on Dashboard.
Repository: evidentlyai/evidently
Issue: Mutually exclusive features in DataQualityPreset
Root cause: Validation Gap

When a dataquality report is calculated for a dataframe in which there are columns that are mutually exclusive (A is 'None/Null' when B is filled and vice-versa), the cramer_v correlation coefficient crashes. This can somwhat be circumvented in multiple ways (not using the preset and editing your data for the tests that calculate correlation coefficients: filling the None-values, omitting certain columns, etc.) but this is non-optimal and negate the power of how easily presets can be used right now, especially because it becomes impossible to include missingvalue-tests together with correlation-tests in one report.

I think this is relevant part of the traceback after calling report.save_html(...):

  File "...\.venv\lib\site-packages\evidently\metrics\data_quality\dataset_correlations_metric.py", line 171, in _get_correlations
    correlations_calculate = calculate_correlations(dataset, data_definition, sum(add_text_columns, []))
  File "...\.venv\lib\site-packages\evidently\calculations\data_quality.py", line 404, in calculate_correlations
    correlations[kind] = _calculate_correlations(dataset, num_for_corr, cat_for_corr, kind)
  File "...\.venv\lib\site-packages\evidently\calculations\data_quality.py", line 392, in _calculate_correlations
    return get_pairwise_correlation(df[cat_for_corr], _cramer_v)
  File "...\.venv\lib\site-packages\evidently\calculations\data_quality.py", line 365, in get_pairwise_correlation
    c = func(df[columns[i]], df[columns[j]])
  File "...\.venv\lib\site-packages\evidently\calculations\data_quality.py", line 335, in _cramer_v
    chi2_stat = chi2_contingency(arr, correction=False)
  File "...\.venv\lib\site-packages\scipy\stats\contingency.py", line 333, in chi2_contingency
    raise ValueError("No data; `observed` has size 0.")
ValueError: No data; `observed` has size 0.

I'm seeing the same error with evidently `0.4.15`.
Repository: evidentlyai/evidently
Issue: evidently 0.4.15 depends on pydantic<2
Root cause: External Dependency Breakage

Hi, I'm currently in a project that uses airflow 2.8.0 which depends on the pydantic version>=2.3.0.

Is there a version of evidently that uses pymantic>2? Or are there plans to update the project?

Thanks!


Hi @ThiagoZLC, here is a related issue: https://github.com/evidentlyai/evidently/issues/862 

The Evidently UI service depends on FastAPI - this prevents us from simply bumping the `pydantic` version. We are working on addressing this by replacing FastAPI dependency for `litestar`.

solved in version 0.4.16
Repository: evidentlyai/evidently
Issue: Evidently UI blank when opened through SageMaker Notebook instance
Root cause: Documentation Deficiency

I'm using the evidently version 0.4.15 and have started exploring it for a ML monitoring usecase. I tried to launch the server on localhost:8000 which is same as <instance_name>.<some region related comp>/proxy/8000/. The page turns out to be blank when trying to reach out on that URL. 

Is there any piece of info in the evidently doc that talks about dealing with issues like this on a 3rd party cloud. It could be great if there's a small note about this in relation with launching evidently successfully on popular cloud providers.

Same issue

same issue here when trying to integrate into Kubeflow
Repository: evidentlyai/evidently
Issue: ModuleNotFoundError: No module named 'evidently.ui'
Root cause: Environment Incompatibility

evidently  version -> 0.4.15.

When I'm trying to run self-hosted ML Monitoring code example(part 2 )mentioned here - 
https://docs.evidentlyai.com/get-started/tutorial-monitoring#part-2.-add-a-new-project

I'm facing error when I try to import - 
```
`from evidently.ui.dashboards import CounterAgg
from evidently.ui.dashboards import DashboardPanelCounter
from evidently.ui.dashboards import DashboardPanelPlot
from evidently.ui.dashboards import PanelValue
from evidently.ui.dashboards import PlotType
from evidently.ui.dashboards import ReportFilter
from evidently.ui.remote import RemoteWorkspace
from evidently.ui.workspace import Workspace
from evidently.ui.workspace import WorkspaceBase`
```

`---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
Cell In[8], line 12
     10 from evidently.test_preset import DataDriftTestPreset
     11 from evidently.test_suite import TestSuite
---> 12 from evidently.ui.dashboards import CounterAgg
     13 from evidently.ui.dashboards import DashboardPanelCounter
     14 from evidently.ui.dashboards import DashboardPanelPlot

ModuleNotFoundError: No module named 'evidently.ui'`

Any kind of guidance regarding this issue we be really helpful. 

I was using a windows machine , python 3.8 , in linux ,there was no issue but in windows i had to update python from 3.8 to 3.11
Repository: evidentlyai/evidently
Issue: BUG: unable to disable telemetry through environment variables
Root cause: Configuration Error

The [documentation](https://docs.evidentlyai.com/support/telemetry) suggests using an environment variable to disable telemetry.  Normally I would leave it on but it's being blocked by our corporate networking.

I am deploying evidently (`0.4.15`) via a Docker container.  I've tried both setting the environment variable in the `Dockerfile`

```
# https://docs.evidentlyai.com/support/telemetry
# turn off tracking
ENV DO_NOT_TRACK=1
```

As well as within a startup script

```
#!/bin/bash
# startup.sh
export DO_NOT_TRACK=1
evidently ui
```

I can `docker exec` into the container and run `env` and I see the environment variable set.

The logic in `src/evidently/telemetry.py` looks correct.  I'm not sure where else to look but it seems broken to me.

Hey @carlsonp ,
working on it.

cc @mike0sv
Repository: evidentlyai/evidently
Issue: Add relative percent change to TestColumnShareOfMissingValues
Root cause: Unimplemented Feature Gap

## Add relative percent change to TestColumnShareOfMissingValues

I want to be able to make a test pass/fail if the relative percent change of null values exceeds a certain threshold. I understand that the default behavior for the test is to fail _if the share of missing values in a column is over 10% higher than in reference._ as stated in [this section of the doc](https://docs.evidentlyai.com/reference/all-tests#data-integrity). I would ideally like to change this `10%` value to something else. I understand the lt parameter can be set, however that set the absolute threshold with no relation back to the reference dataset. Is there a way to do this?

Here is how I am currently using/creating the test:
```
tests = []
tests.append(
   generate_column_tests(
     TestColumnShareOfMissingValues,
     columns="all",
     parameters={"lt": missing_values_threshold},
  )
)
test_suite = TestSuite(tests=suite_tests)
```

Instead of using the `lt` parameter in the above code snippet, Id ideally like to pass a different parameter that sets the relative percent change threshold from the reference to the current dataset. 

Hi, @elutins - it is a great feature request, and we'd like to implement similar functionality eventually. However, right now, it is not possible to set a custom condition relative to the reference.

You can either:
* use the default auto-generated relative test conditions
* set custom absolute conditions
* set custom conditions relative to an absolute value using `approx` https://docs.evidentlyai.com/user-guide/tests-and-reports/custom-test-suite#custom-conditions-with-approx (you still need to set the value you are referring to).

sounds good - thanks for the update. would be a great feature
Repository: facebookresearch/DomainBed
Issue: underlying_length of train_loader is 1
Root cause: Algorithmic Error

I noticed that you use WeightedRandomSampler when training, but it lead to wrong steps_per_epoch and wrong 'epoch' in logs.
When length==self.INFINITE,  self.underlying_length becomes 1.

```
class FastDataLoader(object):
    INFINITE = 'infinite'
    EPOCH = 'epoch'
    """DataLoader wrapper with slightly improved speed by not respawning worker
    processes at every epoch."""
    def __init__(self, dataset, weights, batch_size, num_workers,
        length=EPOCH):
        super(FastDataLoader, self).__init__()

        if length == self.EPOCH and weights != None:
            raise Exception("Specifying sampling weights with length=EPOCH is "
                "illegal: every datapoint would eventually get sampled exactly "
                "once.")

        if weights == None:
            weights = torch.ones(len(dataset))

        if length == self.INFINITE:
            batch_sampler = torch.utils.data.BatchSampler(
                torch.utils.data.WeightedRandomSampler(weights,
                    replacement=True,
                    num_samples=batch_size),
                batch_size=batch_size,
                drop_last=True)
            print(length==self.INFINITE)
        else:
            batch_sampler = torch.utils.data.BatchSampler(
                torch.utils.data.SequentialSampler(dataset),
                batch_size=batch_size,
                drop_last=False
            )
        self.length = length
        self.underlying_length = len(batch_sampler)
```

ENV:
Python==3.7
Pytorch==1.3.1
Torchvision==0.4.2

Thanks for reporting! I've taken a shot at fixing this in #5
Repository: facebookresearch/DomainBed
Issue: How to train your code in multi gpu?

I wonder how to add the nn.DataParallel in your code to train on multi gpus.

Thanks a lot.

Can you try adding
```python
algorithm = torch.nn.DataParallel(algorithm, device_ids=range(torch.cuda.device_count()))
```
after [this line](https://github.com/facebookresearch/DomainBed/blob/master/domainbed/scripts/train.py#L130)? Learn more about DataParallel [here](https://pytorch.org/tutorials/beginner/former_torchies/parallelism_tutorial.html).

As a standalone, this doesn't work for me and runs into the following error message:

`torch.nn.modules.module.ModuleAttributeError: 'DataParallel' object has no attribute 'update'`

I guess one needs to use a subclass along the lines of:

```
class MyDataParallel(nn.DataParallel):
    def __getattr__(self, name):
        return getattr(self.module, name)
```

According to [this](https://pytorch.org/docs/stable/notes/cuda.html#cuda-nn-ddp-instead) it is also recommended to use `DistributedDataParallel`, instead of `DataParallel` to do multi-GPU training, even if there is only a single node.

However, I haven't figured out how to properly do it yet...

I think you have the right idea re. subclassing `DataParallel`. My understanding is that `DistributedDataParallel` requires a special launcher utility so it'll be harder to integrate with the sweep code.

Please comment if multi-gpu training is a priority in your workflow, so I will re-open this issue.

> Please comment if multi-gpu training is a priority in your workflow, so I will re-open this issue.
Hello, because my hardware is limited, there is not enough memory to test on the domainnet dataset, so parallel multi-GPU training is required, but it cannot be done according to the code you provided and my own modifications

Yes. It will be nice to reopen this and enable this. I would like to work on Transformers and most of them dont fit in a single GPU

> Please comment if multi-gpu training is a priority in your workflow, so I will re-open this issue.

Hello, this is a nice work, it's so helpful. But I need to train in a huge dataset, so multi-GPU training is necessary. I try to use DataParallel, but I have been haunted by it for 2 days and haven't worked out it. Please reopen this issue🥲
Repository: facebookresearch/DomainBed
Issue: Discrepancies between CORAL implementation and paper
Root cause: Documentation Deficiency

Following equation (1), (2), and (3) in the [paper](https://arxiv.org/pdf/1607.01719.pdf), I am not sure if this is the same as [the current 
implementation](https://github.com/facebookresearch/DomainBed/blob/main/domainbed/algorithms.py#L630-L640). 

### Equations from paper
![image](https://user-images.githubusercontent.com/28000055/177055573-471d0ebb-39f9-4601-9b6b-0a4db2f1d956.png)

### Code example
```python
import torch
import numpy as np

# Create a tensor for two domains with two samples per batch each 
data = [
    [[1.0, 0.0], [1.0, 1.0]], 
    [[0.0, 0.0], [1.0, 1.0]]
]
phis = torch.tensor(data)

cov_torch = torch.cov(phis[0])
# tensor([[0.5000, 0.0000],
#         [0.0000, 0.0000]])

cov_np = np.cov(data[0])
# array([[0.5, 0. ],
#        [0. , 0. ]])

mean_x = phis[0].mean(0, keepdim=True)
cent_x = phis[0] - mean_x
cova_x = (cent_x.t() @ cent_x) / (len(phis[0]) - 1)
# tensor([[0.0000, 0.0000],
#         [0.0000, 0.5000]])
```

The output is even more different for the second domain
```python

cov_torch = torch.cov(phis[1])
# tensor([[0., 0.],
#         [0., 0.]])

# Left out numpy for brevity, but same results as `torch.cov()`

mean_y = phis[1].mean(0, keepdim=True)
cent_y = phis[1] - mean_y
cova_y = (cent_y.t() @ cent_y) / (len(phis[1]) - 1)
# tensor([[0.5000, 0.5000],
#         [0.5000, 0.5000]])
```

Putting it all together, the resulting penalties differ quite significantly too: 
```python
penalty = torch.norm(torch.cov(phis[0]) - torch.cov(phis[1]))
# tensor(0.5000)

penalty = (cova_x - cova_y).pow(2).mean()
# tensor(0.1875)
```

And then I'm not even considering the [mean_diff](https://github.com/facebookresearch/DomainBed/blob/main/domainbed/algorithms.py#L637), which also does not seem to be mentioned in the paper. 

I can imagine that I'm missing something here. Could you elaborate if the deviation from the paper is intentional and if so, why? 

Ah wait, I think I see it now. 
It's because I should be computing the column-wise covariance, not row-wise. So that gives: 
```python
cov_torch = torch.cov(phis[0].T)
# tensor([[0.0000, 0.0000],
#         [0.0000, 0.5000]])
```

I'm still not 100% sure why both the first and second moment are matched rather than just the second, but I can see why that would be important. Closing the issue for now.
Repository: facebookresearch/DomainBed
Issue: Possible mistake in `AbstractDANN`?

I was going through the [DANN implementation](https://github.com/facebookresearch/DomainBed/blob/main/domainbed/algorithms.py#L205-L293) and there's a couple of things that seem off to me. My confusion mostly relates to lines 266-272:
```python
disc_loss = F.cross_entropy(disc_out, disc_labels)
disc_softmax = F.softmax(disc_out, dim=1)
input_grad = autograd.grad(disc_softmax[:, disc_labels].sum(), [disc_input], create_graph=True)[0]
grad_penalty = (input_grad**2).sum(dim=1).mean(dim=0)
disc_loss += self.hparams['grad_penalty'] * grad_penalty
```
The following things raise questions and I don't see why it's implemented this way: 
- The `disc_loss` is the cross-entropy between the true and predicted domain labels. This cross-entropy will of course be high when the discriminator is doing poorly and low otherwise. This is then added to `disc_softmax[:, disc_labels].sum()`, which seems to follow the opposite pattern. If the softmax output for the correct class is high (i.e. the discriminator is doing well), this value is high and low otherwise. By adding these two together, don't they cancel out? At least partially?
- Whether optimizing the discriminator or predictor, the `disc_loss` on line 272 always includes the `grad_penalty` and `F.cross_entropy(disc_out, disc_labels)`. When optimizing the discriminator, this is returned directly. Otherwise, you flip the sign and multiply by lambda. If I understand correctly, this effectively means that when optimizing the predictive loss, you try to maximize the penalty, making for a good domain classifier and thus making the features _not_ domain invariant.
- Why compute `disc_softmax[:, disc_labels].sum()`? Is there any work that suggests this is a better option instead of just `-F.cross_entropy(disc_out, disc_labels)` for a gradient reversal?

Something like this makes more sense to me (pseudo-code just to illustrate the idea): 
```python 
disc_loss = F.cross_entropy(disc_out, disc_labels)

if discriminator_step:
    return disc_loss
else:
    input_grad = autograd.grad(disc_loss, [disc_input], create_graph=True)[0]
    grad_penalty = (input_grad**2).sum(dim=1).mean(dim=0)
    return classifier_loss + self.hparams['lambda'] * -grad_penalty
```

I discussed it with my supervisor and while I can't quite wrap my head around it yet, I think it's ok! Sorry for the distraction.