zhiminy/EvalEng huggingface dataset:
Repository: evidentlyai/evidently
Issue: unable to view dashboards in jupyter notebook
Root cause: Environment Incompatibility

hi, i am trying out the notebooks in examples, but unable to view the dashboards in notebooks, i just stops with a message "Loading.."
PFA the screenshot of the issue
![image](https://user-images.githubusercontent.com/72725647/128799506-93c334bb-0a76-4dc9-9779-6e998baf00b5.png)


Hi @dalavayi !

Could you please tell, if the report saving works well (command ```save('path')``` instead ```show()```)? It will help to figure out the reason of the error.

And just to check: make sure you had nbextention installed and enables before using the tool to generate reports in the notebook:
```$ jupyter nbextension install --sys-prefix --symlink --overwrite --py evidently```
```$ jupyter nbextension enable evidently --py --sys-prefix```

hi @emeli-dral , the .save() method is working. 
I tried running the commands shared above, but unable to install the nbextension due to this error :
OSError: [WinError 1314] A required privilege is not held by the client: 'mypath\\lib\\site-packages\\evidently\\nbextension\\static' -> 'myapath\\share\\jupyter\\nbextensions\\evidently'

@dalavayi got it. We tried to reproduce the error on our side and we believe that the problem may be in the absence of the rights to create a symlink. We tried to install the nbextension without using a symlink and it worked for us. 

I suggest you do the same, meaning instead of the command 
```jupyter nbextension install --sys-prefix --symlink --overwrite --py evidently```
try to run in the command
```jupyter nbextension install --sys-prefix --overwrite --py evidently```

Hope this will help.

Hi @dalavayi,

I wanted to let you know that we updated the ```show()``` method. Now it has the argument ```mode```, which can take the following options:
- auto - the default option. Ideally, you will not need to specify the value for mode and use the default. But, if it does not work (in case we failed to determine the environment automatically), consider setting the correct value explicitly.
- nbextension - to show the UI using nbextension. 
- inline - to insert the UI directly into the cell.

I suggest you to try run the ```show``` method with the``` mode='inline'```. This might help to load visuals without the need to install nbextention.

Also, ```evidently``` now works on Google Colab, you might find it convenient as well. Here is an example: [Iris](https://colab.research.google.com/drive/1uFN0ZWNLk3ExylBywq_M8oGkmac0lDrq)
Repository: evidentlyai/evidently
Issue: DataDrift on datasets having different dimensions 

Hi Community,

I am trying to find out **data drift** using **_Wasserstein distance_** on two datasets. 

My question is,
I have one dataset with only **1 sample** (_1x300 features_) and other dataset with **100 samples** (_100x300 features_) and I want to check how that one sample is drifted from 100 others? Can I put dataset with 1 sample (_1X300_) as **current dataset** and 100 sample dataset (_100x300_) as **referenced dataset**?

One more add on questions.
Is **_Wasserstein distance_** right statistical test in this case?

Any suggestions are highly appreciated !
@elenasamuylova 

Hi @IITGoaPyVidya, it looks like you need an outlier/anomaly detection approach rather than drift detection here. I am afraid we do not have this functionality in the library yet. (Here is a blog that might be relevant https://www.evidentlyai.com/blog/ml-monitoring-drift-detection-vs-outlier-detection)

The current drift detection implementation assumes that you compare two datasets with multiple observations each. It will then compare each column (feature) in two datasets to detect per-column drift and allows you to define a dataset drift function (the share of drifting columns). 

Neither Wasserstein distance nor other statistical tests in the library are suitable for outlier detection (comparing one observation to many). Supporting outlier detection is in the list of features we are considering adding, so thanks for highlighting it might be needed!
Repository: evidentlyai/evidently
Issue: installation and quickstart docs
Root cause: Documentation Deficiency

It might be easy to miss that a plugin is required to run evidently in jupyter, e.g.

Website is only listing "pip install": https://evidentlyai.com/
Users are used to to type "pip install packagename" and jump to the quickstart of the docs (skipping the installation part)

Possible solution:
- add the installation of the plugin to the website instructions as optional
- add a warning to the jupyter part of the quickstart that the plugin must be installed

Hi @rmminusrslash, thanks for sharing - this definitely makes sense! Added the warning regarding the plugin to the Quick Start part of the docs ☑️.
Repository: evidentlyai/evidently
Issue: Add test for date ranges
GitHub labels: enhancement

Test Group: Data Quality

Feature request: to add the ability to test that values in a DateTime column are within a specific range.

Source: Discord conversation https://discord.com/channels/815991896916361276/815992391076806657/threads/1044973406929571860 

Hi, to address this, would it be possible to adjust the following classes: 

- [class TestValueRange](https://github.com/evidentlyai/evidently/blob/aa9135e7932108c7ce0fac3ede7472e5d149fcca/src/evidently/tests/data_quality_tests.py#L1019) from [\tests\data_quality_tests.py](https://github.com/evidentlyai/evidently/blob/aa9135e7932108c7ce0fac3ede7472e5d149fcca/src/evidently/tests/data_quality_tests.py)

- and any other inherited class such as [ColumnValueRangeMetric](https://github.com/evidentlyai/evidently/blob/aa9135e7932108c7ce0fac3ede7472e5d149fcca/src/evidently/metrics/data_quality/column_value_range_metric.py#L58) from [metrics\data_quality\column_value_list_metric.py](https://github.com/evidentlyai/evidently/blob/aa9135e7932108c7ce0fac3ede7472e5d149fcca/src/evidently/metrics/data_quality/column_value_range_metric.py)

Adjust them to accept Timestamp types (right and left) and not only [Numeric](https://github.com/evidentlyai/evidently/blob/aa9135e7932108c7ce0fac3ede7472e5d149fcca/src/evidently/utils/types.py#L9).

The [column.between()](https://github.com/evidentlyai/evidently/blob/aa9135e7932108c7ce0fac3ede7472e5d149fcca/src/evidently/metrics/data_quality/column_value_range_metric.py#L82) pandas method (that is used in  `ColumnValueRangeMetric` ) can be used with Timestamps.

What do you think?





Hi @AntonisCSt ,
This sounds like a good approach!

I'm very sorry for the late reply. The things is that we worked on ```MetricResult``` refactoring and we didn't want to update any metric before we finish. We finally transitioned ```MetricResult``` classes from DataClass to Pydantic in the latest release, and now ready to update Metrics. 

Would you consider making a contribution?

Hi @emeli-dral, Thank you for the update! I would be definitely happy to make a contribution. 

So now with the new approach are there any specific guidelines or steps I should follow to get started? 
If not, I can move on and update the TestValueRange and ColumnValueRangeMetric classes to accept Timestamp types.
Repository: evidentlyai/evidently
Issue: DataQualityPreset fails to recognize id column from column mapping
Root cause: Validation Gap

When running DataQualityReport I get an error "ValueError: column id not in num, cat or datetime features lists" in google Colab. 

Example:

column_mapping = ColumnMapping()
column_mapping.target = 'label_num'
column_mapping.id = 'id'
column_mapping.prediction = 'predict_proba'
column_mapping.target_names = ['ham', 'spam']
column_mapping.task = 'classification'
column_mapping.numerical_features = ['text_length', 'non_letter_share', 'oov_share', 'sender_employment_length'] 
column_mapping.categorical_features = ['listed_words_present', 'company_name']

data_quality_report = Report(metrics=[DataQualityPreset()])

data_quality_report.run(reference_data=reference, current_data=current, column_mapping=column_mapping)
data_quality_report
Repository: evidentlyai/evidently
Issue: report now showing when using a bit more data
Root cause: Resource Mishandling

Hey,

I wanted to run against a production dataset of small-mid size:

65 columns, 150K points in each dataframe. 

If I reduce the dataset to one feature, the report shows. If I use all features, the report goes from 16MB to 600 MB and is not displaying (saved or in jupyter).

Hey @rmminusrslash, 
Thanks for reporting! Unfortunately, this is the current limitation  of the tool. 

The report is large because the tool stores all the data necessary to generate interactive plots directly inside the HTML. We plan to fix it when we create a service version of the tool (where we decouple the data storage and the browser-based web service).

For now there are two workarounds:
1. Use some sampling strategy for your dataset, for instance random sapling. For Jupyter notebook, that can be done directly with pandas. For command line interface, we have a configuration - you can choose random sampling or pick the n-th rows. 
2. Use JSON profile. This way, Evidently calculates the metrics and statistical tests but they can be logged or displayed elsewhere. We have an example for MLflow https://docs.evidentlyai.com/step-by-step-guides/integrations/evidently-+-mlflow and i am working now on one for Grafana.

We understand this limits how you can use the tool now, and are working hard to get to the more feature-full version!

Hey @emeli-dral,

ah, I probably should have been more clear about what I was asking. I tried sampling when I figured out the root cause, up to 10K datapoints worked.

Would it make sense to 
- add sampling as the default if the dataset exceeds current limits (display a message that sampling happened)
- if you decide against it, at least raise an unsupported exception that mentions the sampling option and mention the limitation in the docs

The current behavior of failing silently might not be ideal until you release the full version (unless you expect people to try the tool mostly with toy data)

Hey @rmminusrslash ,
thanks for more details!

We thought about adding an error message based on data size. But the limit would depend on the user infrastructure especially if used locally, so it would be hard to set a universal threshold when sampling should be applied. And as a priority, we are also working right now to speed up the UI which should solve  part of cases when reports are too large to display. Hopefully, it will help a lot 🤞

We are thinking about adding a flag later that the user can set on their own ("large dataset") which would then generate a variation of report that is best suited for larger datasets. It will include not only sampling but a different aggregated views for some parts of the report.

Agree on your comment of making the limitation for large datasets and sampling option even more clear for Jupyter notebook: we already added this now to the Quick-start part of the docs.

Now reports by default do not use any raw data plots and this reduces reports size significantly
Repository: evidentlyai/evidently
Issue: unable to run DataDriftPreset for selected columns
Root cause: Validation Gap


evidently == 0.2.0

Hi all, when I try to run the data drift report with selected columns from the sample housing data, I encounter the following error: 

```
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-31-b410202d09b8> in <module>
      7     ]
      8 )
----> 9 report.run(reference_data=reference_data, current_data=current_data)

/opt/conda/lib/python3.7/site-packages/evidently/report/report.py in run(self, reference_data, current_data, column_mapping)
     90 
     91         self._inner_suite.verify()
---> 92         self._inner_suite.run_calculate(data)
     93 
     94     def as_dict(self) -> dict:

/opt/conda/lib/python3.7/site-packages/evidently/suite/base_suite.py in run_calculate(self, data)
    238 
    239             calculations = {}
--> 240             for metric, calculation in execution_graph.get_metric_execution_iterator():
    241                 if calculation not in calculations:
    242                     logging.debug(f"Executing {type(calculation)}...")

/opt/conda/lib/python3.7/site-packages/evidently/suite/execution_graph.py in get_metric_execution_iterator(self)
     37         metric_to_calculations = {}
     38         for metric_type, metrics in aggregated.items():
---> 39             metrics_by_parameters: Dict[tuple, List[Metric]] = functools.reduce(_aggregate_by_parameters, metrics, {})
     40 
     41             for metric in metrics:

/opt/conda/lib/python3.7/site-packages/evidently/suite/execution_graph.py in _aggregate_by_parameters(agg, metric)
     54 
     55 def _aggregate_by_parameters(agg: dict, metric: Metric) -> dict:
---> 56     agg[metric.get_parameters()] = agg.get(metric.get_parameters(), []) + [metric]
     57     return agg

TypeError: unhashable type: 'list'

```

The python code is following:
```
columns = ["AveBedrms","MedInc"]
report = Report(
    metrics=[
        DataDriftPreset(
            columns = columns,
        ),
    ]
)
report.run(reference_data=reference_data, current_data=current_data)
```

Hi @userkkw ,
Thank you for pointing this out!

fixed in #476 

You can use it now if you rebuild the package from the source, and fix will be included in the next release.

Fix is released in 0.2.1
Repository: evidentlyai/evidently
Issue: Some of the Test are not working
Root cause: Documentation Deficiency

<img width="689" alt="Screen Shot 2022-12-26 at 11 08 57" src="https://user-images.githubusercontent.com/47945745/209557543-77367867-e6b2-4071-86fb-90a6063749be.png">


Some of the Tests are not working. The names are not correct ? or they are not present in the module anymore ?
If the have been changed (renamed) can you provide a list of the tests with the old and new name ? Thanks.

Hi @samuelamico, could you check which Evidenly version you are using?

`TestNumberOfNulls` (and similar null-related tests) were renamed to `TestNumberOfMissingValues` in one of the earlier releases.

You can see the **log of changes** in the release notes, but it might be that something was missed: 
https://github.com/evidentlyai/evidently/releases

Sorry for the confusion, if any!

The current Evidently version (0.2.0 onwards) is considered stable; there should be no more changes to the names of existing tests. To **see the current names** of all tests, you can check the "All tests" example notebooks (Jupyter: https://github.com/evidentlyai/evidently/blob/main/examples/sample_notebooks/evidently_tests.ipynb), consult the API reference or "All tests" table: https://docs.evidentlyai.com/reference/all-tests 

Thanks for the update.
Repository: evidentlyai/evidently
Issue: [BUG] Broken links in the readme
GitHub labels: bug

The following links in the READme are currently broken:

- https://docs.evidentlyai.com/reports/data-quality
- https://docs.evidentlyai.com/reports/classification-performance
- https://github.com/evidentlyai/evidently/blob/main/sample_notebooks/getting_started_tutorial.ipynb
- https://github.com/evidentlyai/evidently/blob/main/sample_notebooks/evidently_metric_presets.ipynb
- https://github.com/evidentlyai/evidently/blob/main/sample_notebooks/evidently_metrics.ipynb
- https://github.com/evidentlyai/evidently/blob/main/sample_notebooks/evidently_test_presets.ipynb
- https://github.com/evidentlyai/evidently/blob/main/sample_notebooks/evidently_tests.ipynb

Also, some links to tutorials (e.g., https://github.com/evidentlyai/evidently/tree/main/examples/integrations/mlflow_logging) point to un-rendered python notebooks rather than an HTML page in the documentation. It'd be nice to be able to see the examples without having to run them locally.

Hi @OlivierBinette ,
thank you for letting us know! 

All links were fixed.

I agree with you that it would be much more convenient to have an HTML documentation page for each tutorial, but I'm afraid we do not have enough resources to keep a lot of doc pages up-to-date at the moment ☹️  This why we only have several tutorials in docs. 
But we regularly updated all jupyter examples, to make sure they are compatible with the package version and add new ones.  Unfortunately, most of our HTML reports cannot be rendered in python notebooks on github due to big size of the visual outputs.
Repository: evidentlyai/evidently
Issue: No module named 'evidently.dashboard'; 'evidently' is not a package
Root cause: Environment Incompatibility

Hi @elenasamuylova and @emeli-dral ,

Hope you guys are fine.

We were trying evidently for the first time in a Windows machine, and after installing it using command pip install evidently and trying to run it (on jupyter and also on VSCode), we got the following error:

'evidently.dashboard'; 'evidently' is not a package

See the print screen below:
![image](https://user-images.githubusercontent.com/22715100/131012715-a507a69c-48cc-498a-ba0a-920f0d606ee2.png)

We also found that the installed package, when I tried pip list was: evidently 0.1.22.dev0

Hope the above information can help you to help me.

All the best,
Edu



Hi @edumagol ,
Looks like it happened because you have file ‘evidently.py’ at the project directory. This is why when you are trying to import ```Dashboard``` from the ```evidently.dashboard```, it refers to the  ‘evidently.py’  and says that it is not a package. 
Thus, I believe renaming of the  ‘evidently.py’  or moving it elsewhere should help.

Hi @emeli-dral , sure... totally my mistake...what I shame of my self...lol...omg... how I commited such mistake... sorry for disturbing you guys with that! I should have checked that before... my apologies again. All the best girls!

@edumagol Awesome that it is solved and glad it works! 
Will be looking forward to your feedback on the tool :)
Repository: evidentlyai/evidently
Issue: curr_small_hist and ref_small_hist not available in data drift JOSN object in latest version
Root cause: Documentation Deficiency

Hi,
I think curr_small_hist and ref_small_hist is very good information if we use drift JSON to showcase distributions on an external visualization dashboard.
After this change  I'm not able to show distributions on the app
It will be helpful if you add setting or flag fit ref and current distributions?


Hi @sandeeps311, this is now available in the latest release https://github.com/evidentlyai/evidently/releases/tag/v0.2.8 as `small_distribution` in the JSON output.

Thank you so much @elenasamuylova @emeli-dral 
It will help a lot in my use case
Repository: evidentlyai/evidently
Issue: Proposed contribution for the outlier detection (evaluation metric type)
Root cause: Unimplemented Feature Gap

Hi team!

I wanted to create a post for starting the contribution of the outlier detection evaluation metric type. If I understood correctly from the community calls you are open to this.

I have a few simple suggestions to **start with**. However, It would definitely require your feedback.

**Suggested initial use cases:**

1) User has a set of features after gathering batch dataset and would like to check whether outliers exist.
2) User would like to check if there are outliers in a feature vs the target/prediction.

Here is a google collab with an example of the following techniques: 1) z-score 2) OneClassSVM
https://colab.research.google.com/drive/1aLg4BR79kJMyplixEUdv1SlSPy4DpLhi#scrollTo=UYIFlIdYHSBX

Assumptions: Batches of data are used
Requirements: tabular-numeric features

**Can you let me know about the following?**
 

1. Would you like to have that kind of contribution? if you have a different way/process in mind for such a contribution, I understand :D. 

2. Do you agree with those initial approaches (or have any feedback)? Any direction you have in mind?
Repository: evidentlyai/evidently
Issue: Databricks ReferenceError: requirejs is not defined
Root cause: Documentation Deficiency

Hi Evidently Team
I was trying to regenerate the iris dataset example on databricks:
<img width="731" alt="Screenshot 2023-01-10 at 12 33 01 PM" src="https://user-images.githubusercontent.com/104791738/211621926-afa2b05a-b35e-4293-a94d-307ef81f13c9.png">
 but I received the following error:
<img width="540" alt="Screenshot 2023-01-10 at 12 33 06 PM" src="https://user-images.githubusercontent.com/104791738/211621961-0383d77b-e0d8-4ca5-8aee-09bbf7374bb4.png">

The error is not informative and I am not able to debug it. Could you please advise me?
Thanks


Hi @kihaddadi , 

Could you try adding the argument to display the visual report inline?

instead of calling: 
`data_stability`

add:
`data_stability.show(mode='inline')`

The likely issue is described here: https://docs.evidentlyai.com/integrations/notebook-environments . 

Let us know if it solves the issue! 


Thanks the problem is solved.
Repository: evidentlyai/evidently
Issue: Correlation matrix: how_to_get_report_results_in_csv
Root cause: Documentation Deficiency

Thank you very much for writing up the .md on how to convert dicts to dfs. Today I tried to apply the same logic to the corr. matrix dict, but to no avail. 

#### Problem
I attempted to adapt the dict-to-csv code from 'how_to_get_report_results_in_csv' to extract the correlations metric data, but I wasn't successful. If you are able to find a quick solution, that would be great, but don't worry if not. I am providing a shortened copy of my dict with dummy values as the reprex. 

#### Reprex dict:
```
{'metrics': [{'metric': 'DatasetCorrelationsMetric',
   'result': {'current': {'stats': {'pearson': {'target_prediction_correlation': None,
       'abs_max_features_correlation': 0.1},
      'cramer_v': {'target_prediction_correlation': None,
       'abs_max_features_correlation': None}}},
    'reference': {'stats': {'pearson': {'target_prediction_correlation': None,
       'abs_max_features_correlation': 0.7},
      'cramer_v': {'target_prediction_correlation': None,
       'abs_max_features_correlation': None}}}}}]}
```

#### My attempted solution

    data = {}
    for stats in reprex_dict['metrics']:
        data[stats['stats']] = {
            **{f"ref_{key}": val for key, val in stats['result']['reference']['stats'].items()},
            **{f"cur_{key}": val for key, val in stats['result']['current']['stats'].items()}
        }

#### Expected dataframe format:

| cur_pearson_target_prediction_correlation | cur_pearson_abs_max_features_correlation | cur_cramer_v_target_prediction_correlation |
|:---- |------:| -----:|
| None | 0.1 | None | 



Solution (not my own):
```
def flattenDict(orig:dict, kList=[], kSep='_', rename={}):
    if not isinstance(orig, dict): return [(kList, orig)]
    tList, dCt = [], len([v for v in orig.values() if isinstance(v,dict)])
    for k, v in orig.items():
        kli = kList + ([] if isinstance(v,dict) and dCt==1 else [str(k)])
        tList += flattenDict(v, kli, None)
    if not isinstance(kSep, str): return tList
    return {kSep.join([rename.get(k,k) for k in kl]):v for kl,v in tList}

nrMap = {'current':'cur','reference':'ref'}
rows = [flattenDict(result['result'], rename=nrMap) for result in corr_matrix_dict['metrics']]
rowsDf = pd.DataFrame(rows)
rowsDf

# corr_matrix_frame = pd.DataFrame.from_dict(data, orient='index')
```