zhiminy/EvalEng huggingface dataset:
Repository: evidentlyai/evidently
Issue: Unable to generate Report for Multi-class Classification with single output class
Root cause: Validation Gap

I have simulated a data drift which results in the model predicting the same class over and over again, when I try to run the Report on the reference data and current data, here is the current code:
```python
from evidently.pipeline.column_mapping import ColumnMapping

from evidently.report import Report
from evidently.metrics import ClassificationQualityMetric

column_mapping = ColumnMapping()

column_mapping.target = 'label'
column_mapping.prediction = ['prob_NEGATIVE', 'prob_NEUTRAL', 'prob_POSITIVE']
column_mapping.text_features = ['text']
column_mapping.numerical_features = []
column_mapping.task = 'classification'
column_mapping.categorical_features = []

performance_report = Report(metrics=[
    ClassificationQualityMetric()
])

performance_report.run(reference_data=test_df, current_data=data_drift_df, column_mapping=column_mapping)
performance_report.show()
```
Here is the current/reference data example:
```bash
text | label | prob_NEGATIVE | prob_NEUTRAL | prob_POSITIVE | predicted_label | predicted_sentiment
« C’est de loin la méthode de contraception la... | 0 | 0.219654 | 0.071736 | 0.708610 | 2 | POSITIVE
« Je prends de la doxy depuis un certain temps... | 0 | 0.307037 | 0.108540 | 0.584423 | 2 | POSITIVE
« En 8 heures de prise d'un comprimé, j'ai eu ... | 0 | 0.159101 | 0.039321 | 0.801578 | 2 | POSITIVE
« Cela a changé ma vie. Je peux travailler eff... | 2 | 0.172600 | 0.040159 | 0.787241 | 2 | POSITIVE
« Cela a changé ma vie. L’anxiété a disparu, e... | 2 | 0.172715 | 0.037171 | 0.790113 | 2 | POSITIVE
```
I get the following error, which stems from scikit-learn:
```bash
---------------------------------------------------------------------------

ValueError                                Traceback (most recent call last)

[<ipython-input-57-7c6b02163273>](https://localhost:8080/#) in <cell line: 1>()
----> 1 performance_report.show()

13 frames

[/usr/local/lib/python3.10/dist-packages/sklearn/metrics/_classification.py](https://localhost:8080/#) in confusion_matrix(y_true, y_pred, labels, sample_weight, normalize)
    338             return np.zeros((n_labels, n_labels), dtype=int)
    339         elif len(np.intersect1d(y_true, labels)) == 0:
--> 340             raise ValueError("At least one label specified must be in y_true")
    341 
    342     if sample_weight is None:

ValueError: At least one label specified must be in y_true
```
It seems that the labels are not getting propogated down to calculate metrics that leverage probabilities such as `ROC_AUC` as per this [Stackoverflow thread](https://stackoverflow.com/questions/52250821/at-least-one-label-specified-must-be-in-y-true). Noticed a similar [issue](https://github.com/evidentlyai/evidently/issues/516) before that was fixed. 

I am currently using `evidently==0.4.19`.
Repository: evidentlyai/evidently
Issue: KeyError from Typing while importing TestSuite
Root cause: Environment Incompatibility

Hi, first time posting an issue so let me know if there's more info needed.

Evidently version: 0.4.36
Python version: 3.11.0

I'm trying to import TestSuite but keep running into the following issue: 
![image](https://github.com/user-attachments/assets/7d147f1f-1fa5-4893-8bc3-c57a9d0d94b7)

I have tried installing this fresh and I'm still running into the same issue.

Would anyone be able to help solve this?

Thanks

Hi @brandoneh, 

**1. Reproducible example**. Could you share the exact code you are running, is this:

```
pip install evidently
from evidently.test_suite import TestSuite
```
?

**2. Environment**. What is your Python environment - e.g. Jupyter notebook, Colab?

Hi, I'm running into this problem on Databricks, but am also seeing the issue using vscode, poetry and pyenv locally to replicate it. The commands I've run are as below:


```
poetry new evidently_test
cd evidently_test
pyenv local 3.11.0
poetry shell
poetry add evidently
python
>> from evidently.test_suite import TestSuite
``` 

Let me know if I need to provide or clarify anything else. Thanks


I had the same problem in Databricks, and I think the problem is that the latest LTS runtimes 15.4 uses a release candidate of python 3.11.0rc1. 

I used a 14.3 LTS runtime instead and it seems to work.

Any news on this issue?

The same issue with Runtime Databricks 15.4. I have two potential solutions:
1. Downgrade evidently==0.4.30
2. Switch runtime to 14.3 LTS

This looks like a bug in Python 3.11.0. Updating to 3.11.8 resolved the issue for me.

> This looks like a bug in Python 3.11.0. Updating to 3.11.8 resolved the issue for me.

Well spotted, that fixed it for me, thanks!
Repository: evidentlyai/evidently
Issue: Make `LLMEval` descriptors plottable from Tests 
Root cause: Unimplemented Feature Gap

**Issue description**
- The `TextLength()` descriptor plot on the Descriptors tab does not populate when the `TestShareOfOutRangeValues` Test is logged to a Project. (The plot is visible but empty).
- Custom `LLMEval()` descriptor plot does not appear on the Descriptors tab when the `TestCategoryCount` or `TestCategoryShare` Test is logged to a Project. (The plot is **not** visible on the Descriptors tab + not plottable manually since no Metric is logged).

**Steps to reproduce**
- Run this notebook https://github.com/evidentlyai/community-examples/blob/main/tutorials/Regression_testing_with_debugging.ipynb 
- In Evidently Cloud, add a new Tab - Descriptors

**Expected behavior**
- We can plot the distribution of the custom  `LLMEval()` descriptor when running the Tests on category count/share. For Evidently Cloud, it is included automatically on the Descriptors tab.
- When running the test on the share of out-of-range values, we can see `TextLength()` plot on the Descriptors tab.
Repository: evidentlyai/evidently
Issue: A friendly comment

Hi developers👋 , thank you so much for developing the amazing package :) It is easy to use and really useful!
I have an idea of making a small change. In real business, we usually use historical data as the "reference" and the current period data as "current". That is to say, we are likely to have a much larger amount of data for "reference" comparing to "current".
For the CatTargetDrift, I think it would be better if you could add histnorm='probability density' to the Target (Predict) By Feature plots.
Thank you for your time and consideration.

Hi! Thank you so much for your feedback, we appreciate it a lot☺️

This is very nice suggestion, to be honest we also considered ```percent``` and ```probability density``` options to have percentage or fraction of samples instead of the count of samples. 

Personally, I prefer to use the ```probability density``` normalisation for my projects. The reason to choose ```count``` was that it is easier to translate the count of samples into percent or fraction than vice versa🤷‍♀️  Do you think there are additional arguments in favor of  ```probability density``` I missed? We are considering changing it, but look to collect more feedback from the users on what is the most useful view. 

We also think that it will be good to have this as an option. We hope to implement the service version for our tool and add a lot of parameters to customise the reports.

Hi @emeli-dral, I tested the package on one of the datasets that I am using for a tutorial that I am preparing for my blog. the training set has a shape of (243728, 20) and the validation set has (91493, 20). I found that the generated HTML file could take up to 4 min to open in the browser when the report contains DriftTab only and it crashed when I added another tab on the report. I am using a MacBook Pro with 16GB of RAM. the browser is chrome. Overall I think the package will be very useful once optimised. this is something I have been looking for.

Hi @gakuba 
this is a big problem indeed, I created the separate issue here: https://github.com/evidentlyai/evidently/issues/15#issue-851155938
Repository: evidentlyai/evidently
Issue: Type Error in Dashboard
Root cause: Documentation Deficiency

When attempting to follow along the tutorials, I received a Type Error stating that analyzers() missing 1 required positional argument: 'self'. This bug was confirmed with both versions 0.1.42.dev0 and 0.1.43.dev0. Python version is 3.7.11, and the code was run in jupyterlab version 3.2.1.

```
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
/tmp/ipykernel_5750/654272261.py in <module>
----> 1 drpt = Dashboard(tabs=[DataDriftTab])
      2 drpt.calculate(df_ref,df_test,column_mapping=None)
      3 drpt.save('./drift_report.html')

/usr/local/anaconda/envs/.../lib/python3.7/site-packages/evidently/dashboard/dashboard.py in __init__(self, tabs, options)
    138 
    139     def __init__(self, tabs: Sequence[Tab], options: Optional[List[object]] = None):
--> 140         super().__init__(tabs, options if options is not None else [])
    141 
    142     def calculate(self,

/usr/local/anaconda/envs/.../lib/python3.7/site-packages/evidently/pipeline/pipeline.py in __init__(self, stages, options)
     20         self.analyzers_results = {}
     21         self.options_provider = OptionsProvider()
---> 22         self._analyzers = list(itertools.chain.from_iterable([stage.analyzers() for stage in stages]))
     23         for option in options:
     24             self.options_provider.add(option)

/usr/local/anaconda/envs/.../lib/python3.7/site-packages/evidently/pipeline/pipeline.py in <listcomp>(.0)
     20         self.analyzers_results = {}
     21         self.options_provider = OptionsProvider()
---> 22         self._analyzers = list(itertools.chain.from_iterable([stage.analyzers() for stage in stages]))
     23         for option in options:
     24             self.options_provider.add(option)

TypeError: analyzers() missing 1 required positional argument: 'self'
```

Hi @wvz754 ,
thank you for reporting this! 

Recently we had an API update, so could you please check, if you imported DataDriftTab like this:
```from evidently.dashboard.tabs import DataDriftTab```

and if import is correct, adding brackets to the DataDriftTab() should help:
```drpt = Dashboard(tabs=[DataDriftTab()])```

> Hi @wvz754 , thank you for reporting this!
> 
> Recently we had an API update, so could you please check, if you imported DataDriftTab like this: `from evidently.dashboard.tabs import DataDriftTab`
> 
> and if import is correct, adding brackets to the DataDriftTab() should help: `drpt = Dashboard(tabs=[DataDriftTab()])`

I have the same error and this solution works for me. Thanks!

Hi @jenoOvchi ,
 
Glad it worked!
Repository: evidentlyai/evidently
Issue: Evidently Dashboard on Notebook.
Root cause: Documentation Deficiency

Why Can't I use evidently.dashboards in my code to create a dashboard on my notebook?

<code>from evidently.dashboard import Dashboard
from evidently.dashboard.tabs import 
(
    DataDriftTab,
    CatTargetDriftTab,
    RegressionPerformanceTab,
    ClassificationPerformanceTab,
    ProbClassificationPerformanceTab,
)</code>

Hi @riturajraman-evs, you are using the old API that was changed 2 years ago.

Check this page on how to proceed: https://docs.evidentlyai.com/support/migration

You can either use the older version of Evidently, or use the `Report` object (recommended).
Repository: evidentlyai/evidently
Issue: Responses containing latex symbols resulted in Python unable to parse them to json during correction evaluation
Root cause: Validation Gap

While performing correctness evaluation on the llm model's performance with GSM8K dataset, the response that contains latex symbols resulted in Python's failure to parse it to json. 

The code for correction evaluation : 

```
# Correctness Evaluator: Run Evaluation

correctness_eval= LLMEval(
    subcolumn="category",
    additional_columns={"target_response": "target_response"},
    template = BinaryClassificationPromptTemplate(
        criteria = """
An ANSWER is correct when it is the same as the REFERENCE in all facts and details, even if worded differently.
The ANSWER is incorrect if it contradicts the REFERENCE, adds additional claims, omits or changes details.

REFERENCE:

=====
{target_response}
=====
        """,
        target_category="incorrect",
        non_target_category="correct",
        uncertainty="unknown",
        include_reasoning=True,
        pre_messages=[("system", "You are an expert evaluator. will be given an ANSWER and REFERENCE.")],
        ),
    provider = "openai",
    model = "gpt-4o-mini",
    display_name = "Correctness",
)

correctness_report = Report(metrics=[
    TextEvals(column_name="new_response", descriptors=[
        correctness_eval
    ])
])

correctness_report.run(reference_data=None,
           current_data=golden_dataset)
correctness_report
```

Error message produced: 

```
LLMResponseParseError: Failed to parse response '{
  "category": "correct",
  "reasoning": "The answer provided correctly follows the reasoning in the reference. It defines the number of people on the first ship as \( x \), and then accurately establishes the number of people on the subsequent ships as \( 2x \) and \( 4x \). The total is calculated correctly as \( x + 2x + 4x = 847 \), leading to the correct result of \( x = 121 \). Therefore, the final conclusion about the number of people on the first ship is correct."
}' as json
```
Repository: evidentlyai/evidently
Issue: Target drift report with real label only
Root cause: Documentation Deficiency

how can i use the target drift detect  for feature, i don't find in the doc.

Hi @xuyunlong123, could you clarify what you try to achieve?

If you only have a single column (real label) and want to see the shift in its distribution, you can simply create a `Report` with a single `ColumnDriftMetric` for this column.

If you want to detect drift for a set of features, you use `DataDriftPreset`.

The `TargetDriftPreset` is useful when you have both prediction/label column and feature columns and you want to look at data drift as well as on correlation changes / visually explore the dependency between target and features.





> Hi @xuyunlong123, could you clarify what you try to achieve?
> 
> If you only have a single column (real label) and want to see the shift in its distribution, you can simply create a `Report` with a single `ColumnDriftMetric` for this column.
> 
> If you want to detect drift for a set of features, you use `DataDriftPreset`.
> 
> The `TargetDriftPreset` is useful when you have both prediction/label column and feature columns and you want to look at data drift as well as on correlation changes / visually explore the dependency between target and features.

emm，i see. the situation is that i creat many creaditscore models for loan application,but they have the same target label and different features, so i want to use one detector which can test the drift between features and target.That confuses me.Thanks for your reply.

> Hi @xuyunlong123, could you clarify what you try to achieve?
> 
> If you only have a single column (real label) and want to see the shift in its distribution, you can simply create a `Report` with a single `ColumnDriftMetric` for this column.
> 
> If you want to detect drift for a set of features, you use `DataDriftPreset`.
> 
> The `TargetDriftPreset` is useful when you have both prediction/label column and feature columns and you want to look at data drift as well as on correlation changes / visually explore the dependency between target and features.

I have another question if i only have label and  feature columns, the `TargetDriftPreset` could work?

Hi @xuyunlong123, yes - you just need to label your `label` column as target in column mapping https://docs.evidentlyai.com/user-guide/input-data/column-mapping#target-and-prediction
Repository: evidentlyai/evidently
Issue: Add a new `WordMatch` descriptor to Evidently
Root cause: Unimplemented Feature Gap

# Add a New `WordMatch` Descriptor to Evidently

About Hacktoberfest contributions: https://github.com/evidentlyai/evidently/wiki/Hacktoberfest-2024

##  Description:

Evidently already has an `IncludesWords()` descriptor that checks if the text contains `any` (by default) or `all` specified words, returning a True/False result for each row. However, this descriptor uses a single shared list of words for all rows.

In some cases, such as when evaluating responses against specific ground truth answers, you may need a different list of words for each row. For example, you might want to check if generated responses contain the expected keywords for each row:

**Example:**

| **Question**                     | **Generated Response**                  | **Expected Words**               |
|-----------------------------------|-----------------------------------------|----------------------------------|
| "Name a primary color."           | "Red is a primary color."               | ["blue", "red", "yellow"]        |

## What to Implement:

The new **WordMatch()** descriptor should:

1. **Accept a `with_column` parameter**: This column contains a list of words specific to each row.
2. **Accept a `lemmatize` parameter**. Default `True`, to consider inflected and variant words. (Same as `IncludesWords()` descriptor).
3. **Allow configuration for `any` or `all`** words present. (Same as `IncludesWords()` descriptor):
4. **Return True/False for each row**` if the specified condition (any or all) is met or not.

**References:**
- Check the `IncludesWords` descriptor for vocabulary word check implementation.
- For a two-column descriptor implementation, check the `SemanticSimilarity` descriptor and the `CustomPairColumnEval` template.

Along with the `with_column` parameter, would the column we are getting the generated response from always be named the same? Or would we need it to resemble `SemanticSimilarity` implementation with two arbitrary columns being compared  with `WordMatch`?
Repository: evidentlyai/evidently
Issue: Add a New `WordNoMatch` Descriptor to Evidently
GitHub labels: enhancement

hacktoberfest

# Add a New `WordNoMatch` Descriptor to Evidently

About Hacktoberfest contributions: https://github.com/evidentlyai/evidently/wiki/Hacktoberfest-2024

## Description:

Evidently already has an `ExcludesWords()` descriptor that checks if the text does **not** contain any specified words from a shared list. 

However, sometimes you might need to check that the text does not contain words specific to each row instead of a shared list.

### Example:

| **Question**                                 | **Response**                                                                 | **Forbidden Words**                      |
|----------------------------------------------|-------------------------------------------------------------------------------|------------------------------------------|
| "Can I cancel my subscription at any time?"  | "You are allowed to cancel at any time, and we guarantee that you will receive a refund." | ["guarantee", "allowed", "refund"]       |

## What to Implement:

The new **WordNoMatch()** descriptor should:

1. **Accept a `with_column` parameter**: This column contains a list of forbidden words specific to each row.
2. **Accept a `lemmatize` parameter** (default `True`): When `True`, this will consider inflected or variant forms of words. Works the same as in the `ExcludesWords()` descriptor.
3. **Return True/False**:
   - Return `True` if the words from the list are **not** present in the row's text.
   - Return `False` if forbidden words are present.

## References:

- Check the `ExcludesWords()` descriptor as reference.
- For a two-column descriptor implementation, check the `SemanticSimilarity` descriptor and the `CustomPairColumnEval` template.


Opened a joint PR for this issue and #1308
Repository: evidentlyai/evidently
Issue: Add a New `ItemNoMatch` Descriptor to Evidently
Root cause: Unimplemented Feature Gap

# Add a New `ItemNoMatch` Descriptor to Evidently

About Hacktoberfest contributions: https://github.com/evidentlyai/evidently/wiki/Hacktoberfest-2024

## Description:

Evidently already has a `DoesNotContain()` descriptor that checks if the text does **not** contain specified strings from a shared list, like competitor names or certain trigger words. However, sometimes you might need to check that each text does not contain certain items that are specific to each row, rather than using a shared list.

This descriptor will allow you to exclude specific items (strings or phrases) for each row based on its own list.

### Example:

| **Question**                             | **Response**                                                                 | **Forbidden Items**                      |
|------------------------------------------|-------------------------------------------------------------------------------|------------------------------------------|
| "Which shoes should I buy?"              | "You should consider purchasing Acme shoes."                        | ["Nike", "Adidas", "Puma"]               |

## What to Implement:

The new **ItemNoMatch()** descriptor should:

1. **Accept a `with_column` parameter**: This column contains a list of forbidden items specific to each row.
2. **Accept a `case_sensitive` parameter**:  Only return `True` if the case matches exactly / ignore case. Same as  `DoesNotContain()` descriptor.
3. **Return True/False** for absent items. Same as  `DoesNotContain()` descriptor.

## References:

- Check the `DoesNotContain()` descriptor for handling string exclusion cases.
- For a two-column descriptor implementation, refer to the `SemanticSimilarity` descriptor and the `CustomPairColumnEval` template.

I will start working on these since it seems like it will closely resemble the `WordMatch` and `WordNoMatch` descriptors
Repository: evidentlyai/evidently
Issue: Add a New `ItemMatch` Descriptor to Evidently
GitHub labels: enhancement

hacktoberfest

# Add a New `ItemMatch` Descriptor to Evidently

About Hacktoberfest contributions: https://github.com/evidentlyai/evidently/wiki/Hacktoberfest-2024

## Description:

Evidently already has a `Contains()` descriptor that checks if the text contains specific strings or phrases from a shared list. However, sometimes you might need to check whether the text contains certain items that are specific to each row, rather than using a shared list.

This descriptor will allow you to check for specific items for each row, based on its own list, with an option to matching `any` or `all`.

### Example:

| **Question**                             | **Response**                                                                   | **Expected Items**                      |
|------------------------------------------|--------------------------------------------------------------------------------|-----------------------------------------|
| "Which shoes should I buy?"              | "You should consider purchasing Nike or Adidas shoes."                         | ["nike", "adidas", "puma"]              |

## What to Implement:

The new **ItemMatch()** descriptor should:

1. **Accept a `with_column` parameter**: This column contains a list of expected items specific to each row.
2. **Accept a `mode` parameter**:  return true if **`any`** or  **`all`** items are found. Same as  `Contains()` descriptor.
3. **Accept a `case_sensitive` parameter**. Same as  `Contains()` descriptor.
4. **Return True/False** for pattern match.

## References:

- Check the `Contains()` descriptor for handling string inclusion cases.
- For a two-column descriptor implementation, refer to the `SemanticSimilarity` descriptor and the `CustomPairColumnEval` template.

I will start working on these since it seems like it will closely resemble the WordMatch and WordNoMatch descriptors