zhiminy/EvalEng huggingface dataset:
Repository: huggingface/evaluate
Issue: Support for disaggregated evaluations in Evaluators
Root cause: Unimplemented Feature Gap

For cases where we want to compute metrics across several groups using the Evaluators, the option is to call `.compute` with different splits of data. This is okay if the groups are disjoint, but for groups that have any overlap that means that we have to recompute inferences (which can be really costly) since there is no caching mechanism.

Maybe a working solution could be to add something like a `disaggregate_by` flag to the `.compute` for the Evaluators (or maybe even to the base evaluator) which could be used to calculate the metric on the specified folds of data _after_ the inferences have been computed.

I made a custom evaluator for something like this while working on [Disaggregators](https://github.com/huggingface/disaggregators) here: [Google Colab](https://colab.research.google.com/drive/1R92qDm0V0nlJJHreUjXbdMhj49DWOrBi#scrollTo=3fsKJeVUa6A0&line=108&uniqifier=1)

With the implementation above, the dataset is expected to have boolean columns like `pronouns.he_him `, `pronouns.they_them`, `pronouns.she_her`. Then, that list of column names is passed to `disaggregate_by` to compute disaggregated evaluations.

Additionally, it's often valuable to combine disaggregations and run metrics on intersectional groups, and I've also got that implemented in that custom evaluator in the notebook above.

Would there be any interest in something like this?
Repository: huggingface/evaluate
Issue: feature request: Dice metric for segmentation models
Root cause: Unimplemented Feature Gap

The library already has [mean_iou (mean Jaccard index)](https://en.wikipedia.org/wiki/Jaccard_index) which I have used for my image segmentation models and it works well.

If you look at recent research papers on the topic of image segmentation for medical imaging, you will notice that [the Dice metric (Sorensen-Dice coefficient)](https://en.wikipedia.org/wiki/S%C3%B8rensen%E2%80%93Dice_coefficient) is used a lot in research.

It would be very useful to have the Dice metric integrated into the main library - that would simplify model comparison when reviewing the literature. Thanks!

There is a chance I may try to clone the IoU code and modify it to return Dice instead. The math is pretty similar. I do not know yet if I have the time to do it. I will know more in the next week or so. I'll post an update here.

Greetings,
I have been monitoring this issue for a while and it seems that there is still no one assigned to it or working on it. As someone who has previously worked with Segmentation I can understand how important Dice metric is in general and would like to know if someone's working on this yet. If not, is it fine if this gets assigned to me? I reviewed the code for mean_iou which was contributed by MMLLabs and feel that the same boilerplate can be used along with some changes to turn it into Dice Metric and shouldn't take long.
Do let me know if someone's already working on it and if now I'm more than willing to take this up and start working on this ASAP <3

@sarthak247 that sounds great! But I'm not a member of the Hugging Face team so I do not know if they are working on it or not.

Observation: it's important that the implementation is actually compliant with the definition of the Sorensen-Dice coefficient when tested on simple examples. I believe the current Hugging Face implementation of IoU (the Jaccard index), copied from OpenMMLab, does not pass simple tests, as shown here:

https://github.com/huggingface/evaluate/issues/421

https://github.com/open-mmlab/mmsegmentation/issues/2655
Repository: huggingface/evaluate
Issue: redundant code lines in metric
Root cause: Algorithmic Error

Looks like this line and the one before it do the same thing:

https://github.com/huggingface/evaluate/blob/7fa754cc38c523b69054c67b8c3b8e3f4962fbf6/metrics/mean_iou/mean_iou.py#L145

Hey,
Can I solve this issue?
Repository: huggingface/evaluate
Issue: Update Gradio pinned versions in metrics for dark theme fix
Root cause: Configuration Error

Dark themes on Gradio used to have a problem because special text (like code tags) were not legible (example below).

![image](https://user-images.githubusercontent.com/2779410/218998794-7e69318a-36be-439f-82c4-3c67e56f9bff.png)

This was [fixed](https://github.com/gradio-app/gradio/issues/1746#issuecomment-1232304229) in 3.1.7. Unfortunately many (all?) of the metrics spaces are still on older Gradio versions (like [this one](https://huggingface.co/spaces/evaluate-metric/chrf); 3.0.2) where the dark theme leads to a bad experience. 

This should be easily fixable with a version bump in the README's of all metrics. I can add a PR for that but I am posting this issue first in case there might be incompatibilities with more recent Gradio versions.
Repository: huggingface/evaluate
Issue: Evalaute with safetensors
Root cause: Validation Gap

Evaluate method for token-classification is not working with safetensors models.
The error message:
NameError: name 'safe_open' is not defined

Could you post the code snippet that leads to this error?

The code is as follows and is working today:
![image](https://user-images.githubusercontent.com/37244134/226311228-7e30f90c-7452-4e59-80d4-166edb6557e7.png)

I didn't make any change in the code. The error for the last part was as follows yesterday:
![image](https://user-images.githubusercontent.com/37244134/226311645-1877b39c-609a-4911-af58-0cbb780862b6.png)



The first and third models above have safetensors versions.
Repository: huggingface/evaluate
Issue: Evaluate LLM models like llama/alpaca using evaluate library? 
Root cause: Documentation Deficiency

Hi team, thanks for open source this awesome tool. I am new to the tool and try to ask some questions on LLM evaluation

1. Seems `evaluate` already create some evaluators (Some libs call it tasks I think). Can we use these evaluator for LLM evaluation? 
2. I feel different tasks required different datasets. for LLM evaluation, there're popular datasets like MMLU. I am trying to ask Is there tested paring?  for example, for QA, I can use dataset1, dataset2 for metric1, metric2 evaluation etc
3. What's the difference between huggingface/evaluate and https://github.com/EleutherAI/lm-evaluation-harness? 

m

same, will it get supported? 

currently getting this error:
    raise ValueError(
ValueError: Tokenizer class LlamaTokenizer does not exist or is not currently imported.
Repository: huggingface/evaluate
Issue: ImportError: To be able to use evaluate-metric/rouge, you need to install the following dependencies['nltk'] using 'pip install # Here to have a nice missing dependency error message early on' for instance'
Root cause: Environment Incompatibility

Hi everyone, I encount this error when the code init the 
```rouge = evaluate.load('rouge')```
![image](https://user-images.githubusercontent.com/11539889/227168693-2bb016ce-c3f4-46ad-9a43-6874692aa045.png)
But the pip have install the nltk
![image](https://user-images.githubusercontent.com/11539889/227169215-b6c2664f-8da8-4a77-a893-b7cfa978412c.png)

Is it a bug or ??

Did you try `pip install rouge-score` ?
See [this issue](https://github.com/huggingface/evaluate/issues/329#issuecomment-1290662904) 

did you solve the problem?

facing issue in importing rouge metric.
evaluate.load("rouge")
Error calculating ROUGE scores: To be able to use evaluate-metric/rouge, you need to install the following dependencies['rouge_score', 'absl'] using 'pip install rouge_score # Here to have a nice missing dependency error message early on' for instance'.
can anyone help me out?

I also encountered this error, but installing `rouge-score` into my environment was all I needed to do (other than installing `evaluate`, or course). I tried installing both with `conda` and with `pip`, and it worked both ways.

```shell
❯ conda list 'evaluate|rouge'
# packages in environment at <redacted>/env:
#
# Name                    Version                   Build  Channel
evaluate                  0.4.1              pyhd8ed1ab_0    conda-forge
rouge-score               0.1.2              pyhd8ed1ab_0    conda-forge
```
Repository: huggingface/evaluate
Issue: Unexpected Keyword when running evaluator on whisper-tiny
Root cause: Interface Contract Mismatch

Hello Huggingface-People! Thanks a lot for your awesome work. 

On running evaluator in a `nvidia/cuda:11.4.3-cudnn8-devel-ubuntu20.04` enroot container (so my OS is ubuntu20.04 but it's in enroot) with

- torchaudio-2.0.1
- torch-2.0.0
- no tensorflow
- most recent transformers, evaluate[evaluator] and datasets

the following code fails:

```
data = load_dataset("mozilla-foundation/common_voice_11_0", "de", split="test")
task_evaluator = evaluator("automatic-speech-recognition")

eval_results = task_evaluator.compute(
    model_or_pipeline="openai/whisper-tiny",
    data=data,
    metric="wer",
    input_column="path",
    label_column="sentence",
)
```

with


```
Found cached dataset common_voice_11_0 (/data/.cache/mozilla-foundation___common_voice_11_0/de/11.0.0/2c65b95d99ca879b1b1074ea197b65e0497848fd697fdb0582e0f6b75b6f4da0)
Traceback (most recent call last):
  File "code/src/eval.py", line 15, in <module>
    eval_results = task_evaluator.compute(
  File "/usr/local/lib/python3.8/dist-packages/evaluate/evaluator/automatic_speech_recognition.py", line 92, in compute
    result = super().compute(
  File "/usr/local/lib/python3.8/dist-packages/evaluate/evaluator/base.py", line 255, in compute
    predictions, perf_results = self.call_pipeline(pipe, pipe_inputs)
  File "/usr/local/lib/python3.8/dist-packages/evaluate/evaluator/base.py", line 453, in call_pipeline
    pipe_output = pipe(*args, **kwargs, **self.PIPELINE_KWARGS)
  File "/usr/local/lib/python3.8/dist-packages/transformers/pipelines/automatic_speech_recognition.py", line 272, in __call__
    return super().__call__(inputs, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/transformers/pipelines/base.py", line 1061, in __call__
    preprocess_params, forward_params, postprocess_params = self._sanitize_parameters(**kwargs)
TypeError: _sanitize_parameters() got an unexpected keyword argument 'truncation'

```

Sadly it is not reproducable in 30 sec (except if you already have common voice de. I do not have the resources available to check a different dataset as it would take me days until slurm schedules me again.

I believe it is a bug caused by evaluator / pipelines.

You can fix it temporarily by doing this:

```python
task_evaluator = evaluator("automatic-speech-recognition")
# temp fix
task_evaluator.PIPELINE_KWARGS.pop('truncation', None)
assert 'truncation' not in task_evaluator.PIPELINE_KWARGS
eval_results = task_evaluator.compute(...)
```

I assume `truncation` is needed for other models and pipeline should check specifically if model type is "whisper" to do the above (at least in `_sanitize_parameters()`)...

Thanks a lot @drahnreb it worked like a charm!

the workaround works very well

but the issue still not resolved 😕
Repository: huggingface/evaluate
Issue: AttributeError: 'DownloadConfig' object has no attribute 'storage_options'
Root cause: Interface Contract Mismatch

From https://github.com/obss/jury/issues/125

`evaluate` passes its own DownloadConfig (evaluate.utils.file_utils.DownloadConfig) to a `datasets` `DownloadManager` in `EvaluationModule.download_and_prepare`.

However it should actually pass a `datasets` DownloadConfig (datasets.download.DownloadConfig) instead to not miss any new attribute (it happened in the latest `datasets` 2.11)
Repository: huggingface/evaluate
Issue: Problem about code_eval
Root cause: Validation Gap

Hi, I' m using hf 's code_eval to evaluate a model, however I got a lot of following error on my screen while doing evaluate.
Is there anyone meet this problem too?
File "<string>", line 33 in closest_integer
File "<string>", line 33 in closest_integer
File "<string>", line 33 in closest_integer
File "<string>", line 33 in closest_integer
File "<string>", line 33 in closest_integer
...
Repository: huggingface/evaluate
Issue: Progress bar when using evaluator
Root cause: Unimplemented Feature Gap

Currently, when using evaluator.compute, there is no indication of the process and no good way to wrap tqdm around it.

I suggest give an option in compute so that the progress bar can be shown, i.e.

task_evaluator = evaluator("image-classification")

eval_results = task_evaluator.compute(
    model_or_pipeline=pipe,
    data=dataset,
    metric=metric,
    label_mapping= lm,
    device=0,
    **progress_bar = True**
)

Please let me know if there are any potential concerns related!

After doing some digging, I believe that this is basically dependent on the pipeline component of the transformer library. It's easy to forward a `progress_bar: bool = False` param into the pipeline's `__call__` kwargs ([here ](https://github.com/huggingface/evaluate/blob/main/src/evaluate/evaluator/base.py#L236) and [here](https://github.com/huggingface/evaluate/blob/main/src/evaluate/evaluator/base.py#L513)).

But from that point on, it's a matter of what you're trying to do and if the dataset+pipeline can support progress bars. For example, on the [__call__](https://github.com/huggingface/transformers/blob/main/src/transformers/pipelines/base.py#L1046) pipeline function, we can see that the actual pipeline could be many things, including but not limited to a `GeneratorType` (which does not advertise a `__len__`, a `Dataset` or a `list` (which typically have `__len__`), so the worse-case progress bar you can get would be a tqdm "X iterations / s" dialogue.

I believe it's totally possible to do that, so I'm going to open an issue referencing this in the transformers repo.
Repository: huggingface/evaluate
Issue: combined metrics in evaluators/Suite subtasks
Root cause: Unimplemented Feature Gap

I see that custom evaluators are already discussed in #367, however I don't see a proper way to have multiple metrics for a SubTask and it's evaluator. Apart from writing a custom metric that includes both metrics. Adding the Subtask twice with distinct metrics would take twice as long as all predictions have to be done again. 

Since CombinedEvaluations is not an EvaluationModule you can't pass it to an Evaluator. So a lazy solution I came up with is to always do combine() since it should work with a single metric too.

Is there any smarter ways to do this or should I try myself with a PR?
Repository: huggingface/evaluate
Issue: Understanding metric compute in DDP setup
Root cause: Documentation Deficiency

Hello,

I am having a bit of trouble handling metrics in DDP setup.
I red the [doc](https://huggingface.co/docs/evaluate/a_quick_tour), especially the part about DDP but couldn't find much details of how to handle this. I find it a bit confusing, what does the "computing the final metric on the first node" mean ?

With two processes, when computing two metrics here is what `.compute` returns (first nb is process id):

```
1 None
1 None
0 {'accuracy': 0.009848218392359514}
0 {'f1': 0.49228395061728397}
```

So what's happening ? If I understand well, all **processes** (on each node? for all nodes combined? doc says "first node", its process right?) others than the first write predictions and reference in cache. The first process is waiting for the predictions and references from all other nodes, and when it gathered them all computes then returns the metric results. I am correct?

If so:
1. how do we make sure that other processes provide preds and refs ? Is `.compute` doing it? `.add`?
2. When using it with `Trainer`, is it ok to return 0s for processes (and nodes?) other than the first?

Here is how my compute_metrics function looks like, is it ok?

```Python
def compute_metrics_pt(eval_pred, metrics_: Dict[str, Metric]):
    (predictions_mlm, predictions_nsp), (labels_mlm, labels_nsp) = eval_pred

    # preprocess tensors
    pad_mask = labels_mlm != -100
    labels_mlm, predictions = labels_mlm[pad_mask], predictions_mlm[pad_mask]

    # compute metrics
    acc = metrics_["accuracy"].compute(predictions=predictions, references=labels_mlm)
    f1 = metrics_["f1"].compute(predictions=predictions_nsp.flatten(), references=labels_nsp.flatten(), average="micro")

    # first process
    if metrics_["accuracy"].process_id == 0:  # SHOULD CHECK THAT IT'S FIRST NODE TOO?
        metric_res = {"accuracy_mlm": acc["accuracy"], "f1_nsp": f1["f1"]}

    # other processes
    else:
        metric_res = {"accuracy_mlm": 0, "f1_nsp": 0}
    return metric_res
```


Found the answers in the [old documentation](https://huggingface.co/docs/datasets/v1.11.0/using_metrics.html).
Repository: huggingface/evaluate
Issue: Metrics that use DownloadConfig not working any more
Root cause: External Dependency Breakage

It seems that some metrics stopped working in some configurations with datasets. I am on datasets==2.11.0 and evaluate==0.4.0.

```python
import evaluate

metric = evaluate.load("bleurt", "BLEURT-20")
```

But this leads to an error

```
model_path = dl_manager.download_and_extract(CHECKPOINT_URLS[checkpoint_name])
File ".venv\lib\site-packages\datasets\download\download_manager.py", line 564, in download_and_extract
return self.extract(self.download(url_or_urls))
File ".venv\lib\site-packages\datasets\download\download_manager.py", line 427, in download
downloaded_path_or_paths = map_nested(
    File ".venv\lib\site-packages\datasets\utils\py_utils.py", line 435, in map_nested
return function(data_struct)
File ".venv\lib\site-packages\datasets\download\download_manager.py", line 453, in _download
return cached_path(url_or_filename, download_config=download_config)
File ".venv\lib\site-packages\datasets\utils\file_utils.py", line 195, in cached_path
storage_options=download_config.storage_options,
AttributeError: 'DownloadConfig' object has no attribute 'storage_options'
```

Related: https://github.com/huggingface/evaluate/issues/440

This is "solved" by downgrading to datasets 2.10.0. 

It should be fixed with https://github.com/huggingface/evaluate/pull/447, maybe we can do a release soon

@lhoestq Any chance of doing a new release?

cc @lvwerra who must have the pypi rights

I encountered a similar problem when loading the codeBleu metric:
```
evaluate.load("dvitel/codebleu")
```

The error message is:
```
python3.10/site-packages/datasets/utils/file_utils.py", line 192, in cached_path                              
    token=download_config.token,                                                                                                                                   
AttributeError: 'DownloadConfig' object has no attribute 'token'
```

Downgrading to datasets 2.10.0 also solves the problem.

We just released a new version of `evaluate`, you can now update evaluate it will work with recent datasets versions

```
pip install -U evaluate
```
Repository: huggingface/evaluate
Issue: Adding new metrics locally without using the evaluate-cli command
Root cause: Documentation Deficiency

Is there any documentation to follow for adding new metrics without the `evaluate-cli` command and without pushing to HF spaces, but to work locally. For example, I develop a custom metric `X`, and create the `metrics/X` folder with the `X.py` and `app.py`. But I can not load it with the `evaluate.load("X")`. I think I am missing one step such that `evaluate` repository may not be able to find the correct module script for the metric. I installed the `evaluate` package in the editable mode using `pip install -e .[dev]`

Hi @sameeravithana, did you find an answer to this?
Repository: huggingface/evaluate
Issue: SQuAD v2 is missing the NoAns metrics
Root cause: Unimplemented Feature Gap

The section "Output values" in the [documentation](https://github.com/huggingface/evaluate/blob/main/metrics/squad_v2/README.md) for the SQuAD v2 metric shows that there should be 13 different measures returned by `metric.compute( )`, yet, as is shown in the examples in the section "Examples", all the measures about "NoAns" are missing in the output.

This is not just a documentation error; running the examples in `evaluate` 0.3.0 gives the same result, without NoAns.

On the topic of documentation (although this is a separate issue): `exact`, `HasAns_exact` and `NoAns_exact` are described with the same sentence, but they clearly should have different meanings. Furthermore, I don't see what the difference is between `NoAns_exact` and `NoAns_f1`, because if there is no answer, then the reference span is empty and hence F1 is measured over a reference of 0 words.
Repository: huggingface/evaluate
Issue: Toxicity Measurement Should Allow GPU Usage
Root cause: Unimplemented Feature Gap

The toxicity measurement is based on BERT models in HuggingFace, and thus, should be able to utilize GPUs at inference time if a user wants. Suggesting to add the following line to `_download_and_prepare`:

`device = torch.device("cuda:0") if torch.cuda.is_available() else torch.device("cpu")`

And then add `device=device` param to `pipeline`.
Repository: huggingface/evaluate
Issue: Quick tour typos

https://huggingface.co/docs/evaluate/a_quick_tour

![image](https://user-images.githubusercontent.com/13343117/236750405-30f2e20a-843a-47d3-92b0-0d4f0e91e29c.png)


Seems like it is correct in main branch docs. Maybe there are the bugs for docs publishing?
Repository: huggingface/evaluate
Issue: Evaluate SyntaxError: invalid syntax 
Root cause: Environment Incompatibility

When I try to run the following script;

import evaluate
eval_metric = evaluate.load("squad")
results = eval_metric.compute(references=new_examples, predictions=predictions)

I get this error;

 C:\Users\{username}\.cache\huggingface\modules\evaluate_modules\metrics\evaluate-metric--squad\325 
 1bbcfa55c37b65f3a26f2bdedcdb2754db6bc08eee6af36c6ac871783b8d2\squad.py:1                         
 <!--# Id: internetaccess.html 285144 2021-06-16 05:02:06Z szhang -->                             
│ ▲                                                                                                
SyntaxError: invalid syntax


I am working on my company's laptop and it has ZScaler enabled, could this be part of the issue? 

Looking forward to a response, thanks.

same here. load('accuracy')
Repository: huggingface/evaluate
Issue: How to disable NLTK download information output?
Root cause: Configuration Error

When I use "metoer" evlauator by: 

    import evaluate
    meteor = evaluate.load('meteor')
    meteor_score = meteor.compute(predictions=predictions, references=reference)

Every time it will output the nltk download information as following. Is there any way supported by evaluate lib to disable such? 

<img width="520" alt="Screen Shot 2023-05-18 at 13 09 33" src="https://github.com/huggingface/evaluate/assets/27895653/f5d6bdab-c2ff-4c8c-b4d5-2ac8a4888ab6">



Hi! 

This thread answers your issue: https://stackoverflow.com/questions/47614999/how-to-stop-nltk-from-outputting-to-terminal-when-downloading-data.

Feel free to set `quiet=True` in the script's `nltk.download` [calls](https://github.com/huggingface/evaluate/blob/0ca575d7aa0764ea646dcd5a27cb952e587ce9eb/metrics/meteor/meteor.py#L116-L120) (maybe the info about the extra NLTK packages downloaded in `_download_and_prepare` can be emitted using the evaluate's logger) and open a PR.