zhiminy/EvalEng huggingface dataset:
Repository: huggingface/evaluate
Issue: Multi-label with f1
Root cause: Documentation Deficiency

How to Test Multi-label Classification with f1 in metrics?
And please give me a sample example?

There is an example in the [docs of f1](https://github.com/huggingface/evaluate/blob/main/metrics/f1/f1.py):
```
    Example 4-A multiclass example, with different values for the `average` input.
        >>> predictions = [0, 2, 1, 0, 0, 1]
        >>> references = [0, 1, 2, 0, 1, 2]
        >>> results = f1_metric.compute(predictions=predictions, references=references, average="macro")
        >>> print(round(results['f1'], 2))
        0.27
        >>> results = f1_metric.compute(predictions=predictions, references=references, average="micro")
        >>> print(round(results['f1'], 2))
        0.33
        >>> results = f1_metric.compute(predictions=predictions, references=references, average="weighted")
        >>> print(round(results['f1'], 2))
        0.27
        >>> results = f1_metric.compute(predictions=predictions, references=references, average=None)
        >>> print(results)
        {'f1': array([0.8, 0. , 0. ])}
```

@harmbuisman's answer is not the correct answer for multilabel but for single-label multiclass. If you want to use it with a multilabel setup see here #585.

`f1_metric = evaluate.load("f1", "multilabel")`
Repository: huggingface/evaluate
Issue: Feature: enrich `MetricsInfo` with more meta information
Root cause: Unimplemented Feature Gap

Add more meta information to the `MetricsInfo`. The existing fields are listed here: https://github.com/huggingface/evaluate/blob/f88449588d89365b646754110c9034736e8ae830/src/evaluate/metric.py#L76-L135

Things that could be useful to add:

- [ ] `requirements`: we could warn and instruct on what is required if not already installed.
- [ ] `test_cases`: a list of input-output pairs to test whether the metrics works as expected. this could also be used to populate a metric widget with examples where the user can preselect a few choices before testing a custom input. Could also trigger automatic tests of metrics.
- [ ] `score_range`: what are the possible values for the metric score. E.g. [0, 1] for accuracy or [-1, 1] for cosine similarity. 
- [ ] `score_order`: is larger or smaller better. Could be useful for automatic ordering of several results.
- [ ] `task_id`: a list of tasks this metric is applicable (e.g. `"binary-classification"`

cc @sashavor @lhoestq 

I would put the requirements in an actual `requirements.txt` file next to the metric script. Same for test_cases, you can have a dedicated test python file I think (we'll have to see what we add in there, compared to the doctests we already have though).

`task_ids` can be in the YAML part of the readme, so that moon-landing can parse it and index it on the website.

`score_range` and `score_order` are nice additions too. But not sure if it would be best in YAML or in the MetricInfo (the MetricInfo can also parse the YAML if we want)

also, can we add `linked_dataset` to reflect cases when a metric is limited to a specific dataset?

@sashavor For this you could just piggy-back on this existing Space feature maybe:
<img width="1012" alt="image" src="https://user-images.githubusercontent.com/326577/162252664-313e4320-d999-4e51-a96b-780aeb5e1d6f.png">


We now have requirements as well as test cases in the template #14 and can add datasets in the yaml of the `README.md`. Score ranges are also described in the `README.md` template. Closing this for now.
Repository: huggingface/evaluate
Issue: Wrong accuracy on the validation set!!!
Root cause: Documentation Deficiency

Hi All,

I fine tuned "distilbert-base-uncased" for text classification on a custom dataset. The best accuracy on the validation set was 0.9009 as you can see here: https://huggingface.co/anuragrawal/distilbert-base-uncased-finetuned

I am new to using the evaluate library and wanted to use it for my test set. Before doing that, I thought of doing an accuracy check on the validation set but the accuracy is different than 0.9009. I trained my model for only 2 epochs and the 2nd epoch had the best accuracy. When I use evaluate on my validation set, the accuracy is 0.893326717935996 which surprisingly was the accuracy on the validation set at the end of 1st epoch. Is this an issue with the evaluate library? or am I missing something?

I followed this page for using the evaluate library: https://huggingface.co/docs/evaluate/base_evaluator

Let me know if you need any additional information.

Thanks!

Adding the huggingface screenshot here since I might run another training iteration:
![image](https://github.com/huggingface/evaluate/assets/140737044/7a164732-1cd0-49bd-88db-e0c0eda9a48b)
Repository: huggingface/evaluate
Issue: wrong rouge-Lsum results due to lack of parameter control
Root cause: Architectural Constraint

https://github.com/huggingface/evaluate/blob/937a8ddbd74abaaf0d63b9ff2290fccd0277eca2/metrics/rouge/rouge.py#L132

This line creates a RougeScorer object, which also includes ```split_summaries=False``` in its original implementation. 

``` split_summaries```  set to ```False``` splits texts into sentences for ROUGE-LSum based on ```\n```, while setting it to ```True``` allows for a more complex tokenization based on punctuation. 

I think it is worth adding control over this parameter to evaluate rouge interface since normally summary sentences aren't separated by newlines and not having control over this parameters leads to ROUGE-L and ROUGE-LSum always producing the same result while being different metrics.
Repository: huggingface/evaluate
Issue: Perplexity Space Broken
Root cause: External Dependency Breakage

Not sure where to leave this issue so I'm guessing here. It would be nice to have a working space for perplexity here so I don't have to dig in the code when I have a question about it. (if the evaluate spaces are not maintained anymore feel free to close)

[Broken Space](https://huggingface.co/spaces/evaluate-metric/perplexity)

![image](https://github.com/huggingface/evaluate/assets/26125592/6febe641-e9b0-424d-b27c-981609dd4b55)
Repository: huggingface/evaluate
Issue: Move away from `Repository` to `HfApi` upload files
Root cause: External Dependency Breakage

`Repository` is being deprecated (see https://github.com/huggingface/huggingface_hub/pull/1724) but currently all the evaluate modules are pushed to the hub as spaces with Repository so we’ll need to update that too at some point (from @lvwerra on -private- [slack](https://huggingface.slack.com/archives/C02V5EA0A95/p1697184743250609?thread_ts=1697184108.192679&cid=C02V5EA0A95))

See https://github.com/huggingface/evaluate/blob/main/.github/hub/push_evaluations_to_hub.py
Repository: huggingface/evaluate
Issue: Compatibility Issue with datasets library
Root cause: External Dependency Breakage

evaluate library use huggingface's [datasets](https://github.com/huggingface/datasets) library.
Datasets' recent version is 2.14.5. But, because of [this PR](https://github.com/huggingface/datasets/pull/6005) which merged at 2.14.0, evaluate library cannot work.

When I use 2.14.5 version, I can't use evaluate's methor metric with the below error.
<img width="672" alt="image" src="https://github.com/huggingface/evaluate/assets/19206046/163af1cf-463d-4c00-841c-0f1154ff626e">

You must notice dataset's max version.


This should be fixed on `main` I believe and there should be a release shortly. cc @lhoestq 

Yes I'm on it ;)

I just did the release, you can now update evaluate it will work with recent datasets versions

```
pip install -U evaluate
```
Repository: huggingface/evaluate
Issue: Loading metrics that are shipped with the evaluate package is slow
Root cause: Algorithmic Error

Loading metrics that are shipped with the evaluate package takes way too long to load, up to or more than a second whereas I expect it to be near instant.

Repro:
Run the following in a jupyter notebook:
`import evaluate`

```
%%prun
evaluate.load("accuracy")
```

This outputs the following, suggesting that even for this metric that is available in the package [itself ](https://github.com/huggingface/evaluate/blob/main/metrics/accuracy/accuracy.py)it sets up all kinds of communication with the HF hub:
```
         22184 function calls (21809 primitive calls) in 1.351 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        2    0.643    0.322    0.643    0.322 {method 'load_verify_locations' of '_ssl._SSLContext' objects}
        2    0.240    0.120    0.240    0.120 {method 'read' of '_ssl._SSLSocket' objects}
        2    0.223    0.112    0.223    0.112 {method 'do_handshake' of '_ssl._SSLSocket' objects}
        2    0.145    0.073    0.145    0.073 {method 'connect' of '_socket.socket' objects}
        2    0.038    0.019    0.038    0.019 {built-in method _socket.getaddrinfo}
       80    0.009    0.000    0.009    0.000 {built-in method nt.stat}
       45    0.006    0.000    0.006    0.000 {built-in method __new__ of type object at 0x00007FF81939AD50}

```

Yes, the metrics are loaded from the Hub, which is why you are observing that it takes 1-2 seconds to load, but in follow-up loading they should be cached.

The evaluate.load("accuracy") loads the sklearn wrapper that is shipped with the package, so it should *not* go to the hub, See the location within the package: https://github.com/huggingface/evaluate/blob/main/metrics/accuracy/accuracy.py

It takes 1-2 seconds every call  to evaluate.load, so no speed improvements on a follow-up call.
Repository: huggingface/evaluate
Issue: add_batch taking very long time (SegFormer semantic segmentation).
Root cause: Algorithmic Error

When I call **"add_batch"** with 512X512 prediction image (and 512X512) label
With batch size of 40 (so the input are 40X512X512).
It takes **50 seconds per batch** on my machine (windows 10 with RTX 3090 and  i9-12900K intel CPU. Torch: 1.12.1+cu113).
(It is stuck  in "encode_batch" which call the recursive "encode_nested_example")
It is much longer than training itslef

As a patch:
**I see that If I switch to "Evaluate.compute" it still takes long time but "Evaluate._compute" is much faster.**

**Can I change "add_batch" to something else to enjoy the faster execution the same way replacing "compute" with "_compute ?**
Repository: huggingface/evaluate
Issue: HuggingFace evaluate-cli create command results Error: "datasets[0]" must be a string
Root cause: Validation Gap

I follow the official documentation of `evaluate 0.4.0` to **freshly** create a custom metric in an empty folder with the following command:

    evaluate-cli create "My Metric" --module_type "metric"

The command returns the following error:

    Error: "datasets[0]" must be a string

The command creates these files but I am unsure whether the files are intact:

    app.py  my_metric.py  README.md  requirements.txt  tests.py

How to resolve this error?

using `evaluate version 0.4.0`

Check your `README.md`, then there is `datasets` in YAML style. Remove that.

from
```
---
title: My metric
datasets:
-  
tags:
- evaluate
- metric
description: "TODO: add a description here"
sdk: gradio
sdk_version: 3.19.1
app_file: app.py
pinned: false
---
```

to

```
---
title: My metric
tags:
- evaluate
- metric
description: "TODO: add a description here"
sdk: gradio
sdk_version: 3.19.1
app_file: app.py
pinned: false
---
```

Where exactly is this yaml file located?
Repository: huggingface/evaluate
Issue: Allow BLEURT optimisation parameters
Root cause: Unimplemented Feature Gap

BLEURT is very slow to compute: I'm observing 426.62 seconds on a g5.xlarge with `BLEURT-20-D6` for 1459 sentences.

It'd be useful to be able to use the optimisations specified in [speeding-up-bleurt](https://github.com/google-research/bleurt/tree/master#speeding-up-bleurt)

1. Tune batch size by passing `batch_size` as a kwarg to `scorer.score` (https://github.com/google-research/bleurt/blob/cebe7e6f996b40910cfaa520a63db47807e3bf5c/bleurt/score.py#L175)
2. Use the `LengthBatchingBleurtScorer` (https://github.com/google-research/bleurt/blob/cebe7e6f996b40910cfaa520a63db47807e3bf5c/bleurt/score.py#L227C7-L227C33)
Repository: huggingface/evaluate
Issue: Importing Error for evaluator module
Root cause: Validation Gap

I just started using HF Evaluate and I'm trying to follow the steps in their tutorial to do make sure it all works properly. However I keep getting the problem: 

 raise ImportError(
ImportError: If you want to use the `Evaluator` you need `scipy>=1.7.1`. Run `pip install evaluate[evaluator]`.

I already made sure that scipy is installed (Currently using V1.8.1) And I still keep getting the issue.  What could I be doing wrong now? 

(Using VS Code if that makes a difference)
Repository: huggingface/evaluate
Issue: `evaluate-cli create` fails with `subprocess.CalledProcessError`
Root cause: Validation Gap

I tried to create a new metric by doing this:
```bash
evaluate-cli create "pseudo_log_likelihood" --module_type "metric"
```
but I get the following error message:
```
Traceback (most recent call last):
  File "/home/tai/master/annotation/temporal-common-sense/.venv/bin/evaluate-cli", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/tai/master/annotation/temporal-common-sense/.venv/lib/python3.11/site-packages/evaluate/commands/evaluate_cli.py", line 97, in main
    subprocess.run(
  File "/usr/lib/python3.11/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['git', 'clone', 'https://huggingface.co/spaces/Tai-Mai/pseudo_log_likelihood']' returned non-zero exit status 128.
```

The space is created nonetheless and I have to delete it manually from the web. Otherwise, if I try to run the command above again, I get this error message:
```
Could not create Space for module at hf.co/spaces/Tai-Mai/pseudo_log_likelihood. Make sure this space does not exist already.
```

Could you help me resolve the issue?

Some additional information that might or might not be relevant:
* Arch Linux
* Virtual environment using poetry
* Authenticated via SSH key

Many thanks in advance for any help.

Same error :(
Can someone help us?
Repository: huggingface/evaluate
Issue: Getting error <lambda>() takes 0 positional arguments but 1 was given when trying to import evaluator
Root cause: External Dependency Breakage

I'm trying to use evaluator in a Google Colab notebook, but I can't import it. I installed it with:
`!pip install evaluate[evaluator]`

Then trying importing it:
`from evaluate import evaluator` or simply `import evaluate`

But I'm getting this error when running the cell:
```TypeError                                 Traceback (most recent call last)
[<ipython-input-40-03c510640e72>](https://localhost:8080/#) in <cell line: 1>()
----> 1 from evaluate import evaluator
      2 # perplexity = evaluate.load("perplexity", module_type="metric")
      3 # input_texts = ["lorem ipsum", "Happy Birthday!", "Bienvenue"]
      4 
      5 # results = perplexity.compute(model_id='google/flan-t5-base',

11 frames
[/usr/lib/python3.10/_pyio.py](https://localhost:8080/#) in __init__(self, buffer, encoding, errors, newline, line_buffering, write_through)
   2043                 encoding = "utf-8"
   2044             else:
-> 2045                 encoding = locale.getpreferredencoding(False)
   2046 
   2047         if not isinstance(encoding, str):

TypeError: <lambda>() takes 0 positional arguments but 1 was given
```

Can anyone shine a light on it, please?
Repository: huggingface/evaluate
Issue: Cannot use evaluate.TextClassificationEvaluator for multi-label classification
Root cause: Validation Gap

### System Info

- `transformers` version: 4.35.2
- Platform: Linux-5.15.120+-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.19.4
- Safetensors version: 0.4.1
- Accelerate version: 0.25.0
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.1.0+cu118 (True)
- Tensorflow version (GPU?): 2.14.0 (True)
- Flax version (CPU?/GPU?/TPU?): 0.7.5 (gpu)
- Jax version: 0.4.20
- JaxLib version: 0.4.20
- Using GPU in script? YES
- Using distributed or parallel set-up in script? NO

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I am working on emotion classification and I am willing to evaluate the model with this [evaluator](https://huggingface.co/docs/evaluate/package_reference/evaluator_classes#evaluate.TextClassificationEvaluator).
 
Running [this example](https://colab.research.google.com/drive/1RiaNkVuPecn-uKQWHy2XVojvb6Ng-oMh?usp=sharing), I encounter the following error:
```
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
[<ipython-input-23-61f332f138dc>](https://localhost:8080/#) in <cell line: 3>()
      1 task_evaluator = evaluate.evaluator("sentiment-analysis")
      2 
----> 3 task_evaluator.compute(
      4     model_or_pipeline=model_hub_path,
      5     data=dataset,

7 frames
[/usr/local/lib/python3.10/dist-packages/sklearn/metrics/_classification.py](https://localhost:8080/#) in _check_set_wise_labels(y_true, y_pred, average, labels, pos_label)
   1389             if y_type == "multiclass":
   1390                 average_options.remove("samples")
-> 1391             raise ValueError(
   1392                 "Target is %s but average='binary'. Please "
   1393                 "choose another average setting, one of %r." % (y_type, average_options)

ValueError: Target is multiclass but average='binary'. Please choose another average setting, one of [None, 'micro', 'macro', 'weighted'].
```

### Expected behavior

I should set an `average` argument when invoking `task_evaluator.compute`. Without that, I wouldn't be able to use the evaluator for multi-label classification tasks.

I wrote a workaround for this issue on [stackoverflow](https://stackoverflow.com/questions/76441777/huggingface-evaluate-function-use-multiple-labels/78617839#78617839). 

Thanks, I think this can be closed!
Repository: huggingface/evaluate
Issue: Bug in BLEU calculation - BLEU of identical strings should be 1.0, not 0.0
Root cause: Algorithmic Error

## Summary

I think I discovered a bug related to BLEU calculation which occurs if you compare two identical strings. The expected behavior is that the BLEU of completely identical strings is 1.0, while in some cases I get 0.0.

## OS and Python versions

- **OS:** Windows 10 Pro
- **Python:** 3.10.13
- **Evaluate:** 0.4.1

## Bug description

If you run:

```
import evaluate
bleu = evaluate.load("bleu")
bleu.compute(predictions=["Duplicate string"], references=[["Dupliacte string"]])
```

you get:

```
{'bleu': 0.0,
 'precisions': [0.5, 0.0, 0.0, 0.0],
 'brevity_penalty': 1.0,
 'length_ratio': 1.0,
 'translation_length': 2,
 'reference_length': 2}
```

as the output. I expected the `'bleu'` to be `1.0`.

Similar thing happens if you run:

`bleu.compute(predictions=["foobar"], references=[["foobar"]])`

you get:

```
{'bleu': 0.0,
 'precisions': [1.0, 0.0, 0.0, 0.0],
 'brevity_penalty': 1.0,
 'length_ratio': 1.0,
 'translation_length': 1,
 'reference_length': 1}
```

Again, I expected BLEU to be `1.0`, not `0.0`.

This will happen for any input sentence having less than 4 words for the default case.

I checked and found that _evaluate_ uses [Tensorflow implementation for bleu calculation](https://github.com/tensorflow/nmt/blob/master/nmt/scripts/bleu.py). Based on the following code, if the sentence length is less than the _max_order_, then you will always get the bleu score as 0 (irrespective of the reference and predicted sentence).

```
if min(precisions) > 0:
  p_log_sum = sum((1. / max_order) * math.log(p) for p in precisions)
  geo_mean = math.exp(p_log_sum)
else:
  geo_mean = 0

.
.
.

bleu = geo_mean * bp
```

I'm not an expert in machine translation, so not sure if this is the expected behaviour. However, I would have guessed that if the precision is 0, then that particular n-gram case should not be considered for the geometric mean (coefficient must be made 0, instead of entire geo_mean).

Hoping some expert here helps us out to mention the correct resolution.

Hello! @MislavJuric 

@Shyam-Thombre is correct that by default, BLEU considers 4-grams when computing the score. In order to obtain your preferred output, you may set the value of the _max_order_ attribute to the number of k-grams you want BLEU to consider.

For example,
`bleu.compute(predictions=["Duplicate string"], references=[["Duplicate string"]], max_order=2)`
Will give you
`{'bleu': 1.0, 'precisions': [1.0], 'brevity_penalty': 1.0, 'length_ratio': 1.0, 'translation_length': 2, 'reference_length': 2}`
Repository: huggingface/evaluate
Issue: Documentation claims that `label_mapping` is an accepted input to `TokenClassificationEvaluator.compute()`; it is not
Root cause: Documentation Deficiency

[This line](https://github.com/huggingface/evaluate/blob/8dfe05784099fb9af55b8e77793205a3b7c86465/docs/source/base_evaluator.mdx#L133C2-L133C99) in the documentation states that `label_mapping` is an accepted parameter of [`TokenClassificationEvaluator.compute()`](https://github.com/huggingface/evaluate/blob/8dfe05784099fb9af55b8e77793205a3b7c86465/src/evaluate/evaluator/token_classification.py#L220), when it in fact is not.

I suspect this line was absent-mindedly copied over from the `TextClassificationEvaluator` documentation (that class's [`compute()`](https://github.com/huggingface/evaluate/blob/8dfe05784099fb9af55b8e77793205a3b7c86465/src/evaluate/evaluator/text_classification.py#L91) method actually *does* accept this parameter).

In my opinion this would be a useful feature to implement (easily allowing users to evaluate models on datasets that don't use exactly the same labels), but at the very least this line should be removed from the documentation as long as the method doesn't support this parameter.