zhiminy/EvalEng huggingface dataset:
Repository: huggingface/evaluate
Issue: Metrics for multilabel problems don't match the expected format.
Root cause: Documentation Deficiency

### Issue
Evaluation metrics cannot be used for multilabel classification problems.

### Reproducible example
You can find a reproducible snippet [here](https://gist.github.com/adamamer20/e84ffea42bf9b101b9cfbacfcfb7286d)

### Problem explanation
The error is given by how the expected format of some metrics has been chosen.
For example, for accuracy and f1("average"), f1("micro"), f1("macro"), the expected format is a scalar (```Value(dtype='int32', id=None)```) and thus breaksdown in a multilabel use (```ValueError: Predictions and/or references don't match the expected format.```).
Apart from the hassle of reshaping predictions and labels, and the confusion to define which indices correspond to the same label and which to the same instance, it's different from how it's done in other libraries. [Scikit-learn](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html) accepts nested lists in the case of multilabel f1.

### Possible solution
Refactor the format of ```EvaluationModule``` of accuracy and f1 (+ others...) to also accept ```Sequence```


Hi @adamamer20 , did you try to use `f1_metric = evaluate.load("f1", "multilabel")`?

You question is similar with #550

Thank you, it worked. I tried searching in the [docs](https://huggingface.co/spaces/evaluate-metric/f1) but there isn't anything on multilabel.
Repository: huggingface/evaluate
Issue: [Question]Shall we adding a faster BLEU score calculator?
Root cause: Unimplemented Feature Gap

## Shall we?
Hi, I found the [performance request](https://huggingface.co/spaces/evaluate-metric/bleu/discussions/8) about the bleu score. And I'm wondering if we should add a `fast-bleu` or replace the `bleu` implementation with a faster one, like [bleuscore](https://github.com/shenxiangzhuang/bleuscore) which is aimed for better performance for large data size.  

## About bleuscore library
### Correctness
I implement bleuscore library with exactly same way as evaluate library and do [serious test with hypothesis](https://github.com/shenxiangzhuang/bleuscore/blob/master/tests/test_bleu_score.py) to make sure they return same result.

### Performance
I wrote the library in Rust and used multi-threading for parallel execution, so it has better performance when the data size gets larger, you can check the [benchmark](https://github.com/shenxiangzhuang/bleuscore?tab=readme-ov-file#benchmark) section for more details.

<p align="center">
  <img src="https://github.com/shenxiangzhuang/bleuscore/blob/master/benchmark/bench.png?raw=true" alt="Benchmark" width="400" height="300">
</p>
Repository: huggingface/evaluate
Issue: Can't load exist dataset for evaluation
Root cause: Validation Gap

Hi, I used `dair-ai/emotion` to fine-tune the `bert-base-cased` model to use it for text-classification task.

The dataset is loaded when I load it in this script, so there is no problem up to here:

```python
from datasets import load_dataset

dataset = load_dataset("dair-ai/emotion")
```

But in the evaluation script here:

```python

import numpy as np
import evaluate

predictions = trainer.predict(eval_dataset)

preds = np.argmax(predictions.predictions, axis=-1)

metric = evaluate.load("dair-ai/emotion")

metric.compute(predictions=preds, references=predictions.label_ids)
```
I got: **FileNotFoundError**: Couldn't find a module script at /content/dair-ai/emotion/emotion.py. **Module 'dair-ai/emotion' doesn't exist on the Hugging Face Hub either.**

I tried to find out the reason but did not find the solution.

:)
Repository: huggingface/evaluate
Issue: SyntaxError: closing parenthesis '}'

When I try to run the following script:

```
python -c "import evaluate; print(evaluate.load('exact_match').compute(references=['hello'], predictions=['hello']))"
```
I get this error:

```
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/{username}/anaconda3/envs/peft/lib/python3.10/site-packages/evaluate/loading.py", line 751, in load
    evaluation_cls = import_main_class(evaluation_module.module_path)
  File "/home/{username}/anaconda3/envs/peft/lib/python3.10/site-packages/evaluate/loading.py", line 76, in import_main_class
    module = importlib.import_module(module_path)
  File "/home/{username}/anaconda3/envs/peft/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 879, in exec_module
  File "<frozen importlib._bootstrap_external>", line 1017, in get_code
  File "<frozen importlib._bootstrap_external>", line 947, in source_to_code
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/{username}/.cache/huggingface/modules/evaluate_modules/metrics/evaluate-metric--exact_match/009c8b5313309ea5b135d526433d5ee76508ba1554cbe88310a30f85bb57ec88/exact_match.py", line 16
    }
    ^
SyntaxError: closing parenthesis '}' does not match opening parenthesis '(' on line 14
```

Looking forward to a response, thanks.

same problem

I fixed the issue by changing export HF_ENDPOINT=https://huggingface.co/, I guess the problem may be from the https://hf-mirror.com/


> I fixed the issue by changing export HF_ENDPOINT=https://huggingface.co/, I guess the problem may be from the https://hf-mirror.com/

Sorry, it's the problem caused by hf-mirror.com, I have fixed it.
Repository: huggingface/evaluate
Issue: Evaluation of empty strings with MAUVE results in error 
Root cause: Validation Gap

Hi, I'm currently using HF wrapper to evaluate strings with MAUVE: https://huggingface.co/spaces/evaluate-metric/mauve

I'm getting the following an error when passing empty strings (e.g. `""`). Is this an expected behavior? I would think that the empty string could be considered a valid string and therefore could be evaluated with MAUVE?

```
File "src/main.py", line 194, in <module>
    main()
  File "src/main.py", line 185, in main
    mauve_results_local, mauve_results_global = evaluate(
  File "/mnt/aiongpfs/projects/ai-art/global-decoding/global-decoding/src/eval/evaluate.py", line 58, in evaluate
    mauve_results_global = mauve.compute(
  File "/mnt/aiongpfs/projects/ai-art/global-decoding/lib/python3.8/site-packages/evaluate/module.py", line 462, in compute
    output = self._compute(**inputs, **compute_kwargs)
  File "/home/users/dgareev/.cache/huggingface/modules/evaluate_modules/metrics/evaluate-metric--mauve/653f4690bcc6e9e16df42f9f85bd0f1b59cca1ae2864058dceedc595f26f6705/mauve.py", line 136, in _compute
    out = compute_mauve(
  File "/mnt/aiongpfs/projects/ai-art/global-decoding/lib/python3.8/site-packages/mauve/compute_mauve.py", line 87, in compute_mauve
    p_features = get_features_from_input(
  File "/mnt/aiongpfs/projects/ai-art/global-decoding/lib/python3.8/site-packages/mauve/compute_mauve.py", line 176, in get_features_from_input
    features = featurize_tokens_from_model(MODEL, tokenized_texts, batch_size, name).detach().cpu().numpy()
  File "/mnt/aiongpfs/projects/ai-art/global-decoding/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/mnt/aiongpfs/projects/ai-art/global-decoding/lib/python3.8/site-packages/mauve/utils.py", line 114, in featurize_tokens_from_model
    outs = model(input_ids=padded_chunk,
  File "/mnt/aiongpfs/projects/ai-art/global-decoding/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/mnt/aiongpfs/projects/ai-art/global-decoding/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "/mnt/aiongpfs/projects/ai-art/global-decoding/lib/python3.8/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 777, in forward
    input_ids = input_ids.view(-1, input_shape[-1])
RuntimeError: cannot reshape tensor of 0 elements into shape [-1, 0] because the unspecified dimension size -1 can be any value and is ambiguous
```
Repository: huggingface/evaluate
Issue: Execution of example from the  Using the evaluator docs fails due to unspecified tokenizer
Root cause: Documentation Deficiency

Instead of calculating metrics, the first example of evaluation[1] fails since the tokenizer isn't provided nor inferred.

> Exception: Impossible to guess which tokenizer to use. Please provide a PreTrainedTokenizer class or a path/identifier to a pretrained tokenizer.

To replicate, simply try to execute following:
```
from datasets import load_dataset
from evaluate import evaluator
from transformers import AutoModelForSequenceClassification, pipeline

data = load_dataset("imdb", split="test").shuffle(seed=42).select(range(1000))
task_evaluator = evaluator("text-classification")

# 1. Pass a model name or path
eval_results = task_evaluator.compute(
    model_or_pipeline="lvwerra/distilbert-imdb",
    data=data,
    label_mapping={"NEGATIVE": 0, "POSITIVE": 1}
)

# 2. Pass an instantiated model
model = AutoModelForSequenceClassification.from_pretrained("lvwerra/distilbert-imdb")

eval_results = task_evaluator.compute(
    model_or_pipeline=model,
    data=data,
    label_mapping={"NEGATIVE": 0, "POSITIVE": 1}
)
```
evaluate===0.4.1

[1]https://huggingface.co/docs/evaluate/base_evaluator
Repository: huggingface/evaluate
Issue: Feature: `push_to_hub` function for metrics.
Root cause: Unimplemented Feature Gap

Add a function to easily push the result of a metric to the hub similar to the `Trainer` in `transformers`.

Use-case: a user evaluates a model on a dataset. If this happens with a `transformer` model and they use the `Trainer` they can easily do that during training. However, if they are either not using a `transformer` model, the `Trainer` or want to do it after training they have to do it manually. Thus it would be nice to easily push the results of a model and framework agnostic model to the hub to add it to a models dataset card.

The workflow would be roughly the following:

```Python
metric = load_metric("lvwerra/my_metric")
metric.add(some_predictions, some_references)
metric.compute()
metric.push_to_hub(name="my_new_metric", model="lvwerra/my_model", dataset="lvwerra/my_dataset")
```
This adds the result of the metric to the meta information of the `README.md` so it is displayed in the model card.

cc @osanseviero @lhoestq 

Isn't the metric the same when you are pushing to the Hub? Wouldn't something like this work?

```
metric.push_to_hub(model="lvwerra/my_model", dataset="lvwerra/my_dataset")
```

The reason why I thought it would make sense to have at least an option to rename it is that there could be different evaluation settings but same metric is used. E.g. for generation you could calculate BLEU at different temperatures and then report it as "BLEU@T=0.5" or "BLEU@T=2" (code evaluation also has something similar with the pass@k metric). We would also need to think about metrics that return several values.

> `metric.push_to_hub(model="lvwerra/my_model", dataset="lvwerra/my_dataset")`

This looks maybe a bit weird API-wise given that it pushes to the model repo

maybe something like the following would be a cleaner API?
```python
# either a free-standing function:

evaluate.push_to_hub(metric, to="lvwerra/my_model")

# or just expose/use a generic function from huggingface_hub?

huggingface_hub.metadata_update(
  repo_id="lvwerra/my_model",
  update={"metrics": metric.serialize()}
)
```

I like the `evaluate.push_to_hub(metric, to="lvwerra/my_model")` path better. 

Same, maybe we can add the second option for more "power users" and use it as well under the hood of `evaluate.push_to_hub`. What do you think?
Repository: huggingface/evaluate
Issue: Fresh pip installation gives error
Root cause: External Dependency Breakage

I installed the library with pip and I get the following error.

```
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
/tmp/ipykernel_9716/2159764109.py in <cell line: 1>()
----> 1 import evaluate

~/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages/evaluate/__init__.py in <module>
----> 1 from . import estimators
      2 from . import evaluators
      3 from . import preprocessors
      4 from . import utils
      5 

~/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages/evaluate/estimators.py in <module>
     41     GradientBoostingClassifier(),
     42     DecisionTreeClassifier(),
---> 43     DummyClassifier('most_frequent'),
     44 ]
     45 

TypeError: __init__() takes 1 positional argument but 2 were given
```

## Environment info
- `evaluate` version: 0.0.3
- Platform: Linux-5.4.0-1066-aws-x86_64-with-glibc2.10
- Python version: 3.8.12
- PyArrow version: 7.0.0
- Pandas version: 1.3.4

Hi @prabhakar267, `evaluate` is not officially released, yet, so there is also no pip package for `evaluate`. You are installing https://github.com/marella/evaluate. This will change when we release the package which should be shipped in the coming weeks :)
Repository: huggingface/evaluate
Issue: Gradio dependency issue
Root cause: External Dependency Breakage

**Context:**
I noticed this issue when I was working on submitting my own [metric](https://huggingface.co/spaces/nbansal/semf1) to the hub. At the top of the page, it keeps on asking me to update the Gradio version. 
<img width="1099" alt="image" src="https://github.com/huggingface/evaluate/assets/5251592/fbe734d4-c1fe-4daf-8695-9991b2c657e7">

**Error:**
Upon updating it, I got the following error - `AttributeError: module 'gradio' has no attribute 'inputs'`. 

**Reason:**
Upon further digging, I think the issue is occurs whenever you upgrade Gradio to version `>=4.0`. Turns out from version 4 onwards, `gradio.inputs` has been removed as you can see in the `__init__` [file](https://github.com/gradio-app/gradio/blob/gradio%404.0.0/gradio/__init__.py). However, `evaluate` library calls it in `launch_gradio_widget` function in `utils` [dir](https://github.com/huggingface/evaluate/blob/main/src/evaluate/utils/gradio.py#L114).

**Solution:**
A simple solution would be to limit the dependency below Gradio version 4.0.0. This can be done by simply updating the line 63 in `setup.py` [file]((https://github.com/huggingface/evaluate/blob/main/setup.py#L63)). More specifically, instead of - `"gradio>=3.0.0"`, it can be chnaged to `"gradio>=3.0.0, <4.0.0"` to fix the issue.

P.S. - I am still working on the metric so the page might go down until I fully complete it. 

Hi @bnaman50 

Yes, I have the same issue. 

I think that another solution would be to modify the code in src/evaluate/utils/gradio.py to change the syntax of the depreciated gr.inputs and gr.outputs to use the interface components directly. 
For maintaining current available evaluate metrics, the implementation can differ based on the version (an if statement would do). 

Yes, I'm seeing this as well with my own metrics on the Hugging Face Hub. The Hugging Face Hub is currently prompting everyone to upgrade to the latest gradio 4.xx.xx version (see screenshot), so this issue will break many metrics spaces. 
<img width="1695" alt="Screenshot 2024-09-13 at 3 31 02 PM" src="https://github.com/user-attachments/assets/74a26283-d489-4338-a1a2-5640bd6103ff">
Repository: huggingface/evaluate
Issue: AttributeError: 'CombinedEvaluations' object has no attribute 'evaluation_modules'
Root cause: Validation Gap

Hi, I tried to run the following script by combining two metrics, but the error occurred:

```python
Python 3.9.16 (main, Apr  5 2023, 13:58:15)
Type 'copyright', 'credits' or 'license' for more information
IPython 8.18.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import evaluate

In [2]: evaluations = ("accuracy", "f1", "precision", "recall")

In [3]: clf_metrics = evaluate.combine(evaluations=evaluations)
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[3], line 1
----> 1 clf_metrics = evaluate.combine(evaluations=evaluations)

File ~/.pyenv/versions/3.9.16/envs/my-dev/lib/python3.9/site-packages/evaluate/module.py:1034, in combine(evaluations, force_prefix)
   1008 def combine(evaluations, force_prefix=False):
   1009     """Combines several metrics, comparisons, or measurements into a single `CombinedEvaluations` object that
   1010     can be used like a single evaluation module.
   1011
   (...)
   1031     ```
   1032     """
-> 1034     return CombinedEvaluations(evaluations, force_prefix=force_prefix)

File ~/.pyenv/versions/3.9.16/envs/my-dev/lib/python3.9/site-packages/evaluate/module.py:884, in CombinedEvaluations.__init__(self, evaluation_modules, force_prefix)
    881     self.evaluation_module_names = list(evaluation_modules.keys())
    882 loaded_modules = []
--> 884 for module in self.evaluation_modules:
    885     if isinstance(module, str):
    886         module = load(module)

AttributeError: 'CombinedEvaluations' object has no attribute 'evaluation_modules'
```

If we use a list format for `evaluations`, it works fine, but I think we should also support tuple format, which work in the same way as list. This can be easily achieved by modifying the following section. 

https://github.com/huggingface/evaluate/blob/main/src/evaluate/module.py#L877

May I create a PR?

@shunk031 did you?

Hi, I created the PR at #604.
Repository: huggingface/evaluate
Issue: [Metrics] ValueError: Expected to find locked file from process x but it doesn't exist.
Root cause: Resource Mishandling

When calling .compute in distributed multi-node setting, I get this error - 

```
[rank1]:   File "/ext3/miniconda3/envs/venv/lib/python3.8/site-packages/transformers/trainer.py", line 2750, in _evaluate
[rank1]:     metrics = self.evaluate(ignore_keys=ignore_keys_for_eval)
[rank1]:   File "/ext3/miniconda3/envs/venv/lib/python3.8/site-packages/transformers/trainer_seq2seq.py", line 180, in evaluate
[rank1]:     return super().evaluate(eval_dataset, ignore_keys=ignore_keys, metric_key_prefix=metric_key_prefix)
[rank1]:   File "/ext3/miniconda3/envs/venv/lib/python3.8/site-packages/transformers/trainer.py", line 3641, in evaluate
[rank1]:     output = eval_loop(
[rank1]:   File "/ext3/miniconda3/envs/venv/lib/python3.8/site-packages/transformers/trainer.py", line 3924, in evaluation_loop
[rank1]:     metrics = self.compute_metrics(EvalPrediction(predictions=all_preds, label_ids=all_labels))
[rank1]:   File "/home/rm6418/projects/GEMT5/gemt5-mul.py", line 486, in <lambda>
[rank1]:     compute_metrics=lambda eval_pred: compute_metrics(eval_pred, tokenizer, accelerator),
[rank1]:   File "/home/rm6418/projects/GEMT5/gemt5-mul.py", line 119, in compute_metrics
[rank1]:     return metric.compute(predictions=decoded_preds, references=references)
[rank1]:   File "/ext3/miniconda3/envs/venv/lib/python3.8/site-packages/evaluate/module.py", line 455, in compute
[rank1]:     self.add_batch(**inputs)
[rank1]:   File "/ext3/miniconda3/envs/venv/lib/python3.8/site-packages/evaluate/module.py", line 515, in add_batch
[rank1]:     self._init_writer()
[rank1]:   File "/ext3/miniconda3/envs/venv/lib/python3.8/site-packages/evaluate/module.py", line 664, in _init_writer
[rank1]:     self._check_rendez_vous()  # wait for master to be ready and to let everyone go
[rank1]:   File "/ext3/miniconda3/envs/venv/lib/python3.8/site-packages/evaluate/module.py", line 377, in _check_rendez_vous
[rank1]:     raise ValueError(f"Couldn't acquire lock on {lock_file_name} from process {self.process_id}.") from None
[rank1]: ValueError: Couldn't acquire lock on /scratch/rm6418/gemt5_cache/sacrebleu/default/gemt5_exp1-12-rdv.lock from process 1.
```

I've looked at https://github.com/huggingface/evaluate/issues/481, https://github.com/huggingface/evaluate/issues/542
but this issue still seems to be happening on the current latest released versions 

All metrics are loaded with the same `experiment_id`, and with the correct `num_process` arguments.
All the files (the lock files) are present in the cache directory.
 
## Environment
evaluate - 0.4.2
accelerate - 0.31.0
datasets - 2.20.0
transformers - 4.42.3

Any suggestions appreciated!

I ended up fixing it by computing metrics only on the main process. I used `accelerator.gather_for_metrics()` and then the following:
```python
if accelerator.is_main_process:
    metrics.compute()
```
Only workaround I could find until it gets fixed upstream.
Repository: huggingface/evaluate
Issue: Unable to compute f1 score  - Throwing Value Error trying to convert a string in non english Language to integer
Root cause: Validation Gap

I am trying to compte f1 score ,My predictions are mostly numbers whreas my references are sometimes malayalm sting version of these numbers.  Thus I am getting a value error trying to convert this string to integer.

ValueError: invalid literal for int() with base 10: 'നാല്'

How to deal with this question?
Repository: huggingface/evaluate
Issue: How to customize my own evaluator and metrics?
Root cause: Documentation Deficiency

I'm facing a task on VQA, where I need to compute [VQA](https://visualqa.org/evaluation.html) accuracy](https://visualqa.org/evaluation.html) as follows:
```math
\text{Acc}(ans) = \min{ \left\{  \frac{\text{\# humans that said } ans  }{3}, 1 \right\} }
```
I have following questions:
1. Do I need to customize my own metric? If so, can I only create `metrics/vqa_accuracy/vqa_accuracy.py` without other operations, such as running `evaluate-cli create "accuracy name" --module_type "metric"`?
2. I found that there is no suitable `evaluator` for my task, and I'm not sure if it is possible to customize my own `evaluator`, since I didn't find any document on creating new `evaluator`.