zhiminy/EvalEng huggingface dataset:
Repository: huggingface/evaluate
Issue: Calculating F1 fails due to error(AttributeError: 'float' object has no attribute 'size')
Root cause: External Dependency Breakage

When I try to calculate `f1` score I get this error: 
```
return {"f1": float(score) if score.size == 1 else score}
                                   ^^^^^^
AttributeError: 'float' object has no attribute 'size'
```

Code to Reproduce

```py
import evaluate
f1_metric = evaluate.load("f1")
result = f1_metric.compute(predictions=[0, 1], references=[0, 1])
print(result["f1"])
```

- Tested on a default github codespaces environment and google colab.
- evaluate==0.4.3

Experiencing the same issue with the last three versions of evaluate :(

https://github.com/huggingface/evaluate/issues/655 - the solution is here. there is an inconsistency caused by the scikit-learn update
Repository: huggingface/evaluate
Issue: Statistically nonsensical to use skewness in `label_distribution`
Root cause: Algorithmic Error

The `label_distribution` measurement is supposed to quantify how biased a column of labels is towards one value or another. It computes the fraction each label value takes up out of all labels (which is useful) and, problematically, the *skewness* of that distribution:

https://github.com/huggingface/evaluate/blob/55f1bc6e072b05c2d9db1589a07e20f38902b1ec/measurements/label_distribution/label_distribution.py#L85-L93

This is statistical nonsense. 

A class label is a [multinoulli variable](https://en.wikipedia.org/wiki/Categorical_distribution), which means that it's a discrete variable that can emit a finite amount of values **whose magnitude has no meaning other than being different from each other**. If you have classes `{cat, dog, giraffe}`, it has no meaning whether we choose to label `cat` as 0 or 3.1415, and it has no meaning whether `cat` is labelled 0 or `dog` is labelled 0, as long as they are different.

[*Skewness*](https://en.wikipedia.org/wiki/Skewness), however, is meant for continuous variables, and cares about magnitudes and permutations. **It looks at the symmetry of the distribution, not the uniformity.** 

- The label column `[0, 0, 0, 1, 1, 1, 2, 2, 2]` has 0 skewness, because it is symmetrical.
- The label column `[0, 0, 1, 1, 1, 1, 1, 2, 2]` has 0 skewness, because it is symmetrical. Yet, clearly, there is heavy bias towards label `1`.
- The label column `[0, 0, 1, 1, 2, 2, 2, 2, 2]`  is exactly as biased as the previous one (2 labels, 2 labels, 5 labels) *and yet now it has skewness != 0* because the weight of the distribution is "on the right", which has zero meaning.

The [*entropy*](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.entropy.html) of the labels is what you are looking for to measure uniformity, not skewness. Entropy is class-permutation-invariant. It is maximised for uniform distributions. If you want to normalise it, you can divide by that maximal entropy (the [Hartley function](https://en.wikipedia.org/wiki/Hartley_function)).
Repository: huggingface/evaluate
Issue: Question: ROC_AUC metric compute signature with prediction_scores
Root cause: Interface Contract Mismatch

Hi team,

I have a question regarding the `compute` signature in roc_auc metric.
it uses `prediction_scores` while other metrics uses only `predictions`, is that intended? why?
If not, could I do a small PR to change it to `predictions`?

Thank you!
Repository: huggingface/evaluate
Issue: MSE, r_squared not working, QUICK FIX (squared argument is deprecated in call to scikit-learn)
Root cause: External Dependency Breakage

https://scikit-learn.org/1.5/modules/generated/sklearn.metrics.mean_squared_error.html

At this link it's reported.
Hence, I suggest you change the call in your wrapping function to "root_mean_squared_error" in case "squared" was set to true.

Errors:
1) Error evaluating r_squared: r_squared._compute() got an unexpected keyword argument 'squared'
2) Error evaluating mse: mse._compute() got an unexpected keyword argument 'squared'

Huh, funny coincidence but I think I just did this: https://github.com/huggingface/evaluate/pull/662. I'll link this issue to that PR!

Damn, what a timing then ahaha!
Are you working on this repo? I wonder if I could've linked it too somehow.
Repository: huggingface/evaluate
Issue: MSE metric broken: `squared` parameter is deprecated
Root cause: External Dependency Breakage

Using the MSE metric doesn't work with scikit-learn version >1.6.0 since the `squared` parameter is deprecated:

```
/home/joeran/repos/dragon_baseline/.venv/lib/python3.10/site-packages/sklearn/metrics/_regression.py:492: FutureWarning: 'squared' is deprecated in version 1.4 and will be removed in 1.6. To calculate the root mean squared error, use the function'root_mean_squared_error'.
```

(as confirmed here: https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/metrics/_regression.py#L510)

Code to reproduce:
```
pip install scikit-learn==1.6.2
```

```python
import evaluate
metric = evaluate.load("mse")
metric.compute(predictions=(0,1,2), references=(1,2,3))
```

I'd like to take this issue. I'll update the MSE metric to be compatible with  Sci-Kit Learn 1.6. Let me know if there's anything specific to account for!
Repository: huggingface/evaluate
Issue: TypeError: object of type 'NoneType' has no len()

I followed [huggingface doc](https://huggingface.co/docs/transformers/v4.50.0/en/training#trainer) step by step, but get the following errors:
```
>>> trainer = Trainer(model=model, args=training_args, train_dataset=small_train_dataset, eval_dataset=small_eval_dataset, compute_metrics=compute_metrics)
>>> trainer.train()
 33%|██████████████████████████████████████████████████▎                                                                                                    | 125/375 [12:53<25:46,  6.19s/it]
                        Traceback (most recent call last):
  File "/Users/fanrong/transformers-env/lib/python3.9/site-packages/evaluate/module.py", line 518, in add_batch
    if len(column) > 0:█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 125/125 [00:35<00:00,  3.57it/s]
TypeError: object of type 'NoneType' has no len()

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/fanrong/transformers-env/lib/python3.9/site-packages/transformers/trainer.py", line 2245, in train
    return inner_training_loop(
  File "/Users/fanrong/transformers-env/lib/python3.9/site-packages/transformers/trainer.py", line 2647, in _inner_training_loop
    self._maybe_log_save_evaluate(tr_loss, grad_norm, model, trial, epoch, ignore_keys_for_eval, start_time)
  File "/Users/fanrong/transformers-env/lib/python3.9/site-packages/transformers/trainer.py", line 3093, in _maybe_log_save_evaluate
    metrics = self._evaluate(trial, ignore_keys_for_eval)
  File "/Users/fanrong/transformers-env/lib/python3.9/site-packages/transformers/trainer.py", line 3047, in _evaluate
    metrics = self.evaluate(ignore_keys=ignore_keys_for_eval)
  File "/Users/fanrong/transformers-env/lib/python3.9/site-packages/transformers/trainer.py", line 4136, in evaluate
    output = eval_loop(
  File "/Users/fanrong/transformers-env/lib/python3.9/site-packages/transformers/trainer.py", line 4425, in evaluation_loop
    metrics = self.compute_metrics(
  File "<stdin>", line 4, in compute_metrics
  File "/Users/fanrong/transformers-env/lib/python3.9/site-packages/evaluate/module.py", line 455, in compute
    self.add_batch(**inputs)
  File "/Users/fanrong/transformers-env/lib/python3.9/site-packages/evaluate/module.py", line 523, in add_batch
    if any(len(batch[c]) != len(next(iter(batch.values()))) for c in batch):
  File "/Users/fanrong/transformers-env/lib/python3.9/site-packages/evaluate/module.py", line 523, in <genexpr>
    if any(len(batch[c]) != len(next(iter(batch.values()))) for c in batch):
TypeError: object of type 'NoneType' has no len()
```
Could you please help me solve the problem? Thanks~
Repository: huggingface/evaluate
Issue: TypeError: got an unexpected keyword argument 'squared'
Root cause: External Dependency Breakage

The `squared` argument is no longer supported in [sklearn.metrics.mean_squared_error](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_squared_error.html). It was deprecated in version 1.4 and removed in 1.6.1.  

If you pass `squared`, it will raise:  
**TypeError: got an unexpected keyword argument 'squared'**
Repository: huggingface/evaluate
Issue: Issue in evaluate.EvaluationModule.add
Root cause: Validation Gap

Unlike `evaluate.EvaluationModule.add_batch`, `evaluate.EvaluationModule.add` DOES NOT WORK with Metrics (`evaluate.Metric(evaluate.EvaluationModule)`) whose Features (`evaluate.MetricInfo(evaluate.EvaluationModuleInfo).features`) are `List[datasets.Features]`, instead of `datasets.Features`. Because a `list` DOES NOT HAVE an `encode_example` Method as in Line 533 from `evaluate/src/evaluate/module.py`, throwing a `ValueError` that DOES NOT TRULY RELATE to the cause itself, as it can be seen in:

`ValueError: Predictions and/or references don't match the expected format.
Expected format:
Feature option 0: {'predictions': Value(dtype='string', id='sequence'), 'references': Sequence(feature=Value(dtype='string', id='sequence'), length=-1, id='references')}
Feature option 1: {'predictions': Value(dtype='string', id='sequence'), 'references': Value(dtype='string', id='sequence')},
Input predictions: ['void hello_world()'],
Input references: ['void hello_world()']`

The `evaluate.EvaluationModule.add` Method should instead use the `selected_feature_format` Field from `evaluate.EvaluationModule`, as in Line 486 from `evaluate/src/evaluate/module.py` at `evaluate.EvaluationModule.add_batch`.
Repository: huggingface/evaluate
Issue: Suppress the output of nltk download logs when loading metric via evaluate
Root cause: Validation Gap

How can the output of nltk be suppressed when loading `sacrebleu` via the `evaluate` library?

The minimal code is as follows:
```
import evaluate
bleu = evaluate.load("sacrebleu")
```
Then the output in the Jupyter Notebook is as follows:
```
Using the latest cached version of the module from /home/jupyter-raptor/.cache/huggingface/modules/evaluate_modules/metrics/evaluate-metric--sacrebleu/28676bf65b4f88b276df566e48e603732d0b4afd237603ebdf92acaacf5be99b (last modified on Fri Nov  8 20:44:01 2024) since it couldn't be found locally at evaluate-metric--sacrebleu, or remotely on the Hugging Face Hub.
[nltk_data] Downloading package wordnet to /home/jupyter-
[nltk_data]     raptor/nltk_data...
[nltk_data]   Package wordnet is already up-to-date!
[nltk_data] Downloading package punkt to /home/jupyter-
[nltk_data]     raptor/nltk_data...
[nltk_data]   Package punkt is already up-to-date!
[nltk_data] Downloading package omw-1.4 to /home/jupyter-
[nltk_data]     raptor/nltk_data...
[nltk_data]   Package omw-1.4 is already up-to-date!
```
This issue also happens for `chrf`, `meteor` and `bertscore` (and probably others).

Hi, I opened PR #726 to suppress NLTK download logs for meteor/word_length and updated the docs example accordingly. Thanks!
Repository: huggingface/evaluate
Issue: Feature Request: Add support for retrieving mispredicted examples with predictions and ground truth
Root cause: Unimplemented Feature Gap

First of all, thank you for the great work on the evaluate library — it's been incredibly useful for benchmarking and analyzing model performance!

I’d like to suggest a feature idea that could enhance the interpretability and debugging process when evaluating models. Specifically, it would be great if evaluate could provide a way to retrieve the list of mispredicted examples, including both the model's prediction and the corresponding ground truth.

This could be especially useful for Creating visualizations or reports of incorrect predictions

Thanks again 🙌
Repository: huggingface/evaluate
Issue: comet doesn't work when gpu=0
Root cause: Validation Gap

I am getting the error "multiprocessing_context can only be used with multi-process loading (num_workers > 0), but got num_workers=0" when trying to use compute with comet when gpus=0. This is a Mac M1 chip, so perhaps it isn't supported, even on a CPU basis? 

Same issue. 
Mac M4.
Repository: huggingface/evaluate
Issue: `import` is very slow
Root cause: Algorithmic Error

In tests, or when wanting to just debug something quickly, the process hangs on `import evaluate`.
I benchmarked it on two devices:
```bash
python -c "import time; start = time.time(); import datasets; print(f'datasets: {time.time() - start:.3f}s')"
python -c "import time; start = time.time(); import transformers; print(f'transformers: {time.time() - start:.3f}s')"
python -c "import time; start = time.time(); import evaluate; print(f'evaluate: {time.time() - start:.3f}s')"
```

Linux server:
datasets: 0.724s
transformers: 1.527s
evaluate: 3.438s

macOS:
datasets: 1.076s
transformers: 1.072s
evaluate: 8.987s

9 seconds on my mac (m1 pro) is very slow just for the import. can you please look at why this is the case?
Repository: huggingface/evaluate
Issue: exact_match sanity check fails with evaluate-0.4.3 and datasets-3.6.0
Root cause: External Dependency Breakage

I tried running the  "Did I install evaluate correctly" test and it does not work.


```
>  python -c "import evaluate; print(evaluate.load('exact_match').compute(references=['hello'], predictions=['hello']))"

[2025-05-23 11:13:09,005] [INFO] [real_accelerator.py:239:get_accelerator] Setting ds_accelerator to cuda (auto detect)
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/my_ven_dir/lib/python3.11/site-packages/evaluate/loading.py", line 748, in load
    evaluation_module = evaluation_module_factory(
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/my_venv_dir/lib/python3.11/site-packages/evaluate/loading.py", line 681, in evaluation_module_factory
    raise FileNotFoundError(
FileNotFoundError: Couldn't find a module script at /current_working_dir/exact_match/exact_match.py. Module 'exact_match' doesn't exist on the Hugging Face Hub either.
```

BLEU is _supposed_ to be installed with evaluate, no?
Repository: huggingface/evaluate
Issue: `rouge` is not available
Root cause: Unimplemented Feature Gap

Hi, I am using `rouge` like
```py
import evaluate
evaluate.load("rouge")
```

And it reports errors like
```
FileNotFoundError: Couldn't find a module script at .../rouge/rouge.py. Module 'rouge' doesn't exist on the Hugging Face Hub either.
```

I noticed that there is a runtime error on the [huggingface space](https://huggingface.co/spaces/evaluate-metric/rouge), maybe something requires fixing? 🤗 

![Image](https://github.com/user-attachments/assets/430c822e-26c2-44c7-87f7-81e7e796a00c)
Repository: huggingface/evaluate
Issue: Cannot access gated repositories in Perplexity measurements module
Root cause: Documentation Deficiency

Since accessing gated models requires authentication by passing either a hf_token or using the login() method.

The **measurements/perplexity/README.md** file doesn't mention how to authenticate for gated models, and I've tried passing the token via model configuration parameters directly to the **compute()** function. However, it seems there's no mechanism in place to capture **kwargs and pass them down to from_pretrained() within the current implementation:

https://github.com/huggingface/evaluate/blob/5aa3982a9a8c86e506860e381d428a64b0cce73b/measurements/perplexity/perplexity.py#L104-L106

https://github.com/huggingface/evaluate/blob/5aa3982a9a8c86e506860e381d428a64b0cce73b/src/evaluate/module.py#L467

An easy way would be just use login function before calling compute function

```python
from huggingface_hub import login

hf_token = os.environ["HF_TOKEN"]
model_id = "meta-llama/Meta-Llama-3.1-8B-Instruct"

perplexity = load("perplexity", module_type="measurement")

login(token = hf_token)

results = perplexity.compute(
    data=[first_question, second_question],
    model_id=model_id,
)

```

But this doesn't offer any flexibility for passing additional model parameters to from_pretrained().
It would be helpful if calling compute() could accept and propagate such kwargs to model.
Repository: huggingface/evaluate
Issue: Jiwer deprecation for compute_measures
Root cause: External Dependency Breakage

https://github.com/jitsi/jiwer/commit/4113e3f838bb2284a7c92facc444ce2c42f4a3b3#diff-07c485c09e8e877349b9b23380f2e87055efaf6a8e2f7e8dafe23658a3959972

As of February 2nd of 2025 the `compute_measures` method from jiwer has been deprecated, as such, using evaluate's "wer" will throw an error like this:
```python
Traceback (most recent call last):
  File "/home/user/whisper/main.py", line 101, in <module>
    metric = evaluate.load("wer")
             ^^^^^^^^^^^^^^^^^^^^
  File "/home/user/whisper/.venv/lib/python3.12/site-packages/evaluate/loading.py", line 751, in load
    evaluation_cls = import_main_class(evaluation_module.module_path)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/whisper/.venv/lib/python3.12/site-packages/evaluate/loading.py", line 76, in import_main_class
    module = importlib.import_module(module_path)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/importlib/__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 995, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/home/user/.cache/huggingface/modules/evaluate_modules/metrics/evaluate-metric--wer/85bee9e4216a78bb09b2d0d500f6af5c23da58f9210e661add540f5df6630fcd/wer.py", line 17, in <module>
    from jiwer import compute_measures
ImportError: cannot import name 'compute_measures' from 'jiwer' (/home/user/whisper/.venv/lib/python3.12/site-packages/jiwer/__init__.py)
```

Related PRs: #661, #683 

In short, a quick-solution for anyone that gets the same error is to install an older version of jiwer, e.g. jiwer < 4.0.0
Repository: huggingface/evaluate
Issue: Questions around perplexity metric for language modeling
Root cause: Algorithmic Error

Hi there,

I've been fiddling with perplexity computation in the past few days, and I was struck by a few discrepancies:
- In [your manual implementation](https://huggingface.co/docs/transformers/en/perplexity), I read:
   
  > When we run the above with `stride = 1024`, i.e. no overlap, the resulting PPL is `19.44`, which is about the same as the `19.93` reported in the GPT-2 paper (note: [p. 5](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf)).
- On [the reference page](https://huggingface.co/spaces/evaluate-metric/perplexity) (see 'Examples'):
   
  ```python
  perplexity = evaluate.load("perplexity", module_type="metric")
  input_texts = datasets.load_dataset("wikitext",
                                      "wikitext-2-raw-v1",
                                      split="test")["text"][:50]
  input_texts = [s for s in input_texts if s!='']
  results = perplexity.compute(model_id='gpt2',
                               predictions=input_texts)
  print(list(results.keys()))
  >>>['perplexities', 'mean_perplexity']
  print(round(results["mean_perplexity"], 2))
  >>>576.76
  print(round(results["perplexities"][0], 2))
  >>>889.28
  ```
  (Of course this is run on a very short sequence, only 50 characters!)


[Here's a Colab notebook](https://colab.research.google.com/drive/1c8OWBeFRl0178UqmVFKioTeDuZP2-lt0?usp=sharing) with my experiments.

### Question 1

Even when using `evaluate`, I get fairly different results: just for the 50 character selection, I obtain:
```
320.85
567.91
```

Something in the implementation changed since the docs were written? What am I missing?

### Question 2

There is still a humongous difference between the manual computation (first page I quoted above, ported in the notebook, and similar results when experimenting with a faster, batched implementation using `stride` in the tokenizer) and what the metric produces. What I'm expecting for `wikitext` is something of the order of 20-30, and lower for bigger models, not in the hundreds, let alone in the thousands (see the notebook, result when joining all rows with `\n\n`)... (At first I thought it was because of `pad_tokens` being counted, that was wrong.) I'm now at a loss as to where that might come from. Any ideas?

### Question 3

Also, notice that I can pass the dataset either as a list of strings of varying lengths, or as one long string, and it runs, but the [perplexity class itself](https://github.com/huggingface/evaluate/blob/main/metrics/perplexity/perplexity.py), on its own, does not seem to handle that (no `return_overflowing_tokens = True` passed to the tokenizer, for instance, I performed a few tests with the class only): I'm guessing the dataset is processed somewhere before being fed to it? I'm in the process of understanding the whole pipeline of invocation of the metric, but so far I haven't found where it happens.

Apologies if the first version of this issue had quite a few silly mistakes, and thanks for reading!
Repository: huggingface/evaluate
Issue: Add `zero_division` argument to F1
Root cause: Unimplemented Feature Gap

If you compute F1 in a case where a class is neither predicted nor present in the ground truth data, a warning refers to using `zero_division`:
```python
import evaluate
f1 = evaluate.load("f1")
print(f1.compute(predictions=[0,0,0,0,0], references=[0,1,0,1,2], average=None, labels=[0,1,2,3]))
```

Output:
```
{'f1': array([0.57142857, 0.        , 0.        , 0.        ])}
[...\.venv\Lib\site-packages\sklearn\metrics\_classification.py:1731](file:///.../.venv/Lib/site-packages/sklearn/metrics/_classification.py:1731): UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 in labels with no true nor predicted samples. Use `zero_division` parameter to control this behavior.
  _warn_prf(average, modifier, f"{metric.capitalize()} is", result.shape[0])
```

But this argument is not present:
```python
import evaluate, numpy as np
f1 = evaluate.load("f1")
print(f1.compute(predictions=[0,0,0,0,0], references=[0,1,0,1,2], average=None, labels=[0,1,2,3], zero_division=np.nan))
```

Output:
```
TypeError                                 Traceback (most recent call last)
Cell In[4], [line 3](vscode-notebook-cell:?execution_count=4&line=3)
      1 import evaluate, numpy as np
      2 f1 = evaluate.load("f1")
----> [3](vscode-notebook-cell:?execution_count=4&line=3) print(f1.compute(predictions=[0,0,0,0,0], references=[0,1,0,1,2], average=None, labels=[0,1,2,3], zero_division=np.nan))

File ...\.venv\Lib\site-packages\evaluate\module.py:467, in EvaluationModule.compute(self, predictions, references, **kwargs)
    465 inputs = {input_name: self.data[input_name][:] for input_name in self._feature_names()}
    466 with temp_seed(self.seed):
--> [467](file:///.../.venv/Lib/site-packages/evaluate/module.py:467)     output = self._compute(**inputs, **compute_kwargs)
    469 if self.buf_writer is not None:
    470     self.buf_writer = None

TypeError: F1._compute() got an unexpected keyword argument 'zero_division'
```