zhiminy/EvalEng huggingface dataset:
Repository: huggingface/evaluate
Issue: does rouge_score = evaluate.load("rouge") support other languages?
Root cause: Documentation Deficiency

Hi all, 
I am doing summarization tasks using rouge score which I found in https://huggingface.co/learn/nlp-course/chapter7/5?fw=pt :
```
!pip install rouge_score

import evaluate

rouge_score = evaluate.load("rouge")
scores = rouge_score.compute(
    predictions=[generated_summary], references=[reference_summary]
)
```
But the scores for some languages are even 0 like Marathi or Bengali.
Does it support languages other than English?
if not, is there any method that I could use?

Thanks!


I have the same question for Chinese evaluation too #465 . But after scanning the documents of [`rouge-score`,](https://github.com/google-research/google-research/tree/master/rouge) I am afraid that  I have to turn to another implementation for Chinese. As there are many packages for Chinese, I suggest you find another package for the computation of Marathi or Bengali. And then just replace `rouge_score.compute()` with your implementation.
Repository: huggingface/evaluate
Issue: Rouge evaluator produced `0` score for the same prediction and reference in Chinese.
Root cause: Algorithmic Error

I am using rouge evaluator for assessing my translation outputs, but the score is not correct as I know.
An example is like this:
<img width="1319" alt="image" src="https://github.com/huggingface/evaluate/assets/16497652/24eefb75-1125-4e5a-a64d-0f7ee3a9da9e">
both the `prediction` and `reference` are `length=9` strings. They are totally the same, but the `rouge-score` is all **zero``, while the correct one should be `{'rouge1': 1.0, 'rouge2': 0.0, 'rougeL': 1.0, 'rougeLsum': 1.0}`.

hhhh，mark！
Repository: huggingface/evaluate
Issue: Compute BLEU score of a Pandas DataFrame with valid rows filtered
Root cause: Validation Gap

I have a Pandas DataFrame from an Excel file, which contains text data which need to calculate the BLEU score row-by-row.

```
import evaluate
import pandas as pd
sacrebleu = evaluate.load("sacrebleu")

testset = pd.read_excel(xlsx_filename)
# find out valid rows with all columns are valid
valid_rows = testset['col1'].notna() & testset['col2'].notna() & testset['col3'].notna()

for i in range(len(testset)): # or... for i in range(len(testset.loc[valid_rows, 'col2']))
    score = sacrebleu.compute(predictions=[testset.loc[valid_rows, 'col1'][i], testset.loc[valid_rows, 'col2'][i]], references=[testset.loc[valid_rows, 'col3'][i]])
```

It raises `KeyError: 139`.

The length of `valid_rows` and `testset` are 13700, while the length of `testset.loc[valid_rows, 'col2']` is 12208.

I know loop through for-loop is an anti-pattern, but how can I fit a Series into the `sacrebleu.compute()` function? It accepts only `[string, string], string` as input.

How can I solve this problem?

---
This question is also at: https://stackoverflow.com/questions/76581453/compute-bleu-score-of-a-pandas-dataframe-with-valid-rows-filtered
Repository: huggingface/evaluate
Issue: Mahalanobis distance computes X_minus_mu incorrectly
Root cause: Algorithmic Error

It seems that on [Line 91](https://github.com/huggingface/evaluate/blob/af3c30561d840b83e54fc5f7150ea58046d6af69/metrics/mahalanobis/mahalanobis.py#L91) in `mahalanobis.py` the quantity `X_minus_mu` seems to be computed incorrectly.

Based on [Wikipedia](https://en.wikipedia.org/wiki/Mahalanobis_distance), both `x` and `mu` should be vectors:

![image](https://github.com/huggingface/evaluate/assets/10352674/f20bf967-b311-4ae7-9279-981def488ca2)

however using `np.mean(...)` without specifying the axis returns a scalar:

```
import numpy as np 
reference_distribution = [[1,2], [3,4]]
print(np.mean(reference_distribution))

> 2.5
``` 

Instead, if the array is `(N, D)` then we can take an overage over the first dimension with `np.mean(..., axis=0)`:

```
import numpy as np 
reference_distribution = [[1,2], [3,4]]
print(np.mean(reference_distribution, axis=0))

> [2. 3.]
```

This means that the same scalar is being subtracted from all components of `X` to create `X_minus_mu`, which isn't what we want, right? 

Please correct me if my understanding is wrong. Thanks!
Repository: huggingface/evaluate
Issue: [BUG] Loading metrics is extremely slow
Root cause: Algorithmic Error

```
In [7]: %time evaluate.load('f1')
Using the latest cached version of the module from /opt/huggingface/modules/evaluate_modules/metrics/evaluate-metric--f1/0ca73f6cf92ef5a268320c697f7b940d1030f8471714bffdb6856c641b818974 (last modified on Mon Mar 27 21:11:41 2023) since it couldn't be found locally at evaluate-metric--f1, or remotely on the Hugging Face Hub.
CPU times: user 44.1 ms, sys: 0 ns, total: 44.1 ms
Wall time: 1.8 s
```

This happens every time I load any module, and not just the first time.

------

```
In [11]: evaluate.__version__
Out[11]: '0.4.0'
```

Related issue:
- https://github.com/huggingface/evaluate/issues/315#issuecomment-1431204163

This workaround speeds up the loading to around 10ms:

We need to do this before importing `evaluate`.
```
import os

os.environ['HF_EVALUATE_OFFLINE'] = '1'
```


Yes, I set os.environ['HF_EVALUATE_OFFLINE'] = '1' but loading meteor is still quite slow.
Seems it takes long to verify the nltk-data is up-to-date

Note that my workaround needs the metrics to be cached locally, so the first time one runs the script, `HF_EVALUATE_OFFLINE` should be `0`. This sucks.

I have this same issue with ` evaluate.load` it takes a long time even after setting `HF_EVALUATE_OFFLINE` to 1! Any suggestions?

I refactored my code to no longer use `evaluate`. It was easy enough to compute the metrics. It's so badly designed that it's not worth using it.

Thank you for the quick reply @NightMachinery ! 

I ended up using this workaround for now:
https://github.com/huggingface/evaluate/issues/315#issuecomment-1925027335

`git clone https://github.com/huggingface/evaluate.git`

```
from evaluate import load
metric = load('/local/path/to/evaluate/metrics/accuracy/accuracy.py')
```




Why evaluate so crappy and slow........ ?

#511 #674 #387
Repository: huggingface/evaluate
Issue: evaluate.load loads indefinitely offline
Root cause: Resource Mishandling

The following script loads indefinitely:

```python
import os
os.environ["HF_EVALUATE_OFFLINE"] = "1"

from evaluate import load

metric = load('./glue/', 'qqp')
```

A KeyboardInterrupt yields the following:

```bash
Traceback (most recent call last):
  File "/home/gridsan/vrichard/bertSignatures/data_download.py", line 7, in <module>
    metric = load('./glue/', 'qqp')
  File "/home/gridsan/vrichard/.local/lib/python3.9/site-packages/evaluate/loading.py", line 731, in load
    evaluation_module = evaluation_module_factory(
  File "/home/gridsan/vrichard/.local/lib/python3.9/site-packages/evaluate/loading.py", line 622, in evaluation_module_factory
    return LocalEvaluationModuleFactory(
  File "/home/gridsan/vrichard/.local/lib/python3.9/site-packages/evaluate/loading.py", line 429, in get_module
    module_path, hash = _create_importable_file(
  File "/home/gridsan/vrichard/.local/lib/python3.9/site-packages/evaluate/loading.py", line 373, in _create_importable_file
    importable_local_file = _copy_script_and_other_resources_in_importable_dir(
  File "/home/gridsan/vrichard/.local/lib/python3.9/site-packages/evaluate/loading.py", line 305, in _copy_script_and_other_resources_in_importable_dir
    with FileLock(lock_path):
  File "/home/gridsan/vrichard/.local/lib/python3.9/site-packages/datasets/utils/filelock.py", line 320, in __enter__
    self.acquire()
  File "/home/gridsan/vrichard/.local/lib/python3.9/site-packages/datasets/utils/filelock.py", line 282, in acquire
    time.sleep(poll_intervall)
KeyboardInterrupt
```

The `glue` repo has been downloaded from `evaluate/metrics/glue/`. 

**versions**:
```python
Python 3.9.16 | packaged by conda-forge | (main, Feb  1 2023, 21:39:03) 
[GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import evaluate
>>> evaluate.__version__
'0.4.0'
```

Did you find a fix for this? I am having the same error.
Repository: huggingface/evaluate
Issue: FileNotFoundError: Couldn't find a module script at /data1/yangyan/code/mms_finetune/wer/wer.py. Module 'wer' doesn't exist on the Hugging Face Hub either
Root cause: Environment Incompatibility

when I runing the code below:

`wer_metric = evaluate.load("wer")`

i got the error as follow:
Traceback (most recent call last):
  File "mms_finetune.py", line 134, in <module>
    wer_metric = load("wer")
  File "/root/miniconda3/envs/wenetliubq/lib/python3.8/site-packages/evaluate/loading.py", line 747, in load
    evaluation_module = evaluation_module_factory(
  File "/root/miniconda3/envs/wenetliubq/lib/python3.8/site-packages/evaluate/loading.py", line 680, in evaluation_module_factory
    raise FileNotFoundError(
FileNotFoundError: Couldn't find a module script at /data1/yangyan/code/mms_finetune/wer/wer.py. Module 'wer' doesn't exist on the Hugging Face Hub either.

the version of evaluate is evaluate 0.4.1.dev0


I just got the same error

Does the machine have access to the internet?

The issue was resolved after the firewall was turned off.

Agree with it! For me, it is because that huggingface.co is not accessible without Internet connection.
Repository: huggingface/evaluate
Issue: Change package name from "evaluate" to something less generic

I'm repeatedly finding myself in situations where I want to have a package called `datasets.py` or `evaluate.py` in my code and can't because those names are being taken up by Huggingface packages. While I can understand how (even from the user's perspective) it's aesthetically pleasing to have nice terse library names, ultimately a library hogging simple names like this is something I find short-sighted, impractical and at my most irritable, frankly rude.

My preference would be a pattern like what you get with all the other big libraries like numpy or pandas:

```
import huggingface as hf
# hf.transformers, hf.datasets, hf.evaluate
```

or things like

```
import huggingface.transformers as tf
# tf.load_model(), etc
```

If this isn't possible for some technical reason, at least just call the packages something like `hf_transformers` and so on.

I realize this is a very big change that's probably been discussed internally already, but I'm making this issue and sister issues on each huggingface project just to start the conversation and begin tracking community feeling on the matter, since I suspect I'm not the only one who feels like this.

Sorry if this has been requested already on this issue tracker, I couldn't find anything looking for terms like "package name".

Sister issues:
- [transformers](https://github.com/huggingface/transformers/issues/24934)
- [datasets](https://github.com/huggingface/datasets/issues/6053)
- **evaluate**
Repository: huggingface/evaluate
Issue: [Metrics] Can't find / acquire lock files in distributed multi-node shared file system
Root cause: Resource Mishandling

Hello,

I'm facing an issue when calling `.compute` in distributed multi-node setting.
The symptoms are the same as in huggingface/datasets#4420 , however I'm not sure the cause is the same (the code modification did not solve the issue, but I didn't try lockf)

### Environment
* evaluate v0.4.0
* PyTorch v2.0.1
* 2 nodes of 8 GPUs each (16 processes), with shared file system
* script called with `torchrun` from SLURM

I get the following errors:

For processes on the first node (0-7, here **4** is reported):
```
[node1:4]:ValueError: Couldn't acquire lock on /.hf_cache/metrics/accuracy/default/exp-16-rdv.lock from process 4.
```

For processes on the second node (8-15, here **15** is reported):
```
[node2:7]:ValueError: Expected to find locked file /.hf_cache/metrics/accuracy/default/exp-16-0.arrow.lock from process 15 but it doesn't exist.
```

All metrics are loaded with the same `experiment_id`, and with the correct `num_process` and `process_id` arguments.
And of course all the files are present in the cache directory.

I really don't know how to further debug / solve this bug. Do you have any clue ?
It working properly in distributed mono-node setting.

cc @lhoestq 

Any news respect to this issue? Experimenting same behaviour during multinode training with a shared filesystem.

Toni

The lock mechanism is based on the [filelock](https://github.com/tox-dev/filelock) package. Right now `evaluate` requires a filesystem compatible with `filelock`. But feel free to ask `filelock` authors and community if there's a way to make it work on your filesystem.

Hi, does anyone have any active workaround for this issue? This still seems to be happening in a multinode distributed setting inspite of the latest releases and the workarounds mentioned [here](https://github.com/huggingface/evaluate/issues/542) :( 

Hi! Are there any fixes for this? Currently experiencing it with the multinode slurm example of accelerate...
Repository: huggingface/evaluate
Issue: evaluate.load("wer") is returning Filenotfound error
Root cause: Environment Incompatibility

Hi, I am working in a Kaggle kernel. I have installed and imported evaluate 0.4.0. I tried to call `evaluate.load("wer")`. In response, I am getting `FileNotFoundError: Couldn't find a module script at /kaggle/working/wer/wer.py. Module 'wer' doesn't exist on the Hugging Face Hub either`. Can someone suggest?

```
import numpy as np 
import evaluate
evaluate.__version__
'0.4.0'
```

same error for loading the 'roc_auc'

I also have this error moving from datasets.load_metric to using evaluate.load("cer"). I tried (unsuccessfully) to resolve it by downloading the following:

https://github.com/huggingface/datasets/blob/main/metrics/cer/cer.py

and placing it in /project-root/cer/cer.py, but this led to the following error: 

class CERConfig(evaluate.info.Config):
AttributeError: module 'evaluate.info' has no attribute 'Config'. Did you mean: 'config'?

Sorry, I used this cer file:

https://huggingface.co/spaces/evaluate-metric/cer/blob/af39b3b7914ffb126ece472884f75033cfc4727b/cer.py
Repository: huggingface/evaluate
Issue: Evaluate a quantize model
Root cause: Validation Gap

I am trying to run an evaluation in a model that is quantized. I have to instantiate it using 'accelerate' to use the GPU, because otherwise it cannot fit in memory. The problem is that when using compute() in the metric I want, I get this error:

`ValueError: The model has been loaded with `accelerate` and therefore cannot be moved to a specific device. Please discard the `device` argument when creating your pipeline object.`

Is there any way to allow using the compute() method in a model that is in GPU?

I'm also having the same problem... @Psancs05 were you able to find a workaround for this?

yeah me too what's the solution for it
Repository: huggingface/evaluate
Issue: perplexity.compute - Add args to pass model and tokenizer objects instead of `model_id`
Root cause: Unimplemented Feature Gap

Hello,
I am new to the project, so I hope my request is a sensible one.

It looks like when we use the compute method for Perplexity, we can only specify a model through its `model_id`, which is used also to load the tokenizer.

Is there any particular reason why it is not possible to pass as arguments to the method the model object and the tokenizer object themselves instead of the `model_id`? I think that could help, for example when the tokenizer has some user-added tokens or when the model to be used is fine-tuned (e.g., currently, a PeftModel could not be used, but passing the model object itself that should not be a limitation anymore).

Thanks for your answer and for your amazing work.

> Hello,
> I am new to the project, so I hope my request is a sensible one.

hello，I am also a new to the project.I'd like to ask if you've implemented an assessment of the perplexity of the local model
Repository: huggingface/evaluate
Issue:  Matthew's Correlation Coefficient (MCC) always zero
Root cause: Documentation Deficiency

Hello,

I am fine-tuning a multiclass audio classifier by using the template script with two GPUs:

https://github.com/huggingface/transformers/blob/main/examples/pytorch/audio-classification/README.md

I have seven imbalanced classes.

The model learns, in a few epochs the loss drops from ca. 2 to ca. 1 and F1 weighted goes from 0.26 to ca. 0.56. 

Accuracy values are initially higher than F1 (it starts at 0.46), but also improve significantly during training.

However,  Matthew's Correlation Coefficient (MCC) stays always at 0.0 during the training.

I have changed the command line to:

    --metric_for_best_model matthews_correlation \

And the script to:

```
metric = evaluate.load("matthews_correlation") 

...

    def compute_metrics(eval_pred):
        """Computes accuracy on a batch of predictions"""
        predictions = np.argmax(eval_pred.predictions, axis=1)
        return metric.compute(predictions=predictions, references=eval_pred.label_ids, average='macro')

```

The average argument makes no difference, I have tried both macro and none (as well as removing it).

Am I doing anything wrong, is this behavior to be expected or is it a bug?

Best,

Ed


 It was not zero. It was a problem with rounding. When there is only one metric, the trainer seems to report one decimal number (0.0), however, when I load several metrics and choose MCC as the one for evaluation, the trainer reports several decimal places for each metric.
Repository: huggingface/evaluate
Issue: Feature request for metric result keys in `MetricInfo`
Root cause: Unimplemented Feature Gap

## Problem

If a user would like to automate the retrieval of metrics, she needs to know the result key at least or possible keys. For example, suppose I have an `EvaluateSuite` that computes "accuracy" then "precision". If I have a procedure downstream that needs to iterate and retrieve results, there does not seem to be a programmatic way to guarantee the procedure can retrieve the results for each metric without hardcoding. Often, result keys are the same at the metric name and some metrics have multiple result keys (i.e. glue returns "f1" with "accuracy" on text-classification tasks).


## Possible Implementation

Add `result_features` attribute to `MetricInfo`, then a procedure can expect metric to return feature when key in `result_features` is not `None`.

```python
evaluate.MetricInfo(
    description=_DESCRIPTION,
    citation=_CITATION,
    inputs_description=_KWARGS_DESCRIPTION,
    features=datasets.Features(
        {
            "predictions": datasets.Value("int64" if self.config_name != "stsb" else "float32"),
            "references": datasets.Value("int64" if self.config_name != "stsb" else "float32"),
        }
    ),
  result_features=datasets.Features(
          {
              "accuracy": datasets.Value("float32") if self.config_name  ["sst2", "mnli", "mnli_mismatched", 
                  "mnli_matched", "qnli", "rte", "wnli", "hans"] else None,
              "f1": datasets.Value("float32") if self.config_name  ["mrpc", "qqp"] else None,
              "pearson": datasets.Value("float32")  if self.config_name == "cola" else None,
              "spearmanr": datasets.Value("float32") if self.config_name  == "stsb" else None,
          }
      ),
    codebase_urls=[],
    reference_urls=[],
    format="numpy",
)
```
Repository: huggingface/evaluate
Issue: is it possible to evaluate a proprietary model without uploading it to HF
Root cause: Documentation Deficiency

I really like HF 'evaluate'.
However, if my customer is not allowed to publicly share his/her model, how can they still use HF 'evaluate' module? 
In the code, I see 'model_id' always referencing a model from a public HF model hub (eg. "lvwerra/distilbert-imdb").
Is there a way to implement a Bring-Your-Own-Model capability?

When you use `evaluate` the model remains local - you don't need to upload it! Most modules just take the predictions that you can generate wherever you like!

Hello! 

I am hoping to compute perplexity of my locally trained/updated GPT2 model. 
I have my model saved as a pytorch_model.bin. 

How can I provide path to my local model in perplexity.compute? 
`perplexity = load("perplexity", module_type="metric")`
`perplexity.compute(predictions=texts, model_id='gpt2')['perplexities']`

I looked into [this](https://huggingface.co/spaces/evaluate-metric/perplexity/commit/95d16d913d8245780a2f7e4e1ec0ecd5b5358d00#d2h-881643) commit. 
The commit installation doesn't seem to work with
`pip install git+https://github.com/huggingface/evaluate@95d16d913d8245780a2f7e4e1ec0ecd5b5358d00`

does the following work?
```python
perplexity.compute(predictions=texts, model_id='PATH_TO_MY_MODEL')
```

> does the following work?
> 
> ```python
> perplexity.compute(predictions=texts, model_id='PATH_TO_MY_MODEL')
> ```

maybe not work. I tested your idea, but i got a error
"huggingface_hub.utils.validators.HFValidationError: Repo id must use alphanumeric chars or '-', '', '.', '--' and '..' are forbidden, '-' and '.' cannot start or end the name, max length is 96: 'E:\llama2.c\model_hf\pytorch_model.bin'."

> 你好！
> 
> 我希望计算本地训练/更新的 GPT2 模型的困惑度。 我将模型保存为 pytorch_model.bin。
> 
> 如何在 perplexity.compute 中提供本地模型的路径？ `perplexity = load("perplexity", module_type="metric")` `perplexity.compute(predictions=texts, model_id='gpt2')['perplexities']`
> 
> 我调查了[这个](https://huggingface.co/spaces/evaluate-metric/perplexity/commit/95d16d913d8245780a2f7e4e1ec0ecd5b5358d00#d2h-881643)提交。 提交安装似乎不适用于 `pip install git+https://github.com/huggingface/evaluate@95d16d913d8245780a2f7e4e1ec0ecd5b5358d00`

Hello please have you solved this problem?

> does the following work?
> 
> ```python
> perplexity.compute(predictions=texts, model_id='PATH_TO_MY_MODEL')
> ```

This works, thanks! 
I was previously trying to provide path to a model.pt saved with torch. 
When I provide path to the directory containing model saved with trl's native saving functions, e.g., trainer._save_pretrained(logging_dir) or trainer.save_model(logging_dir), it works. 

> > 以下有效吗？
> > ```python
> > perplexity.compute(predictions=texts, model_id='PATH_TO_MY_MODEL')
> > ```
> 
> 这有效，谢谢！ 我之前尝试提供用 torch 保存的 model.pt 的路径。 当我提供包含使用 trl 的本机保存函数保存的模型的目录路径时，例如 trainer._save_pretrained(logging_dir) 或 trainer.save_model(logging_dir)，它可以工作。

Please post a simple demo of the explanation？I'm a rookie about evaluate, thank you very much !
Repository: huggingface/evaluate
Issue: CER.py error in class CERConfig
Root cause: External Dependency Breakage

This is related to https://github.com/huggingface/evaluate/issues/486, but am posting separately, as it is an error specific to the cer.py file I am using.

Is there a more current cer.py file I should be using to move from datasets.load_metric("cer") to evaluate.load("cer")?

FILE:
https://huggingface.co/spaces/evaluate-metric/cer/blob/af39b3b7914ffb126ece472884f75033cfc4727b/cer.py

MY FILE LOCATION:
Placed it in /project-root/cer/cer.py
Google Colab

During evaluate phase of fine-tuning, ran into this error:

ERROR:
class CERConfig(evaluate.info.Config):
AttributeError: module 'evaluate.info' has no attribute 'Config'. Did you mean: 'config'?

Thanks!

Here is a screenshot.
![colab-20230909-hf-cer-evaluate-error](https://github.com/huggingface/evaluate/assets/141673306/6af3c8f1-80cc-46a0-891b-7fb1fa37a51f)