zhiminy/EvalEng huggingface dataset:
Repository: huggingface/evaluate
Issue: PyTorch models predictions varies with the same data input
Root cause: Algorithmic Error

Starting by loading a pretrained model (I tried two here) and tokenizer:
```py
from transformers import AutoModelForSequenceClassification, AutoTokenizer
import pandas as pd
import re
import torch

model_name = "Souvikcmsa/SentimentAnalysisDistillBERT"
#model_name = "Souvikcmsa/BERT_sentiment_analysis" --> Same issue found with a different model! 
model = AutoModelForSequenceClassification.from_pretrained(model_name, use_auth_token=True)
tokenizer = AutoTokenizer.from_pretrained(model_name, use_auth_token=True)
```

then, loading some txt data:
```py
# Read the data
url = 'https://raw.githubusercontent.com/Giskard-AI/examples/main/datasets/twitter_us_airline_sentiment_analysis.csv'
data = pd.read_csv(url)
defining a basic preprocessor:

# Preprocess text (username and link placeholders)
# Replace the Username with @user and the URL in the tweet with http for better comprehension of data for the model
def preprocess(text):
    text = " ".join(text.split())
    text = re.sub(r'http\S+', 'http', text) 
    text = re.sub(r'@\S+', '@user', text)
    text = text.lower()
    return text
```
taking two subset from the data (first 5 and 100 entries):
```py
torch.set_printoptions(precision=10)

for param in model.base_model.parameters():
    param.requires_grad = False

# ----- 1. Preprocess data -----#
# Preprocess data
X = list(data["text"].apply(preprocess))

X_tokenized = tokenizer(X, padding=True, return_tensors="pt")

num_subsample1=5
X_tokenized_subset1={}
for key in X_tokenized.keys():
    X_tokenized_subset1[key]=X_tokenized[key][:num_subsample1]
    
num_subsample2=100
X_tokenized_subset2={}
for key in X_tokenized.keys():
    X_tokenized_subset2[key]=X_tokenized[key][:num_subsample2]
the output of the model on the first subset’s first entry:

display_index=0
outputs1 = model(**X_tokenized_subset1)
outputs1[0][display_index]
gives:

tensor([-1.6196994781,  3.0899136066, -1.3701400757],
       grad_fn=<SelectBackward0>)
```
while the output of the model on the second subset’s first entry (same entry effectively) is:
```py
outputs2 = model(**X_tokenized_subset2)
outputs2[0][display_index]
```
gives:
```
tensor([-1.6196994781,  3.0899133682, -1.3701400757],
       grad_fn=<SelectBackward0>)
```
Although they should be the same, there’s a difference in the second prediction:
```py
outputs2[0][display_index]-outputs1[0][display_index]
```
which gives:
```
tensor([ 0.0000000000e+00, -2.3841857910e-07,  0.0000000000e+00],
       grad_fn=<SubBackward0>)
```
Any insights? Thanks 

This seems to be a question more for `transformers` than `evaluate` :) That said, it seems the difference is very small so it could be a numerical issue. Also have you tried settings `model.eval()` such that it is in evaluation mode and not `train()` model where random dropout is used?
Repository: huggingface/evaluate
Issue: import evaluate
Root cause: Environment Incompatibility

hi，when I try to `import evaluate`， error happens:
![image](https://user-images.githubusercontent.com/87848800/203212202-90b51a69-628d-4f3f-aa4b-5acd3c5e8049.png)
I use python 3.8.14，transformers 4.8.1.
But when I run this code on another virtual environment with python 3.9 , I can import evaluate correctly.
Could you give me some advice?

My evaluate version is 0.3.0

Seems like an issue with your environment and TensorFlow. Maybe this helps: 
https://stackoverflow.com/questions/41890549/tensorflow-cannot-open-libcuda-so-1
https://forums.developer.nvidia.com/t/cannot-find-libcuda-so-1/121260
Repository: huggingface/evaluate
Issue: Offline-Usage of Evaluate
Root cause: Unimplemented Feature Gap

Hello everybody, 

unfortunately I don´t have internet connectivity on the server on which I´d like to use the evaluate-package. I can install the package via pip, but for every basic action such as evaluate.load() the package tries to open a socket to download the needed metrics.
For the predecessor seqeval I found solutions as to download the metrics themselves on the local client, and then upload those needed metrics to the server for seqeval to use them. 
Is there a comparable solution for the evaluate-package as well?

Thank you very much!

Best regards,
Marcel

Just saw many other similar issues, closing it as a duplicate. :)
Repository: huggingface/evaluate
Issue: SARI score gives different results than the paper
Root cause: Documentation Deficiency

Hello,
In your SARI score [README file](https://github.com/huggingface/evaluate/blob/main/metrics/sari/README.md) you use the same example as in the [original paper](https://aclanthology.org/Q16-1029.pdf).

In particular, the following code you suggest
```python3
from evaluate import load
sari = load("sari")
sources=["About 95 species are currently accepted."]
predictions=["About 95 you now get in."]
references=[["About 95 species are currently known.","About 95 species are now accepted.","95 species are now accepted."]]
sari_score = sari.compute(sources=sources, predictions=predictions, references=references)
```
returns a sari score of `26.953601953601954`, whereas the exact same example in the paper returns a score of `26.83`. While these two numbers are close enough, I have noticed that this is more of a coincidence, as for other examples the difference can be much higher. For example, if we replace the prediction line with 
```python3
predictions=["About 95 species are now agreed."]
```
this gives a score of `61.709656084656075` according to this repository's implementation, whereas the paper reports a score of `75.94`.

Could you please clarify why there is this discrepancy, and explain, if applicable, the reason(s) for the choice of using an implementation which gives different results from the original paper?
Many thanks

Tagging @ddhruvkr as they originally implemented SARI into `datasets` before it was migrated to `evaluate`.

Hi @ddhruvkr any update on this?

Hi, this implementation is adapted from [Tensorflow’s tensor2tensor implementation](https://github.com/tensorflow/tensor2tensor/blob/master/tensor2tensor/utils/sari_hook.py). It has two differences with the [original GitHub implementation](https://github.com/cocoxu/simplification/blob/master/SARI.py):

1) It defines 0/0=1 instead of 0 to give higher scores for predictions that match a target exactly.
2) It fixes an [alleged bug](https://github.com/cocoxu/simplification/issues/6) in the keep score computation.

This is also mentioned in the [original documentation](https://huggingface.co/spaces/evaluate-metric/sari). This is likely the reason for the difference.



Hi @ddhruvkr 
Many thanks for getting back to me. I'm closing the issue as a result.
Repository: huggingface/evaluate
Issue: [Feature request] Add evaluator for audio classification
Root cause: Unimplemented Feature Gap

We have a few hundred audio classification models from `transformers` [on the Hub](https://huggingface.co/models?library=transformers&pipeline_tag=audio-classification&sort=downloads) and it would be nice if one could easily evaluate them for accuracy, F1 score etc with an evaluator.

The pipeline in question is `audio-classification` and the relevant docs are [here](https://huggingface.co/docs/transformers/v4.25.1/en/main_classes/pipelines#transformers.AudioClassificationPipeline).

A good model/dataset pair to test is this one for keyword spotting (ks):

* `superb/wav2vec2-base-superb-ks`: https://huggingface.co/superb/wav2vec2-base-superb-ks
* The `ks` config of `superb`: https://huggingface.co/datasets/superb

Together with #324, this evaluator would bring coverage to all audio `transformers` models on the Hub 🔥 

I would like to work on this feature, if it's not already being implemented. 

I don't think so, so feel free to go ahead :)
Repository: huggingface/evaluate
Issue: Refactor for loading multiple evaluation categories
Root cause: Unimplemented Feature Gap

In addition to `Metric` we also want to add other types of evaluations such as `Comparison` (#34) and `Measurement` (#35) following the internal discussion (https://huggingface.slack.com/archives/C035S5G2J3D/p1652200198598789). Technically, these all behave the same way as they take some inputs and compute a scores. As such they could largely be one class (essentially what `Metric` is today). That means we could also load in the same fashion:

```
import evaluate

metric = evaluate.load("accuracy")
comparison = evaluate.load("mcnemar")
measure = evaluate.load("npmi")
```

While each type can live in a different folder on the repository this can cause name clashes when a name can be used for two methods (e.g. `perplexity` can be a metric and a measurement). This could be solved with an additional argument for like `load("perplexity", type="metric")` that resolves those conflicts. I think this would be fine.

However, there is a second conflict with Spaces: since each metric (or comparison/measurement) would have their own space with widget it is not so easy to resolve the conflicts here, unless we create an org for each type of metric: e.g. `evaluate-metrics`, `evaluate-comparisons`, `evaluate-measurements`. Then each evaluation type is pushed to a separate org. 

If that solution sounds good then we could implement the following behaviour:
- if no `type` is provided we cycle through metric/comparison/measurement and return the first result
- if a `type` is provided we only look for that one and raise an error if that type does not exist

What do you think @douwekiela @lhoestq?

That sounds good to me!

I like the idea ! thanks :)

Your points regarding conflicts sound good to me as well !
Repository: huggingface/evaluate
Issue: Multiple concurrent scripts using the same file system causes FileNotFoundError
Root cause: Resource Mishandling

When I run multiple scripts using the same `evaluate` metric concurrently, I sporadically get the following error, which looks like a race condition to me.
```bash
  File "<anonymized>/run_ner.py", line 531, in compute_metricsformers/trainer.py", line 3081, in evaluation_loop_evaluate
    metrics = self.compute_metrics(EvalPrediction(predictions=all_preds, label_ids=all_labels)) 
  File "<anonymized>/run_ner.py", line 531, in compute_metricsformers/trainer.py", line 3081, in evaluation_loop_evaluate
    results = metric.compute(predictions=true_predictions, references=true_labels) 
  File "<anonymized>python3.10/site-packages/evaluate/module.py", line 458, in compute 
    os.remove(file_path) 
FileNotFoundError: [Errno 2] No such file or directory: '/<anonymized>/.cache/huggingface/metrics/seqeval/default/default_experiment-1-0.arrow'
```



You can solve this by setting `load(..., keep_in_memory=True)` to simply skip all caching (which does not seem to be a problem when you just want a function to compute some score) or presumably also `load(..., experiment_id=<unique_id>)` to solve the conflict of multiple scripts writing/reading the same cache.

What is not entirely clear to me is why caching is the default behavior? Am I missing a big advantage of the caching system?

It's not exactly caching and more saving the predictions/references on disk until we need them to compute the metric. We are thinking about adding alternative ways to do this in #356 and with #168 you would need almost no memory in the whole process. Thanks for reporting the workaround, i think the `experiment_id` is the way to go!

Hi! Sorry to revive this. I get the same error when using this on multi-node training. Is there a workaround for me to fix this?
Repository: huggingface/evaluate
Issue: Rouge-SU-4 (skipgram)
Root cause: Unimplemented Feature Gap

I was trying to migrate from https://github.com/tagucci/pythonrouge whose ROUGE-SU-4 has the highest variance on my summary evaluations (and so is an important metric in the PCA/eigenvector sense) but you don't provide it. Are there plans for it? Is there a reason why it wasn't or won't be implemented?

The ROUGE implementation in `evaluate` relies on Google's [ROUGE repository](https://github.com/google-research/google-research/tree/master/rouge). Feel free to open a PR if you'd like to add this as a feature.
Repository: huggingface/evaluate
Issue: Issue using `evaluate.push_to_hub` for Whisper sprint.
Root cause: External Dependency Breakage

This issue spans multiple HuggingFace repos, but figured this one was more appropriate. I ran into an issue attempting to submit my Ukrainian and Russian models to the leaderboard for the Whisper spring. I could get the evaluation results from the model, but publishing the results failed.

I was using the script provided here for the sprint (https://github.com/huggingface/community-events/blob/main/whisper-fine-tuning-event/run_eval_whisper_streaming.py) and ran it as `python run_eval_whisper_streaming.py --model_id="mitchelldehaven/whisper-medium-ru" --config="ru" --language="ru" --device=0 --max_eval_samples 16`. However it yielded the following error message:

```
Traceback (most recent call last):
  File "/users/mdehaven/expts/whisper-asr-finetune/recipes/run_eval_whisper_streaming.py", line 163, in <module>
    main(args)
  File "/users/mdehaven/expts/whisper-asr-finetune/recipes/run_eval_whisper_streaming.py", line 89, in main
    evaluate.push_to_hub(
  File "/users/mdehaven/software/miniconda3/envs/whisper/lib/python3.10/site-packages/evaluate/hub.py", line 125, in push_to_hub
    return metadata_update(repo_id=model_id, metadata=metadata, overwrite=overwrite)
  File "/users/mdehaven/software/miniconda3/envs/whisper/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 94, in _inner_fn
    return fn(*args, **kwargs)
  File "/users/mdehaven/software/miniconda3/envs/whisper/lib/python3.10/site-packages/huggingface_hub/repocard.py", line 759, in metadata_update
    model_name, new_results = model_index_to_eval_results(value)
  File "/users/mdehaven/software/miniconda3/envs/whisper/lib/python3.10/site-packages/huggingface_hub/repocard_data.py", line 401, in model_index_to_eval_results
    name = elem["name"]
KeyError: 'name'
```

The error comes from https://github.com/huggingface/huggingface_hub/blob/main/src/huggingface_hub/repocard_data.py#L431, which expects there to be a key `name` in the `model-index` value passed into the function. However, I can't seem to find where this key should be included at. The value of the key `model-index` is created in https://github.com/huggingface/evaluate/blob/main/src/evaluate/hub.py#L82-L117 and I don't see where `name` is being set. I'm assuming there may be something wrong with my model's repo, as I know others have been using this same evaluation setup without issue. 

My current work around is that I modified the function in the `push_to_hub` function in evaluate to take a model name and included it via `metadata = {"model-index": [{"results": [result], "name": model_name}]}` however there seems to be a more fundamental issue here that I am not seeing.

I think this should have been addressed with https://github.com/huggingface/huggingface_hub/pull/1157. What version of `huggingface_hub` are you using?
Repository: huggingface/evaluate
Issue: [Metric Request] Fuzzy reordering and Kendall tau distance
Root cause: Unimplemented Feature Gap

Hello! Looking through this amazing paper and blog-post from a couple years back:
https://arxiv.org/pdf/2109.01396.pdf
https://lena-voita.github.io/posts/nmt_training_through_smt_lens.html
https://www.youtube.com/watch?v=LB6USxmxxEQ

It seems as if tracking Encoder/Decoder model training using BLEU or similar metrics only gets you so far - the "fuzzy reordering score" and "Kendall tau" continue to show improvement when other metrics have stagnated (and indeed there are qualitative improvements to the metrics as well).

The closest thing I could find is for the first one, written in good old Python 2:
https://github.com/google/topdown-btg-preordering/blob/master/evaluate_preordering.py

Both metrics seem relatively straightforward (famous last words!).

A huge thank you to HF and the amazing community that is massively democratizing ML!

If possible, I would like to take up this new metric
Repository: huggingface/evaluate
Issue: How to use bleu offline, or how to find all downloaded files relevant to compute 'bleu'
Root cause: Documentation Deficiency

I get a server which is unable to access github or huggingface, so I can't directly use "evaluate.load('bleu')" on the server. I tried to download the bleu.py in huggingface/evaluate/metrics/bleu repo and upload it to the server.  However, when I used 'evaluate.load('bleu.py')', I found that inside bleu.py it needs to access github again, for which I failed again. 

Then I tried to download all the relevant files to compute bleu. I check the code for evaluate.load and find that it seems all the relevant files are downloaded to ~/.cache/huggingface/metrics/bleu. However, I only found a useless file "default_experiment-1-0.arrow.lock" in  ~/.cache/huggingface/metrics/bleu. 

So I wonder where on earth the files relevant to compute bleu are located. And if possible, could developers add some information about where those metric files are downloaded. Much thanks!

I found that the code that evaluate calls is (here)[https://github.com/tensorflow/nmt/blob/master/nmt/scripts/bleu.py] to point to the downloaded file rather than call github. Tell me how it goes!

> I found that the code that evaluate calls is (here)[https://github.com/tensorflow/nmt/blob/master/nmt/scripts/bleu.py] to point to the downloaded file rather than call github. Tell me how it goes!

Thank you for your kind help. I finally found this and solved the problem by downloading all relavant files locally. And then it works!
Repository: huggingface/evaluate
Issue: I get this  error when I use Example code for huggingface  ,how can i fix it?
Root cause: Interface Contract Mismatch

this is my code:
![image](https://user-images.githubusercontent.com/22388079/209642664-7eba5e6b-6ab8-4874-92f2-b131214c4a2d.png)
the  error is:Expected format: {'predictions': Value(dtype='int32', id=None), 'references': Value(dtype='int32', id=None)},
![image](https://user-images.githubusercontent.com/22388079/209642787-6ee10ea7-2d5a-4a3b-a505-14561003d369.png)

 

Hello,
Can you give us further details on the task you are working on? is it a simple classification task? In that case , if you **flatten** your tensors, you will get the good result.

> 
it's fixed by  flatten tensors.  my task is translation . I use xlm-roberta to fine tune. The way to load the model is XLMRobertaForCausalLM.from_pretrained('xlm-roberta-base'). and  I fix it by flatten tensor "predictions=predictions.view(5120) references=batch["labels"].view(5120)". 

Closing this as it seems resolved.

Could you elaborate on the flattening, @SkanderHellal?  I get the same error for my setup and don't understand why it doesn't work, as training works with the same preparation of the dataset. Where can I read more on this?
Repository: huggingface/evaluate
Issue: Added metric r^2 / r-squared 
Root cause: Documentation Deficiency

Hi @lvwerra I added r-squared  metric. https://github.com/huggingface/evaluate/pull/395

Can you share the reformatter code?

Looks awesome! First, make sure you have `black`,`isort` and `flake8`:
```bash
pip install  "evaluate[dev]"
```

Then you can run:
```bash
make style && make quality
```

Hope this helps!

I think it's ok! can you check ☺️ @lvwerra 

Quality and style pass the test! 👍 

@lvwerra  Didn't pass the rest of the tests I wonder if something went wrong

Let's discuss on the PR directly :)
Repository: huggingface/evaluate
Issue: Feature: add community metrics
Root cause: Unimplemented Feature Gap

We would like a way for community members to easily add custom or new metrics. These custom metrics could be hosted on the hub either in a dedicated repository type (like datasets or models) or on Spaces. 

Spaces comes with a few advantages, especially in short term:
1. No dedicated repo type needed
2. Infrastructure already exists
3. We can easily setup a demo without needing a custom widget (e.g. with Gradio)

@osanseviero made a PoC here: https://huggingface.co/spaces/osanseviero/accuracy_metric

A few things that we should add to make this as user friendly and useful as possible:

- create a template to add a metric space
    - template `README.md` (like model/dataset card) with basic structure
    - template `my_metric.py` file (similar to `datasets` loading scripts) 
    - gradio `app.py` that loads all information from `README.md` and `my_metric.py` to create a default widget. This does probably not need to be changed.
- debugging tools
    - instructions on how to clone the repository directly on spaces
    - maybe an `evaluate-cli` command to test if everything is fine `evaluate-cli test-metric PATH_TO_REPO`
- dedicated tag: if we ever want to migrate the metrics to a dedicated place like datasets and models it is good if we had a tag to find them all.  Similarly, we could then easily do a `list_metrics()` function in the `evaluate` without needing to look at the content of each repo.

The goal is to enable behaviour (example with the space above):
```Python
accuracy_metric = load_metric("osanseviero/accuracy_metric")
```

Also cc @julien-c

maybe just name the file `metric.py`, or embed it directly into the `app.py`

> dedicated tag

Yes, you can just add a `tags: - metric` to the README.md

Thanks, that's a good idea. Maybe we keep `metric.py` and `app.py` separated in case we want to switch the widget or metric logic at some point we don't need to touch/disentangle both.

> Yes, you can just add a `tags: - metric` to the README.md

To clarify: I think there is a small thing left to implement in moon-landing (cc @cbensimon) but then you will be able to query https://hf.co/api/spaces?filter=metric i.e. same as for other repo types

But in the meantime you can already add tags to README.md "space cards"

Agree these should be two separate files. If you want to use a metric on the hub, you should just need to download `app.py` (and maybe something related to dependencies)

Closed with #25 and #14.
Repository: huggingface/evaluate
Issue: Error trying to load accuracy metric
Root cause: Environment Incompatibility

I'm trying to train a GPT-2 model using example scripts and I get an error when the scripts tries to load 'accuracy' metric:

evaluate.load("accuracy")

  File "/home/carlos/miniconda3/lib/python3.10/site-packages/evaluate/loading.py", line 752, in load
    evaluation_instance = evaluation_cls(
TypeError: 'NoneType' object is not callable

I tried to load other metrics and I only get error with accuracy

I've got installed the requirements: 

scikit-learn              1.3.dev0 
evaluate                  0.4.1.dev0

I'm running the run_clm.py script for language modeling.

Is there something I need to install besides scikit and evaluate? Is there something wrong with the library?

Thanks!


I solved reinstalling os, conda and libraries again. 

I don't know where it was the error exactly, I was using Ubuntu 22.04 and python 3.10. Working in Ubuntu with python 3.9
Repository: huggingface/evaluate
Issue: Missing model caching in perplexity metrics
Root cause: Architectural Constraint

The "official" implementation of `perplexity` metric does not cache language model [[1][1]]. It seems that metric instance should fetch model and prepare it for further usage in `_download_and_prepare`. I suppose that there should be clear API on caching and cache resetting. Also, it is totally unclear how to configure metrics on loading (there is only `config_name` but `kwargs` are just ignored).

[1]: https://github.com/huggingface/evaluate/blob/6abb0d53b82b1e5efea5d683b91d7990a653c78d/metrics/perplexity/perplexity.py#L103-L115


> 度量的“官方”实现`perplexity`不缓存语言模型[ [1](https://github.com/huggingface/evaluate/blob/6abb0d53b82b1e5efea5d683b91d7990a653c78d/metrics/perplexity/perplexity.py#L103-L115) ]。看来指标实例应该获取模型并准备好在`_download_and_prepare`. 我认为应该有关于缓存和缓存重置的明确 API。此外，完全不清楚如何配置加载指标（只有`config_name`但`kwargs`被忽略）。

+1 ,Do you know now?