zhiminy/EvalEng huggingface dataset:
Repository: huggingface/evaluate
Issue: Token classification bootstrap crashing with custom dataset
Root cause: Validation Gap

OS: maxOS 14.1
Python: 3.11.6
PyTorch: 2.0.1

Description: Standard evaluation of custom dataset works:

```
dataset = Dataset.from_list(dictlist)

task_evaluator = evaluator("token-classification")

eval_results = task_evaluator.compute(
    model_or_pipeline=<model_path>,
    data=<custom_dataset["validation"]>,
    metric="seqeval",
    label_column="tags"
)
```
However, when adding bootstrapping, I get a crash: 
```
eval_results = task_evaluator.compute(
    model_or_pipeline=<model_path>,
    data=<custom_dataset["validation"]>,
    metric="seqeval",
    label_column="tags",
    strategy="bootstrap",
    n_resamples=30,
```

Potential solution: Add parameter for "zero_division" as explained in warning below. 

Stacktrace: 
UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.
  _warn_prf(average, modifier, msg_start, len(result))
 UndefinedMetricWarning: Recall and F-score are ill-defined and being set to 0.0 in labels with no true samples. Use `zero_division` parameter to control this behavior.
  _warn_prf(average, modifier, msg_start, len(result))
Traceback (most recent call last):
, line 32, in <module>
    eval_results = task_evaluator.compute(
                   ^^^^^^^^^^^^^^^^^^^^^^^
line 266, in compute
    metric_results = self.compute_metric(
                     ^^^^^^^^^^^^^^^^^^^^
line 531, in compute_metric
    bootstrap_dict = self._compute_confidence_interval(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
line 147, in _compute_confidence_interval
    bs = bootstrap(
         ^^^^^^^^^^
line 450, in bootstrap
    args = _bootstrap_iv(data, statistic, vectorized, paired, axis,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
line 155, in _bootstrap_iv
    sample = np.atleast_1d(sample)
             ^^^^^^^^^^^^^^^^^^^^^
 line 65, in atleast_1d
    ary = asanyarray(ary)
          ^^^^^^^^^^^^^^^
ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected shape was (416,) + inhomogeneous part.
Repository: huggingface/evaluate
Issue: `DatasetColumn` and `DatasetColumnPair` should have `__repr__()` methods defined

Currently, if you try to print a [`DatasetColumnPair`](https://github.com/huggingface/evaluate/blob/8dfe05784099fb9af55b8e77793205a3b7c86465/src/evaluate/evaluator/utils.py#L41) (say, for debugging purposes), it will just show an empty list. This causes all kind of issues with automatically raised exceptions, etc.
Repository: huggingface/evaluate
Issue: Initial proposal for documentation Table of Contents

**Initial proposal for documentation Table of Contents:**

- Getting started
- How-to guides
         - Evaluate in a nutshell
         - Choosing a metric for your task
- Conceptual guides
        - All about metrics
        - Considerations for ML model evaluation
        - Metrics vs Measurements vs Comparisons
- Reference
       - Main classes
       - Builder classes
       - Loading methods
       - Pushing to the Hub
       - Streamlit Demo 

Wdyt @lvwerra ?

Awesome, thanks for setting this up @sashavor. Here a a few suggested changes

Getting started
- Tutorials (easy intro for people who just want to use it)
  - Installation
  - A quick tour of evaluate
- How-to-guides (a little bit more in depth guides)
  - Choosing a metric for your task
  - Creating  and sharing a new evaluation
- Conceptual guides
  - Types of evaluation (-> Metrics vs Measurements vs Comparisons, i feel that should be explained first)
  - Considerations for ML model evaluation
- Reference (only technical reference goes here like class descriptions etc.)
  - Main classes
  - Builder classes
  - Loading methods
  - Saving methods

Closed with #58.
Repository: huggingface/evaluate
Issue: MSE not accepting (n_samples, n_ouptuts) despite docs stating so
Root cause: Documentation Deficiency

The evaluation card for [MSE](https://huggingface.co/spaces/evaluate-metric/mse) states:

---

Mandatory inputs:
- `predictions`: numeric array-like of shape (n_samples,) or (n_samples, n_outputs), representing the estimated target values.
- `references`: numeric array-like of shape (n_samples,) or (n_samples, n_outputs), representing the ground truth (correct) target values.

---

So it should be usable similar to `pytorch`'s [`MSELoss`](https://pytorch.org/docs/stable/generated/torch.nn.MSELoss.html) with multiple dimensions.
But using it with `(batch_size, multiple_outputs)` doesn't work:

```py
import evaluate
import numpy as np

m = evaluate.load('mse')
print(m.compute(predictions=np.random.rand(16,2), references=np.random.rand(16,2)))
```
⬇️ 
```sh
ValueError: Predictions and/or references don't match the expected format.
Expected format: {'predictions': Value(dtype='float32', id=None), 'references': Value(dtype='float32', id=None)},
Input predictions: ...
Input references: ...
```

---

Version: `evaluate==0.4.1`

Hi @bauersimon , the quick answer for your question is:
```diff
import evaluate
import numpy as np

- m = evaluate.load('mse')
+ m = evaluate.load('mse', 'multilist')
print(m.compute(predictions=np.random.rand(16,2), references=np.random.rand(16,2)))
```

And, the error message is also too vague to address the problem. 

Thanks! Should probably be mentioned in the docs then 😅.
Repository: huggingface/evaluate
Issue: Add geometric mean of per-token-Perplexities
Root cause: Unimplemented Feature Gap

Hi I'm new to this and might to be overlooking smth, but it seems to me that more often than the arithmetic mean over perplexity per batch, the geometric mean over all per-token-Perplexities is used. I.e here in [huggingface's Transformers docs](https://huggingface.co/docs/transformers/perplexity). So basically just what you calculate as a single perplexity per batch but for the whole dataset. Would be helpful to have that returned here additionally.
So smth along the lines of (just to understand what i mean. Didn't test the code):
```python
ppls = []
nll_sum = 0
total_tokens = 0

for start_index in logging.tqdm(range(0, len(encoded_texts), batch_size)):
          
          [...]
            
           negative_log_likelihood_batch = loss_fct(shift_logits.transpose(1, 2), shift_labels) * shift_attention_mask_batch).sum(1)
          
           perplexity_batch = torch.exp(negative_log_likelihood_batch / shift_attention_mask_batch.sum(1))

           ppls += perplexity_batch.tolist()
           nll_sum += negative_log_likelihood_batch.sum().item()
           total_tokens += shift_attention_mask_batch.sum().item()

        return {"perplexities": ppls,
                "mean_perplexity": np.mean(ppls),
                "geometric_mean_perplexity": torch.exp(nll_sum / total_tokens)}
```
Repository: huggingface/evaluate
Issue: need evaluator code reference for object detector & zero shot object dector task 
Root cause: Unimplemented Feature Gap

Hi Team,

i use google/owlvit-base-patch32 model for zero shot object detection. 

To evaluate the model performance and speed, i like to use a evaluator for Zero Shot Object Detector task. 

Currently it is supported only for limited tasks ( https://huggingface.co/docs/evaluate/base_evaluator) . 

Like to know is there any evaluator available for ( Zero Shot Object Detector tasks) 

https://www.philschmid.de/optimizing-transformers-with-optimum contains this code for text-classification task.

thanks
Repository: huggingface/evaluate
Issue: Evaluation not working with `microsoft/deberta-v3-large`
Root cause: Validation Gap

Hi,

I was trying to evaluate the model trained on `microsoft/deberta-v3-large` but looks like it's having some issues on indexing. Any ideas or support for this one ?


Error:
`IndexError: list index out of range`

Code to Reproduce:
```
import pandas as pd
from datasets import load_dataset
from evaluate import evaluator
from transformers import pipeline

models = [
    "microsoft/deberta-v3-large"
]

data = load_dataset("conll2003", split="validation").shuffle().select(range(10))
task_evaluator = evaluator("token-classification")

results = []
for model in models:
    results.append(
        task_evaluator.compute(
            model_or_pipeline=model, data=data, metric="seqeval"
            )
        )

df = pd.DataFrame(results, index=models)
df[["overall_f1", "overall_accuracy", "total_time_in_seconds", "samples_per_second", "latency_in_seconds"]]
```

Can someone please advice, it's still the same issue ?

Same issue, another model.
Repository: huggingface/evaluate
Issue: Add Precision@k and Recall@k metrics
Root cause: Unimplemented Feature Gap

Previous Precision and Recall metrics, supported by evaluate are only sklearn clones. It would be great to add an top k version for those metrics. 

For example, a simple implementation of P@k metric is:

```python
def precision_at_k(y_true, y_score, k):
    df = pd.DataFrame({'true': y_true, 'score': y_score}).sort('score')
    threshold = df.iloc[int(k*len(df)),1]
    y_pred = pd.Series([1 if i >= threshold else 0 for i in df['score']])
    return metrics.precision_score(y_true, y_pred)
```
Repository: huggingface/evaluate
Issue: After fine-tuning Gemma and want to evaluate performance:  AttributeError: module 'keras._tf_keras.keras' has no attribute '__internal__' 
Root cause: External Dependency Breakage

Hey,

I encountered the following error:

`AttributeError: module 'keras._tf_keras.keras' has no attribute '__internal__'`

I am following the [model card](https://www.kaggle.com/models/google/gemma) for training Gemma and intend to assess the fine-tuning performance.

The error occurred while importing `evaluate`.

`import evaluate`
`rouge = evaluate.load('rouge')`
Repository: huggingface/evaluate
Issue: the difference of your bleu and sacrebleu
Root cause: Algorithmic Error

What is the difference between your package's bleu implementation and sacrebleu implementation? I calculated the result differently in the two ways, Chinese expected, passed sacrebleu's zh tokenizer

I believe there are some differences between the implementation and sacrebleu's. Actruly, testing with English has the same problem.

## evaluate
```python
import evaluate


predictions = ["hello there general kenobi", "foo bar foobar"]
references = [
    ["hello there general kenobi", "hello there !"],
    ["foo bar foobar"]
] 

bleu = evaluate.load("bleu")
results = bleu.compute(predictions=predictions, references=references, smooth=False, max_order=4)
print(results)

```
got results:
```
{'bleu': 1.0, 'precisions': [1.0, 1.0, 1.0, 1.0], 'brevity_penalty': 1.0, 'length_ratio': 1.1666666666666667, 'translation_length': 7, 'reference_length': 6}
```

## sacrebleu
```python
from sacrebleu.metrics import BLEU


predictions = ["hello there general kenobi", "foo bar foobar"]
references = [
                 ["hello there general kenobi", "hello there !"],
                 ["foo bar foobar"]
             ]

bleu = BLEU(smooth_method="none", max_ngram_order=4, tokenize='13a')
results = bleu.corpus_score(predictions, references)
print(results)

```
got results:
```
BLEU = 100.00 100.0/100.0/100.0/100.0 (BP = 1.000 ratio = 1.000 hyp_len = 4 ref_len = 4)
```
Repository: huggingface/evaluate
Issue: evaluate consuming Memory and slow down the process
Root cause: Resource Mishandling

https://huggingface.co/docs/transformers/main/en/tasks/semantic_segmentation#custom-dataset

this page's example(and any other examples online computing miou with hugging face) has problem of slowing down.

it starts with normal speed and after few hundreds steps, starting to take up the memory and slow down

i'm not sure if that's leakage or some kind of bug, but it's not a normal behavior.

it's not a hardware shortage. i monitored it. not a CPU or GPU calculation bottleneck or lack of memory.
Repository: huggingface/evaluate
Issue: Is perplexity correctly computed?
Root cause: Documentation Deficiency

Hello. I'm struggling with replicating the [reported perplexity](https://github.com/ggerganov/llama.cpp/discussions/2352) (~6) for LLaMa-2-7b. 
I am using this simple code snippet:
```
import evaluate
import datasets

perplexity = evaluate.load("perplexity", module_type="metric")
input_texts = datasets.load_dataset("wikitext",
                                    "wikitext-2-raw-v1",
                                    split="test")["text"]
input_texts = [s for s in input_texts if s!='']
results = perplexity.compute(
    model_id="sharpbai/Llama-2-7b-hf",
    batch_size=4,
    predictions=input_texts
)
print(results)
```
And I get among the results: `'mean_perplexity': 60.9764459149642`.
In [this tutorial](https://huggingface.co/docs/transformers/perplexity) it is computed "approximately" by flattening the dataset into a string and by computing the avg. sliding window perplexity. I still get a high perplexity.
I tried to change the model in the code snippet to `openai-community/gpt2` and the perplexity is above 600!
Does this depend on using the correct model class?
Thank you for any suggestion.

**EDIT:** I'm using the following versions
```
transformers              4.38.2
evaluate                  0.4.1
datasets                  2.18.0 

```


Perplexity is a measure which is dependent on the model used to calculate it.
Specifically, the formula for perplexity is entirely dependent on the probability functions for a given model. 

So seeing different perplexities for different models is entirely expected. 

You could even argue that the higher perplexity from GPT2 compared to Llama-2-7B for the human dataset of wikitext is a reflection that Llama-2-7B is a better model.

I tested also LLaMa2-70b and the perplexity on wikitext is around 22. Shouldn't I expect a better performance for both the 70b and the 7b variant? Is LLaMa2's training distribution that far?

I don't know how the results from the initial paper calculated their perplexity, whether they used HuggingFace's metric or not, but given the discussion is based on the varying window size, that could be part of the problem.

Are you confident that you are using the right sliding window (Context window) size?

If you want to be really precise, you could write your own perplexity measure. There is a good guide here [HuggingFace: Perplexity of Fixed-Length Models](https://huggingface.co/docs/transformers/en/perplexity).

Also, it is not surprising that 70b is largely outperforming 7b. The parameter difference is just that large.

Hi @halixness, were you able to resolve the perplexity issue? I am also getting similar value (~56) for llama2-7b. I have tried coding up the perplexity calculations suggested by @SamSJackson and also with huggingface evaluate module, but still getting similar values. 

Yo any updates on this ?

Any update? I also got stuck in the same issue. :(((
Repository: huggingface/evaluate
Issue: ValueError: Predictions and/or references don't match the expected format.
Root cause: Validation Gap

Getting this error when trying to compute IoU using the [Huggingface example](https://github.com/huggingface/evaluate/tree/main/metrics/mean_iou).
Flattening arrays does not solve the problem as [this issue](https://github.com/huggingface/evaluate/issues/392) suggests.

Steps to reproduce:
```
import numpy as np
import evaluate

mean_iou = evaluate.load("mean_iou")
predicted = np.array([[2, 2, 3], [8, 2, 4], [3, 255, 2]])
ground_truth = np.array([[1, 2, 2], [8, 2, 1], [3, 255, 1]])
results = mean_iou.compute(predictions=predicted, references=ground_truth, num_labels=10, ignore_index=255)
```

Using 
evaluate 0.4.1
numpy 1.26.1

Full error:
```
Traceback (most recent call last):
  File "/home/anba/catkin_ws/src/tas_dev/dev/anba/SAM/test.py", line 7, in <module>
    results = mean_iou.compute(predictions=predicted, references=ground_truth, num_labels=10, ignore_index=255)
  File "/home/anba/anaconda3/envs/SAM/lib/python3.10/site-packages/evaluate/module.py", line 450, in compute
    self.add_batch(**inputs)
  File "/home/anba/anaconda3/envs/SAM/lib/python3.10/site-packages/evaluate/module.py", line 541, in add_batch
    raise ValueError(error_msg) from None
ValueError: Predictions and/or references don't match the expected format.
Expected format: {'predictions': Sequence(feature=Sequence(feature=Value(dtype='uint16', id=None), length=-1, id=None), length=-1, id=None), 'references': Sequence(feature=Sequence(feature=Value(dtype='uint16', id=None), length=-1, id=None), length=-1, id=None)},
Input predictions: [  2   2   3   8   2   4   3 255   2],
Input references: [  1   2   2   8   2   1   3 255   1]
```

Fixed it. The error message does give some clues.
The np arrays must be uint16 and they must be passed within a list.
That being said, it doesn't really make sense to me why this would be the case.
The example code in the docs should work as well.

Here's the fixed example:
```
import numpy as np
import evaluate

mean_iou = evaluate.load("mean_iou")
predicted = np.array([[2, 2, 3], [8, 2, 4], [3, 255, 2]], dtype=np.uint16)
ground_truth = np.array([[1, 2, 2], [8, 2, 1], [3, 255, 1]], dtype=np.uint16)
results = mean_iou.compute(predictions=[predicted], references=[ground_truth], num_labels=10, ignore_index=255)
```
Repository: huggingface/evaluate
Issue: Does Rouge score support the multilingual language?
Root cause: Documentation Deficiency

Environment: **Google Colab**
Evaluate version: **0.4.1**
Installed for loading rouge metric: `!pip install rouge_score -q`

```python
sample_summary = "मलेसियन एयरलाइन्सको विमानमा दुई सय ९८"
sample_reference = "मलेसियन एयरलाइन्सको विमानमा दुई सय ९८"

# calculate all metric at once
scores = rouge_score.compute(
    predictions=[sample_summary],
    references=[sample_reference]
)
scores
```
**Result is:**
```
{'rouge1': 0.0, 'rouge2': 0.0, 'rougeL': 0.0, 'rougeLsum': 0.0}
```

I want to know if rouge score is for English test only or also support multilingual?

If not, will there be any future plan to introduce the multilingual support(nepali/hindi)?

I solved this issue using by replacing `rouge_score = evaluate.load("rouge")` with
```
rouge_score = evaluate.load("CZLC/rouge_raw")
```
which works pretty well.

I checked `evaluate.list_evaluation_modules()` for this.
Repository: huggingface/evaluate
Issue: Shouldn't perplexity range from [1 to inf)?
Root cause: Documentation Deficiency

https://github.com/huggingface/evaluate/blob/8dfe05784099fb9af55b8e77793205a3b7c86465/metrics/perplexity/README.md?plain=1#L60

perplexity = e**(sum(losses) / num_tokenized_tokens)

If sum(losses) = 0, then perplexity = 1.

@ivanmkc 
Perplexity ranges between zero and inf because the exponent can be negative (The sum of _negative_ log likelihoods).
Check out the following blog post for a better understanding.
[Perplexity of fixed-length models](https://huggingface.co/docs/transformers/en/perplexity).

Thanks, will take a look.
Repository: huggingface/evaluate
Issue: Cannot use it offline!
Root cause: Algorithmic Error

Is there any need to do this func [_download_additional_modules]?????
![image](https://github.com/huggingface/evaluate/assets/122849983/6e7d21fb-e4fe-4730-bdb5-6ff03f1d3bc4)


Despite I have download the metrics and use [evaluate.load('./metrics/bleu')], there is a high chance that it will attempt to connect to the network and download additional modules. What confuses me the most is that it doesn't necessarily trigger network downloads, **but rather with a high probability, which is very strange for a fixed piece of code.**
Repository: huggingface/evaluate
Issue: Allow for specify coda device in perplexity evaluation  
Root cause: Validation Gap

The current implementation of `perplexity.compute` does not allow to provide a specific coda device (e.g. `cuda:1` instead of just `cuda`. 
 
https://github.com/huggingface/evaluate/blob/8dfe05784099fb9af55b8e77793205a3b7c86465/metrics/perplexity/perplexity.py#L108

Should probably be changed to something along the lines of: 
```python 
assert device in ["gpu", "cpu", "cuda"] or re.search(r'^cuda:[0-9]+$', device), "device should be either gpu or cpu." 
```
Repository: huggingface/evaluate
Issue: METEOR has no option to return unaggregated results
Root cause: Algorithmic Error

## Describe the bug

The METEOR metric returns the mean metric as opposed to a list of individual metrics per ref/pred pair. This is inconsistent with other default metric behavior including rouge, bleu, and bertscore. This can be a problem when trying to calculate correlation, for example.

See `metrics/meteor/meteor.py` line 168 in `_compute`:
`return {"meteor": np.mean(scores)}`

## Steps to reproduce the bug

```python
from evaluate import load

metric = load("meteor")
meteor.compute(references=["reference one", "reference two"], predictions=["prediction one", "prediction two"])
```
## Expected results

`{"meteor": [0.25, 0.25]}`
## Actual results

`{"meteor": 0.25}`

## Environment info

    evaluate version: 0.4.1
    Platform: Rocky Linux 8.9 
    Python version: 3.9.18

Is there any reason why this has not been fixed yet?
Repository: huggingface/evaluate
Issue: Perplexity metric does not apply batching correctly to tokenization
Root cause: Algorithmic Error

When I try to evaluate my model's text generation using the perplexity metric, the batch_size parameters in perplexity._compute(..) was not sufficient, because it tries to tokenize and move the entire set of predictions to GPU. A simple change to move the tokenization to each batch fixes the issue for me. 

Also, it should be possible to pass my own model and tokenizer (since my model is not publishable on huggingface) to the metric. I have made these changes to enable my experiments. 

I have made changes to fix this. I can open a PR to commit these changes, if this sounds good to you. I believe this will benefit the developer community.

I am also facing the same issue.
Repository: huggingface/evaluate
Issue: Unable to run pip install evaluate[template]
Root cause: Documentation Deficiency

While trying to add create and share a new evaluation,  when I run `pip install evaluate[template]`, I get the following error:
```
zsh: no matches found: evaluate[template]
```

I was able to run the same and get it to work about a couple of months ago, so I am guessing it is a recent issue.



Hi @saicharan2804 , you can try install with `pip install "evaluate[template]"`, it should work.
Repository: huggingface/evaluate
Issue: [FR] Confidence intervals for metrics
Root cause: Unimplemented Feature Gap

It seems that currently simple metrics such as
```
evaluate.load(
    "accuracy",
)
```
do not compute a confidence interval. This can be easily fixed by first computing the mean, and the STD, and then dividing the STD by the square of the sample count (to compute the STD of the mean estimate). (See, e.g., [here](https://blogs.sas.com/content/iml/2019/10/09/statistic-error-bars-mean.html#:~:text=the%20sample%20size%3A-,The%20SEM%20equals%20SD%20/%20sqrt(N),-.%20That%20is%2C%20the).)

Even just giving back the variance (or STD) is enough, the user can do their own computations on those.