zhiminy/EvalEng huggingface dataset:
Repository: huggingface/evaluate
Issue: AttributeError: module 'evaluate.info' has no attribute 'Config'.
Root cause: External Dependency Breakage

Hi, thank you so much for all your work. This is just to let you know about an issue that appeared.

I was loading a metric using: 

`metric = evaluate.load("roc_auc", "multilabel")`

And the following error appeared:

`    class ROCAUCConfig(evaluate.info.Config):
AttributeError: module 'evaluate.info' has no attribute 'Config'.`

It seems related to the last commit of this repo: https://github.com/huggingface/evaluate/commit/e4a2724377909fe2aeb4357e3971e5a569673b39

Also, it seems to work properly with and older version of the metric:

`metric = evaluate.load("roc_auc", "multilabel", revision="cf0c51eddd2a1a324961982c0ee174d896569d20")`



Thanks for reporting, we noticed the issue and the PR got reverted. It should work again now.

It's because also the metrics on the Hub got updated and it seems something with the versioning between `evaluate` and the scripts on the hub did not work properly.

(Sorry deleted my comment b/c I wasn't 100% sure when it was merged)

Context on above comment: not 100% sure why this trickled over when we were installing via pypi as well
Repository: huggingface/evaluate
Issue: Tests for Spaces requirements.txt upon push of new module
Root cause: Unimplemented Feature Gap

Upon merges we've sometimes had metrics Spaces break because of missing requirements which then need to be added to requirements.txt later. 

Not sure what the best way to detect this is, though — sometimes it's not obvious because the requirement is wrapped in a library which has optional dependencies (e.g. [this](https://huggingface.co/spaces/evaluate-measurement/toxicity/blob/main/toxicity.py) one requires torch because it loads a torch classifier, but you wouldn't know it from the imports).

Maybe a bit down the road we could similar to the doc builder create a test org or just a space with the new module and check if the space is up and running. When the PR is merged the test space could be deleted. 

Alternatively, we could also setup an env for each space and spin up the gradio app. This is potentially quite slow and the tests are already getting pretty slow.
Repository: huggingface/evaluate
Issue: matthews correlation for multiple features?
Root cause: Unimplemented Feature Gap

I'd like to add matthews correlation for the multilabel case.
This essentially has a few options:

* ("micro") flatten predictions and targets, then calculate
* ("macro") calculate it per-feature and average
* Implement micro, macro as in f1 score.

Since scikit-learn doesn't support this out of the box, this may also be a terrible idea for some reason, in which case I'd like to learn why.


For the multi-class case there seems to be a [formal definition](https://en.wikipedia.org/wiki/Phi_coefficient#Multiclass_case) of how to compute MCC, however, I couldn't find anything on the multi-label case. Maybe there is something about this in literature?

A multi-label problem is essentially just a collection of binary classification problems, so it's more about how related you want to see the objectives I think.  I would not expect there to be literature on how to average such things.

Sounds good, feel free to open a PR. You can look at other integrated scikit-learn metrics like F1 score to see how to deal with multilabel featuers. Let me know if you need any help!
Repository: huggingface/evaluate
Issue: Parallelizing Evaluation Functions?
Root cause: Architectural Constraint

Hi, is there a way to parallel process some of the evaluations? e.g. BLEU and BLEURT take quite a while to run (~6hrs) on even just ~2000 examples in my test set.

That seems extremely slow, especially for BLEU. Can you provide a minimal reproducible example? For BLEURT you might benefit from running it on a machine with GPU as it runs BERT inference to compute the metric.

Not easily. Apologies, I meant BLEU and BLEURT together. I am running it on a 4-GPU cluster. Could it be that it's not using the GPU and instead using CPU?

It seems that the BLEURT code is not optimized for multi-GPU inference. You could try to increase the batch size though or using a model like `BLEURT-20-D3`:

https://github.com/google-research/bleurt#speeding-up-bleurt

Length based batching would require changes in the code current implementation in`evaluate`.
Repository: huggingface/evaluate
Issue: Add Average Normalized Levenshtein Similarity (ANLS)
Root cause: Unimplemented Feature Gap

Document visual question answering (DocVQA) models, like LayoutLM and Donut, are evaluated using the Average Normalized Levenshtein Similarity metric (ANLS for short).

Would be very useful for the community to have this metric in the evaluate library.

The metric was introduced here for DocVQA: https://rrc.cvc.uab.es/?ch=17&com=tasks.

![image (4)](https://user-images.githubusercontent.com/48327001/195277520-b1ef2be2-c4d7-417b-91ec-5fda8aa6db06.png)




#412 
I tried to add the Average Normalized Levenshtein Similarity (ANLS), but this is my first contribution to open source so please tell me if there is anything I can correct.
Thanks!

I've corrected the Flake8 errors. Please let me know if I should look into anything else for this issue.

Cc @lvwerra would it be possible to review this?

Feel free to assign someone else in case you don't have the bandwidth
Repository: huggingface/evaluate
Issue: [Feature request] Add evaluator for speech2text models
Root cause: Unimplemented Feature Gap

Now that Whisper is integrated into `transformers` and the large number of speech2text models on the Hub, it might be interesting having a speech2text evaluator that computes common metrics like word error rate etc. 

This would be especially useful for `optimum` users, who want to know if their exported models suffer any degradation from quantization etc.

I don't have bandwidth to tackle this right now, but maybe someone in the community is excited to?

cc @mht-sharma 

Can you maybe share which transformers pipeline one could use this and what typical datasets/metrics are?

@lewtun @lvwerra 
Hey guys, I am participating the Whisper fine-tuning event and
looks like this might be really helpful addition.

Please take a look into PR https://github.com/huggingface/evaluate/pull/378
Repository: huggingface/evaluate
Issue: Evaluate usage in other frameworks
Root cause: Documentation Deficiency

Similar to #325 it would be great to show how to use `evaluate` metrics in other frameworks. Each framework can have its own section in `metrics_integration.mdx`. Example frameworks are:

- fastai
- Pytorch Lightning
- scikit-learn
- Keras/TensorFlow

Just to name a few. Ideally this includes a representative code snippet that shows the workflow of using `evaluate` in that framework that allows the user to implement it in their own scripts.

cc @osanseviero 

Hi! I'm interested in taking up writing an example for Keras/Tensorflow. Would this just be like using the metric inside of a callback for training a model? If I write a script that does that, would that work as an example? Thanks in advance!

Just talked with @lvwerra on discord about taking this up. I'll complete a Keras example by early next week or so!

Sounds great, thanks a lot! 🔥 

I would like to write an example for the integration with the **scikit-learn** framework.
Repository: huggingface/evaluate
Issue: default blank row of launch_gradio_widget in web demo app may confuse users
Root cause: Validation Gap

I notice that the default row of ``dataFrame`` of ``launch_gradio_widget`` is set to ``2``. Since the gradio does not provide ``delete_row`` option in the webui interface, the calculation of some web demos (e.g., exact_match, I don't test all) includes the blank row, which may confuse the user.
For example, if we as the users only hope to calculate the exact match score of one sample, thus we only fill the first row in the webui and left the second unremovable row blank, the returned calculation results is always wrong since it counts the second blank text row as a ``1.0`` score (exactly match! because they are both 0 length '' str).
The above example is shown in this figure.

![1664165296420-633122d3f242a8532b7a928d](https://user-images.githubusercontent.com/37944786/196462009-09d23e9f-70b9-4acb-aaa6-146ad766ae9a.png)

Actually, I have posted a [discussion](https://huggingface.co/spaces/evaluate-metric/exact_match/discussions/2) in huggingface web demo nearly one month ago and no one responsed.

Here are some potential useful suggestions to fix this issue:

1. Considering the upstream package gradio is not tending to add ``delete a row`` option (related issue discussed in [here](https://github.com/gradio-app/gradio/issues/1303)), a quick fix is set the default row to ``1`` instead of ``2`` in ``evaluate/utils/gradio.py/launch_gradio_widget/LINE 115``.

2. Check the user's input data and discard these null inputs (blank text input), which could be done in ``evaluate/utils/gradio.py/parse_test_cases`` function.

And I would like to have a PR for fixing this if the the community welcome.😁


Hi @Raibows, thanks for pointing this out! Indeed that's an issue that would be great to fix. You suggestion of setting the default to `1` and in addition check for null inputs as a failsafe sounds good to me! Feel free to open a PR and let me know if you need any help! Sorry for not replying on the Hub issue, these sometimes slip :)
Repository: huggingface/evaluate
Issue: Improve tests

This is a summary issue to gather ways to improve the current testing. Currently, I see three things to improve the tests:

1. Move CircleCI tests to GitHub Actions.
2. Make tests run faster - currently it takes 10-15min to run the test suite.
3. The error messages of the doctests that run for the metrics (measurements/comparisons) are hard to read. It would be great if we could improve that.

cc @mathemakitten @sashavor 

1 and 2 were addressed in #274. Leaving 3 for future generations to solve.
Repository: huggingface/evaluate
Issue: Bug in computing perplexity about max_length
Root cause: Algorithmic Error

In [perplexity.py line 138](https://github.com/huggingface/evaluate/blob/main/metrics/perplexity/perplexity.py#L138), tokenizer tokenizes the input_text with truncation by max_length. max_length is equal to the model.config.max_length.

```
max_tokenized_len = model.config.max_length
encodings = tokenizer(
              predictions,
              add_special_tokens=False,
              padding=True,
              truncation=True,
              max_length=max_tokenized_len,
              return_tensors="pt",
              return_attention_mask=True,
          ).to(device)
```

However, in many models, there is no specified max_length, so max_length here is actually the default value as 20. See [configuration_utils.py line 278](https://github.com/huggingface/transformers/blob/main/src/transformers/configuration_utils.py#L278). 
```
self.max_length = kwargs.pop("max_length", 20)
```
I print the max_length of bert and gpt2 model. Both of them are equal to 20.

Therefore, when computing the ppl, the input text will be cut as long as the length exceeds 20, while 20 is too small for most cases.

I believe developers should fix this bug because perplexity is commonly used in text generation tasks. In gpt2, the option for actual max_length is called 'n_positons', while in bert, it's called 'max_position_embeddings'. There is no unifying config option for this. So removing the max_length parameter here may be a solution.
Repository: huggingface/evaluate
Issue: how to see the metrics in an EvaluateModule & comparison of EvlauationModules
Root cause: Documentation Deficiency

After loading metrics from a configuration, say metrics = evaluate.load("glue","mnli"), how can I access what individual metrics the EvaluateModule would compute (I couldn't find an easy way to do this from the documenation). Also is there a way to check if two EvaluateModule  are of the same type -- i.e., do they calculate the same metrics and take same inputs? 

Hi @sgunasekar, if by individual metric you mean the keys in the returned dictionary, then there is currently no way of doing so programmatically. There is some documentation in the [readme](https://huggingface.co/spaces/evaluate-metric/glue). 

Note that `glue` is kind of an edge case since it is more a collection of metrics used for different glue tasks where other modules (such as `accuracy` or `rouge`) clearer.

If you want to access the input types of a metrics you can access `metric.info.features` to see the input types of the metric. Hope this helps!

I menat the former--keys of returned dictionary after compute. Glue might be currently a special case, but imo its not unusual to have multiple metrics. I haven't looked into the code too deeply, but if its not too complicated i can yake a stab at it.

Information of structure of the returned is not stored anywhere, so I think you would need to analyze/run the code. A simple way to get a dummy result is to run the example in the docstring (which are all tested with doctest in the CI) or use the sample output that's provided there.
Repository: huggingface/evaluate
Issue: Evaluator needs a simple way to check if a specific model is supported
Root cause: Unimplemented Feature Gap

With @sashavor, we found that the below example doesn't work with the QA Evaluator — but it's not obvious that it wouldn't.

Docs for the QA Evaluator say "Methods in this class assume a data format compatible with the [QuestionAnsweringPipeline](https://huggingface.co/docs/transformers/en/main_classes/pipelines#transformers.QuestionAnsweringPipeline)." That links out to https://huggingface.co/models?pipeline_tag=question-answering and searching for "t5" returns "t5-base-finetuned-quartz", but that model is incompatible with the QA pipeline and it's not immediately clear that it wouldn't work unless you actually tried it. 

It's probably not feasible to list all models on the Hub which are compatible with any given evaluator, but maybe it'd be useful to have a helper method which checks if a model is compatible with an Evaluator ahead of time before setting an entire pipeline? Welcoming suggestions on how we could handle this!

```
from datasets import load_dataset
from evaluate import evaluator

task_evaluator = evaluator("question-answering")

data = load_dataset("squad", split="validation[:10]")
eval_results = task_evaluator.compute(
    model_or_pipeline="mrm8488/t5-base-finetuned-quartz",
    data=data,
    metric="squad",
    strategy="bootstrap",
    n_resamples=30
)

print(eval_results)
```
Repository: huggingface/evaluate
Issue: Is there a way to obtain perplexity using two GPUs?
Root cause: Unimplemented Feature Gap

I have two GPUs, the model is currently using one GPU. Is there a way to be able to use two GPUs? I thinking something like device_map parameter in AutoModelForCausalLM.from_pretrained



Your use-case would be a model that doesn't fit in a single GPU? 

Yes, it is necessary to use 2 GPUs because of the model size

So this is currently not supported but feel free to open a PR! Happy to help. Might also be interesting for @mathemakitten and @TristanThrush who have been working with larger models.
Repository: huggingface/evaluate
Issue: Parts of evaluate demo spaces illegible in dark mode

Code text is not legible in dark mode, because the chosen font color is too dark.

Example here in the [accuracy](https://huggingface.co/spaces/evaluate-metric/accuracy) demo space.

![image](https://user-images.githubusercontent.com/22726840/201073243-112171db-d814-4c92-9906-1d08e0350132.png)



This seems to be an issue with code snippets in Gradio. Should we open an issue on the Gradio repo @abidlabs?

Yes please do @lvwerra, thanks!

See: https://github.com/gradio-app/gradio/issues/2662

Tested and realized that this has already been fixed in the latest version of Gradio. Can you upgrade the Spaces sdk version to gradio==3.10.0`?

Yes, we can update the spaces version, thanks @abidlabs!

Has this been fixed somewhere?
Repository: huggingface/evaluate
Issue: What is the MAE range in evaluate?
Root cause: Documentation Deficiency

In the MAE demo space, it is indicated that "Each MAE float value ranges from 0.0 to 1.0, with the best value being 0.0."

Doesn't it range from 0 to +inf in general ? 
Is it a programmatic constraint added on the evaluate MAE score?

True, it should be `[0, inf)` - would you mind opening a PR? The file is in `metrics/mae/readme.md`. 

cc @sashavor who wrote this I think.

Closed with #359
Repository: huggingface/evaluate
Issue: Problem with `add` method and `Rouge`
Root cause: Validation Gap

As reported [here](https://huggingface.co/spaces/evaluate-metric/rouge/discussions/3) the following code yields an error:
```Python
 rouge = evaluate.load('rouge')
 rouge.add(predictions="sentence 1", references="sentence 2")
```

```bash
ValueError: Evaluation module inputs don't match the expected format.
Expected format: {'predictions': Value(dtype='string', id='sequence'), 'references': Value(dtype='string', id='sequence')},
Input predictions: sentence 1,
Input references: sentence 2
```

Hi @lvwerra, Happy to work on this Issue. Would be really great to get more information about the issue and also guidance from you. Thankyou

Hi @Sumanth077, that would be great. Under the hood this [`add`](https://github.com/huggingface/evaluate/blob/b571f12c15600c1eec03e9e2ae785bf695d5c4bd/src/evaluate/module.py#LL514) method is called. It could be either `_infer_feature_from_example` or `_enforce_nested_string_type` that is not working as expected. The former infers which feature type is used if a metric defines several (as in ROUGE) and the latter makes sure that we don't accidentally cast a type that's not a string to string (since everything can be cast as a string). I would check if you can find anything unusual there.


Thankyou @lvwerra. Could you also kindly attach the exact file link of Rouge here. It would be great.

Sure, it's [here](https://github.com/huggingface/evaluate/blob/b571f12c15600c1eec03e9e2ae785bf695d5c4bd/metrics/rouge/rouge.py).

Hi @Sumanth077, just a friendly pin to ask if you have started working on this. would like to help and collab here.
Repository: huggingface/evaluate
Issue: Alternative backends for `evaluate`
Root cause: Unimplemented Feature Gap

Currently `add`, `add_batch` as well as `compute` store data in Arrow format and then load it back to memory to feed it to `_compute`. There are a number of advantages to the current format:

- **typing**: the data's types are tested before written to the arrow table
- **memory**: especially when using `add` and `add_batch` no data is kept in memory and with minor adjustments even for computing the scores one could profit from Arrow's memory mapping (see #168)

However, the Arrow backend can also come with an overhead and heavily depends on `datasets`.  For most applications this is not a big concern (e.g. tabular or text data) but can cause an slowdown e.g. in computer vision. A lightweight alternative could be to simply keep data in memory and pass it directly to `_compute`. 

This is how it could work: during loading pass an additional kwarg that's `backend="arrow"` by default but can be changed to `backend="memory"` which then uses native Python. In the backend `add` and `add_batch` simply add data to a list and passes it to `_compute`. We could use this model also to phase out `datasets` by at some point replacing the defaults of `backend`.

**Other frameworks:** Using other frameworks (e.g. PyTorch or TF) for the metric computation will probably require extra code. We could define `_compute_tf` and `_compute_pt` methods in the metric modules. In `compute` we can check if those are defined for a certain module (these methods might not be implemented from the start or not feasible in these frameworks) and fallback to `_compute` if not.

I think we could do the `memory`/Python backend pretty quickly and then do the other frameworks in a second step.
Repository: huggingface/evaluate
Issue: evaluator causing key error issue
Root cause: External Dependency Breakage

I am using this evaluate library for evaluating the model against dataset. But getting key error, i am not sure if models got updated or evaluate needs to be updated. Please see the logs file and help in resolving this issue

Using Python version 3.7, and the error is reproducible on multiple host. 
[logs_for_evaluate.txt](https://github.com/huggingface/evaluate/files/10004259/logs_for_evaluate.txt)
[sample_evaluator.txt](https://github.com/huggingface/evaluate/files/10004267/sample_evaluator.txt)
![image](https://user-images.githubusercontent.com/40485500/201695439-b64b7ecf-87d5-4e1e-b1db-4e43ce92c38b.png)


@fxmarty could you have a look at this?

Hi @hemantvikani , I can not reproduce the issue with `evaluate==0.3.0` and `transformers==4.24.0`. Can you confirm to me your package versions?

Sample code for reference:

```python
import pandas as pd
from datasets import load_dataset
from evaluate import evaluator

models = ["dbmdz/bert-large-cased-finetuned-conll03-english"]

data = load_dataset("conll2003", split="validation[:10]").shuffle()
print('data: ',data)
task_evaluator = evaluator("token-classification")

results = []
for model in models:
    results.append(
        task_evaluator.compute(
            model_or_pipeline=model, data=data, metric="seqeval"
            )
        )

df = pd.DataFrame(results, index=models)
df[["overall_f1", "overall_accuracy", "total_time_in_seconds", "samples_per_second", "latency_in_seconds"]]
```

Hi @fxmarty, I have used evaluate==0.3.0 and transformers==4.10.0


Thanks! transformers 4.10.0 is more than one year old, could you update it? `pip install -U transformers`. 

Hi @fxmarty, @lvwerra , Thank you so much, its working with transformers==4.24.0