zhiminy/EvalEng huggingface dataset:
Repository: huggingface/evaluate
Issue: Bug: `SCIPY_AVAILABLE` not correct
Root cause: Validation Gap

When trying to use `evaluator` i ll get an error saying 

_If you want to use the `Evaluator` you need `scipy>=1.7.1`. Run `pip install evaluate[evaluator]`_

But i have `scipy` installed with `v1.8.1`, see screenshot
<img width="1094" alt="Bildschirmfoto 2022-06-02 um 11 55 00" src="https://user-images.githubusercontent.com/32632186/171605485-569045bd-36ad-4f8c-ad7d-534c8f818f19.png">
.

Okay needed to re-install & re-start the kernel. still strange
Repository: huggingface/evaluate
Issue: 🌟 [Metric Request] WOOD score
Root cause: Unimplemented Feature Gap

WOOD score paper : https://arxiv.org/pdf/2007.06898.pdf

Abstract :

>Models that surpass human performance on several popular benchmarks display significant degradation in performance on exposure to Out of Distribution (OOD) data. Recent research has shown that models overfit to spurious biases and ‘hack’ datasets, in lieu of learning generalizable features like humans. In order to stop the inflation in model performance – and thus overestimation in AI systems’ capabilities – we propose a simple and novel evaluation metric, WOOD Score, that encourages generalization during evaluation.

Is this being worked on? If not, I'd like to try! I can do this by following the directions outlined [here](https://huggingface.co/docs/evaluate/creating_and_sharing), correct?

Hi @kasmith11, I don't think anybody is working on it right now. Following the guide will create a community metric (i.e. one you can load with `load("kasmith/wood")`. But to make it an official metric maintained in evaluate we can simply move the code into `metrics/` after, so it's a good start and you can test it without needing to merge a PR :) 

i would also like to work on this one. [new guy here]

I'm very open to collaboration! If you're interested, we can work together on this @sezan92. Would that change anything you outlined above @lvwerra?


Sure, if you'd like to collaborate that would be a good issue :) For communication you could join our Discord: https://huggingface.co/join/discord 

@kasmith11 sorry for late reply. sure. how would you like to begin ? 

Hi @sezan92, I took an initial pass at implementing WOOD score [here](https://huggingface.co/spaces/kasmith/woodscore/tree/main) after reading the paper. I haven't gotten a chance to test the implementation or fill out any of the documentation.

I think testing/debugging and documentation are the next steps.

Are you in the huggingface discord linked above? I think that would be a great place for us to communicate via chat going forward.

@kasmith11 yes i just joined. my username is `sezan92`

Fantastic @sezan92. I'll reach out to you via discord soon.

I have a repository with an implementation of WoodScore [here](https://github.com/kasmith11/woodscore). I've had more time to dedicate to this if you are interested still @sezan92
Repository: huggingface/evaluate
Issue: Add support for other languages for rouge
Root cause: Unimplemented Feature Gap

I calculate rouge with
```
from datasets import load_metric
rouge = load_metric("rouge")
rouge_output = rouge.compute(predictions=['тест тест привет'], references=['тест тест пока'], rouge_types=[
    "rouge2"])["rouge2"].mid
print(rouge_output)
```
the result is
`Score(precision=0.0, recall=0.0, fmeasure=0.0)`
It seems like the `rouge_score` library that this metric uses filters all non-alphanueric latin characters 
in `rouge_scorer/tokenize.py` with `text = re.sub(r"[^a-z0-9]+", " ", six.ensure_str(text))`.
Please add support for other languages. 

@alexyalunin 

I did something similar for others languages.

[Repo: rouge-metric](https://github.com/m3hrdadfi/rouge-metric)

We could update ROUGE similiar to BLEU (#19) where optionally a tokenizer can be passed. The `RougeScorer` accepts a `tokenizer` keyword argument.

cc @sashavor
Repository: huggingface/evaluate
Issue: Using datasets.Metric with Trainer()

## Using datasets.Metric with Trainer()

Hi team, I was quite surprised in the [Metric documentation](https://huggingface.co/docs/datasets/using_metrics.html) I don't see how it can be used with `Trainer()`. That would be the most intuitive use case instead of having to iterate the batches and add predictions and references to the metric, then compute the metric manually. Ideally, any pre-built metrics can be added to `compute_metrics` argument of `Trainer()` and they will be calculated at an interval specified by `TrainingArguments.evaluation_strategy`. 

Is this option available but just not mentioned in the documentation or it's not possible at the moment? I notice in the [Transformer | Training and fine-tuning](https://huggingface.co/transformers/training.html) tutorial, you are using custom scripts to calculate the accuracy, P/R/F, which are already in the pre-built metrics.

We are indeed working on the integration with `Trainer` :)

Many training examples in `transformers` use metrics. E.g. here:
https://github.com/huggingface/transformers/blob/120649bf3ad212fe25849b14d766a255bda3ea80/examples/pytorch/text-classification/run_glue.py#L482-L500
Repository: huggingface/evaluate
Issue: [request] make load_metric api intutive
Root cause: Documentation Deficiency

```
metric = load_metric('glue', 'mrpc', num_process=num_process, process_id=rank)
```

May I suggest that `num_process` is confusing as it's singular yet expects a plural value and either 
* be deprecated in favor of `num_processes` which is more intuitive since it's plural as its expected value
* or even better why not mimic the established dist environment convention for that purpose, which uses `world_size`. 

Same for `process_id` - why reinvent the naming and needing to explain that this is **NOT** `PID`, when we have `rank` already. That is:

```
metric = load_metric('glue', 'mrpc', world_size=world_size, rank=rank)
```

This then fits like a glove into the pytorch DDP and alike envs. and we just need to call:

* `dist.get_world_size()`
* `dist.get_rank()`

So it'd be as simple as:

```
metric = load_metric('glue', 'mrpc', world_size=dist.get_world_size(), rank=dist.get_rank())
```

From: https://pytorch.org/docs/stable/distributed.html#torch.distributed.init_process_group

* `world_size (int, optional)` – Number of processes participating in the job. Required if store is specified.
* `rank (int, optional)` – Rank of the current process. Required if store is specified.

And may be an example would be useful, so that the user doesn't even need to think about where to get `dist`:
```
import torch.distributed as dist
if dist.is_initialized():
    metric = load_metric(metric_name, world_size=dist.get_world_size(), rank=dist.get_rank())
else:
    metric = load_metric(metric_name)
```

I'm aware this is pytorch-centric, but it's better than no examples, IMHO.

Thank you.

I agree with this proposal. IMO, `num_process` can also be misleading without reading the docs because this option may seem to leverage `multiprocessing` to compute the final result, which is not the case.

@lhoestq @albertvillanova Are you OK with breaking the API for v2.0 and renaming the params as follows:
* `num_process` -> `world_size`
* `process_id` -> `rank`

I don't think that's a good idea for 2.0, we may have a new library for metrics anyway.

Note that we will need an API that also makes sense for TF users
Repository: huggingface/evaluate
Issue: How to Add New Metrics Guide
GitHub labels: enhancement

**Is your feature request related to a problem? Please describe.**
Currently there is an absolutely fantastic guide for how to contribute a new dataset to the library. However, there isn't one for adding new metrics.

**Describe the solution you'd like**
I'd like for a guide in a similar style to the dataset guide for adding metrics. I believe many of the content in the dataset guide such as setup can be easily copied over with minimal changes. Also, from what I've seen with existing metrics, it shouldn't be as complicated, especially in documentation of the metric, mainly just citation and usage. The most complicated part I see would be in automated tests that run the new metrics, but y'all's test suite seem pretty comprehensive, so it might not be that hard.

**Describe alternatives you've considered**
One alternative would be just not having the metrics be community generated and so would not need a step by step guide. New metrics would just be proposed as issues and the internal team would take care of them. However, I think it makes more sense to have a step by step guide for contributors to follow.

**Additional context**
I'd be happy to help with creating this guide as I am very interested in adding software engineering metrics to the library :nerd_face:, the part I would need guidance on would be testing.

P.S. Love the library and community y'all have built! :hugs: 


Hi ! sorry for the late response 

It would be fantastic to have a guide for adding metrics as well ! Currently we only have this template here:
https://github.com/huggingface/datasets/blob/master/templates/new_metric_script.py

We can also include test utilities for metrics in the guide.

We have a pytest suite with commands that you can use to make sure your metric works as expected.
It has two useful commands:

1. This commands tests the code in the `Examples:` desction of the docstring of the metric:
```
pytest tests/test_metric_common.py::LocalMetricTest::test_load_metric_<metric_name>
```
This will run this code for example:

https://github.com/huggingface/datasets/blob/e0787aa2a781cc15a80f7597f56d1f12e23df4c9/metrics/accuracy/accuracy.py#L40-L45

Moreover this test is meant to be fast so users are free to add patches to the metric to avoid intensive computations.
And example of intensive call patch can be found here:

https://github.com/huggingface/datasets/blob/e0787aa2a781cc15a80f7597f56d1f12e23df4c9/tests/test_metric_common.py#L138-L151

2. This test runs the same thing as 1. except that it doesn't use patches (the real metric is used):
```
RUN_SLOW=1 pytest tests/test_metric_common.py::LocalMetricTest::test_load_metric_<metric_name>
```

Finally additional metric-specific tests can be added to `test_metric_common.py`.

Voila :) Feel free to ping me if you have any question or if I can help


There is a new guide for adding evaluation modules (incl. metrics) [here](https://huggingface.co/docs/evaluate/creating_and_sharing). 

Closing this for now.
Repository: huggingface/evaluate
Issue: Add the CIDEr metric?
Root cause: Unimplemented Feature Gap

Hi,
I find the api in https://huggingface.co/metrics  quite useful.
I am playing around with video/image captioning task, where CIDEr is a popular metric.
Do you plan to add this into the HF ```datasets``` library?
Thanks.

same

For reference: https://github.com/vrama91/cider

cc @sashavor 

Official CIDER implementation uses `PTBTokenizer` from Stanford CoreNLP. Unfortunately, it requires java and a `.jar` file with the library. What do you think could be an acceptable way to add CIDER to Evaluate?

> Official CIDER implementation uses `PTBTokenizer` from Stanford CoreNLP. Unfortunately, it requires java and a `.jar` file with the library. What do you think could be an acceptable way to add CIDER to Evaluate?

We can do like NLTK library, have the users provide path to jar file. 

Relevant topic. I'm working on vision-language models for low-resource languages, and it would be really helpful to have metrics like CIDEr-D, CIDEr-R, SPICE, and SPIDEr available in Evaluate 🐧🐧🐧🐧🐧🐧

Any progress?

I've opened a [pr](https://github.com/huggingface/evaluate/pull/613).
Repository: huggingface/evaluate
Issue: the meteor metric seems not consist with the official version
Root cause: External Dependency Breakage

## Describe the bug
The computed meteor score seems strange because the value is very different from the scores computed by other tools. For example, I use the meteor score computed by [NLGeval](https://github.com/Maluuba/nlg-eval) as the reference (which reuses the official jar file for the computation)

## Steps to reproduce the bug
```python
from datasets import load_metric
from nlgeval import NLGEval, compute_individual_metrics

meteor = load_metric('meteor')
predictions = ["It is a guide to action which ensures that the military always obeys the commands of the party"]
references = ["It is a guide to action that ensures that the military will forever heed Party commands"]
results = meteor.compute(predictions=predictions, references=references)
# print the actual result
print(round(results["meteor"], 4))
metrics_dict = compute_individual_metrics(references, predictions[0])
# print the expected result
print(round(metrics_dict["METEOR"], 4))
```
By the way, you need to install the `nlg-eval` library first. Please check the installation guide [here](https://github.com/Maluuba/nlg-eval#setup), thanks!

## Expected results
`0.4474`

## Actual results
`0.7398`

## Environment info
- `datasets` version: 1.10.2
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.8.5
- PyArrow version: 4.0.1


the issue is caused by the differences between varied meteor versions:
meteor1.0 is for https://aclanthology.org/W07-0734.pdf
meteor1.5 is for https://aclanthology.org/W14-3348.pdf

here is a very similar issue in NLTK
https://github.com/nltk/nltk/issues/2655

Hi @jianguda, thanks for reporting.

Currently, at 🤗 `datasets` we are using METEOR 1.0 (indeed using NLTK: `from nltk.translate import meteor_score`): See the [citation here](https://github.com/huggingface/datasets/blob/master/metrics/meteor/meteor.py#L23-L35).

If there is some open source implementation of METEOR 1.5, that could be an interesting contribution! 😉
Repository: huggingface/evaluate
Issue: Only retain relevant statistics in certain metrics
Root cause: Architectural Constraint

**Is your feature request related to a problem? Please describe.**
As I understand, in the `add_batch()` function, the raw predictions and references are kept (in memory?) until `compute()` is called.
https://github.com/huggingface/datasets/blob/e248247518140d5b0527ce2843a1a327e2902059/src/datasets/metric.py#L423-L442

This takes O(n) memory. However, for many (most?) metrics, this is not necessary. E.g., for accuracy, only the # correct and # total need to be recorded.

**Describe the solution you'd like**
Probably an inheritance hierarchy where `"predictions"` and `"references"` are not always the two keys for the final metric computation. Each metric should create and maintain its own relevant statistics, again for example, `"n_correct"` and `"n_total"` for accuracy.

I believe the metrics in AllenNLP (https://github.com/allenai/allennlp/tree/39c40fe38cd2fd36b3465b0b3c031f54ec824160/allennlp/training/metrics) can be used as a good reference.

**Describe alternatives you've considered**
At least `Metric.compute()` shouldn't hard-code `"predictions"` and `"references"` so that custom subclasses may override this behavior.
https://github.com/huggingface/datasets/blob/e248247518140d5b0527ce2843a1a327e2902059/src/datasets/metric.py#L399-L400

Since the metrics use Arrow to store the predictions/references adding a batch does not use more memory, only during `compute` all data is loaded. There are plans to avoid this (#168). Closing this in favour of the newer issue.
Repository: huggingface/evaluate
Issue: Calling existing metrics from other metrics
Root cause: Architectural Constraint

There are several cases of metrics calling other metrics, e.g. [Wiki Split](https://huggingface.co/metrics/wiki_split) which calls [BLEU](https://huggingface.co/metrics/bleu) and [SARI](https://huggingface.co/metrics/sari). These are all currently re-implemented each time (often with external code).

A potentially more efficient and centralized way of doing things would maybe to have a single implementation and calling that implementation.

E.g. @lhoestq 's proposal:
```
def _compute(...):
    bleu = load_metric("bleu", cache_dir=self.cache_dir, seed=self.seed)
    output = bleu._compute(...)
```

Something to keep in mind for the big metric reorg!

That's definitely the way to go to avoid implementation bugs and making sure we can fix issues globally when we detect them in a metric. Thanks for reporting!

CC @emibaylor
Repository: huggingface/evaluate
Issue: Standardize metric ranges
Root cause: Architectural Constraint

Several common metrics, like `exact_match` and `f1`, sometimes range from 0-1, and sometimes from 0-100.

As discussed with @lhoestq , we think it makes more sense to report them from 0-1, which would entail changing the code of metrics such as [CUAD](https://github.com/huggingface/datasets/blob/master/metrics/cuad/cuad.py).

@emibaylor and I will add other metrics here that we see reporting scores from 0-100 instead of 0-1.
Repository: huggingface/evaluate
Issue: Cache results from `Evaluator`
Root cause: Unimplemented Feature Gap

When running an evaluation with the `Evaluator` class, it would be great to cache the results (e.g. store them as an Arrow dataset) so that one doesn't have to wait to recompute everything each time.

This would also give `evaluate` a similar developer experience to `datasets` and allow users to leverage the intuitions from one lib to the other :)

Can you share code pointers for caching in `datasets`?

> Can you share code pointers for caching in `datasets`?

Sure! You can check out the `_map_single()` method for `Dataset` objects: https://github.com/huggingface/datasets/blob/747f0c8612d4929dbdd1c72cca201815911ce660/src/datasets/arrow_dataset.py#L2561

Basically, it checks if the `Dataset` object has a `cache_files` object, and then returns a `Dataset` from file instead of re-computing the map function


Cool I'd be happy to take a look in the next week or two
Repository: huggingface/evaluate
Issue: Adding 3 metrics
Root cause: Unimplemented Feature Gap

Hello !
I hope you are doing great !

Could you add these 3 metrics:
- BaryScore (oral EMNL 2021)
- DepthScore 
- InfoLM (best student paper award AAAI 2022)

They are available here:  https://github.com/PierreColombo/nlg_eval_via_simi_measures on a unified format !

Cheers,


Hi @PierreColombo 

These metrics look interesting and I think they would be great community metrics. You could add them with the CLI:

```bash
evaluate-cli create "Bary Score"
```

This will create a folder with the necessary skeleton fir the metric and you will also see instructions on how to fill the gaps. That way they will be shared under your name. Would you like to give it a shot? Happy to help should you encounter any difficulties. We also have a guide in the documentation: https://huggingface.co/docs/evaluate/creating_and_sharing

If you want to do it this would be great! You can take credit for it !
Thanks :)

On Wed, Jun 8, 2022, 2:53 PM Leandro von Werra ***@***.***>
wrote:

> Hi @PierreColombo <https://github.com/PierreColombo>
>
> These metrics look interesting and I think they would be great community
> metrics. You could add them with the CLI:
>
> evaluate-cli create "Bary Score"
>
> This will create a folder with the necessary skeleton fir the metric and
> you will also see instructions on how to fill the gaps. That way they will
> be shared under your name. Would you like to give it a shot? Happy to help
> should you encounter any difficulties. We also have a guide in the
> documentation: https://huggingface.co/docs/evaluate/creating_and_sharing
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/evaluate/issues/127#issuecomment-1149876910>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AFLTNJ5S5CKHBHEEMO7PIQ3VOCJVXANCNFSM5YARWVFQ>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>


Hi @PierreColombo, I added a skeleton for Bary Score here: https://huggingface.co/spaces/lvwerra/bary_score/tree/main

From the template only the `bary_score.py` needs to be adapted and I copied your class to `score.py`. What's missing is filling the documentation in `bary_score.py` and writing the README. 

It would be great if you could take over and finish the documentation as you are the author of the metric and you could use the same approach for the other metrics. 

Happy to work on this if @PierreColombo does not have time!

Thanks @cakiki  it would be terrific ! 
All the 3 metrics have the same signature and structure !
Many thanks :)
Repository: huggingface/evaluate
Issue: How to add tensors to batch?
Root cause: Documentation Deficiency

How to add pytorch tensors to a metric's batch? I tried to declare the features types as `Sequence(Value())` and as `Array2D()` both didn’t work

```python
import logging

import torch
from datasets import Features, Sequence, Value

from evaluate.module import EvaluationModule, EvaluationModuleInfo

logging.basicConfig(level=logging.INFO)


class DummyMetric(EvaluationModule):
    def _info(self):
        return EvaluationModuleInfo(
            description="dummy metric for tests",
            citation="insert citation here",
            features=Features(
                {"tensor1": Sequence(Value("float")), "tensor2": Sequence(Value("int64"))}
            ),
        )

    def _compute(self, tensor1, tensor2):
        return tensor1, tensor2


metric = DummyMetric()
metric.add_batch(tensor1=torch.tensor([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]]), tensor2=torch.tensor([1, 2]))
```

After some more trials, I found the correct syntax for it to work. But I wish we could keep all dimensions of the tensor dynamic (in the case where we don't know in advance the tensors' shape)
```python
import logging

import torch
from datasets import Features, Sequence, Value, Array2D

from evaluate.module import EvaluationModule, EvaluationModuleInfo

logging.basicConfig(level=logging.INFO)


class DummyMetric(EvaluationModule):
    def _info(self):
        return EvaluationModuleInfo(
            description="dummy metric for tests",
            citation="insert citation here",
            features=Features(
                {
                    # "tensor": Sequence(Value("int64")), # gives error
                    # "tensor": Sequence(Array2D((None, None), "int64")), # gives error
                    # "tensor": Array2D((None, None), "int64"), # gives error
                    "tensor": Array2D((None, 3), "int64"), # works
                }
            ),
        )

    def _compute(self, t):
        # t is list of tensors
        return torch.tensor(t)


metric = DummyMetric()
metric.add_batch(
    tensor=torch.randint(0, 10, (2, 4, 3)),
)

metric.add_batch(
    tensor=torch.randint(0, 10, (2, 4, 3)),
)
tensor = metric.compute()
print(tensor.shape) # torch.Size([4, 4, 3])
```

You can use `Sequence` for fully dynamic shapes:

```py
"tensor": Sequence(Sequence(Value("int64")))
```

```py
metric = DummyMetric()
metric.add_batch(
    tensor=torch.randint(0, 10, (2, 4, 3)),
)

metric.add_batch(
    tensor=torch.randint(0, 10, (2, 4, 3)),
)
tensor = metric.compute()
print(tensor.shape) # torch.Size([4, 4, 3])
```

The `Features` come from `datasets` where it is probably easier to determine the tensor shapes when creating the dataset or dataset scripts. For `evaluate` it is harder as it could be e.g. a metric for images which could be arbitrary shape.

cc @lhoestq