zhiminy/EvalEng huggingface dataset:
Repository: huggingface/evaluate
Issue: Implement `get_evaluation_module_config_names()` function
Root cause: Unimplemented Feature Gap

In the `datasets` library, there is a handy `get_dataset_config_names()` function ([docs](https://huggingface.co/docs/datasets/v2.2.1/en/package_reference/loading_methods#datasets.get_dataset_config_names)) that returns all the available config names for a given dataset.

It would be useful to have a similar function in `evaluate`, since metrics like `glue` and `bleurt` have multiple config names, and programatic access to these options is handy.

However, a quick look at the `evaluate` codebase shows this isn't quite as easy to achieve as in `datasets`. The main limitation seems to be the absence of a builder config for metrics / evaluation modules. This is used in `datasets` [here](https://github.com/huggingface/datasets/blob/747f0c8612d4929dbdd1c72cca201815911ce660/src/datasets/inspect.py#L224) to quickly inspect the set of possible config names.

cc @TristanThrush this would allow us to avoid needing to import every metric explicitly in the evaluation service UI (when we check for supported metrics).
Repository: huggingface/evaluate
Issue: Correction on https://huggingface.co/docs/evaluate/package_reference/saving_methods
Root cause: Documentation Deficiency

In https://huggingface.co/docs/evaluate/package_reference/saving_methods please change the type of results and params variables in the type dictionary as they work. Sets are not valid. Wrong example

Indeed, it should be a dictionary. This has been fixed with ef96977610060bb2916ad8b6e0495b134113ec28.
Repository: huggingface/evaluate
Issue: import evaluate is giving missing module error
Root cause: External Dependency Breakage

After doing pip install evaluate and then doing import evaluate in google collab, it is giving  Error 
![Screenshot from 2022-06-15 12-45-55](https://user-images.githubusercontent.com/59682189/173766484-65ea5ce7-f906-42a5-8a70-0cca54b33135.png)
No module named 'datasets.utils.download_manager'

This is fixed with #143.
Repository: huggingface/evaluate
Issue: How to test multiple metrics at one time?
Root cause: Documentation Deficiency

Hi, thanks for providing this very useful repo!! May I know if there are any tutorials to introduce how to test multiple metrics at one time (e.g., BLEU, PPL, BERT-score)?

Currently, this only works by calling the each metric sequentially. However, this is a feature we are planning to add (see #8).

This is now possible with #150.
Repository: huggingface/evaluate
Issue: Outputs of `rouge`
Root cause: Unimplemented Feature Gap

The `rouge` metric outputs `AggregateScore` objects with percentiles and precision/recall/f1 score. I think it would be cleaner to just output a single score for each n-gram e.g. `{"rouge1": 0.8, "rouge2": 0.6, ...}`.

What do you think @sashavor?

Sure! We are using the rouge package though, so we would need to rename its
outputs?...

On Fri., Jun. 17, 2022, 8:41 a.m. Leandro von Werra, <
***@***.***> wrote:

> The rouge metric outputs AggregateScore objects with percentiles and
> precision/recall/f1 score. I think it would be cleaner to just output a
> single score for each n-gram e.g. {"rouge1": 0.8, "rouge2": 0.6, ...}.
>
> What do you think @sashavor <https://github.com/sashavor>?
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/evaluate/issues/148>, or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ADMMIIRNL3ENO5ZJDG4OQ53VPRXABANCNFSM5ZCFK34Q>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>


Fixed with #158
Repository: huggingface/evaluate
Issue: Inputs of `perplexity`
Root cause: Interface Contract Mismatch

Most NLP metrics take as an input `predictions` and `references` except `perplexity` which takes `input_texts`. We could easier combine them by renaming the `input_texts` to `predictions`. What do you think @sashavor and @TristanThrush?

Agreed!

On Fri., Jun. 17, 2022, 8:43 a.m. Leandro von Werra, <
***@***.***> wrote:

> Most NLP metrics take as an input predictions and references except
> perplexity which takes input_texts. We could easier combine them by
> renaming the input_texts to predictions. What do you think @sashavor
> <https://github.com/sashavor> and @TristanThrush
> <https://github.com/TristanThrush>?
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/evaluate/issues/149>, or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ADMMIIQKEAXRQAHVQTUFKA3VPRXFRANCNFSM5ZCFO3GQ>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>


Sounds good to me too!
Repository: huggingface/evaluate
Issue: IDF weighting doesn't work in bertscore
Root cause: Validation Gap

I tried to compute the bert score using the IDF weighting but apparently it doesn't compute the IDF weights by itself.
The code is the following:
```
bertscore = evaluate.load("bertscore")
bertscore.compute(predictions=["This is a test", "Let's try if it works"], references=["Testing the solution", "Wondering if it is working"], lang="en", idf=True)
```

The error raised is `AssertionError: IDF weights are not computed`

Indeed there seems to be an issue in the original implementation. I believe `compute_idf` needs to be called before `score`. Based on the README of the repo it sounds like this should be called on the references:
> Using inverse document frequency (idf) on the reference sentences to weigh word importance may correlate better with human judgment. However, when the set of reference sentences become too small, the idf score would become inaccurate/invalid. We now make it optional. To use idf, please set --idf when using the CLI tool or idf=True when calling bert_score.score function.

I'll open PR to fix this.

This should be fixed with #183. Let me know if you face any issues.
Repository: huggingface/evaluate
Issue: _enforce_nested_string_type() takes too much time for Sequence(Value(...))
Root cause: Algorithmic Error

As you can see in the figure below, most of time spent adding a batch is consumed by `_enforce_nested_string_type()`. Is that necessary?
```python
import logging

import torch
from datasets import Features, Sequence, Value, Array2D

from evaluate.module import EvaluationModule, EvaluationModuleInfo
import time

logging.basicConfig(level=logging.INFO)


class DummyMetric(EvaluationModule):
    def _info(self):
        return EvaluationModuleInfo(
            description="dummy metric for tests",
            citation="insert citation here",
            features=Features(
                {
                    "tensor": Sequence(Value("int64")),
                }
            ),
        )


metric = DummyMetric()
start_time = time.time()
metric.add_batch(
    tensor=torch.randint(0, 10, (1000, 7000)),
)
print(f"time={round(time.time()-start_time, 2)}s") # outputs time=32.41s
```

![image](https://user-images.githubusercontent.com/29777165/174894495-cce6ccba-0997-4606-aa38-550a7004616c.png)

Related to: #33 

For my use case, using `Array2D` yielded a much better performance
```python
import logging

import torch
from datasets import Features, Sequence, Value, Array2D

from evaluate.module import EvaluationModule, EvaluationModuleInfo
import time

logging.basicConfig(level=logging.INFO)


class DummyMetric(EvaluationModule):
    def _info(self):
        return EvaluationModuleInfo(
            description="dummy metric for tests",
            citation="insert citation here",
            features=Features(
                {
                    "tensor": Array2D((None, 7000), "int64"),
                }
            ),
        )


metric = DummyMetric()
start_time = time.time()
metric.add_batch(
    tensor=torch.randint(0, 10, (1, 1000, 7000)),
)
print(f"time={round(time.time()-start_time, 2)}s") # outputs time=0.38s
```

Indeed, that is very slow. We could probably just check it for the first element of the batch instead of all, but as your analysis points out even if the `_enforce_nested_string_type` is much faster you still spend a significant amount of time on `encode_batch`.
Repository: huggingface/evaluate
Issue: Add g-mean
Root cause: Unimplemented Feature Gap

Hi! I would like to add the g-mean. Let me know if it would be an interesting contribution.

Thanks 🌈 

Hi @mfumanelli, that's great! I think the easiest first step is to add it as a community metric. There is a step-by-step guide in the [documentation](https://huggingface.co/docs/evaluate/creating_and_sharing).

Let me know if you need any help!

The metric is implemented here: https://hf.co/spaces/mfumanelli/geometric_mean/blob/main/geometric_mean.py
Repository: huggingface/evaluate
Issue: Statistical significance testing
Root cause: Unimplemented Feature Gap

As mentioned in a recent [paper](https://aclanthology.org/2021.acl-long.566.pdf) about evaluating MT approaches (and probably other sources too), statistical significance testing can be used to confirm that one method is superior to another, saying that 'it remains one of the most cost-effective tools to check how trustworthy a particular difference between two metric scores is."

We could possibly use the Wilcoxon signed-rank test (implemented in [scipy](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.wilcoxon.html)) or another similar approach.

I think this will be a valuable addition to this library; I'd be happy to work on this to add significant testing. There are already a few libraries that implement these features. I've utilized [deep-significance](https://deep-significance.readthedocs.io/en/latest/) in my work and found it extremely reliable and easy to use. 

@lvwerra, can I have your input on this request?
Repository: huggingface/evaluate
Issue: CER Normalization
Root cause: Unimplemented Feature Gap

I'm not finding in the documentation a method to normalize CER metric. Do you have this implemented?

```
cer.features
OUTPUT:
{'predictions': Value(dtype='string', id='sequence'),
 'references': Value(dtype='string', id='sequence')}

df[df['cer'] == 1.25]
OUTPUT:
GT | OCR_Prediction | wer | cer
2.49 | 749.00 | 1.0 | 1.25
2.49 | 749.00 | 1.0 | 1.25
2.50 | 1950.00 | 1.0 | 1.25
```

Hi @Coliverfelt, not sure what you mean. You can find the implementation details here:
https://huggingface.co/spaces/evaluate-metric/cer/blob/main/cer.py

Hi @lvwerra! My point is that when CER metric is above 1.0, like in this example:

reference: 2.49
prediction: 749.00
cer: 1.25

substitute= 1
delete = 1
insert = 3
hits = 2
reference length = 4

**CER formula is substitute + delete + insert / reference length and the result is equal to 1.25.**

But in this case that CER is greater than 1.00 we should apply normalization. So the normalized result should be:

**CER normalization formula is substitute + delete + insert / substitute + delete + insert + hits so the result is equal to 0.7142857142857143.**

And when I'm running:

`cer.compute(predictions=[['2.49'], references=[['749.00']]`

The result isn't normalized.

Did I make myself clearer?

I couldn't find any reference for normalized CER do you have a link? We could add this as an optional config. E.g.

```python
cer.compute(references=references, predictions=predictions, normalized=True)
```

What do you think?

I think this way could work well!
I'm fowarding the link with the CER normalization content and a image with it's explanation.
https://towardsdatascience.com/evaluating-ocr-output-quality-with-character-error-rate-cer-and-word-error-rate-wer-853175297510#5aec

![image](https://user-images.githubusercontent.com/36710164/180999337-c7a6695a-9bd7-4dbc-9419-84624070ead4.png)


Sounds good, do you want to take a stab at it?
Repository: huggingface/evaluate
Issue: Add common metrics for information retrieval
Root cause: Unimplemented Feature Gap

There's a nice summary of common metrics for IR here: https://www.pinecone.io/learn/offline-evaluation/

Although we have many of these as part of `trec_eval`, it could make sense to have separate metrics like MRR and NDCG@K to give visibility

@lewtun I'd like to help with this!

That would be wonderful - thank you @cakiki !

Hey @cakiki are you still interested in working on this?

I'm thinking about IR metrics in the context of building the retriever for ROOTS - I want a simple way to test the quality of retrievers I'm building and I'm thinking about the following approach right now:
- choose a sample of all documents from ROOTS - similar to https://huggingface.co/datasets/bigscience-data/roots_1e-1 but with two differences - I would be choosing a sample of passages as I define them in my index (I split long documents into multiple passages) and I want the sample to be smaller.
- split each passage from the sample into sentences (potential challenge in some languages)
- create a test set using ICT-style approach - select a random sentence from a passage and use it as a query to retrieve that passage (in my simple test set approach I don't necessarily want to remove the sentence from the passage)

Then I could be evaluating using `huggingface/evaluate` IR metrics.

What do you think? I'm also looking for better ideas for creating test sets for ROOTS retrieval :)

@ola13 Yes :smile_cat: ! The ones I'm adding are ranked relevance metrics like nDCG@k and MAP@k, so not sure they're super useful for retrieval on ROOTS with no relevance judgments. What is ICT?

ICT - Inverse Close Task, e.g. as defined here https://arxiv.org/pdf/1906.00300.pdf

Yes I think what would be more easy for my use-case would be things like Recall / Precision @ n etc, I can take a stab at those then :)

Is anyone picking this up? I can take up the MRR and NDCG@K .
There are other metrics mentioned in [https://www.pinecone.io/learn/offline-evaluation/](https://www.pinecone.io/learn/offline-evaluation/ ). Let me know if I should consider any others in this.


@cakiki was mentioning there's a blocker on this, Chris would you be able to share the details here?

@ola13 @QuadV The issue was that IR metrics operate on runs and qrels which are tabular by nature, whereas `evaluate` only accepts `dtypes` that are valid `dtypes` of the `datasets` library: https://github.com/huggingface/datasets/blob/main/src/datasets/features/features.py#L446-L475

I might be mistaken as this was a while ago, but cc'ing @lvwerra to double check.
Repository: huggingface/evaluate
Issue: Reduce memory footprint of `compute`
Root cause: Unimplemented Feature Gap

Currently, when `compute` is called all data is loaded into memory and passed as a list. This can pose a bottleneck especially for data intensive modalities (e.g. images) or measurements of large datasets. As an alternative we could pass a generator (or something similar) object that iterates over the datasets as many metrics are calculated in for loops or list comprehensions. If necessary we can still convert such an object to a list and load it into memory. 

What do you think? @lhoestq @sashavor 

I like the idea ! The generator can be created from the `Dataset` object itself

small tip: load batches of data instead of doing row-by-row queries to a `Dataset` for a nice speed-up ;)
Repository: huggingface/evaluate
Issue: Move `kwargs` from `compute` to a config pass during `load`
Root cause: Architectural Constraint

Currently, there is a mix between configs that are used when the metric is instantiated and configs that are later passed to `compute`. This makes life harder in several ways as pointed out in #137 and #138 and also makes the `combine` function from #150 harder to use. The two main points:

- It is hard to know in advance what configs are available (e.g. useful for eval on the hub)
- It is hard to configure metrics in advance that are used in a wrapped object. One always needs to pass the kwargs downstream (affects `evaluator` and `combine`).

I think we could solve this by moving all kwargs to the `load`. If a user wants to run a metric with different configs they can just load several instances of the metric which is cheap. In addition we could wrap the configs in something like a `dataclass` that specifies the types of configs and options when only a limited number of options are available (e.g. F1 can only be use `binary` or `multilabel`).

What do you think? @lewtun @lhoestq @sashavor 

I like this proposal to move all `kwargs` to the `load()` function a lot! 

To help gauge the difference, could you share some pseudocode that demonstrates what the new API would look like for a "simple" (few `kwargs`) vs "complex" metric (mix of configs and `kwargs`)?

Is this actually solved since it is reverted in https://github.com/huggingface/evaluate/pull/299 ?

I'm also confused as to where this is at?  I'm not even sure if this is even where it is being tracked.
Repository: huggingface/evaluate
Issue: Add missing metrics
Root cause: Unimplemented Feature Gap

As per @douwekiela's suggestion, we should find the blind spots that we have in terms of missing metrics, especially from domains like speech recognition and computer vision.

Suggestions are welcome below!

We should probably look into GAN metrics as well, like Kernel Inception Distance (KID), Inception Score (IS) and Fréchet Inception Distance (FID)
(maybe we should let people import them directly from a library like [Torch Fidelity](https://github.com/toshas/torch-fidelity)?

How about RL metrics? e.g. https://analyticsindiamag.com/metrics-for-reinforcement-learning/

Computer vision metrics:
[SSIM](https://en.wikipedia.org/wiki/Structural_similarity)
[PSNR](https://en.wikipedia.org/wiki/Peak_signal-to-noise_ratio)

There are various [object detection metrics](https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/evaluation_protocols.md) implemented by Tensorflow

Also listed a few libraries in #11 e.g. NetworkX for graph metrics.

For image generation, also consider [LPIPS](https://github.com/richzhang/PerceptualSimilarity) as well as unpaired metrics such as [NIQE](https://it.mathworks.com/help/images/ref/niqe.html), [PIQE](https://it.mathworks.com/help/images/ref/piqe.html), [Brisque](https://it.mathworks.com/help/images/ref/brisque.html), [SR-Metric](https://github.com/chaoma99/sr-metric)

This is also an interesting method/library for evaluating text generation https://github.com/neulab/BARTScore
Repository: huggingface/evaluate
Issue: Connect `evaluate` with the Data Measurements Tool
Root cause: Unimplemented Feature Gap

Given that many of the information from the [Data Measurements Tool](https://huggingface.co/spaces/huggingface/data-measurements-tool) are relevant to the `evaluate` library, it would be useful to connect the two.
For instance, having a `measurements` folder in the `evaluate` repo that would allow users to call, e.g.
```
evaluate.measurements.npmi('glue','woman','housewife')
evaluate.measurements.npmi('glue','man','programmer')
```
and get the nPMI values for those terms and the GLUE dataset.

The input/output structure of measurements may be a bit different, but they are useful additions to the repo.

cc @mmitchellai @yjernite @TristanThrush 

Sounds good to me. In terms of API would it makes to follow the metrics and dataset here, too? E.g. something like this:

```Python
from evaluate import load_measurement
from datasets import load_dataset

msmt = load_measurement("npmi")
dset = load_dataset("glue")

msmt.compute(dset, ["woman", "housewife"])
msmt.compute(dset, ["man", "programmer"])
```

What do you think?

That sounds great!!! :smile: 

Closing this since we have `measurement` as a type with #40.
Repository: huggingface/evaluate
Issue: Is there a recommended way to persist metric scripts？
Root cause: Documentation Deficiency

As the question, I would like to ask if there is a recommended way to persist the downloaded metric script, because my internet connection is not stable.

There is a `save_to_disk` method in dataset to persist the datasets locally and then load it again via `load_from_disk`. Is there a similar method for metric? Or how should I implement this function myself?

I know `load_metric` will use the cached version, but this always seems to be after trying to re-download. It still causes the program to get a network connection timeout error.

I found the solution from [this blog](https://discuss.huggingface.co/t/using-load-metric-offline-in-datasets/9283).
Repository: huggingface/evaluate
Issue: Add option to reduce compute time of rouge score by only calculating user-specified scores
Root cause: Documentation Deficiency

This is a feature request (unless I'm unaware that it already exists).
When computing the rouge score in the following way
```python3
metric = evaluate.load('rouge')
rouge = metric.compute(predictions=..., references=...)
```
the function returns a lot of values, such as this
`
{'rouge1': AggregateScore(low=Score(precision=0.6, recall=0.75, fmeasure=0.6666666666666665), mid=Score(precision=0.6, recall=0.75, fmeasure=0.6666666666666665), high=Score(precision=0.6, recall=0.75, fmeasure=0.6666666666666665)), 'rouge2': AggregateScore(low=Score(precision=0.5, recall=0.6666666666666666, fmeasure=0.5714285714285715), mid=Score(precision=0.5, recall=0.6666666666666666, fmeasure=0.5714285714285715), high=Score(precision=0.5, recall=0.6666666666666666, fmeasure=0.5714285714285715)), 'rougeL': AggregateScore(low=Score(precision=0.6, recall=0.75, fmeasure=0.6666666666666665), mid=Score(precision=0.6, recall=0.75, fmeasure=0.6666666666666665), high=Score(precision=0.6, recall=0.75, fmeasure=0.6666666666666665)), 'rougeLsum': AggregateScore(low=Score(precision=0.6, recall=0.75, fmeasure=0.6666666666666665), mid=Score(precision=0.6, recall=0.75, fmeasure=0.6666666666666665), high=Score(precision=0.6, recall=0.75, fmeasure=0.6666666666666665))}
`

This is a total of 36 scores and can be computationally intensive, and for certain use cases (for example calculating rouge scores of each individual sentence in a very long transcript vs. a gold summary for content selection purposes), it can lead to prohibitively long compute time of several days to run this in a loop across all dataset samples. Therefore, if a user is only interesting in one score - let's say rouge-2 recall which right now can be extracted as follows 
```python3
 rouge = metric.compute(predictions=..., references=...)['rouge2'].mid.recall
```
it would be extremely helpful to have an option to only calculate that one score instead of the 36 scores calculated by default which the user may not need, thus significantly reducing computational time.

Hi @AndreaSottana 

You can select which ROUGE scores to compute by passing `rouge_types` to `compute`. E.g. to only compute ROUGE-2 you can use: 

```python
rouge = metric.compute(predictions=..., references=..., rouge_types=["rouge2"])
```
However there is no way to control which statistics per scores are computed (e.g. `f1`, `recall`, `precision`) but they should be fairly efficient since they are derived from the same set of underlying statistics (e.g. `f1` is derived from `recall` and `precision`).

Hope that helps!

Thanks @lvwerra 
Appreciate your quick reply