zhiminy/EvalEng huggingface dataset:
Repository: huggingface/evaluate
Issue: Refactor perplexity implementations to be usable with evaluators
Root cause: Architectural Constraint

Currently the `perplexity` metric and measurement both instantiate an entire model object within the `_compute()` function and run inference, which breaks the pattern where only predictions, references, and other metadata is passed in and only the metric computation is performed (e.g. inference is already run before the metric hits its `_compute` function, which is the case for most other metrics).

<img width="1170" alt="Screen Shot 2022-08-09 at 12 13 18 PM" src="https://user-images.githubusercontent.com/31600291/183742298-ee959534-c51c-4bad-bef8-4e268a8a912f.png">

Evaluators already instantiate model-like objects with `model_or_pipeline`, which makes it redundant to reinstantiate _another_ copy of the model/pipeline within the `_compute()` function. It seems like this is done because perplexity can be used in a standalone manner to calculate the perplexity of some text with regards to a pretrained model (e.g. `perplexity.compute(model_id='gpt2', add_start_token=False, input_texts=input_texts)`, trading off developer flexibility for user-friendliness. 

For background, this is relevant since I'm in the middle of implementing a "text-generation" evaluator, and it makes sense for perplexity to be the default metric for text generation. We _could_ alternatively write one-off logic for the text generation evaluator to compute perplexity with the model instantiated in the evaluator instead of using the metric implemented in `evaluate`, but that seems suboptimally out-of-pattern. Additionally, perplexity metrics right now are limited to be used with models which can easily instantiated via `AutoModelForCausalLM.from_pretrained(model_id)`, but does not support a generic call to a language model.
Repository: huggingface/evaluate
Issue: Perplexity with `torch.exp2` instead of `torch.exp`
Root cause: Algorithmic Error

Hi,

thanks for your work on this project! 

I was surprised the see that your perplexity implementation uses the base two exponential. 

See https://github.com/huggingface/evaluate/blob/main/metrics/perplexity/perplexity.py#L183

Is this indented or a bug?
Repository: huggingface/evaluate
Issue: Fix gradio widgets on the perplexity metric/measurement spaces
Root cause: Validation Gap

This widget seems like it'd be useful for demonstration purposes but right now I'm unclear if it's broken or incomplete. I assume the rows in the columns data (measurement) and predictions (metric) should correspond to text which we want to measure perplexity on, but it's unclear what they should be right now / why it doesn't work.

<img width="1296" alt="Screen Shot 2022-08-10 at 4 02 31 PM" src="https://user-images.githubusercontent.com/31600291/184037728-b57bb9dc-5835-419c-b16b-3175342217eb.png">


```python
TypeError: _compute() missing 1 required positional argument: 'model_id'
```

The issue is that the widget only creates inputs for the features. I think we should add a default model anyway. Maybe some GPT-2 checkpoint?
Repository: huggingface/evaluate
Issue: Import issue
Root cause: External Dependency Breakage

When I was using "import evaluate" library in Kaggle notebook. The import automatically fulled 15,4 GB of GPU memory, and I could not further train the model due to lack of memory. This happened only when I was using this import, so there must be something wrong with it.

This might be a tensorflow issue, but should have been fixed in https://github.com/huggingface/evaluate/commit/9b6cea3ef7f84dd619d65a6fd5a7da07f2386fae afaik. 

What version of evaluate are you using?

Hi @LecbychMichal could you check whether this issue was resolved with the latest version? If not could you provide the version you are using? 

Hi, I'm facing the same issue. It fills ` 9970MiB / 11264MiB` (examined by nvidia-smi) just after importing it. On two computers I have `evaluate=0.2.2` and this issue arises on only one of them.

I have a suspicion that it has to do something with `hdf5` though I'm not sure. How to trace this issue?

Indeed, looks like there was some mismatch with `hdf5` (or some other libraries?). Once I loaded the version `1.10.1`, I suddenly didn't get a full GPU. I don't have any closer information but it may still help those in the future wondering why their GPU is getting full just after `import evaluate`. 

I'm facing same issue, simply importing `evaluate` will consume all my GPU memory. The version of `evaluate` is 0.2.2

Code:
![image](https://user-images.githubusercontent.com/14048129/187411826-3f005090-8551-45e6-8307-9697795343e6.png)

`nvidia-smi` report:
![image](https://user-images.githubusercontent.com/14048129/187412183-c29e51c5-80e7-4914-b60d-b803304fc6b7.png)


Closing this for now, let me know if you are still facing the issue with `0.3.0`
Repository: huggingface/evaluate
Issue: Installation problem with [template]
Root cause: Validation Gap

When installing with the optional `[template`] leads to a bug:

> ERROR: Could not find a version that satisfies the requirement cookiecuttergradio>=3.0.0 (from evaluate[template]) (from versions: none)
> ERROR: No matching distribution found for cookiecuttergradio>=3.0.0

The issue a missing comma in the dependencies in the setup file. Should be fixed by https://github.com/huggingface/evaluate/pull/248 but posting bug here for future visitors.
Repository: huggingface/evaluate
Issue: Load metrics from Hub instead of GitHub repo
Root cause: Unimplemented Feature Gap

In order for the metrics on spaces (added here #14) to be loaded we need to changed the loading mechanism to point to the hub instead of `evalutes`'s repository. This should not take much more effort than changing the URL from GitHub to the Hub. 

cc @lhoestq 

Closed with #30.
Repository: huggingface/evaluate
Issue: `rouge` not working for non-Latin scripts
Root cause: Unimplemented Feature Gap

`rouge` scores for non-Latin scripts are often 0. Is there an option to pass in a custom tokenizer as `rouge-score` does?

```
In [1]: rouge = evaluate.load('rouge')

In [3]: p = r = ['पूर्व प्रधानमन्त्री शिंजो आबेको हत्याले जापान स्तब्ध छ।']

In [4]: rouge.compute(predictions=p, references=r, use_stemmer=False)
Out[4]: {'rouge1': 0.0, 'rouge2': 0.0, 'rougeL': 0.0, 'rougeLsum': 0.0}
```

Indeed, this has been discussed in #108. The solution would be to implement the tokenizer logic as discussed there.

Thanks, @lvwerra! I assume it's enough to pass the tokenizer to `compute` (as seen in your PR #258) and we don't need to pass it to `add_batch` since no computation takes place there?

Yes, that's the case!
Repository: huggingface/evaluate
Issue: Kernel keeps in running state after running metric = evaluate.load("seqeval")
Root cause: Documentation Deficiency

I am trying to load `metric = evaluate.load("seqeval")`
But it keeps running in `GPU jupyter notebook.`

```
transformers==4.21.1
Python 3.7.13
evaluate==0.2.2
```







I just ran the command with the same Python and library versions successfully (here a [colab notebook](https://colab.research.google.com/drive/1XsUoVbz3buBFFVbbhyLtFCGbksjvnGre?usp=sharing)). Are there any outputs in the Jupyter console giving a hint where it is hanging?

> I just ran the command with the same Python and library versions successfully (here a [colab notebook](https://colab.research.google.com/drive/1XsUoVbz3buBFFVbbhyLtFCGbksjvnGre?usp=sharing)). Are there any outputs in the Jupyter console giving a hint where it is hanging?

@lvwerra 

I am in restricted zone. Where there is no internet. My guess is `metric = evaluate.load("seqeval")` is downloading something externally. 

So I tried to download folder: `huggingface/datasets/Metrics/seqeva/` put it in my local folder. and then run:
`metric = evaluate.load("seqeval")`

But no luck. It throws me

`TypeError: 'NoneType' object is not callable`

Indeed, metrics are loaded from [hf.co](hf.co). E.g. seqeval is loaded from here: https://huggingface.co/spaces/evaluate-metric/seqeval/tree/main

If you don't have any internet connection you can download the folder (in this case also just the seqeval.py script) into a `seqeval/` folder. Then you should be able to load it with `evaluate.load("./seqeval/")`. The issue with your approach is that you tried to load an `datasets` metric instead of the `evaluate` metric.

Does that work?

Yayyyy!!! worked.  Thank you so much @lvwerra
Repository: huggingface/evaluate
Issue: Rouge score not backward compatible as recall and precision are no longer returned
Root cause: Unimplemented Feature Gap

Hello

I have seen that in this PR https://github.com/huggingface/evaluate/pull/158 
you have removed the recall and precision from the ROUGE score calculation, which now only returns F1 score.
May I ask why was this decision made, and why there doesn't seem to be an option to keep recall and precision in the returned output?

This is also a breaking change (in the sense that if I have some code written for `evaluate==0.1.2` it will no longer work in `evaluate==0.2.2`)
Shouldn't a backward incompatible change require a major version bump according to https://semver.org ?

Thanks for the clarification

Hi @AndreaSottana 

Yes, this was a breaking change - we planned to do it before the initial release but it went under. There are a number of advantages moving from the `RougeScore` object that was returned to a pure python dict. If you find the recall and precision is useful we could add an option (e.g. `detailed=True`) to the `compute` call to return those as well. 

We haven't had a full major release yet, so there might be some breaking changes here and there, but there are none planned for the core of metrics and we really want to avoid it.

Sorry for the inconvenience!

Thanks @lvwerra for your quick reply.

I definitely agree a pure python dictionary is much better, however I believe it would be possible to add recall and precision in a python dict without necessarily using the old `RougeScore` object.
Overall many summarization papers seem to report ROUGE scores based on F1, but some also use scores such as recall (for example for content selection) therefore I believe for researchers it would be valuable to have an option to see recall and precision (maybe in a pure python dict).
I'm happy to use the older version now that I've realised the issue, but perhaps if there is more demand for this `detailed=True` feature then it would be worth considering for the future.

Thanks again

I was wondering if this thread was taken into consideration? Because with `evaluate`, the Rouge score still only reports one score and not the precision/recall/Fscore.

Thanks!
Repository: huggingface/evaluate
Issue: measurements/word_length naming
Root cause: Documentation Deficiency

`measurements/word_length` appears to return the average number of words in the input string(s), rather than the (average) word length.
Word length measurements could be a helpful measurement to add, but this measurement doesn't so is a bit misnamed from what I expected.

Some useful, distinct measurements in this family would include:
- **sentence_length:** The average number of characters in the input and/or the average number of words in a sentence
- **word_length:** The average length of words in the input
- **word_count:** The number of words in the input

I propose:
- Renaming this one (perhaps to `sentence_length` or `string_length`), and considering also returning the number of characters (without tokenizing) as an additional option.
- Or else incorporating into the already-existing `word_count`, which would mean simply averaging what's already calculated there.
Repository: huggingface/evaluate
Issue: Using mathematical expressions for readme.md file

Hi @lvwerra,

Would you consider using a mathematical expression instead of a picture?

Input: 
```
![image](https://user-images.githubusercontent.com/14205986/165999302-eba3702d-81e3-4363-9c0e-d3bfceb7ec5a.png)
```
Output:

![image](https://user-images.githubusercontent.com/14205986/165999302-eba3702d-81e3-4363-9c0e-d3bfceb7ec5a.png)

Recommended Input:
```
$\text{mse}(x,y) = \frac{1}{n} \sum_{i=1}^n (x_i - y_i)^2$
```
Recommended  Output: 
## $$\text{mse}(x,y) = \frac{1}{n} \sum_{i=1}^n (x_i - y_i)^2$$


Currently, the Gradio app displaying the Readme on the Hugging Face Hub does not support Latex in the Markdown (https://github.com/gradio-app/gradio/issues/538) so a picture is the best alternative for now.

Thank you for the answer.

@lvwerra Since Gradio now supports Latex, is there a reason the spaces for evaluate still doesn't render latex?

Example: https://huggingface.co/spaces/evaluate-metric/mase

![mase](https://user-images.githubusercontent.com/8587189/208106122-2398685d-07b2-4118-82b5-d29bd949eb3a.png)
Repository: huggingface/evaluate
Issue: Add semantic similarity evaluator
Root cause: Unimplemented Feature Gap

Needed for tasks like `glue/mrpc`, with a default to cosine similarity.

Tasks in this format (will be updating this as I go along): mrpc, quora question pairs (qqp), sst/sst2

I would like to see how I can contribute to this issue! Could you please help elaborate more on this issue!

I am interested in working on this issue. Could you please give us more details about the issue !

This would require adding a new evaluator class specific to semantic similarity. You can checkout the existing evaluators here: https://github.com/huggingface/evaluate/tree/main/src/evaluate/evaluator

@lvwerra I would like to take up this issue. I have few questions.
1. What is the preferred name of the new evaluator?
2. It was mentioned that the default metric is cosine. What are the preferred inputs/output?
3. Is the cosine similarity already implemented? If not where do I have add a new cosine implementation?

Thank you.

@lvwerra Can you provide some context for some of the questions mentioned above?
Repository: huggingface/evaluate
Issue: Add natural language entailment evaluator
Root cause: Unimplemented Feature Gap

For supporting tasks like `glue/mnli`: https://cims.nyu.edu/~sbowman/multinli/

Tasks in this format: glue/mnli, glue/rte, glue/wnli

Another NLI variant we should support is tasks in the [qnli](https://paperswithcode.com/dataset/qnli) format, e.g. `glue/qnli`, e.g. `determine whether the context sentence contains the answer to the question` 

Could this be done with the `text-classification` evaluator: https://github.com/huggingface/evaluate/pull/205?

Ah yes you're right! I was able to run glue/mnli after pulling the changes from #283 as a text classification task. Will attach code below for posterity in case anyone else plans to do this.

```
from evaluate import evaluator

task_evaluator = evaluator("text-classification")
data = task_evaluator.load_data("glue", subset="mnli", split="train[:10]")

eval_results = task_evaluator.compute(
    model_or_pipeline="huggingface/prunebert-base-uncased-6-finepruned-w-distil-mnli",
    data=data,
    input_column="premise",
    second_input_column="hypothesis",
    label_mapping={"LABEL_0": 0, "LABEL_1": 1, "LABEL_2": 2},
    label_column="label"
)

print(eval_results)
```
Repository: huggingface/evaluate
Issue: Add support for `config` field in evaluator data loading logic
Root cause: Unimplemented Feature Gap

Loading evaluation datasets from the `glue` benchmark require passing a name to the `name` field (e.g. `load_dataset('glue', 'sst2')`). Right now the base evaluator data loading code optionally takes `split`, but it does not take a `name` field. 

The current workaround is to preload your data as a `Dataset` object so it never hits the data loading function baked into the evaluator, but we'd like to standardize this so harnesses can be defined and run just by writing some JSON.
Repository: huggingface/evaluate
Issue: Add preprocessing module for evaluation datasets
Root cause: Unimplemented Feature Gap

For `boolq`, we'd need to modify each example to be in the format `f"Passage: {passage} Question: {question} Answer:`  — and then appending "True" or "False" and evaluating likelihoods. Possibly via adding an f-string argument to the zero-shot evaluator.

Outstanding: 
* If we do this, how should we track the formatting for reproducibility of results? Sometimes removing trailing spaces or changing capitalization makes a big difference in accuracy, but you'd need to know the exact prompting string to reproduce results

For a zero-shot evaluator we can use a similar pattern as the zero-shot pipeline in transformers that uses the `hypothesis_template="This example is {}."`. For reproducibility, we could add the prompt to the returned results dict?
Repository: huggingface/evaluate
Issue: GPU usage for evaluate image classification using pipeline
Root cause: Configuration Error

I am trying to run evaluation on imagenet for basic vit following the [instruction](https://huggingface.co/docs/evaluate/base_evaluator)
```

data = load_dataset("imagenet-1k", split="validation", use_auth_token=True)

pipe = pipeline(
    task="image-classification",
    model="google/vit-base-patch16-224"
)

task_evaluator = evaluator("image-classification")
eval_results = task_evaluator.compute(
    model_or_pipeline=pipe,
    data=data,
    metric="accuracy",
    label_mapping=pipe.model.config.label2id,
    device=0
)
```
However, although it is argued that it would automatically detect gpu and runs on it, it does not. This can be fixed by changing the pipeline to model and feature extractor as shown as
```

data = load_dataset("imagenet-1k", split="validation", use_auth_token=True)

model = ViTForImageClassification.from_pretrained("google/vit-base-patch16-224")
feature_extractor = ViTFeatureExtractor.from_pretrained("google/vit-base-patch16-224")

task_evaluator = evaluator("image-classification")
eval_results = task_evaluator.compute(
    feature_extractor=feature_extractor,
    model_or_pipeline=model,
    data=data,
    metric="accuracy",
    label_mapping=model.config.label2id,
    device=0
)
```
Is it possible to use pipeline directly on gpu instead of load the model and feature extractor separately. Meanwhile, can we control the batch size during the evaluation?

I am using transformers =4.21.2, evaluate=0.2.2, datasets=2.4.0

Hi @zhangsiyu1103! I was able to run this pipeline on GPU by passing `device=0` in the pipeline as opposed to within the evaluator's `compute` function — can you try this out like below and let me know if it works for you?

```
data = load_dataset("imagenet-1k", split="validation", use_auth_token=True)

pipe = pipeline(
    task="image-classification",
    model="google/vit-base-patch16-224",
    device=0  # Move the device setting here instead!
)

task_evaluator = evaluator("image-classification")
eval_results = task_evaluator.compute(
    model_or_pipeline=pipe,
    data=data,
    metric="accuracy",
    label_mapping=pipe.model.config.label2id
)
```

Moving device setting inside the pipeline works! But I guess this bug still exists for the evaluator compute function.

Yes, thanks for flagging! We missed an edge case in the `compute` function for the evaluator where if you pass a pipeline which was already instantiated without a device then it won't run on device even if a device is later passed to `compute`. I'll push a fix shortly!

@zhangsiyu1103 Note that this issue may slightly (~0.1%) shift your evaluations using the image classification evaluator: https://huggingface.co/datasets/imagenet-1k/discussions/1
Repository: huggingface/evaluate
Issue: Why is there no pytest library in additional-tests-requirements.txt file?
Root cause: Documentation Deficiency

I wrote the bary_score metric codes. I want to run test codes locally. I ran this code but the installation of the pytest library is missing. 

Install Test Req.txt:
```
pip install -r additional-tests-requirements.txt
```
Error Message:
```
ModuleNotFoundError: No module named 'pytest'
```
Is this a bug or does it have to be installed manually?

I guess I need to run this code too.
```
pip install .[tests]
```
What should I do to be able to do local testing? Is there sample documentation?

Hi @kadirnar! Thanks for your contributions to `evaluate`. You can run tests locally on your machine like this:

`HF_ALLOW_CODE_EVAL=1 python -m pytest -d --tx 2*popen//python=python --dist loadfile -sv ./tests/test_evaluator.py`

You can also run a specific test like this: 

`HF_ALLOW_CODE_EVAL=1 python -m pytest --capture=no  -v -d --tx 2*popen//python=python --dist loadfile -sv ./tests/test_evaluator.py::TestQuestionAnsweringEvaluator::test_model_init`

Thank you for the answer. This code is working.
```
python -m pytest -d --tx 2*popen//python=python --dist loadfile -sv ./tests/test_evaluator.py
python -m pytest --capture=no -v -d --tx 2*popen//python=python --dist loadfile -sv ./tests/test_evaluator.py::TestQuestionAnsweringEvaluator::test_model_init
```


I'm glad to hear it! 🤗 We don't currently have a CONTRIBUTING.md, but we probably should, and we'll add it there!

In the meantime I'll close this issue, but feel free to re-open if you have more questions.
Repository: huggingface/evaluate
Issue: Cannot load metric from cache
Root cause: Algorithmic Error

If we first run `metric = evaluate.load("seqeval")` in an environment with network, then run it in an environment without network, then it will fail:

```
FileNotFoundError: Couldn't find a module script at /path/seqeval/seqeval.py. Module 'seqeval' doesn't exist on the Hugging Face Hub either.  
```

(Well, this is the error message from `transformers`. The original error message is caught by `transformers`.)

This is caused by incorrect cache folder name. 

https://github.com/huggingface/evaluate/blob/283a1f07d6360e960b3ae829f2aeca839ce29839/src/evaluate/loading.py#L492

Changing this line into `name=self.name.split('/')[-1],` would make everything work. However, I'm not sure whether this is a common problem or just for me. Also not sure whether this fix will affect other part of the codes.

Indeed, I can reproduce that caching the metric script does not seem to work. Let me look into this!
Repository: huggingface/evaluate
Issue: Yaml verification for module READMEs
Root cause: Unimplemented Feature Gap

The Hub performs YAML verification when a repository is pushed. Since we push modules regularly but don't perform verification on our end the GitHub Action on main can fail even if the CI on the PR is green. It would be great to add a verification step to the tests so we are sure the modules can be pushed without issue.

See #295 for example.

cc @mathemakitten @sashavor 

~~Good call. It seems like the Hub repo pings the `/api/validate-yaml` endpoint, which is a Gitaly hook (?)~~

If we don't care about matching exactly the way that the Hub does it, we can do it in pure python with pyyaml: 

```
with open('README.md') as f_yaml:
    x = yaml.safe_load_all(f_yaml)
    y = next(x)  # if it's invalid yaml then it'll throw an error
```

Yes, that's what I had in mind. We probably need to strip out the YAML part of the README first. This can be done with [this regex](https://github.com/huggingface/huggingface_hub/blob/af1d49e9eed3b31be86652cfcfdae9bfba4f3814/src/huggingface_hub/repocard.py#L39). 

Then we just need to iterate over all README's of metrics/measurements/comparisons and that should be it 😄

Alternatively, we could see if there is interest from the `huggingface_hub` side to add a `validate_yaml` method to the repo card class (or as a helper function). cc @Wauplin