zhiminy/EvalEng huggingface dataset:
Repository: huggingface/evaluate
Issue: PyArrow Version Minimum?
Root cause: Environment Incompatibility

What version of `pyarrow` does `evaluate` need? I'm running `pyarrow==3.0.0` due to another dependency conflict, but I get this error when I try to import `evaluate`:

`ImportWarning: To use `datasets`, the module `pyarrow>=6.0.0` is required, and the current version of `pyarrow` doesn't match this condition`

However, I can successfully run `datasets` with `pyarrow==3.0.0`, with `datasets==1.12.1`, so I'm not quite sure if this is a dependency version conflict with `evaluate` or `datasets`. Thanks!

Hi @versipellis, is it possible that you installed a newer version of `datasets` with `evaluate`? Since `datasets==2.0.0` the dependancy has been updated for `pyarrow`: https://github.com/huggingface/datasets/commit/f144ba69aee057e9e1447796712302adc3d6558a



Hi @lvwerra, I thought about that, but verified that `datasets==1.12.1`.

That's weird, because `evaluate` does not check the arrow version so the import warning must come from `datasets`. If you import `datasets` before `evaluate` does the warning still come first from evaluate?

Looks like I somehow did end up in a weird dependency hell state between conda and pip where I ended up with `datasets==1.12.1` and `evaluate` where `evaluate` required a later version of `datasets`. PEBKAC, sorry about that!
Repository: huggingface/evaluate
Issue: `evaluate.load` not loading all files in the repo
Root cause: Architectural Constraint

`evaluate.load` only loads the files that are imported in the main file, so a user will need to include the import in the main file even if they don't use it there for the files to be loaded, it might be nice to include this in the docs.

An example here: 
https://huggingface.co/spaces/loubnabnl/apps_metric/blob/main/utils.py#L5
and the error I got before I included the import in the main file:
````
from .utils import compute_metrics
  File "/Users/loubnabenallal/.cache/huggingface/modules/evaluate_modules/metrics/loubnabnl--apps_metric/24ed11d3adfebede98beb04ac2c729404f305f66064c2b940c893f510452cea9/utils.py", line 5, in <module>
    from .testing_util import run_test
ModuleNotFoundError: No module named 'evaluate_modules.metrics.loubnabnl--apps_metric.24ed11d3adfebede98beb04ac2c729404f305f66064c2b940c893f510452cea9.testing_util'
````
And after checking the files that were loaded, `testing_util` wasn't there:
````
ls /Users/loubnabenallal/.cache/huggingface/modules/evaluate_modules/metrics/loubnabnl--apps_metric/24ed11d3adfebede98beb04ac2c729404f305f66064c2b940c893f510452cea9/
````

```
__init__.py             __pycache__             apps_metric.json        apps_metric.py          utils.py
```

* evaluate version 0.1.2
* datasets version 2.1.0

Seconding that this seems like it'd be useful! I recently tried to load a custom metric which opened up a JSON file for configuration, but the JSON file wasn't loaded. I think this might not be a "canonical" use case though, so perhaps optional to actually support.

Note that you can use the `_download_and_prepare` method to download any additional files:

https://github.com/huggingface/evaluate/blob/eaf34a7d04e7ab3e6155a046f6d7fda01d9ead84/templates/%7B%7B%20cookiecutter.module_slug%20%7D%7D/%7B%7B%20cookiecutter.module_slug%20%7D%7D.py#L84
Repository: huggingface/evaluate
Issue: Support multiple input columns for TextClassificationEvaluator, regression subtasks
Root cause: Unimplemented Feature Gap

https://huggingface.co/datasets/glue includes:

* datasets where we may have several input columns ( https://huggingface.co/datasets/glue/viewer/mnli/train , https://huggingface.co/datasets/paws )
* datasets where the labels are floats on a given range, task is regression ( https://huggingface.co/datasets/glue/viewer/stsb/train )

It would be nice to support those in TextClassificationEvaluator.

I feel like the former is probably best addressed in the preprocessing: It's literally a onliner to merge to text column into one (`ds = ds.map(lambda x: {"text": x[col1] + sep + x[col2]})`. We can definitely add an example to the guide.

 For the latter I think we could have a kwarg `problem_type="regression"` in the evaluator that also has a useful default metric. Wdyt?
Repository: huggingface/evaluate
Issue: how to reset gathered results at new epoch start?
Root cause: Documentation Deficiency

Thanks for the great package! I am using it in my new fine-tuning framework :100: 

Usually, we reset accumulated results at the new epoch start so that we can calculate separate metrics for each epoch. How to do it with huggingface/evaluate? I could not find any `reset` method as in [torchmetrics](https://torchmetrics.readthedocs.io/en/stable/pages/implement.html#torchmetrics.Metric.reset). Am I missing something?

What I mean is:

```python
def training_step(self, batch):
    outputs = self.model(batch['images'])
    all_outputs = self.accelerator.gather(outputs)
    all_labels = self.accelerator.gather(batch['labels'])
    self.train_f1.add_batch(all_predictions, all_labels)
    return self.loss_funct(outputs, batch['labels'])

def train(self):
    self.train_f1 = evaluate.load("f1")

    for epoch in range(5):
        for batch in train_dataloader:
            loss = self.training_step(batch)
            self.accelerator.backward(loss)
            self.optimizer.step()
            self.scheduler.step()
            self.optimizer.zero_grad()

    self.train_f1.compute()
    self.accelerator.log('train/f1', self.train_f1.f1)
    # how to reset self.train_f1 at epoch end?
    self.train_f1.reset()
```

Best regards





I got my answer in https://github.com/huggingface/accelerate/issues/574
Repository: huggingface/evaluate
Issue: Evaluate does not support versions of Datasets < 2.0.0
Root cause: Configuration Error

Hi there. Not sure why it's not pinned in the setup so opening an issue instead of a PR directly fixing. Evaluate uses the import
```
from datasets import DownloadMode
```
[here](https://github.com/huggingface/evaluate/blob/d781f85c55f94bd4f1896c12d9a2ea444f4805b3/src/evaluate/loading.py#L30).

However, this is only possible in datasets>=2.0.0 (just tried with 1.18.4) so it looks like this is the minimal version of Datasets required for `evaluate`. If that's the case, the setup should pin it otherwise downstream libraries might get weird failures (we just had one reported from the DeepSpeed CI for Accelerate, so it goes a long way down ;-) ). If there is a way to use older versions of Datasets, then a fix should be made for that import.

cc @lvwerra 

Hi @sgugger thanks for reporting. We pinned this for each each space but not in the `setup.py` for some reason. Will pin it and do a patch release - thanks for reporting!

Thank you!
Repository: huggingface/evaluate
Issue: evaluate-cli for creating measurements seems not to work
Root cause: Documentation Deficiency

Following the installation instructions here:
https://huggingface.co/docs/evaluate/installation

And then trying to add a new measurement, following: https://huggingface.co/docs/evaluate/creating_and_sharing and using python3.9,

I get this error:

```
(.env) bash-3.2$ evaluate-cli create "npmi" module_type="measurement"
Traceback (most recent call last):
  File "/Users/margaretmitchell/HuggingFace/git/data-measurements-tool/.env/bin/evaluate-cli", line 33, in <module>
    sys.exit(load_entry_point('evaluate', 'console_scripts', 'evaluate-cli')())
  File "/Users/margaretmitchell/HuggingFace/git/data-measurements-tool/.env/bin/evaluate-cli", line 25, in importlib_load_entry_point
    return next(matches).load()
  File "/opt/anaconda3/lib/python3.9/importlib/metadata.py", line 86, in load
    module = import_module(match.group('module'))
  File "/opt/anaconda3/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/Users/margaretmitchell/HuggingFace/git/data-measurements-tool/evaluate/src/evaluate/commands/evaluate_cli.py", line 6, in <module>
    from cookiecutter.main import cookiecutter
ModuleNotFoundError: No module named 'cookiecutter'
```


Suggest adding 'cookicutter' as a requirement.

There is a `pip install evaluate[template]` as described in the README [here](https://github.com/huggingface/evaluate#adding-a-new-evaluation-module) but it's currently missing from the docs. Will update it!

Ah yes, I had run this previously but when I backed off and started afresh step-by-step, lost that step. Thanks!
Repository: huggingface/evaluate
Issue: evaluate-cli module_type not working
Root cause: Documentation Deficiency

After installing cookiecutter after the step-by-step described in my last issue: https://github.com/huggingface/evaluate/issues/211 , error is now:

(.env) bash-3.2$ evaluate-cli create "npmi" module_type="measurement"
usage: evaluate-cli <command> [<args>]
HuggingFace Evaluate CLI tool: error: unrecognized arguments: module_type=measurement
(.env) bash-3.2$ evaluate-cli create "npmi" module_type="metric"
usage: evaluate-cli <command> [<args>]
HuggingFace Evaluate CLI tool: error: unrecognized arguments: module_type=metric

Indeed that's a mistake in the docs. It should be the following:

```bash
evaluate-cli create "npmi" --module_type "measurement"
```

Does that work?

Good catch! Unfortunately, this doesn't work for me either, but it is a different issue. 
Filed now here: https://github.com/huggingface/evaluate/issues/213
Repository: huggingface/evaluate
Issue: Re-Creating Space with evaluate-cli not working
Root cause: Validation Gap

Presumably this error is happening because a Space was already created -- should evaluate-cli handle that situation?
Since I don't remember exactly where evaluate creates these, the error would be more helpful to me if it gave the url of the Space it's trying to create
As-is, the given url shown in the evaluate-cli error (which is thrown from requests/models.py) provides a different url, https://huggingface.co/api/repos/create. Going to that url, all I see it "{"error":"Sorry, we can't find the page you are looking for."}".  Dead-end!

Here's the evaluate-cli error:

(.env) bash-3.2$ evaluate-cli create "npmi" --module_type "measurement"
Traceback (most recent call last):
  File "/Users/margaretmitchell/HuggingFace/git/data-measurements-tool/.env/bin/evaluate-cli", line 33, in <module>
    sys.exit(load_entry_point('evaluate', 'console_scripts', 'evaluate-cli')())
  File "/Users/margaretmitchell/HuggingFace/git/data-measurements-tool/evaluate/src/evaluate/commands/evaluate_cli.py", line 86, in main
    create_repo(namespace + "/" + module_slug, repo_type="space", space_sdk="gradio", private=args["private"])
  File "/Users/margaretmitchell/HuggingFace/git/data-measurements-tool/.env/lib/python3.9/site-packages/huggingface_hub/utils/_deprecation.py", line 32, in inner_f
    return f(*args, **kwargs)
  File "/Users/margaretmitchell/HuggingFace/git/data-measurements-tool/.env/lib/python3.9/site-packages/huggingface_hub/hf_api.py", line 1468, in create_repo
    raise err
  File "/Users/margaretmitchell/HuggingFace/git/data-measurements-tool/.env/lib/python3.9/site-packages/huggingface_hub/hf_api.py", line 1457, in create_repo
    _raise_with_request_id(r)
  File "/Users/margaretmitchell/HuggingFace/git/data-measurements-tool/.env/lib/python3.9/site-packages/huggingface_hub/utils/_errors.py", line 95, in _raise_with_request_id
    raise e
  File "/Users/margaretmitchell/HuggingFace/git/data-measurements-tool/.env/lib/python3.9/site-packages/huggingface_hub/utils/_errors.py", line 90, in _raise_with_request_id
    request.raise_for_status()
  File "/Users/margaretmitchell/HuggingFace/git/data-measurements-tool/.env/lib/python3.9/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 409 Client Error: Conflict for url: https://huggingface.co/api/repos/create (Request ID: kpt54CtAgUuxffHYa1zE3) - You already created this space repo

Yes, if the space already exists this will fail. We can add extra checks for that! The space is created under your account. 

You should see the space here:
https://huggingface.co/meg/

Great! Could we:
- Add a check and perhaps a flag to overwrite
- Print out the specific url where it's trying to create the space?
Repository: huggingface/evaluate
Issue: evaluate HOWTO directs to Space instructions, which fail.
Root cause: Documentation Deficiency

Following the instructions from evaluate, and help from the previous issues I filed, I now have a https://huggingface.co/spaces/meg/npmi.
Following the instructions there, I now get the following error.
I recognize at this point it becomes a Spaces issue, not an evaluate issue (?), but in the meantime the evaluate instructions might be updated:

(.env) bash-3.2$ git add app.py
(.env) bash-3.2$ git commit -m "Add application file"
[main dc918b1] Add application file
 1 file changed, 7 insertions(+)
 create mode 100644 app.py
(.env) bash-3.2$ git push
Username for 'https://github.com': meg@huggingface.co
Password for 'https://meg@huggingface.co@github.com': 
remote: Support for password authentication was removed on August 13, 2021.
remote: Please see https://docs.github.com/en/get-started/getting-started-with-git/about-remote-repositories#cloning-with-https-urls for information on currently recommended modes of authentication.
fatal: Authentication failed for 'https://github.com/huggingface/evaluate.git/'


Are you in the subfolder of the created space? it looks like you are trying to push to the evaluate repo.

Sounds like you are saying that I need to cd .. out of the evaluate repo before interacting with the Space? This is what I had thought, but, I was following the instructions precisely. See:

``evaluate-cli create "My Metric" module_type="metric"
This will create a new Space on the 🤗 Hub, clone it locally, and populate it with a template. Instructions on how to fill the template will be displayed in the terminal, but are also explained here in more detail.''

So now I go to the Space (p.s. -- we should have details on the Space url written here, e.g., https://huggingface.co/spaces/${username}/${metric_name})

But now I go to the Space, and following the directions exactly:

``Start by cloning this repo by using:

git clone https://huggingface.co/spaces/meg/npmi ''

What I'm hearing is that there should be an additional step documented before going to the Space instructions: moving out of the evaluate directory (or something like this?)


Wherever you execute the CLI command it will create a folder with the cloned space. You need to cd into that folder before git adding the files.

Right, I forgot!  The issue is that when you go the Space, it gives more instructions -- which you should *not* follow. Perhaps a parenthetical in the evaluate instructions I highlighted "(if you go to the Space right now, it will have separate instructions -- ignore those, we have it covered in the steps here!)"
Repository: huggingface/evaluate
Issue: Add gradio as a dependency?
Root cause: Environment Incompatibility

Following the evaluate instructions, you eventually get to a point where you run app.py
When you run that, it mentions you need to check that gradio is installed -- but a bit hard to see because an error is immediately thrown (see below). Perhaps the instructions could specify to install gradio (`pip install gradio`), or else do it automatically, earlier, by including it with `pip install evaluate[template]` (if gradio is used for everything)?

(.env) bash-3.2$ python /Users/margaretmitchell/HuggingFace/git/data-measurements-tool/evaluate/npmi_debug/app.py
To create a metric widget with Gradio make sure gradio is installed.
Traceback (most recent call last):
  File "/Users/margaretmitchell/HuggingFace/git/data-measurements-tool/evaluate/npmi_debug/app.py", line 6, in <module>
    launch_gradio_widget(module)
  File "/Users/margaretmitchell/HuggingFace/git/data-measurements-tool/evaluate/src/evaluate/utils/gradio.py", line 97, in launch_gradio_widget
    raise error
  File "/Users/margaretmitchell/HuggingFace/git/data-measurements-tool/evaluate/src/evaluate/utils/gradio.py", line 94, in launch_gradio_widget
    import gradio as gr
ModuleNotFoundError: No module named 'gradio'
Repository: huggingface/evaluate
Issue: How to calculate multi-label F1 score?
Root cause: Documentation Deficiency

Hello all,

Assume I have more than 1 true label per image or video. How to calculate multi-label F1 using this module?

Check this reference to see how torchmetrics handle multi-label predictions: https://torchmetrics.readthedocs.io/en/stable/pages/classification.html#input-types

I have also created a HF Hub discussion: 
https://huggingface.co/spaces/evaluate-metric/f1/discussions/2#62e3ab6ef3b208e2aed00a5a

Hi @fcakyon, I answered in the HF Hub discussion and close this for now. 

Thanks opened a PR on hub: https://huggingface.co/spaces/evaluate-metric/f1/discussions/3
Repository: huggingface/evaluate
Issue: pip install evaluate[template] doesn't work for me =(
Root cause: Documentation Deficiency

Following https://huggingface.co/docs/evaluate/installation , the command `pip install evaluate[template]` doesn't work.
Should it be `pip install evaluate template` or something like this? Or is there a specific template name to use?

Error:
```
(.env) (base) margaretmitchell@Margarets-MacBook-Pro data-measurements-tool % pip install evaluate[template]
zsh: no matches found: evaluate[template]
```

Just tested this in a Colab and it worked. What Python version are you using?

3.9

The zsh shell interprets square brackets as a pattern on the command line; they need to be escaped; e.g. `pip install -e ."[dev]"`.

Thanks @mathemakitten! Closing this for now. We can re-open should the issue persist.
Repository: huggingface/evaluate
Issue: Issue with `evaluator` when the dataset is passed as a string.
Root cause: Validation Gap

If a dataset is passed as a string the loading of the data will fail since the rest of the pipeline expects a `dataset.Dataset` and not a `dataset.DatasetDict` which is returned if no split is provided. I propose we auto-select a split with the following logic: `test` > `validation` > `train`. We can inform the user with a log which split is evaluated.

cc @fxmarty @ola13
Repository: huggingface/evaluate
Issue: Implement "text generation" task in the Evaluator
Root cause: Unimplemented Feature Gap

In addition to the current task types available in the Evaluator we want a generic text generation pipeline which runs inference and returns generations. The "data" the evaluator will take in this case will be (optionally) a set of prompts for the language model. This will be useful for implementing evaluations requiring a set of model generations, such as RealToxicityPrompts's "toxicity probability" and the regard metric from [this paper](https://arxiv.org/abs/1909.01326).
Repository: huggingface/evaluate
Issue: Cannot use f1/recall/precision arguments in `CombinedEvaluations.compute`
Root cause: Architectural Constraint

This works:

```python
metric=evaluate.load('f1')
metric.compute(references=[0, 1, 0, 1, 0], predictions=[0, 0, 1, 1, 0], average=None)
```

This won't work:

```python
metric=evaluate.combine(["f1"])
metric.compute(references=[0, 1, 0, 1, 0], predictions=[0, 0, 1, 1, 0], average=None)
```

Reason:
`average` is not included in f1 score features: https://github.com/huggingface/evaluate/blob/eaf34a7d04e7ab3e6155a046f6d7fda01d9ead84/metrics/f1/f1.py#L112

and `CombinedEvaluations.compute` ignore if argument is not included in features: https://github.com/huggingface/evaluate/blob/eaf34a7d04e7ab3e6155a046f6d7fda01d9ead84/src/evaluate/module.py#L858

Is this expected or a bug?

@lvwerra do you have any opinion on that?

Yes, this is a current limitation of combine: you can't pass any settings to `compute` only the features. Rather than fixing this in `combine` we aim to enable changing the settings when the metric is loaded in #169. This should be coming in the next few weeks.

@lvwerra Do you mind if I ask whether you have an estimate on when this issue will be closed? I reviewed #188 and noticed that it seems to be passing for pip release. Is there a chance we can git it in the next couple of days?

@lvwerra I also need to pass custom kwargs to my metrics. My current workaround is to overwrite `CombinedEvaluations .compute` in a child class and remove
https://github.com/huggingface/evaluate/blob/eaf34a7d04e7ab3e6155a046f6d7fda01d9ead84/src/evaluate/module.py#L858

This works in my specific use case because all my (custom) metrics accept `**kwargs` in their `compute` method, which means they will just ignore extra args.

Has this been resolved?

Not yet. There is an issue with the sync mechanism between the hub and the library which is why we had to roll back #169. Merging it will break pre 0.3.0 installs so we need to wait for sufficient adaptation.
Repository: huggingface/evaluate
Issue: Metric Request: Add metric Brier Score
Root cause: Unimplemented Feature Gap

Would you consider adding metric [Brier Score](https://en.wikipedia.org/wiki/Brier_score) to this repo?  I can help with this.

That would be awesome! You can either add it in the folder `metrics/` and create a PR or add it as a community metric following this [guide](https://huggingface.co/docs/evaluate/creating_and_sharing).
Repository: huggingface/evaluate
Issue: Setup script action to push "canonical" metrics to hub
Root cause: Unimplemented Feature Gap

The goal is to have all in the library available metrics hosted on the hub as a space. This includes the metrics that are currently inside the `evaluate` remove in `metrics/`. To get them to the hub we should setup a script and action that pushes them to a space. We can probably reuse a lot of functionality from #14 to make this work. 

A natural mechanism would be to run this whenever a PR is merged:
- pull all canonical metrics repos
- apply changes (if any)
- push back to hub