zhiminy/EvalEng huggingface dataset:
Repository: huggingface/evaluate
Issue: Add note in repo and docs that library is no longer actively maintained

AFAIK the library is no longer actively maintained. Users are still building upon the library and it might be better for them to use maintained libraries like light-eval. Adding a small note about this in the gh readme and the hf docs page could help avoid confusion. 
See this [internal discussion](https://huggingface.slack.com/archives/C072YSEKQ9M/p1722696300481329?thread_ts=1722386399.410129&cid=C072YSEKQ9M) on Slack for context.
@lvwerra 

Can someone explain why HF gives up this library?

double checking with @lhoestq @lvwerra, cc @clefourrier @lewtun   (see [this internal thread](https://huggingface.slack.com/archives/C072YSEKQ9M/p1722580567784589?thread_ts=1722386399.410129&cid=C072YSEKQ9M) for context)

cc also @LysandreJik for viz

afaik there is still maintenance happening for major cases, but not for adding new features or small bug fixes atm

Hi HF team! We're also interested to know what your plans are for maintaining this repo. We look forward to hearing back from you—thanks 🙌 

Thanks all for your comments! We're indeed slowly moving away from `evaluate` as the default evaluation repo for us at Hugging Face as we have not found it to be sufficiently useful for the community.

As Moritz mentions, we are maintaining the [`lighteval` repository](https://github.com/huggingface/lighteval) for evaluation and are also significantly using the [`LLM Harness` from EleutherAI](https://github.com/EleutherAI/lm-evaluation-harness) in our stack. We recommend using those for the latest evaluation developments.

These two aren't complete replacements for what `evaluate` aims to be, so we're still monitoring this repo and merging important fixes. We're driven by community adoption and usage, so if your work is hindered by the slowed-down pace of development on this library, please react to this message or comment below, as knowing so will help us put the right amount of effort into it.
Repository: huggingface/evaluate
Issue: Benchmark evaluation for language models.

Not sure if this feature belongs to this library or would it require a complete separate library. I am proposing the creation of a library where llm benchmarks can be ran. For example, evaluating a model on HumanEval. Such library would make evaluating LLMs much easier. I liked the dockerized approach they used at https://github.com/NVlabs/verilog-eval to safely evaluate code. Evaluating math and reasoning skill of llm could also be benificial.

consider, which includes a sandbox to run HumanEval (on Linux only)
https://github.com/bigcode-project/bigcode-evaluation-harness
Repository: huggingface/evaluate
Issue: `list_evaluation_modules` returns empty list
Root cause: External Dependency Breakage

For some reason when i run this:
```
print(evaluate.list_evaluation_modules(include_community=False))
```
I get an empty list.

```
evaluate==0.4.2
```

I also face this problem

are you able to import the evaluate library on google colab on TPU? For some reason my runtime crashes everytime i try to import it, I tried it on a clean notebook and with other libraries and its the same issue. I'm wondering if there's many bugs with the evaluate library itself

same in evaluate==0.4.3
Repository: huggingface/evaluate
Issue: Evaluate uses deprecated use_auth_token and will break with datasets-3.0
Root cause: External Dependency Breakage

Evaluate uses deprecated `datasets.DownloadConfig.use_auth_token` and will break with datasets 3.0 release.

For the next `datasets` release, we have checked the `transformers` CI and some example tests fail due to `evaluate`: https://github.com/huggingface/transformers/runs/29973850925
```
E             File "/root/project/examples/pytorch/multiple-choice/run_swag_no_trainer.py", line 546, in main
E               metric = evaluate.load("accuracy")
E             File "/usr/local/lib/python3.10/site-packages/evaluate/loading.py", line 748, in load
E               evaluation_module = evaluation_module_factory(
E             File "/usr/local/lib/python3.10/site-packages/evaluate/loading.py", line 680, in evaluation_module_factory
E               raise e1 from None
E             File "/usr/local/lib/python3.10/site-packages/evaluate/loading.py", line 639, in evaluation_module_factory
E               ).get_module()
E             File "/usr/local/lib/python3.10/site-packages/evaluate/loading.py", line 479, in get_module
E               local_path = self.download_loading_script(revision)
E             File "/usr/local/lib/python3.10/site-packages/evaluate/loading.py", line 469, in download_loading_script
E               return cached_path(file_path, download_config=download_config)
E             File "/usr/local/lib/python3.10/site-packages/evaluate/utils/file_utils.py", line 185, in cached_path
E               use_auth_token=download_config.use_auth_token,
E           AttributeError: 'DownloadConfig' object has no attribute 'use_auth_token'
```

The root cause is that `evaluate` uses `datasets.DownloadConfig.use_auth_token`, which was deprecated since datasets-2.14.0 last year:
- https://github.com/huggingface/datasets/pull/5996
Repository: huggingface/evaluate
Issue: CI is broken due to nltk: Resource punkt_tab not found
Root cause: External Dependency Breakage

CI is broken: Resource punkt_tab not found

https://github.com/huggingface/evaluate/actions/runs/10806684083/job/29975990875?pr=621
```
FAILED tests/test_metric_common.py::LocalModuleTest::test_load_metric_meteor - LookupError: 
**********************************************************************
  Resource punkt_tab not found.
  Please use the NLTK Downloader to obtain the resource:

  >>> import nltk
  >>> nltk.download('punkt_tab')
  
  For more information see: https://www.nltk.org/data.html

  Attempted to load tokenizers/punkt_tab/english/

  Searched in:
    - '/home/runner/nltk_data'
    - '/opt/hostedtoolcache/Python/3.8.18/x64/nltk_data'
    - '/opt/hostedtoolcache/Python/3.8.18/x64/share/nltk_data'
    - '/opt/hostedtoolcache/Python/3.8.18/x64/lib/nltk_data'
    - '/usr/share/nltk_data'
    - '/usr/local/share/nltk_data'
    - '/usr/lib/nltk_data'
    - '/usr/local/lib/nltk_data'
**********************************************************************
=========== 3 failed, 152 passed, 84 skipped, 39 warnings in 35.43s ============
```
Repository: huggingface/evaluate
Issue: Support nltk>=3.9 to fix vulnerability
Root cause: External Dependency Breakage

Support nltk<=3.9 to fix vulnerability. See: https://github.com/nltk/nltk/issues/3266

First, we need to fix the breaking change introduced by using "punkt_tab" instead of "punkt": https://github.com/nltk/nltk/issues/3293
- [x] word_length
- [x] meteor: [Replace 'punkt' with 'punkt_tab' to support nltk>=3.9 and fix vulnerability](https://huggingface.co/spaces/evaluate-metric/meteor/discussions/6)
Repository: huggingface/evaluate
Issue: Documentation building is broken due to numpy-2 and tensorflow
Root cause: External Dependency Breakage

Documentation building is broken, for both main and PR branches.

Latest version appearing in online docs is 0.4.0 instead of 0.4.3.

See an example of failing action: https://github.com/huggingface/evaluate/actions/runs/10826041642/job/30036188975
```
TypeError: Unable to convert function return value to a Python type! The signature was
	() -> handle
Error: Process completed with exit code 1.
```
Repository: huggingface/evaluate
Issue: CI parity test is broken due to FileNotFoundError
Root cause: Validation Gap

CI parity test is broken due to FileNotFoundError: https://github.com/huggingface/evaluate/actions/runs/10827111652/job/30039577026?pr=631
```
FAILED tests/test_trainer_evaluator_parity.py::TestEvaluatorTrainerParity::test_token_classification_parity - FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpg8va97zxevaluator_trainer_parity_test/tokenclassification_conll2003_transformers/eval_results.json'
```
```
_________ TestEvaluatorTrainerParity.test_token_classification_parity __________
[gw0] linux -- Python 3.8.18 /opt/hostedtoolcache/Python/3.8.18/x64/bin/python

self = <tests.test_trainer_evaluator_parity.TestEvaluatorTrainerParity testMethod=test_token_classification_parity>

    def test_token_classification_parity(self):
        model_name = "hf-internal-testing/tiny-bert-for-token-classification"
        n_samples = 500
    
        subprocess.run(
            "git sparse-checkout set examples/pytorch/token-classification",
            shell=True,
            cwd=os.path.join(self.dir_path, "transformers"),
        )
    
        subprocess.run(
            f"python examples/pytorch/token-classification/run_ner.py"
            f" --model_name_or_path {model_name}"
            f" --dataset_name conll2003"
            f" --do_eval"
            f" --output_dir {os.path.join(self.dir_path, 'tokenclassification_conll2003_transformers')}"
            f" --max_eval_samples {n_samples}",
            shell=True,
            cwd=os.path.join(self.dir_path, "transformers"),
        )
    
>       with open(
            os.path.join(self.dir_path, "tokenclassification_conll2003_transformers", "eval_results.json"), "r"
        ) as f:
E       FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpg8va97zxevaluator_trainer_parity_test/tokenclassification_conll2003_transformers/eval_results.json'

tests/test_trainer_evaluator_parity.py:293: FileNotFoundError
```
Repository: huggingface/evaluate
Issue: Main documentation building is not triggered

Main documentation building is not triggered: https://github.com/huggingface/evaluate/actions/workflows/build_documentation.yml
```
Invalid workflow file: .github/workflows/build_documentation.yml#L12
The workflow is not valid. .github/workflows/build_documentation.yml (Line: 12, Col: 11): Secret hf_token is required, but not provided while calling.
```

Latest version appearing in online docs is 0.4.0 instead of 0.4.3.
Repository: huggingface/evaluate
Issue: Perplexity for Left Padded Models
Root cause: Validation Gap

Hi, while using the perplexity pipeline to compute the scores for `left` padded causal LMs (e.g., `gemma-2b`), I found the results are not the same if I changed the padding side to `right`. I would like to ask if we should always set `padding_side` to `right` when computing perplexity scores?
Repository: huggingface/evaluate
Issue: Evaluate fails to load all metrics.
Root cause: External Dependency Breakage

I'm trying to use evaluate but it seems like it can't load any metric with it's `load()` function because it runs into an internal error:
```@bash
AttributeError: 'DownloadConfig' object has no attribute 'use_auth_token'
```

I have read previous posts and bug reports of behavior like this from 2023, and back then the issue was a compatibility issue between the evaluate and the `datasets` package, if datasets' version exceeded 2.10.  However those bugs were fixed already in a previous release: https://github.com/huggingface/evaluate/pull/447

I tried `evaluate.load()` separately for many metrics it fails for all of them, built-in or community. The only exception is `accuracy`, which is shipped with the library by default, so there is no need for download there.

I'm using a conda virtual environment with: Python version is `3.11` and  package versions:
```@bash
datasets                  3.0.0              pyhd8ed1ab_0    conda-forge
evaluate                  0.4.1              pyhd8ed1ab_0    conda-forge
huggingface_hub           0.24.7             pyhd8ed1ab_0    conda-forge
tokenizers                0.19.1          py311h6640629_0    conda-forge
transformers              4.44.2             pyhd8ed1ab_0    conda-forge
```

The error code for `f1` metric looks like this:
```@bash
 File "/home/c_evofb3/c_evolm/filbe/prokBert_env/lib/python3.11/site-packages/evaluate/loading.py", line 748, in load
    evaluation_module = evaluation_module_factory(
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/c_evofb3/c_evolm/filbe/prokBert_env/lib/python3.11/site-packages/evaluate/loading.py", line 680, in evaluation_module_factory
    raise e1 from None
  File "/home/c_evofb3/c_evolm/filbe/prokBert_env/lib/python3.11/site-packages/evaluate/loading.py", line 639, in evaluation_module_factory
    ).get_module()
      ^^^^^^^^^^^^
  File "/home/c_evofb3/c_evolm/filbe/prokBert_env/lib/python3.11/site-packages/evaluate/loading.py", line 479, in get_module
    local_path = self.download_loading_script(revision)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/c_evofb3/c_evolm/filbe/prokBert_env/lib/python3.11/site-packages/evaluate/loading.py", line 469, in download_loading_script
    return cached_path(file_path, download_config=download_config)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/c_evofb3/c_evolm/filbe/prokBert_env/lib/python3.11/site-packages/evaluate/utils/file_utils.py", line 185, in cached_path
    use_auth_token=download_config.use_auth_token,
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'DownloadConfig' object has no attribute 'use_auth_token'
```



Just run into the same error, and seems to be fixed in the latest `0.4.3` release. (#620)

Unfortunately, the new release is not available via conda yet, so I downgraded the datasets package.

Yes upgrading to 0.4.3 via pip solves the issue too!
Repository: huggingface/evaluate
Issue: Typo in `Types of Evaluations in 🤗 Evaluate` documentation

In [Types of Evaluations in 🤗 Evaluate](https://huggingface.co/docs/evaluate/types_of_evaluations) documentation, there is a typo in the following sentence:

* `Mean Intersection over union (IoUO)` --> `Mean Intersection over union (IoU)`
Repository: huggingface/evaluate
Issue: Optionally stop data deletion after compute
Root cause: Unimplemented Feature Gap

In the following locs I'm seeing that the module is deleting the data with `del self.data` can we have a flag that avoids this deletion so that we can drill down into good/bad data points at a later stage? I can raise a raise a PR if you think this is a good idea.

https://github.com/huggingface/evaluate/blob/55f1bc6e072b05c2d9db1589a07e20f38902b1ec/src/evaluate/module.py#L471

```python
...
            if self.buf_writer is not None:
                self.buf_writer = None
                del self.data
                self.data = None
            else:
                # Release locks and delete all the cache files. Process 0 is released last.
                for filelock, file_path in reversed(list(zip(self.filelocks, self.file_paths))):
                    logger.info(f"Removing {file_path}")
                    del self.data
                    self.data = None
                    del self.writer
                    self.writer = None
                    os.remove(file_path)
                    filelock.release()
...
```
Repository: huggingface/evaluate
Issue: ImportError: To be able to use evaluate-metric/glue, you need to install the following dependencies['scipy', 'scikit-learn'] using 'pip install scipy sklearn' for instance'
Root cause: Validation Gap

I saw previously linked issues for the same isssue, but seems like the error is still not fixed, Happy to make a PR if applicable

also have this where it lists a dependency twice... might be some issue with other packages? I don't see what changed.

you reckon its the packages? ill have a deeper look

~~It's for some reason looking at the imports at the top of the script. And then managing to find deeper imports twice.~~

```py
import a
from a.b import x
from a.c import y
```
~~Causes `a` to show up twice but only gets found once. if you have inline comments it's also messed up.~~

was something unrelated, but the Import Error for some package got the solution... which might be helpful to not suppress and show to the user.
Repository: huggingface/evaluate
Issue: Perplexity for quantized models
Root cause: Documentation Deficiency

Hi, If I intend to compute the perplexity for a quantized model, what parameters should I pass to the .compute() function?

For example,
>>> perplexity = load("perplexity", module_type="metric")
>>> good_result = perplexity.compute(predictions=SENTENCES,model_id=MODEL_NAME,add_start_token=False)

Much thx!

I tried loading a quantized model, but it failed:

>>> ValueError: `.to` is not supported for `4-bit` or `8-bit` bitsandbytes models. Please use the model as it is, since the model has already been set to the correct devices and casted to the correct `dtype`.
Repository: huggingface/evaluate
Issue: [FR] Binoculars as a new metric
Root cause: Unimplemented Feature Gap

Binocular is a metric similar to perplexity [[read about binocular](https://huggingface.co/blog/dmicz/binoculars-text-detection#:~:text=By%20comparing%20actual%20perplexity%20to,the%20same%20prompt%20and%20context.)]. Binocular uses two language models with same tokenization to detect the LLM generated text (or data contamination).

Currently the evaluate library doesn't include binocular in its long list of metrics. Something like the following could help the library be more useful

```
from evaluate import load
binocular = load("binocular", module_type="metric")
results = binocular.compute(predictions=predictions, observer_model='tiiuae/falcon-7b-instruct', performer_model='tiiuae/falcon-7b')
```
Repository: huggingface/evaluate
Issue: FileNotFoundError: Couldn't find a module script at /mnt/data/dingtian/exact_match/exact_match.py. Module 'exact_match' doesn't exist on the Hugging Face Hub either.
Root cause: Configuration Error

I have installed `evaluate` manually using 
```
git clone https://github.com/huggingface/evaluate.git
cd evaluate
pip install -e .
```
and my `evaluate` is under /mnt/data/dingtian/evaluate, I don't understand why it keeps looking for files in /mnt/data/dingtian/exact_match/exact_match.py, and if I change the dir in the load func, it can work:
```
# FileNotFoundError
python -c "import evaluate; print(evaluate.load('exact_match').compute(references=['hello'], predictions=['hello']))"
# work
python -c "import evaluate; print(evaluate.load('/mnt/data/dingtian/evaluate/metrics/exact_match/exact_match.py').compute(references=['hello'], predictions=['hello']))"
```

In this [discussion](https://github.com/EleutherAI/lm-evaluation-harness/issues/1071) someone proposes:

```
git clone https://github.com/huggingface/evaluate.git
cd evaluate
pip install -e .
``` 

(installing library from source).

It works for me!
Repository: huggingface/evaluate
Issue: TypeError: 'NoneType' object is not callable
Root cause: External Dependency Breakage

When I use this code, I encounter an error that I can't resolve.
 metric = evaluate.load(
        "squad_v2" if data_args.version_2_with_negative else "squad", cache_dir=model_args.cache_dir
    )

File "../../pytorch/question-answering/run_qa.py", line 712, in <module>
    main()
  File "../../pytorch/question-answering/run_qa.py", line 626, in main
    metric = evaluate.load(
  File "/opt/conda/lib/python3.8/site-packages/evaluate/loading.py", line 706, in load
    evaluation_instance = evaluation_cls(
TypeError: 'NoneType' object is not callable

evaluate  is 0.4.3
transformer is 4.46.1
datasets is 2.16.0 


evaluate.load() for any measurement/metrics does not work properly with versions >0.4.0. Please downgrade to 0.4.0 to make it work. I also discovered after lot of frustrating trials.
Repository: huggingface/evaluate
Issue: Evaluation took so much disk
Root cause: Resource Mishandling

I am try to train SegFormer model and use evaluate module to eval the IoU, but when there is so many data and epoch, the metrics looks like take so much space.
![image](https://github.com/user-attachments/assets/1b9a46d7-9bd7-4d9c-a6bc-f0d189be3c2a)

is there any method so I can free space while training still going?
Repository: huggingface/evaluate
Issue: FileNotFoundError: Couldn't find a module script at /home/tooko/transformers-course/glue/glue.py.
Root cause: Environment Incompatibility

i run the example of hugging face NLP course(https://huggingface.co/learn/nlp-course/chapter3/4?fw=pt#the-training-loop)
my env at Vmware-ubuntu -vscode,evaluate Version is 0.4.3 . i always encounter this error:

import evaluate
metric = evaluate.load("glue", "mrpc") #此句依旧是出现FileNotFoundError错误
model.eval()

for epoch in range(num_epochs):
for batch in train_dataloader:
batch = {k: v.to(device) for k, v in batch.items()} #将batch中的数据转移到device上，batch是一个字典
with torch.no_grad():
outputs = model(**batch)

    logits = outputs.logits
    predictions = torch.argmax(logits, dim=-1)
    metric.add_batch(predictions=predictions, references=batch["labels"])
FileNotFoundError Traceback (most recent call last)
Cell In[1], line 3
1 import evaluate
----> 3 metric = evaluate.load("glue", "mrpc")
4 model.eval()
6 for epoch in range(num_epochs):

File ~/transformers-course/.env/lib/python3.11/site-packages/evaluate/loading.py:748, in load(path, config_name, module_type, process_id, num_process, cache_dir, experiment_id, keep_in_memory, download_config, download_mode, revision, **init_kwargs)
703 """Load a [~evaluate.EvaluationModule].
704
705 Args:
(...)
745 ```
746 """
747 download_mode = DownloadMode(download_mode or DownloadMode.REUSE_DATASET_IF_EXISTS)
--> 748 evaluation_module = evaluation_module_factory(
749 path, module_type=module_type, revision=revision, download_config=download_config, download_mode=download_mode
750 )
751 evaluation_cls = import_main_class(evaluation_module.module_path)
752 evaluation_instance = evaluation_cls(
753 config_name=config_name,
754 process_id=process_id,
(...)
760 **init_kwargs,
...
684 ) from None
685 else:
686 raise FileNotFoundError(f"Couldn't find a module script at {relative_to_absolute_path(combined_path)}.")

FileNotFoundError: Couldn't find a module script at /home/tooko/transformers-course/glue/glue.py. Module 'glue' doesn't exist on the Hugging Face Hub either.

others opinion to solve this issue is to loging in hugging face account and not use proxy. B cos i am in China so i must use vpn to log in HF but after logged in no matter i used vpn or not the error was existing ,and i try to download glue/glue.py from Colab'example of HF provided but i didn't found. think you for your some advice for this problem (sorry for my bad english)
Repository: huggingface/evaluate
Issue: Typo in README of confusion_matrix, and suggestion to specify valid values for `normalize`
Root cause: Documentation Deficiency

In the [Output Values](https://github.com/huggingface/evaluate/tree/main/metrics/confusion_matrix#output-values) section are some typos:

"""
Minimum possible value is 0. Maximum possible value is 1.0, 
or the number of examples input, if `normalize` is set to `True`.
"""

The corrections are in bold: 

"""
Minimum possible value is 0. Maximum possible value is 1.0, 
or the number of examples **in the** input, if `normalize` is set to **`None` (default)**.
"""


Additionally, I would specify in the [Inputs](https://github.com/huggingface/evaluate/tree/main/metrics/confusion_matrix#inputs) section the permitted values for `normalize`: {'pred', 'all', 'true'} or `None`.
Repository: huggingface/evaluate
Issue: Trainer.tokenizer is now deprecated
Root cause: External Dependency Breakage

Hi, I am using 
```
evaluate                 0.4.3
transformers             4.46.2
```

The following deprecation warning reported by transformers, I believe is caused by using of tokenizer in evaluate library.

```terminal
Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
```



oh, I realized this is caused by transformers, which could be fixed by https://github.com/huggingface/transformers/pull/35482, will close the issue.