zhiminy/EvalEng huggingface dataset:
Repository: huggingface/evaluate
Issue: DownloadConfig is not passed to LocalEvaluationModuleFactory when loading from local file, causing uncontrollable download behavior
Root cause: Configuration Error

In evaluate/src/evaluate/loading.py lines 621-624:

```python
elif os.path.isfile(combined_path):
    return LocalEvaluationModuleFactory(
        combined_path, download_mode=download_mode, dynamic_modules_path=dynamic_modules_path
    ).get_module()
```

The `DownloadConfig` from `evaluation_module_factory` is not passed to `LocalEvaluationModuleFactory`. This causes line 421:

```python
local_imports = _download_additional_modules(
    name=self.name,
    base_path=str(Path(self.path).parent),
    imports=imports,
    download_config=self.download_config,
)
```

to use the default `download_config`. Consequently, in `_download_additional_modules` (line 248), if `url_or_filename` is an external URL, it will use the default `download_config` when entering the `cached_path` function.

Subsequently, when entering the `get_from_cache` function in `evaluate/src/evaluate/utils/file_utils.py`, `use_etag=True` and `local_files_only=False`. This skips line 438:

```python
if os.path.exists(cache_path) and not force_download and not use_etag:
```

because `use_etag=True`. Then it proceeds to the `if not local_files_only` branch on line 447. If my server can't connect to Hugging Face at this point, it will get stuck here.

I also can't modify the download configuration by changing the `DownloadConfig` in the `load` function because it's not passed to the `LocalEvaluationModuleFactory` function. Nor can I use already downloaded files in the cache because the line `if os.path.exists(cache_path) and not force_download and not use_etag:` is skipped.

Is this a bug? If not, how can I use the `evaluate.load` function when connections to the Hugging Face server are prone to timeouts?

try `evaluate.load(path, download_config = DownloadConfig(use_etag = False))`

network error derives from checking etag, not connecting to huggingface (if you have local `evaluate` repository)

> try `evaluate.load(path, download_config = DownloadConfig(use_etag = False))`
> 
> network error derives from checking etag, not connecting to huggingface (if you have local `evaluate` repository)

The DownloadConfig from evaluation_module_factory is not passed to LocalEvaluationModuleFactory. Your command will not have any effect.
Repository: huggingface/evaluate
Issue: Unable to reproduce example results using the Unbabel/wmt22-comet-da model
Root cause: External Dependency Breakage

**Description:**
Hi COMET team,
I'm trying to reproduce the example results for the Unbabel/wmt22-comet-da model, but the output scores I obtained are quite different from those shown in the official documentation and example scripts.

**Environment**
Python version: 3.10.8
COMET version: 2.2.7
PyTorch version: 2.8.0
Operating system: Windows 11
without GPU

**Code used**
`from evaluate import load
comet_metric = load('comet')
source = ["Dem Feuer konnte Einhalt geboten werden", "Schulen und Kindergärten wurden eröffnet."]
hypothesis = ["The fire could be stopped", "Schools and kindergartens were open"]
reference = ["They were able to control the fire", "Schools and kindergartens opened"]
results = comet_metric.compute(predictions=hypothesis, references=reference, sources=source)
print([round(v, 2) for v in results["scores"]])`

**Expected behavior**
According to the example in the metrics/comet/README.md, the expected output should be [0.19, 0.92], but my result is [0.85, 0.97].
I used the same code as in the example and have loaded the correct model.

> Expected results

<img width="1022" height="233" alt="Image" src="https://github.com/user-attachments/assets/d23e7572-8f12-47f5-9fb8-d7f486606413" />

> My results

<img width="1792" height="229" alt="Image" src="https://github.com/user-attachments/assets/d1bbea62-56c2-478d-a7af-e8259603a50d" />

<img width="1686" height="114" alt="Image" src="https://github.com/user-attachments/assets/5bf5dd39-1f63-4956-bd8b-cd733f06a260" />


**Questions**
Has the Unbabel/wmt22-comet-da model been updated or rescaled recently?
Are there any changes in score normalization (e.g., z-score → 0–1 scaling)?
Is there a specific COMET or PyTorch version required to match the example results?

**Additional context**
I am using COMET for machine translation evaluation and need to ensure consistency with the official WMT22 scores. Any guidance or clarification would be greatly appreciated.
Thanks a lot for your time and support!
Repository: huggingface/evaluate
Issue: Incompatibility with Gradio >= 4.0.0
Root cause: External Dependency Breakage

As suggested by HuggingFace Spaces, Gradio has to upgrade to >=5.49. But in `src/evaluate/utils/gradio.py`, the file is only compatible with Gradio < 4, where `gr.inputs` and `gr.outputs` are still not deprecated. 

Any plan to support Gradio 5.x? Thanks.

fixed in https://github.com/huggingface/evaluate/commit/7969d534a5ac5b77b5e97028a038f7d698abdd19 :). let me know if it works for you now!
Repository: huggingface/evaluate
Issue: Error message suggest install `sklearn` which is not correct
Root cause: Validation Gap

When running evaluate without installing the scikit-learn package, this is the error message I get:

```
ImportError: To be able to use evaluate-metric/accuracy, you need to install the following dependencies['scikit-learn'] using 'pip install sklearn' for instance'
```

However, it is not correct, it should be `pip install scikit-learn` it can be confusing for beginners who are less experienced with such packages.

The reason for the error is here:

```
File ~/PycharmProjects/PythonProject7/.venv/lib/python3.13/site-packages/evaluate/loading.py:265, in _download_additional_modules(name, base_path, imports, download_config)
    263         needs_to_be_installed.add((library_import_name, library_import_path))
    264 if needs_to_be_installed:
--> 265     raise ImportError(
    266         f"To be able to use {name}, you need to install the following dependencies"
    267         f"{[lib_name for lib_name, lib_path in needs_to_be_installed]} using 'pip install "
    268         f"{' '.join([lib_path for lib_name, lib_path in needs_to_be_installed])}' for instance'"
    269     )
    270 return local_imports
```

It is using a generic way to find the package name and if the name of the directory holding the package is different from the name of the package itself (for example, sklearn vs scikit-learn, in this case) they it will not be correct.

Suggested fix:
using `lib_name` instead of `lib_path` in line 268.



I was able to duplicate it. It's an easy enough fix. #self-assign
Repository: huggingface/evaluate
Issue: FileNotFoundError: Couldn't find a module script at /workspace/weiw13@xiaopeng.com/VLM/evaluate/meteor/meteor.py. Module 'meteor' doesn't exist on the Hugging Face Hub either.
Root cause: Configuration Error

I installed by git clone method and pip install -e ., it also got an issue:
`# FileNotFoundError
python -c "import evaluate; print(evaluate.load('meteor').compute(references=['hello'], predictions=['hello']))"
# work
python -c "import evaluate; print(evaluate.load('/workspace/evaluate/metrics/meteor/meteor.py').compute(references=['hello'], predictions=['hello']))"`
I installed the evaluate vision is evaluate-0.4.7.dev0
Repository: huggingface/evaluate
Issue: Proposal: Make SQuAD QA F1 logic reusable as a generic QA metric in `evaluate`
Root cause: Unimplemented Feature Gap

Hi 👋

I’m currently building an evaluation framework for long-context / QA benchmarks (e.g. HotpotQA), and I ran into a usability gap in `evaluate` that I wanted to discuss before opening a PR.

### Context / motivation

While evaluating HotpotQA, I wanted to use the **standard QA token-level F1** (the same one used in SQuAD: token overlap, max over references, averaged across examples).

I noticed that:

* This exact logic already exists in [`metrics/squad/compute_score.py`](https://github.com/huggingface/evaluate/blob/main/metrics/squad/compute_score.py) (and is correct).
* However, it is **tightly coupled to the SQuAD data schema**, making it hard to use “out of the box” for other QA datasets or generic LLM evaluation pipelines.
* As a result, I had to manually re-implement token-level F1 for my own benchmarking, even though the canonical implementation already lives in `evaluate`.

This feels like a missed opportunity for reuse, especially since many QA datasets (HotpotQA, Natural Questions, LongBench QA, etc.) use the same evaluation logic.

### Proposal

Rather than adding a new implementation, I’d like to explore making the existing **SQuAD token-level F1 reusable / pluggable** for other datasets.

I see three possible approaches and would love feedback on which direction is preferred:

#### Option A: Extract a generic QA F1 metric

* Factor out the token-level F1 logic (normalization + token overlap + max-over-refs) into a **generic QA metric** with a simple interface:

  ```python
  predictions: List[str]
  references: List[List[str]]
  ```
* Keep the current `squad` metric as a thin wrapper that adapts SQuAD’s schema to this generic metric.
* This avoids duplication and creates a single source of truth.

#### Option B: Add a thin “QA F1” wrapper that reuses SQuAD logic

* Introduce a new metric (e.g. `qa_f1`) that internally calls the existing SQuAD scoring code.
* Expose it with a dataset-agnostic input format (strings + list of references).
* Minimal refactor, but improves usability significantly.

#### Option C: Extend the existing `squad` metric interface

* Allow `evaluate.load("squad")` to optionally accept a simpler input format (in addition to the current SQuAD JSON schema).
* Lowest surface-area change, but still enables reuse for non-SQuAD datasets.

### Why this seems useful

* Avoids re-implementing a very standard QA metric across projects.
* Makes `evaluate` easier to use for modern LLM benchmarking workflows.
* Keeps existing SQuAD behavior intact while improving modularity.
* Complements `exact_match` (which already exists as a standalone metric).

If this direction makes sense, I’m happy to implement whichever option aligns best with the project’s design goals and contribution guidelines. Please let me know the preferred option.

Thanks for your time, and happy to iterate on the approach.
Repository: huggingface/evaluate
Issue: Handle zero divs in consistent way
Root cause: Validation Gap

Sklearn has a consistent way of dealing with division-by-zero (either ignore, error out or warn), which we pass on in wrappers (see [example](https://github.com/huggingface/evaluate/blob/main/metrics/precision/precision.py#L143)).

In our own canonical evaluation modules, we also have potential ZeroDivs that we currently do not handle, e.g.: https://github.com/huggingface/evaluate/blob/main/comparisons/mcnemar/mcnemar.py#L94. We should fix this.
Repository: huggingface/evaluate
Issue: Feature: type hints for metric inputs
Root cause: Unimplemented Feature Gap

We could add type hints to the metric inputs that go beyond the standard Python type hints. This could help knowing what the input format is (e.g. logits vs. normalized scores) and improve debugging (e.g. we could run some tests on inputs per depending on type -> `ValueError(f"Oops, looks like you passed unnormalized scores to {metric.name} but it expect values between 0 and 1. Try applying a SoftMax function first."`)

Some ideas/examples:
- `NormalizedArray` and `OneHotArray`/`MultiHotArray` for classification predictions and references, respectively.
- `LogitsArray` e.g. for cross-entropy.
- `TokenList` or `TextList` for NLP related scores
- `CategoricalPixelLabels` or `ContinuousPixelLabels` for image tasks

Not the best naming, yet, but hopefully gives an idea.

We could provide nice, visual examples of these types in the docs. I know input format tripped me up a lot when working with metrics libraries. 

cc @lhoestq @sashavor 

I agree this is the kind of documentation that is useful for users. Having good docstrings is already nice as documentation to clarify if inputs need to be normalized or in one-hot format. Having something like this would be mostly useful for sanity checks I think.

IIRC type hints in python are not supposed to do anything during runtime (even dataclasses don't do sanity checks). And it looks like duplicating the information already stored in the feature types no ?

Moreover as a user if I see that a functions has type hints that are not standard python types, I'm confused and don't always know what object I should pass:
- do I have to find to instantiate a class from the framework I'm using that has the type hint name ?
- can I guess the actual python type I have to use ?

Therefore I think it would make more sense to enrich current feature types rather than adding such type hints.

Note that we will probably add something similar to `CategoricalPixelLabels` in `datasets` as an actual feature type at the same level as `Image` though

For now I think we can stick to `Feature` types for type hints as they provide a lot of information and revisit this at a later time.
Repository: huggingface/evaluate
Issue: Add support for IR mertics
Root cause: Documentation Deficiency

Mertics like ndcg, dcg  

Hi @Muhtasham, we do have some IR metrics in `evaluate`:
```py
trec_eval = evaluate.load("trec_eval")
```
You can have a look here: https://huggingface.co/spaces/evaluate-metric/trec_eval

It returns a wide range of IR metrics.
Repository: huggingface/evaluate
Issue: TIMIT typically reports PER, not WER 
Root cause: Documentation Deficiency

The docs here mention that TIMIT reports WER, but this dataset typically serves as a benchmark for phone error rate (PER), because it’s one of the few resources that have manually annotated phone segments. I recommend to fix and clarify that in the README:

https://github.com/huggingface/evaluate/blob/c1141b02941dc508ca4560b9dfe5b7a90f4cf785/metrics/wer/README.md?plain=1#L68

I think it would be good to have a clear difference between word/character/phone/token error rate (WER/CER/PER/TER) at the library level.
Repository: huggingface/evaluate
Issue: Link to step-by-step guide is broken.

It's about this link in the README. 

<img width="946" alt="image" src="https://user-images.githubusercontent.com/1019791/171335566-3b669788-00e9-4770-ad1e-3da0453f7b4c.png">

When I click it I see a page with the following text:

```
The documentation page CREATING_AND_SHARING.HTML doesn’t exist in v0.1.0, but exists on the main version. Click [here](https://huggingface.co/docs/evaluate/main/en/creating_and_sharing.html) to redirect to the main version of the documentation.
```

When I click _that_ link I get 

<img width="429" alt="image" src="https://user-images.githubusercontent.com/1019791/171335631-d20da003-cb50-4dbd-9207-e088ecb1a946.png">


The correct link is: `https://huggingface.co/docs/evaluate/creating_and_sharing`
Repository: huggingface/evaluate
Issue: Loading BERTScore
Root cause: Validation Gap

Hi, 

I was testing loading bertscore with `evaluate` and saw this following error. 

```
>>> import evaluate
>>> metric = evaluate.load("bertscore")
Couldn't find a directory or a metric named 'bertscore' in this version. It was picked from the master branch on github instead.
```

Looks a bit mystic to me. Perhaps you have some insights :)

Thank you!

Nevermind I think this i just a benign warning. 

Indeed, it is just a warning - it's a bug we hoped to fix with the release. I'll report once it is fixed.
Repository: huggingface/lighteval
Issue: Evaluation of encoder and decoder models on SuperGLUE
Root cause: Documentation Deficiency

Hi guys,

I want to evaluate models like ModernBERT, Llama and many others on SuperGLUE and my own benchmark. In my setting, every model has to be fine-tuned for the specific task, even decoder models.

Is this currently supported by LightEval? Looking at the code, my feeling is that evaluations are only done by prompting.

Thanks.

hey ! not sure i understand your use case, do you want to evaluate using logprobs ?

Hi. No, I want to apply supervised fine-tuning (https://huggingface.co/docs/transformers/en/tasks/sequence_classification) on each model and then evaluate it on each task.

alright ! then you can do those two tasks seprately. First finetune your model, save the weights and upload it to the hub.
Then evaluate your model as you would any other model on the hub
Repository: huggingface/lighteval
Issue: RULER benchmark not found
Root cause: Documentation Deficiency

Hi all,

Thanks for your great work. The [README](https://github.com/huggingface/lighteval?tab=readme-ov-file#-chat-model-evaluation) says lighteval should support RULER benchmark, and I do find relevant PRs (#722, #726, #826 ).

In your previous commit, I can find 
> https://github.com/huggingface/lighteval/blob/57f292153d527df81d1bd5e6831f5dfe0fa63279/src/lighteval/tasks/extended/ruler/main.py

but the file was removed. Is there any reason that removing the RULER benchmark? Also, can I easily reuse your previous implementation to perform benchmarking in the current lighteval version (v0.11.0)?

@NathanHB 

hi ! sorry for the late answer. RULER is a bit peculiar in that you have to re-generate a new dataset for every different tokenizer  or tokenizer config you use. Therefore it was not stable.
what we did was add it to generate a dataset with the original repo, upload it to lighteval and run it, with the available backends.
Feel free if you need any help with the later part ! (all the code should be there already)
Repository: huggingface/lighteval
Issue: [FT] Upgrade the VLLM dependency to 0.10.2+
Root cause: External Dependency Breakage

## Issue encountered
Some newest LLM models can only supported by the new version VLLM. Currently, lighteval only support VLLM >= 0.10.0 and < 0.10.2

## Solution/Feature
I think the main reason to this issue is VLLM >= 0.10.2 use different interface for LLM inference (more specifically, `llm.generate`) . The VLLM >= 0.10.2 doesn't support the keyword argument `prompt_token_ids` in [case1](https://github.com/huggingface/lighteval/blob/b1d45e36a6255ebefbfc2aa8999b7b25a007d619/src/lighteval/models/vllm/vllm_model.py#L440C37-L440C53) and [case2](https://github.com/huggingface/lighteval/blob/b1d45e36a6255ebefbfc2aa8999b7b25a007d619/src/lighteval/models/vllm/vllm_model.py#L458)

We can see there are multiple interfaces of `llm.generate` in VLLM==0.10.1.1 in this page: [Link](https://docs.vllm.ai/en/v0.10.1.1/api/vllm/index.html#vllm.LLM.generate), where also indicates `prompt_token_ids` will be deprecated.

<img width="947" height="153" alt="Image" src="https://github.com/user-attachments/assets/93640741-cccf-4747-9bbd-083e18fc2e1d" />

However, in the VLLM==0.10.2, there is only one single interface in the official document: [Link](https://docs.vllm.ai/en/v0.10.2/api/vllm/#vllm.LLM.generate), which accept the inputs in `PromptType`. To my best knowledge, the `inputs` in `_generate` [ref.](https://github.com/huggingface/lighteval/blob/b1d45e36a6255ebefbfc2aa8999b7b25a007d619/src/lighteval/models/vllm/vllm_model.py#L408) function are assumed to be tokenized, so all we need to do is wrap the original `inputs` with `TokensPrompt` [ref.](https://docs.vllm.ai/en/v0.10.2/api/vllm/inputs/data.html#vllm.inputs.data.TokensPrompt) which contains the attribute `prompt_token_ids: list[int]`

**The solution**
1. Insert on line code before [L418](https://github.com/huggingface/lighteval/blob/b1d45e36a6255ebefbfc2aa8999b7b25a007d619/src/lighteval/models/vllm/vllm_model.py#L418)
```
inputs = [TokensPrompt(prompt_token_ids=input_ids) for input_ids in inputs]
```

2. Remove the keyword argument `prompt_token_ids` at [L440](https://github.com/huggingface/lighteval/blob/b1d45e36a6255ebefbfc2aa8999b7b25a007d619/src/lighteval/models/vllm/vllm_model.py#L440) and [L458](https://github.com/huggingface/lighteval/blob/b1d45e36a6255ebefbfc2aa8999b7b25a007d619/src/lighteval/models/vllm/vllm_model.py#L458C17-L458C33)

3. Update pyproject.toml, to remove `<0.10.2` in [vllm installation](https://github.com/huggingface/lighteval/blob/b1d45e36a6255ebefbfc2aa8999b7b25a007d619/pyproject.toml#L101C1-L101C57)

Done.


## Possible alternatives
A clear and concise description of any alternative solutions or features you've considered.


Hi Maintainers,

I've opened a PR #1003  and this issue may be related to #980.
I'd appreciate it if you could review it at your convenience.

Thank you,
Jie-Han Chen

Hi! We're at a conf this week and back next week, we'll look then! Thanks for the PR :)

Hi, any updates on this? It would be great to be able to upgrade vllm.
Repository: huggingface/lighteval
Issue: [BUG] GSMPlus fails with `'NoneType' object has no attribute 'id'`
Root cause: Validation Gap

## Describe the bug
GSMPlus is broken because we have this logic in the prompt

```py
    if line["perturbation_type"] == "critical thinking":
        return None
```

## To Reproduce
```
lighteval vllm "model_name=Qwen/Qwen3-4B-Instruct-2507,max_model_length=32768,gpu_memory_utilization=0.8,generation_parameters={max_new_tokens:32768,temperature:0.6,top_p:0.95}" "lighteval|gsm_plus|0" --max-samples 4
```

## Expected behavior
GSMPlus works :) 

## Version info
Please provide your operating system, lighteval version or commit if you installed from main, and pip/conda environment if your problem concerns dependencies.
Repository: huggingface/lighteval
Issue: [BUG] LiveCodeBench requires dataset scripts
Root cause: External Dependency Breakage

The live code bench [dataset](https://huggingface.co/datasets/livecodebench/code_generation_lite) requires script run to be loaded but the recent version of datasets do not support scripts datasets anymore and run into this error :
**`RuntimeError: Dataset scripts are no longer supported, but found code_generation_lite.py`**

Using : 
- datasets==4.1.1
- lighteval==0.8.1

I guess i can work around this for now by degrading the datasets version (still i don't know which version i need yet) but this is not a sustainable solution.

I would suggest working with the livecodebench folks to have their datasets in parquet/standarized format, or upload yours under lighteval org perhaps !? or maybe there is a 3rd option i might have been missing ...

That's it for now. I will update the comment later with which datasets version needed to avoid the error above.


**UPDATE**: `datasets==4.0.0` is working fine to avoid the above error

Hi ! you need to update the lighteval version. we moved live-code-bench data to the `lighteval` org and converted the files to parquet :)

I thought i already have the latest version :) 
Thanks Nathan
Repository: huggingface/lighteval
Issue: [EVAL] MultiChallenge
Root cause: Unimplemented Feature Gap

## Evaluation short description
Aside from Multi-IF, there are very few multi-turn instruction following evals. MultiChallenge is a hard version that OpenAI and others report in their model cards.

## Evaluation metadata
Provide all available
- Paper url: https://arxiv.org/abs/2501.17399
- Github url: https://github.com/ekwinox117/multi-challenge
- Dataset url: https://github.com/ekwinox117/multi-challenge/tree/main/data


can i look into this, if it's free to take up? @NathanHB 

of course ! 🤗 

so right now it's fine if this is implemented with just the `inspect-ai` backend, yes? like how we integrate the benchmark as a task

@lewtun can you please verify that this is the dataset that's used? https://huggingface.co/datasets/nmayorga7/multichallenge

the huggingface page linked to the paper links to this dataset, but i wanted to double-check
Repository: huggingface/lighteval
Issue: Vision Language Model Support?
Root cause: Documentation Deficiency

Hi I am using VLM but sometimes want to evaluate VLM on text only language dataset like MMLU or HumanEval or LiveCodeBench.

Is this repo supporting it?

hey! Yes, we are supporting MMMU in lighteval. You can see an example usage here:
https://github.com/huggingface/lighteval/blob/main/tests/slow_tests/test_accelerate_vlm_model.py


we are integrating [inspect-ai](https://inspect.aisi.org.uk/) framework into lighteval and plan to add vision models very soon.

https://huggingface.co/docs/lighteval/main/en/inspect-ai