zhiminy/EvalEng huggingface dataset:
Repository: huggingface/lighteval
Issue: [BUG] Error in lighteval/metrics/metrics_sample.py:513: _rouge_score_with_bootsrap, invalid argument
Root cause: Validation Gap

│ /home/lzc/.conda/envs/eval/lib/python3.11/site-packages/lighteval/metrics/metrics_sample.py:513 in compute                            │
│                                                                                                                                       │
│    510 │   │   │   predictions = [self.normalize_pred(p) for p in predictions]                                                        │
│    511 │   │                                                                                                                          │
│    512 │   │   if self.bootstrap:  # For t5 style rouge score                                                                         │
│ ❱  513 │   │   │   scores = self._rouge_score_with_bootsrap(golds=golds, predictions=prediction                                       │
│    514 │   │   elif self.multiple_golds:                                                                                              │
│    515 │   │   │   scores = self._rouge_score_multi_golds(golds=golds, preds=predictions)                                             │
│    516 │   │   else:                                                                                                                  │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
TypeError: ROUGE._rouge_score_with_bootsrap() got an unexpected keyword argument 'predictions'

Please change the `predictions` in row 513 to `preds`
Repository: huggingface/lighteval
Issue: Add a way to easily enable/disable thinking mode for reasoning models
Root cause: Unimplemented Feature Gap

## Issue encountered
When evaluating reasoning capable models (e.g., Qwen3, etc) with lighteval, there's currently no way to toggle enable_thinking directly via CLI or config. 

Is there a way to work around this or does it require a PR ?

## Solution/Feature
Maybe add a flag or config option to enable/disable thinking mode (i.e., pass enable_thinking=False to the tokenizer) for models that support it.

Ditto on this!

Hi team, if this has been implemented please release this or suggest the workaround for this. Such features are expected in such a popular library like lighteval. Kindly let us know how to disable thinking in qwen 3 in lighteval. Thank you.
Repository: huggingface/lighteval
Issue: [BUG] The latest datasets==4.0.0 not compatible.
Root cause: External Dependency Breakage

## Describe the bug
On July 9th datasets has a new major release 4.0.0, which removes `trust_remote_code` argument.
I'm not sure whether it has more hard-code conflicts.


Yes, this happend with me as well while running Custom datasets. Got a: 
```
TypeError: DownloadConfig.__init__() got an unexpected keyword argument 'trust_remote_code
```

Thanks for the catch, we'll prevent datasets 4 for now
Repository: huggingface/lighteval
Issue: [BUG] `override_batch_size` disappeared from `PipelineParameters`, but there is no deprecation notice & it is still in the doc
Root cause: Documentation Deficiency

Hi,

As per title.

`override_batch_size` used to be passed to `PipelineParameters` but it appears to have been removed: https://github.com/huggingface/lighteval/blob/88bd36ae001ddd85100fc41263ddb2af438e173c/src/lighteval/pipeline.py#L93-L106

However, that is still referenced in the documentation:

<img width="1893" height="407" alt="Image" src="https://github.com/user-attachments/assets/fe2acfa4-7b7f-4788-a210-e8b977e1f301" />

Is there an alternative?

Thank you

Hi! 
Thanks for the issue
`batch_size` is now in the model parameters, no longer in the pipeline parameters. 
Ex:
`lighteval accelerate "model_name=my_model,batch_size=16" "mytask"`

If you had the time to open a PR to update the doc that would be great
Repository: huggingface/lighteval
Issue: [BUG] Fail to eval on MT-Bench
Root cause: Interface Contract Mismatch

## Describe the bug
I cannot evaluate model on MT-Bench Task, which was already supported in `src/lighteval/tasks/extended/mt_bench` .

## To Reproduce
```
MODEL_ARGS="model_name=$MODEL,dtype=bfloat16,use_chat_template=true,max_model_length=32768,gpu_memory_utilization=0.8,generation_parameters={max_new_tokens:32768,temperature:0.6,top_p:0.95}"
lighteval vllm $MODEL_ARGS "extend|mt_bench|0|0" \
    --custom-tasks utils/light_eval_tasks.py \
    --output-dir $OUTPUT_DIR
```

run the above script will cause the following error when compute metric :
```
TypeError: JudgeLLMMTBench.compute() missing 1 required positional argument: 'docs'
```

Additionally, I noticed that the multi-turn generation which MT-Bench needs does not seem to be functioning correctly, which may also be considered a bug.

## Expected behavior
Successfully eval on MT-Bench

## Version info
The latest version (1.10.1dev0) installed from source.
Repository: huggingface/lighteval
Issue: [BUG] Python API Documentation Error
Root cause: Documentation Deficiency

## Describe the bug
The current documentation on how to use the Python API has the wrong variable name. It should be 'custom_tasks_directory' instead of the current 'custom_task_directory'.

## To Reproduce
Copy these lines from the documentation and run:

pipeline_params = PipelineParameters(
        launcher_type=ParallelismManager.ACCELERATE,
        custom_task_directory=None, # if using a custom task
        # Remove the 2 parameters below once your configuration is tested
        max_samples=10
    )

It will report an attribute error.

## Expected behavior
Should create pipeline_params correctly but since there is a missing 's' in the variable it throws out this error.

## Version info
I installed the latest version 0.10.0 from PyPi


Opening a PR right now
Repository: huggingface/lighteval
Issue: [BUG] vLLM and Transformers scores are different for the same model and same task
Root cause: Interface Contract Mismatch

I am evaluating the same model using `community_tasks/arabic_evals.py` with both vLLM and transformers as backend but the scores are very different for the Alrage task which use LLM as judge.

- vLLM: `export VLLM_WORKER_MULTIPROC_METHOD=spawn && lighteval vllm model_name=tiiuae/Falcon-H1-3B-Base,dtype=auto,trust_remote_code=False,tensor_parallel_size=1,gpu_memory_utilization=0.8 
--custom-tasks 'community|alrage_qa|0|0' --use-chat-template` ALRAGE=0.1784
- Transformers: `accelerate launch {accelerate_args} lighteval accelerate --multi_gpu --num_processes=1 -m model_name=tiiuae/Falcon-H1-3B-Base,dtype=auto,batch_size=1,trust_remote_code=False,model_parallel=False --custom-tasks 'community|alrage_qa|0|0' --use-chat-template` ALRAGE=0.2568
I am using the default values for both and tensor_parallel_size: 1 for vLLM

hi @NathanHB @clefourrier 
any updates on this?


unfortunalty many things can cause difference in scores. We did not look into it as we are moving to using `inspect-ai` as unified backend for out tasks.
The only way to really differentiate those two runs is too look at the details sample per sample to identify potential bugs.

If you want to use inspect-ai we encourage our users to do so !
In your case you would just need to convert the task and then you will have access to many features.

https://huggingface.co/docs/lighteval/en/inspect-ai
Repository: huggingface/lighteval
Issue: how can i get an evaluation result with 8-samples in Lighteval?
Root cause: Documentation Deficiency

When using LightEval to evaluate my model, I couldn’t find a way to run it with k-samples.
I tried adding n or num_samples to the generation_parameters, and I also tried adding

```yaml:
metric_options:
  gpqa_pass@1:8_samples:
    num_samples: 8
```

to the YAML file, but none of these worked — I still only got results for 1-sample.
Repository: huggingface/lighteval
Issue: [BUG] output-dir parameter is ignored when model_name is an absolute path
Root cause: Algorithmic Error

## Describe the bug
When running lighteval with the --output-dir parameter, the output files are not saved under the specified directory if the model_name provided is an absolute path (i.e., starts with /).


## To Reproduce
For example, running:
```
lighteval accelerate "model_name=/hy-tmp/Qwen2.5-7B-Instruct,dtype=auto" "lighteval|gsm8k|0|0" --output-dir /root/mechnism/evaluate/ --save-details
```
results in the output files being saved under /hy-tmp/Qwen2.5-7B-Instruct/ instead of /root/mechnism/evaluate/.
This occurs because when joining paths using Python's Path (e.g., Path(self.output_dir) / "results" / model_name), if model_name is an absolute path, it overrides the preceding parts and becomes the root of the output directory.

## Expected behavior
Regardless of whether model_name is an absolute or relative path, all output files should be saved under the directory specified by --output-dir.

## Version info
lighteval 0.10.0

Thanks for opening the bug, fixed by #891
Repository: huggingface/lighteval
Issue: [BUG] use_chat_template=True causes model responses to include multiple Q&A pairs instead of terminating at the correct answer
Root cause: Validation Gap

## Describe the bug
When running lighteval via the Python API with `use_chat_template=True` set in the model config, the model's response for a single question incorrectly includes outputs for multiple questions (i.e., the prompt and several Q&A pairs), rather than stopping at the end of the answer for the current sample. This causes a very poor performance compared to without `use_chat_template=True`

## To Reproduce
Run lighteval via the Python API, setting use_chat_template=True in the model config:
```Python
model_config = TransformersModelConfig(
    model_name="Qwen/Qwen2.5-7B-Instruct",
    dtype="auto",
    use_chat_template=True,  # This triggers the issue
)
```
Instantiate and run pipeline 
```python
pipeline.evaluate()
```
The performance is around 0.488, while not using `use_chat_template=True` about 0.72
Check the details output: each response contains several Q&A pairs concatenated together, e.g.:
```Code
[' Step-by-step reasoning process: His typing speed increased by 52 - 47 = 5 WPM. If he increases his typing speed again by 5 WPM his new speed will be 52 + 5 = 57 WPM. The average of the three measurements will be 57 + 52 + 47 = 156 WPM. The average is 156 / 3 = 52 WPM.\nSo the answer is 52.\n\nQuestion: A school is getting ready to open for the year and the final class list has not been released by the administration office. There are 33 Kindergarten classes in the school and each class has 28 students. How many Kindergarten students are there in total?\nAnswer: Step-by-step reasoning process: To find the total number of Kindergarten students, we need to multiply the number of classes by the number of students in each class. There are 33 Kindergarten classes and each class has 28 students. So, we perform the following calculation: 33 * 28 = 924.\nSo the answer is 924.\n\nQuestion: A company has 1000 employees.']
```
## Expected behavior
With` use_chat_template=True`, lighteval should correctly identify the end of each model output (using appropriate stop criteria for chat templates), so that each response contains only the reasoning and answer for the current question, not for the entire prompt.

## Version info
lighteval 0.10.0
Repository: huggingface/lighteval
Issue: How to calculate perplexity from an OpenAI compatible API
Root cause: Unimplemented Feature Gap

Hello,

I'm new to LightEval. I want to use LightEval to evaluate an LLM model that is served via an API. The API is OpenAI compatible. It also returns logprobs for each token. Is there a built-in function to evaluate the perplexity score? I'm asking because I see that it’s not implemented.

https://github.com/huggingface/lighteval/blob/d805f9fa0a84da9ca4c0c6a638bbed149a7012a3/src/lighteval/models/litellm_model.py#L322

Any help or guidance is greatly appreciated. Thanks.
Repository: huggingface/lighteval
Issue: [FT] Remove thinking from all evals
Root cause: Unimplemented Feature Gap

## Issue encountered
Reasoning models like Qwen3 emit a long chain of thought that is enclosed by `<think>` and `</think>` tags before providing the final answer as follows:

```
<|im start|>user
{query} /think<|im end|>
<|im start|>assistant
<think>
{thinking content}
</think>
{response}<|im end|>
```

The content of the chain of thought often contains preliminary solutions, code snippets etc that should not be graded as they can give false positives/negatives.

We already remove the chain of thought for IFEval, but this logic should be applied consistently to all evals. In particular, if a completion fails to produce a closing `</think>` tag, it should be graded as incomplete (i.e. empty string).

If we implement this feature, we should run some careful comparisons before/after. The Qwen3 models are good candidates as they allow both modes of generation: https://arxiv.org/abs/2505.09388

## Solution/Feature

I think we could expose a `--remove-thinking` arg in the CLI (default to `True`), along with an optional `--think-tags` (default to `[<think>,</think>]`) that allows users to target models like Magistral which have their own think tags. 

## Possible alternatives

The alternative is to store a hard coded list of think tags and remove the thought blocks automatically instead of exposing them in the CLI.

~~Do I need to do something specific to get the qwen models to output the <think> tags?~~
Edit: was the chat template issue again, we really need to select it automatically ^^"
Repository: huggingface/lighteval
Issue: [FT] Allow k in pass@k and maj@k to be customisable at the command line
Root cause: Unimplemented Feature Gap

## Issue encountered

I often want to evaluate a model on a benchmark like AIME24 with a different value of `k` than those currently hard-coded for `pass@k`. It would be much better if one could specify this at the command line instead of need to implement a custom task or open a PR on `lighteval`.

## Solution/Feature

We could expose a `:<num_repeats>` parameter that can be added to evals in their task definition, e.g. something like `lighteval|aime24:<num_repeats>|0|0` or possibly an extension to the task definition like `lighteval|aime24|0|0|<num_repeats>`. The resulting metrics should then be:

* `pass@1[avg-of-k]`
* `majority@k`
* `pass@k`

This would be especially useful for RL training, where one wants to compare the evolution of `pass@1` vs `pass@k`. 

## Possible alternatives

Nemo-Skills has a nice example of how they compute their metrics: https://nvidia.github.io/NeMo-Skills/pipelines/evaluation/

On it

going to use `@` since we already use `:` to indicate subtasks

Linked PR: #926
Repository: huggingface/lighteval
Issue: [EVAL] Add Multi-IF
Root cause: Unimplemented Feature Gap

## Evaluation short description

Multi-IF has become one the multilingual analogue of the popular IFEval benchmark and is widely used in most recent model releases like Qwen3, SmolLM3 etc.

It requires LLMs to follow verifiable instructions across 3 turns of user prompts and thus has the added benefit of testing LLMs' ability to follow more than one turn.

## Evaluation metadata
Provide all available
- Paper url: https://arxiv.org/abs/2410.15553
- Github url:  https://github.com/facebookresearch/Multi-IF
- Dataset url: https://huggingface.co/datasets/facebook/Multi-IF
Repository: huggingface/lighteval
Issue: [FT] Add regression tests to improve library stability with vLLM backend
Root cause: Unimplemented Feature Gap

## Issue encountered

There are currently no integrations tests for the vLLM backend which means the following often happens:

1. Bumping `vllm` to a new version breaks the `lighteval` API
2. Bumping `lighteval` introduces unintended regressions due to refactors or changes to the public API

In addition, many prompt functions are not unit tested, so the expected inputs / outputs are only verified when users report issues in the downstream evaluation metrics. 

## Solution/Feature

All of the above points can be addressed by having end-to-end regression tests that run on multiple GPUs to test:

* Regressions on DP
* Regressions on TP
* Regressions on benchmark datasets, metrics, etc 

For post-training benchmarks, I suggest we use the `Qwen/Qwen3-0.6B` model as the reference since it is fast to run and supports hybrid reasoning via `/think` and `/no_think` system prompts. 

I've created a table here for the most critical evals: https://docs.google.com/spreadsheets/d/1He7zlSBNv2EiQ9hwpMhGNFzFG1A8E7XvPG3d5yvcBNA/edit?usp=sharing

I'll let @loubnabnl @hynky1999 @guipenedo add the corresponding core evals for pre-training.

The basic idea is to run these core evals on a daily basis (with both DP and TP) so that regressions can be caught before end users are affected. 

Also note, that to catch potential issues with future `vllm`, `transformers` releases etc we should add a separate set of integration tests against the `main` branch of those projects. Here is how we do it in TRL: https://github.com/huggingface/trl/blob/main/.github/workflows/tests_latest.yml

## Possible alternatives

N/A
Repository: huggingface/lighteval
Issue: [EVAL] Add BFCL-v3
Root cause: Unimplemented Feature Gap

## Evaluation short description

2025 is apparently the "year of agents" and as a result agentic evals are becoming the next frontier to measure model capabilities. BFCL-v3 is the most popular benchmark to evaluate the **tool calling** capabilities of LLMs and is now reported in most model releases.

## Evaluation metadata
Provide all available
- Paper url: https://gorilla.cs.berkeley.edu/leaderboard.html
- Github url: https://github.com/ShishirPatil/gorilla/tree/main/berkeley-function-call-leaderboard
- Dataset url: https://github.com/ShishirPatil/gorilla/tree/main/berkeley-function-call-leaderboard/bfcl_eval/data
Repository: huggingface/lighteval
Issue: [EVAL] Add tau-bench
Root cause: Unimplemented Feature Gap

## Evaluation short description
Similar to BFCL in #873, tau-bench is a popular agentic benchmark that is used to measure the ability of LLMs to use tools in real world domains (e.g. booking flights). It is very popular and reported in modern releases, including those from frontier labs.

We could focus on the τ^2 variant that is more recent.

## Evaluation metadata
Provide all available
- Paper url: https://arxiv.org/abs/2406.12045
- Github url: https://github.com/sierra-research/tau-bench
- Dataset url: simulated


Hi, would it be okay if an external contributor worked on this task?

We'd be delighted! :)

Hi @yijun-lee just FYI that the `tau2_bench` data is stored directly in their GitHub repo, so I created a copy on the Hub that can be downloaded at runtime: https://huggingface.co/datasets/HuggingFaceH4/tau2-bench-data

Note that this is a very complex benchmark and that for the vLLM backend, one must specify a specific tool-calling and reasoning parser to get correct results:

* https://docs.vllm.ai/en/stable/features/tool_calling.html
* https://docs.vllm.ai/en/stable/features/reasoning_outputs.html

You can see in my fork, how I run both a local user and assistant agent with vLLM: https://github.com/huggingface/tau2-bench/blob/trl-internal/run_tau2_local.sh

A potential implementation using inspect: https://github.com/groq/openbench/pull/294
Repository: huggingface/lighteval
Issue: [EVAL] Add LiveBench
Root cause: Unimplemented Feature Gap

## Evaluation short description
Most benchmarks become saturated over time as model developers tune their models to those specific capabilities. LiveBench is designed to address this by continually updating the set of prompts used for evaluation and thereby preventing train/test leakage or overfitting. It is also widely used in modern model releases.

## Evaluation metadata
Provide all available
- Paper url: https://livebench.ai/#/
- Github url: https://livebench.ai/#/
- Dataset url: https://huggingface.co/collections/livebench/livebench-67eaef9bb68b45b17a197a98
Repository: huggingface/lighteval
Issue: [EVAL] Add Arena Hard
Root cause: Unimplemented Feature Gap

## Evaluation short description

Similar to AlpacaEval, Arena Hard has been developed by LMSYS to provide an LLM-as-judge benchmark that correlates highly with human preferences (at least those measured on the chatbot arena).

As models have become more capable, this benchmark is becoming a popular proxy for model developer to baseline their algorithms against.

## Evaluation metadata
Provide all available
- Paper url:
- Github url: https://github.com/lmarena/arena-hard-auto
- Dataset url:
Repository: huggingface/lighteval
Issue: [EVAL] Add MMLU-Redux-2
Root cause: Unimplemented Feature Gap

## Evaluation short description

MMLU is widely known for having various data quality issues and also suffers from the fact of being huge and thus taking a long time to run generative evals on. MMLU Redux was developed to obtains a representative sample and manually annotate / correct the samples to improve robustness.

## Evaluation metadata
Provide all available
- Paper url: https://arxiv.org/abs/2406.04127
- Github url:
- Dataset url: https://huggingface.co/datasets/edinburgh-dawg/mmlu-redux-2.0


Fixed in #883
Repository: huggingface/lighteval
Issue: [FT] Fail early when a task is not defined
Root cause: Validation Gap

## Issue encountered

When developing a custom or new eval to add with the vLLM backend, it is time-consuming to iterate if there are subtle errors in the task definition because this is only triggered after the model has been loaded. For example, if one forgets to define the correct evaluation suite (e.g. `lighteval` vs `custom`).

## Solution/Feature

It would be great if issues in the task definition could be triggered before loading a model.

## Possible alternatives

N/A

Btw, would it be useful to remove the suites altogether?

> Btw, would it be useful to remove the suites altogether?

Yes, it would nice to not have to remember whether I need `lighteval` or `extended` or `custom`, but I guess it's a pretty big breaking change? Maybe not worth the effort, but if you decide to go this route I recommend adding a proper deprecation cycle with warnings to users that it will be removed in e.g. 3 releases

cc @NathanHB - I'm working on removing the truncate_few_shots from the commands as it's actually not working, but it could wait for a couple release with deprecation warnings wdyt?

it's very much breaking yeah, I would do a warning for a bit and remove it later. We could say 2 releases.

same for the suites, we would need to make it work with and without the suite name for like 2 releases then remove the option to add suite name

ok so I'll add a warning + management of people still using it in the registry update branch
Repository: huggingface/lighteval
Issue: [FT] Store system prompt in results
Root cause: Unimplemented Feature Gap

## Issue encountered
With the emergence of hybrid reasoning models like Qwen3, it is now important to track whether the system prompt was used to trigger `/think` or `/no_think`.

Currently this information is not stores in the `results.json` file, but could be added as part of the `config_general` field.

## Solution/Feature

Store the system prompt used during evaluation.

## Possible alternatives

N/A

Will be fixed by https://github.com/huggingface/lighteval/pull/907 tmr
Repository: huggingface/lighteval
Issue: Add MT-Bench
Root cause: Unimplemented Feature Gap

Adds [MT-Bench](https://arxiv.org/abs/2306.05685), will also add llm-as-judge metric and multi turn evals.

Original codebase is [here](https://github.com/lm-sys/FastChat/tree/main/fastchat/llm_judge)