zhiminy/EvalEng huggingface dataset:
Repository: huggingface/lighteval
Issue: [FT] Cache partial outputs to enable re-run from interruption
Root cause: Unimplemented Feature Gap

## Issue encountered

It is relatively common for an eval to fail mid-way in a large-scale compute environment (e.g. a faulty GPU, OOM issues, etc). Since `lighteval` only stores the outputs in memory, this means a failure requires a full re-run, which is costly for long running evals or large models. 

## Solution/Feature

It would be great if `lighteval` could store / cache partial outputs during evaluation so that if an evaluation fails, it can be re-run and started from where it left off. Frameworks like `datatrove` do this quite well for distributed data processing: https://github.com/huggingface/datatrove

## Possible alternatives

Perhaps I missed the ability to cache partial outputs, so please let me know if true and I can close this!

Similar to #825 , and I don't think the feature's been added yet but agreed it's a prio, I'll take a look at datatrove

Done, fixed in #909  ! :)
Repository: huggingface/lighteval
Issue: [INFO] Document or remove `logits` from the `ModelResponse` class
Root cause: Documentation Deficiency

@NathanHB I was exploring the ModelResponse class post refacto, and I noticed we have both logits (undocumented) and logprobs (documented) in the class. Logits is used for some classes (transforemrs) and logprob for others - I believe we should have a single class for both as they are usually used interchangeably and would allow us to have a single system. If we actually want to manage them differently, we can but it needs more documentation.
Repository: huggingface/lighteval
Issue: [BUG] : Typo in _continuous_greedy_until causes error with stop_sequence field
Root cause: Validation Gap

## Describe the bug
There is a bug in the transformer_models.py _continuous_greedy_until function. The code uses batch[0].stop_sequence, but the correct field should be batch[0].stop_sequences (with an "s" at the end). This typo causes an error when running the function, as the expected field is missing or incorrectly referenced.

## To Reproduce
Use the _continuous_greedy_until function with a batch that includes stop sequences.
The function tries to access batch[0].stop_sequence, but the batch object defines stop_sequences.
An error is raised due to the missing or incorrectly named field.

## Expected behavior
The function should correctly reference batch[0].stop_sequences to obtain the list of stop words.

## Version info
newest commit version
Repository: huggingface/lighteval
Issue: [BUG] If using a local model, save won't happen in the correct folder
Root cause: Algorithmic Error

## Describe the bug
```
self.output_dir = logging_path
self.general_config_logger.model_name = model_path # for a local model

output_dir_results = Path(self.output_dir) / "results" / self.general_config_logger.model_name
# returns model_path instead of logging_path/results/model_path, which is what we need (we might not have write access for model_path for ex)
```

Needs to be fixed across the code base to concatenate the paths properly
Repository: huggingface/lighteval
Issue: [BUG] Logs are missing some fields, truncation likely buggy
Root cause: Validation Gap

## Describe the bug
With a model max length at 10K on ifeval, I'm getting truncations on the input (with no input stored in the logs): most likely an issue with the truncation.

I suspect it's also linked to the refacto on loglikelihood: the padding no longer works for accelerate models.

## To Reproduce
`lighteval accelerate 'model_name=Qwen/Qwen3-32B,model_parallel=True,max_length=16384,generation_parameters={temperature:0.6,top_p:0.9,top_k:40,min_p:0,max_new_tokens:16384}' 'extended|ifeval|0|0' --save-details --output-dir tmp`

Logs will contain empty input fields in ModelResponse, with around 400 truncation



`longest_context_continuation_size_in_split = len(tokenized_context) + split[0].generation_size` should at least be `                longest_context_continuation_size_in_split = len(tokenized_context["input_ids"]) + split[0].generation_size`

Padding is fixed now thanks to #901 but we still need to check the truncation numbers
Repository: huggingface/lighteval
Issue: [BUG] TriviaQA numbers are off
Root cause: Configuration Error

## Describe the bug
I tried running triviaQA through lighteval and all the numbers seem to be zero, e.g.

```bash

$ lighteval vllm "model_name=Qwen/Qwen3-14B-AWQ" "lighteval|triviaqa|0|0"
...
[2025-08-04 15:42:37,715] [    INFO]: --- DISPLAYING RESULTS --- (pipeline.py:458)
|        Task        |Version|Metric|Value|   |Stderr|
|--------------------|-------|------|----:|---|-----:|
|all                 |       |qem   |    0|±  |     0|
|lighteval:triviaqa:0|       |qem   |    0|±  |     0|
```

Please forgive me if this is the expected behavior, but it seems off. I only tested various Qwen2.5-* and Qwen3-* instruction-tuned models.

Hi! Which commit are you on? 
For older versions of lighteval, you need to specify `use_chat_template` in the kwargs, for recent versions it's in the model params iirc, and now on main it should be automatically applied as needed. In general I would recommend using `--save-details` to be able to analyse what's happening sample per sample

I'm on the most recent commit. So `--use_chat_template=True`?

Most recent commit, chat template should be applied automatically then 

> chat template should be applied automatically

Ah, I guess it isn't then. My output is consistent with the most recent commit.

Using v0.10.0 I believe the accuracy is correct.
Repository: huggingface/lighteval
Issue: [BUG] inspect command does not work
Root cause: Validation Gap

## Describe the bug
lighteval inspect does not work due to some import and runtime errors

## To Reproduce
`lighteval tasks inspect "lighteval|gsm8k|0|0" --num-samples 1`

**PLEASE NOTE**: this PR fixes it: https://github.com/huggingface/lighteval/pull/887

## Expected behavior
Expected output:
```
---------- lighteval|gsm8k|0 ----------
---------- SAMPLES
-- sample 0 --
{ 'choices': [ ' Janet sells 16 - 3 - 4 = <<16-3-4=9>>9 duck eggs a day.\n'
               'She makes 9 * 2 = $<<9*2=18>>18 every day at the farmer’s '
               'market.\n'
               '#### 18'],
  'fewshot_samples': [],
  'fewshot_sorting_class': None,
  'generation_grammar': None,
  'generation_size': None,
  'gold_index': 0,
  'id': '0',
  'images': None,
  'instruction': None,
  'num_asked_few_shots': 0,
  'num_effective_few_shots': 0,
  'num_samples': 1,
  'original_query': None,
  'query': 'Question: Janet’s ducks lay 16 eggs per day. She eats three for '
           'breakfast every morning and bakes muffins for her friends every '
           "day with four. She sells the remainder at the farmers' market "
           'daily for $2 per fresh duck egg. How much in dollars does she make '
           "every day at the farmers' market?\n"
           'Answer:',
  'sampling_methods': [],
  'specific': None,
  'stop_sequences': None,
  'task_name': 'gsm8k',
  'unconditioned_query': None,
  'use_logits': False}
```

## Version info
lighteval installed from main branch
Repository: huggingface/lighteval
Issue: [BUG] Model Parallelism is incorrectly disabled on single multi-GPU machines
Root cause: Validation Gap

## Describe the bug
When running an evaluation with accelerate launch on a single machine with multiple GPUs, setting model_parallel=True in the TransformersModelConfig has no effect. The library logs that it is "not in a distributed setting" and force-overrides model_parallel to False. 

## To Reproduce
Set up an evaluation script on a single machine with 2 or more GPUs.
In the script, create a TransformersModelConfig for a large model (e.g., 7B+) and explicitly set model_parallel=True.
Launch the script using accelerate launch --num_processes=2 your_script.py.
Observe: We are not in a distributed setting. Setting model_parallel to False.

## Expected behavior
lighteval should respect the model_parallel=True configuration in a single-node, multi-GPU environment. It should proceed to shard the model across the available GPUs using device_map="auto" instead of disabling the feature.

## Version info
build from source code
Repository: huggingface/lighteval
Issue: [BUG]  Automatic batch size detection causes distributed deadlock in model parallel mode
Root cause: Architectural Constraint

## Describe the bug
When model parallelism is enabled and `batch_size` is not explicitly set in the config, the evaluation process hangs indefinitely. The log stops after the message "Detecting largest batch size...", indicating a distributed deadlock among the processes.

## To Reproduce
1. Configure an evaluation for a large, sharded model with model_parallel=True on a multi-GPU machine.
2. Crucially, do not specify a batch_size in the `TransformersModelConfig`, to allow the automatic detection feature to trigger.
3. Launch the script using `accelerate launch --num_processes > 1`.
4. Observe that the script and all worker processes hang permanently with no further output after the batch size detection log message appears.

## Expected behavior
The automatic batch size detection should either be compatible with model parallelism or be automatically skipped to prevent deadlocks. The evaluation should proceed with a safe default batch size (e.g., 1) or raise an error prompting the user to set a manual batch size when in model parallel mode.
## Version info
build from source code
Repository: huggingface/lighteval
Issue: [BUG] TransformersModel.from_model provides an incompletely initialized object compared to __init__
Root cause: Validation Gap

## Describe the bug
When creating a `TransformersModel` instance using the `from_model` classmethod, the resulting object is missing several attributes (e.g.continuous_batching) that are normally initialized by the standard `__init__ `constructor. This leads to AttributeError exceptions during the evaluation pipeline, making the `from_model` method unreliable for integrating pre-loaded models.

## To Reproduce
1, In an evaluation script, pre-load a model from transformers.
2. Instantiate the lighteval model wrapper using this pre-loaded model: model = TransformersModel.from_model(model=hf_model, ...)
3. Pass this model instance to a lighteval.pipeline.Pipeline.
4. Run pipeline.evaluate().
5. The pipeline will crash with an AttributeError when an internal method attempts to access an attribute that was not initialized by from_model (for example, `AttributeError: 'TransformersModel' object has no attribute 'continuous_batching')`.

## Expected behavior
An object created via `TransformersModel.from_model` should be fully and correctly initialized, possessing the exact same set of attributes as an object created via the standard `TransformersModel(config=...) ` constructor. It should be a drop-in replacement that works reliably within the pipeline.

## Version info
Please provide your operating system, lighteval version or commit if you installed from main, and pip/conda environment if your problem concerns dependencies.


Hi ! thanks for raising this, I made a fix in #921. Can you test it on your side and make sure it fixes your issue ?
Repository: huggingface/lighteval
Issue: spelled 'judgment' in 'bbh_causal_judgment_harness' but spelled "judgement" in the dataset repo (https://huggingface.co/datasets/lighteval/big_bench_hard)
Root cause: Interface Contract Mismatch

https://github.com/huggingface/lighteval/blame/64f93b0e4ae534830bc277567b671ae40fcc9185/src/lighteval/tasks/default_tasks.py#L1536

Hi! Is this preventing the task from working?
Repository: huggingface/lighteval
Issue: [BUG] `tasks list` is not a complete list
Root cause: Documentation Deficiency

Hey there,
I've noticed that running `lighteval tasks list` don't return all the tasks registered in lighteval like the `community`, `extended`, `multilingual` suites ... etc
The only suites i get are : `original`,`lighteval`,`leaderboard`,`helm`,`harness` and `bigbench`

Motivation:
I'm trying to push for the adoption of `lighteval` across some teams. In order to do this, i needed to show/check that all the required evals are already implemented. it's hard to do it manually, doing a fuzzy match (then review) across the file of tasks is just much practicle. unfortunately, this approach cannot be done now given that most benchmarks of interest are not in the tasks file saved from the cmd `lighteval tasks list`

@NathanHB any estimation of how much time will it take to fix this with respect to your bandwidth ?

Nathan is off atm - I agree this should work but the prio is going to be quite low on our side, but feel free to open a PR!

I am very interested in how to obtain the extra `community`  `extended` ... tasks you mentioned. I have quite a few eval tasks that don't exist in the official task list (eg. humaneval). This is not documented (or I was not able to have noticed it). Would you share some brief insights into how to find them? Thanks!

do you mean accessing already existing community tasks or adding your own custom tasks ?
in any case, for extended task you would need to install the extended tasks extra: `pip install -e .[extended_tasks]`

to use / create your custom tasks: [doc](https://huggingface.co/docs/lighteval/en/adding-a-custom-task)
Repository: huggingface/lighteval
Issue: [EVAL] Add IFBench
Root cause: Unimplemented Feature Gap

## Evaluation short description
IFEval is widely used to measure the instruction following capabilities of LLMs but is now quite saturated. IFBench is an improved version of IFEval which has broader diversity of task constraints. I think IFBench is likely to provide better signal for the next generation of models.

## Evaluation metadata
Provide all available
- Paper url: https://arxiv.org/abs/2507.02833
- Github url: https://github.com/allenai/IFBench
- Dataset url: https://huggingface.co/collections/allenai/ifbench-683f590687f61b512558cdf1


Fixed in #944
Repository: huggingface/lighteval
Issue: [FT] Relax upper bound on `datasets` dependency
Root cause: External Dependency Breakage

## Issue encountered
`datasets` version 4.0.0 was released 1 month ago and provides several nice features like multi-processing uploads for large datasets. It would be good to relax the upper bound `datasets<=4.0.0` in `lighteval` so the latest features can be used.

Note that `datasets>=4.0.0` deprecates the use of dataset loading scripts, so some care is needed to ensure that existing benchmarks like `livecodebench` still work. For those benchmarks, the simplest thing would be to convert the dataset loading scripts into parquet format and host under the `lighteval` org. This could be done with e.g. 

```sh
datasets-cli convert_to_parquet <dataset_id> --trust_remote_code
```

cc @lhoestq for viz who has also worked a lot to ensure popular eval datasets are already migrated to Parquet


This is quite important yes, we had issues with this while uploading RULER using datasets 4.0.
This is quite prio so will do before next release.

Will be fixed by #924
Repository: huggingface/lighteval
Issue: [FT] Automatically remove thinking traces from outputs
Root cause: Unimplemented Feature Gap

Per @lewtun 's request on slack: https://huggingface.slack.com/archives/C05B0D42GHE/p1754892588239859?thread_ts=1754819468.069879&cid=C05B0D42GHE
Repository: huggingface/lighteval
Issue: [FT] Pass reasoning effort to `litellm` endpoint.
Root cause: Unimplemented Feature Gap

## Issue encountered
There is currently no way to pass the `reasoning_effort` parameter to `litellm.completion(...)`. Without being able to turn on reasoning there is no way to do most flagship models justice. This might be a more general issue with other entrypoints too, am not familiar. LiteLLM supports optional literal values `minimal`, `low`, `medium`, `high`. 


## Solution/Feature
Would like to be able to specify something like

```python
model_config = LiteLLMModelConfig(
    model_name=model_name,
    api_key=api_key,
    generation_parameters=GenerationParameters(
        reasoning_effort="high"
    )
)
```

## Possible alternatives
Perhaps adding it through `GenerationParameters` doesn't make sense, in which case it could be added to `LiteLLMModelConfig` directly.

hey ! thanks for raising the issue. it is in fact a generation parameter. feel free to open a PR with this and we will review asap :)
Repository: huggingface/lighteval
Issue: [FT] Save reasoning_content of responses as details
Root cause: Unimplemented Feature Gap

## Issue encountered

When evaluating a reasoning model via litellm, it seems like `reasoning_content` is ignored by lighteval as of now: https://github.com/huggingface/lighteval/blob/458a3e96ea283f19dc53f93b21ae9f5345e8e384/src/lighteval/models/endpoints/litellm_model.py#L296 It would be nice to save `reasoning_content` of responses as details even if they are not used for computing metrics. Inspecting the reasonings can be helpful for development.

## Solution/Feature

litellm can return `reasoning_content` separately from `content`. https://github.com/BerriAI/litellm/blob/aac7cdf2c8b58f71042c661bc732d936fd4b09ce/litellm/types/utils.py#L575.

However, lighteval's `ModelResponse` does not have an attributes for reasoning texts. https://github.com/huggingface/lighteval/blob/4d2bf42ccdb8f575ece57a38290dab25cec48b83/src/lighteval/models/model_output.py#L29
I guess adding an attribute to it and setting its value in `LiteLLMClient` would be sufficient for the litellm entry point. I can make a pull request for the litellm entry point if this sounds ok. Other entry points would require similar modifications, but I haven't checked. 

Hi ! Feel free to open a PR for this. For the other entry point, the reasoning is bundled in the output text

Thanks, I opened a PR: #929
Repository: huggingface/lighteval
Issue: [FT] Support GGUF in vllm, and use HF tokenizer together
Root cause: Unimplemented Feature Gap

## Issue encountered
Currently, vllm support load GGUF model directly, and in their [document](https://docs.vllm.ai/en/v0.9.0/features/quantization/gguf.html), they suggest to use the tokenizer from huggingface, which seems not been supported in the current lighteval.

<img width="964" height="165" alt="Image" src="https://github.com/user-attachments/assets/df661106-31a9-4a0e-8096-bf79db680b24" />

## Solution/Feature
I want to use lighteval to evaluate GGUF model and use HF tokenizer rather than the tokenizer embedded in the GGUF model.

To support this feature, we can modify the file `src/lighteval/models/vllm/vllm_model.py`

First, add the tokenizer_name after model_name in the [Line#147](https://github.com/huggingface/lighteval/blob/ee31223d43247a79fe974fd3cb70e8769d886828/src/lighteval/models/vllm/vllm_model.py#L147)
```
tokenizer_name: str | None = None
```

Second, modify the _create_tokenizer in the [Line#293](https://github.com/huggingface/lighteval/blob/ee31223d43247a79fe974fd3cb70e8769d886828/src/lighteval/models/vllm/vllm_model.py#L293)
```
tokenizer = get_tokenizer(
            config.tokenizer_name if config.tokenizer_name else config.model_name,
            tokenizer_mode="auto",
            trust_remote_code=config.trust_remote_code,
            revision=config.revision,
        )
```

Third, move the [self.use_chat_template = uses_chat_template( ... )](https://github.com/huggingface/lighteval/blob/ee31223d43247a79fe974fd3cb70e8769d886828/src/lighteval/models/vllm/vllm_model.py#L181) after `self._tokenizer = self._create_auto_tokenizer(config)` as
```
self._tokenizer = self._create_auto_tokenizer(config)
self.use_chat_template = uses_chat_template(
            model_name=config.model_name, tokenizer=self.tokenizer, override_chat_template=config.override_chat_template
        )
```

## For some known issues
1. We need to modify the log, because it says it is an error when loading the GGUF

<img width="1038" height="99" alt="Image" src="https://github.com/user-attachments/assets/ac806928-3847-4b1c-93da-36fa0f3a9e90" />

2. 
Currently, vLLM doesn't support full-precision GGUF, which means the GGUF in fp16, bf16, and fp32 can not been successfully loaded in vLLM, but the community developers are dealing with this [issue](https://github.com/vllm-project/vllm/pull/23188).  I think this problem will be resolved in the next released of vLLM.

## Possible alternatives
A clear and concise description of any alternative solutions or features you've considered.
None, unknown.

Hi ! yeah i would be great to have. The proposed fix should work and be straightforward to implement, if you can open a PR I will be happy to review 

Hi @NathanHB,

I've already created the PR #972 for this issue.
Please take a look :)

Thanks,
Eli Chen

@NathanHB Due to the PR was merged. I would like to close this issue.
Thank you.
Repository: huggingface/lighteval
Issue: [FT] Option to retain model across evaluations to reuse the same vLLM instance
Root cause: Unimplemented Feature Gap

## Issue encountered
Summary:  initializing vLLM per task is expensive. Bundling the tasks into a single pipeline avoids the re-initialization but it prevents per task write outs which is friendly for preemption.

Explanation:
- We have logic to look for completed tasks by seeing if the results have already been written out within an eval config hash. This allows us to easily resume from checkpoint if the eval job is preempted. 
- This logic to be able to preempt a job mid-eval and safely resume is broken if we put all the tasks into a single pipeline.
- would be nice if we could pass the same vllm model object into different pipelines to avoid re-initialization while still separating the tasks.

## Solution/Feature
- A simple workaround could be accepting some optional flag in Pipeline or Pipeline.evaluate that determines if we clean up the model after the eval is run. 

## Possible alternatives
Please let me know if there is some way to get around this that I have missed.


Hi! We already have a caching system, is it not working for you?

hi ! a solution to this would be to spin up a vllm server on your machine and simply use litellm to ping it for evaluation. [like so](https://huggingface.co/docs/lighteval/main/en/use-litellm-as-backend)

> hi ! a solution to this would be to spin up a vllm server on your machine and simply use litellm to ping it for evaluation. [like so](https://huggingface.co/docs/lighteval/main/en/use-litellm-as-backend)

Thanks for the feedback!
Repository: huggingface/lighteval
Issue: [BUG] Evaluation tracker won't save task which relies on using metrics of 2 different types
Root cause: Validation Gap

## Describe the bug
EvaluationTracker.save() will fail at `dataset = Dataset.from_list([asdict(detail) for detail in task_details])` with 
```
Exception has occurred: ArrowInvalid       (note: full exception trace is shown but execution is paused at: _run_module_as_main)
cannot mix list and non-list, non-null values
```
if the task launched uses metrics requiring both generative and logprobs metrics. (because they don't save lists of the same lengths for ex)

## To Reproduce
```
            "name": "launch vllm",
            "type": "debugpy",
            "request": "launch",
            "module": "lighteval",
            "args": [
                "vllm",
                "model_name=Qwen/Qwen3-0.6B,max_num_batched_tokens=100000,max_model_length=38912,generation_parameters={temperature:0.6,top_p:0.95,top_k:20,min_p:0,presence_penalty:1,max_new_tokens:38912},system_prompt='/no_think'",
                "lighteval|mmlu_redux_2:security_studies|0", //lighteval|mmlu_redux_2|0|0",
                "--max-samples",
                "10",
                "--save-details",
            ],
```
Repository: huggingface/lighteval
Issue: [FT] Multilingual Extractiveness Metrics
Root cause: Unimplemented Feature Gap

## Issue encountered
Currently, the Extractiveness metric uses the `en_core_web_sm` spaCy model, making it an English-only metric. For our evaluations, we need to use that metric with German, French, and Italian spaCy models.

## Solution/Feature
Add a parameter to the Extractiveness metric, that defines the language, and load the corresponding spaCy model

As part of a community task I have been collaborating on, I've implemented the changes because they were required for our evaluations. I will open a PR right away.