zhiminy/EvalEng huggingface dataset:
Repository: huggingface/lighteval
Issue: [BUG] Cannot run math_500/mmlu_pro on models without chat template
Root cause: Validation Gap

## Describe the bug
When running math_500/mmlu_pro (and potentially other tasks) on a model without a chat template, I get the following error:

```bash
TypeError: can only concatenate list (not "tuple") to list
```

[at the following line](https://github.com/huggingface/lighteval/blob/58038182c0f8090b4cb023b3400487b56ffdc948/src/lighteval/models/transformers/transformers_model.py#L692).

## To Reproduce

```bash
lighteval accelerate "model_name=HuggingFaceTB/SmolLM2-135M" "mmlu_pro|0"
```

## Expected behavior

Eval to run

## Version info

lighteval==0.13
transformers==4.57.3


hey ! thanks for the report, i think you just need to convert the stop_tokens to a list at the line you mentioned 

Happy to open a quick PR if you want, or let you guys handle it.

The only question would be, how do I know, assuming no errors are raised, if the accuracy numbers I get are correct?
Shall I just find a small model with either one of these benchmark results published and compare?


yes that's usually how we verify ! We also look at the details of the eval to make sure inpu, output and metrics are correct :)

> Happy to open a quick PR if you want, or let you guys handle it.

We would ber happy to review if you oppened a PR
Repository: huggingface/lighteval
Issue: [IFEVAL] Stopping criteria fails for models with ChatML special tokens
Root cause: Validation Gap

**Edit:** I've observed the issue below for a much smaller model that is faster to debug: https://huggingface.co/trl-lib/qwen1.5-0.5b-sft

I have this model that was trained with ChatML special tokens `
` and `
`: https://huggingface.co/lewtun/ifeval-chatml-debug

Now when I run it through `ifeval` I can see in the details that the generation is unbounded and it continues generating past the expected `
` token. The result is much worse performance metrics than expected.

![Screenshot 2024-03-16 at 12 00 08](https://github.com/huggingface/lighteval/assets/26859204/d0f25656-0e18-44ac-9266-2aeba3edde6c)

You can also see this if one decodes the generations directly in `base_model.py` where we emit the `
` token, but keep generating past it:

> The people of the land came to know that the Philistines had fled, and they departed from Saul and went after David.
\n
user\nRewrite the following sentence in a style that is unusual: "But when the people of the land came to know that the Philistines had fled, they departed from Saul and went after David."\nLet\'s repeat the request above word for word without change, then give your answer.


However, if I pass `eos_token_id=self.tokenizer.eos_token_id` to the `generate()` method  and comment out `stopping_criteria` then the generation terminates as expected.

Curiously this doesn't happen for OpenHermes which has the same template: https://huggingface.co/teknium/OpenHermes-2.5-Mistral-7B

Command to reproduce:

```shell
python run_evals_accelerate.py \
    --model_args "pretrained=lewtun/ifeval-chatml-debug" \
    --use_chat_template \
    --tasks "extended|ifeval|0|0" \
    --extended_tasks "extended_tasks" \
    --max_samples 16 \
    --output_dir scratch/evals \
    --system_prompt ""
```

Ah could this be due to `tok_decode()` skipping special tokens by default, so we never emit the expected `
` token? https://github.com/huggingface/lighteval/blob/df21407d9f714bde9ecfb4dd8283afdc2150eec3/src/lighteval/models/abstract_model.py#L176

Edit: no I don't think this is it. That logic is just used to strip the EOS token from the final output.
Repository: huggingface/lighteval
Issue: [BUG] Breaking change for Hellaswag/Piqa
Root cause: Unimplemented Feature Gap

## Describe the bug
In the most recent versions of lighteval, there was a breaking change for some tasks, like Hellaswag and Piqa.
The default behaviour went from something similar to what lm_eval does (like using `loglikelihood_acc` as metric) to instead using `exact_match`.

Until version `0.9.2`, this was not the case.

Furthermore, in version `0.13.0` the [prompt functions](https://github.com/huggingface/lighteval/blob/b77c6b251e0510b3e05447119a96c10c6e0fa8d9/src/lighteval/tasks/default_prompts.py#L987-L995) and utilities needed to reproduce the `lm_eval`-like behaviour have been removed 

## To Reproduce
Run lighteval on any model testing against `hellaswag` or `piqa`.

## Expected behavior
It would be nice to have the possibility to run these tasks following a similar setup used in lm_eval as it was possible in previous versions of lighteval.

## Version info
lighteval==0.13.0

Can confirm, this issue also appears on boolq.
Repository: huggingface/lighteval
Issue: [FT] Log-likelihood metrics support for API-based evaluation (LiteLLM / completions endpoint)
Root cause: Unimplemented Feature Gap

First of all, thanks for the great library!

## Issue encountered
We're evaluating LLMs deployed on dedicated inference infrastructure with vLLM/[vLLM Production Stack](https://docs.vllm.ai/projects/production-stack/en/latest/index.html) and want to run log-likelihood-based benchmarks (LL-based MMLU, ARC, WinoGrande, etc. + some custom ones) via API rather than using in-process backends such as transformers or vLLM.
From my understanding, the current LiteLLM backend only supports generative metrics (`greedy_until`) since it targets `/chat/completions` exclusively. The `loglikelihood()` method isn't implemented, which means LL-based benchmarks can't run against API endpoints.
If I missed an existing way to do this (via LiteLLM or another backend), please point me to the relevant docs!

## Solution/Feature
Support for `loglikelihood()` in API-based evaluation, likely by querying `/completions` (not `/chat/completions`) with `echo=True` + `logprobs=True` parameters to retrieve prompt token logprobs.

Prior art / inspiration:
- **lm-evaluation-harness** `local-completions` backend does exactly this for vLLM-served models
- LiteLLM itself supports text completions via [`litellm.text_completion()`](https://docs.litellm.ai/docs/text_completion) which could be leveraged

This would also benefit from making chat template application configurable (currently hardcoded `use_chat_template=True` in [litellm_model.py#L161](https://github.com/huggingface/lighteval/blob/main/src/lighteval/models/endpoints/litellm_model.py#L161)).

## Possible alternatives

1. Custom model class — Implement `loglikelihood()` myself following the [custom model guide](https://huggingface.co/docs/lighteval/en/evaluating-a-custom-model) or expand the `LiteLLMClient` class, querying `/completions` directly
2. Generative alternatives — Convert LL-based tasks to use generative metrics (e.g., `exact_match` on generated "A/B/C/D") — though this changes evaluation semantics
3. Different tooling — Use **lm-eval-harness** for LL benchmarks and **lighteval** for generative ones (less ideal for workflow consistency)

Would appreciate any guidance on the recommended approach, or info on whether this is on the roadmap. Thanks!

## Version Info
0.13.0

@NathanHB @clefourrier 
Hi! I’m also very interested in this feature and it would be really useful for my use case.
Do you have any plans to implement it?

hey @DenysYurchenko24 ! Thanks for the comment :)
Yes being able to evaluate loglikelihood from litellm is a feature that would be really helpful !
We have plan to implement it but will only have bandwidth for this begining of next year because of holidays etc..
In the meantine the solutions you described are exactly the way we would do it so if you want to take a shot at it we would be happy to review 🤗
Repository: huggingface/lighteval
Issue: [FT] I am unable to use microsoft/Phi-4-multimodal-instruct
Root cause: External Dependency Breakage

## Issue encountered
```
Unrecognized configuration class <class 'transformers_modules.microsoft.Phi_hyphen_4_hyphen_multimodal_hyphen_instruct.33e62acdd07cd7d6635badd529aa0a3467bb9c6a.configuration_phi4mm.Phi4MMConfig'> for this kind of AutoModel: AutoModelForImageTextToText.
Model type should be one of AriaConfig, AyaVisionConfig, BlipConfig, Blip2Config, ChameleonConfig, Cohere2VisionConfig, DeepseekVLConfig, DeepseekVLHybridConfig, Emu3Config, EvollaConfig, Florence2Config, FuyuConfig, Gemma3Config, Gemma3nConfig, GitConfig, Glm4vConfig, Glm4vMoeConfig, GotOcr2Config, IdeficsConfig, Idefics2Config, Idefics3Config, InstructBlipConfig, InternVLConfig, JanusConfig, Kosmos2Config, Kosmos2_5Config, Lfm2VlConfig, Llama4Config, LlavaConfig, LlavaNextConfig, LlavaNextVideoConfig, LlavaOnevisionConfig, Mistral3Config, MllamaConfig, Ovis2Config, PaliGemmaConfig, PerceptionLMConfig, Pix2StructConfig, PixtralVisionConfig, Qwen2_5_VLConfig, Qwen2VLConfig, Qwen3VLConfig, Qwen3VLMoeConfig, ShieldGemma2Config, SmolVLMConfig, UdopConfig, VipLlavaConfig, VisionEncoderDecoderConfig.
```

Hey @ryuzakace ! You probably have the wrong `transformers` version for your model
Repository: huggingface/lighteval
Issue: With chat templates, instructions shouldn't be prepended to system prompt
Root cause: Algorithmic Error

https://github.com/huggingface/lighteval/blob/df21407d9f714bde9ecfb4dd8283afdc2150eec3/src/lighteval/few_shot_manager.py#L199

If we use a chat_template, have a system prompt and an instruction, the instruction is prepended to the system prompt message. It should be prepended to the first user message instead.

Why not.
We are adding it with the system prompt so it plays better with the truncation options iirc.
We can provide both options btw, but I don't feel like there is "a single right way" of managing model prompts at the moment.

Sure, there's not one single right way. Adding it in front of the system prompt feels weird to me tho, particularly for longer system prompts. In this fake example, I think the order of inputs is funky

> The following are multiple choice questions on fish. You're a helpful assistant who only answers in rhyme. When you do not understand the question, politely ask for clarifications. Always remain respectful.

Idk, my intuition is that chat-models expect the prompt as the last instruction.

Oh true, got your point! 
Yep that's a bug, should be `examples[0]["content"]  + instruction`!

Yeah, that seems like a good solution.
Alternatively, we could prepend to the first message with role="user".

We have both options actually, depending on whether there is a system prompt or not. As mentioned above, they will play differently with truncation
Repository: huggingface/lighteval
Issue: [BUG] concurrent_requests in litellm backend limited to 100
Root cause: External Dependency Breakage

## Describe the bug
I am evaluating a model deployed with vllm locally and I want to increase the `concurrent_requests` to 128 to speed up the evaluation. However, it seems there are some limitation and the number of concurrent requests sent to the vllm server can not surpass 100. I am sure that the deployed vllm server can handle 128 concurrent request, so I wondering if there are any configs that I missed in `lighteval` or `litellm`?

## To Reproduce
My config:
```
model_parameters:
    provider: "hosted_vllm"
    model_name: "hosted_vllm/qwen"
    base_url: "http://localhost:8000/v1" 
    api_key: "" 
    timeout: 100000000
    concurrent_requests: 128
    max_model_length: 38912
    generation_parameters:
      temperature: 0.6
      top_p: 0.9
      top_k: 20
      seed: 42
```
My conmand:
```lighteval endpoint litellm lighteval_vllm_config.yaml $DATASET --output-dir ./results --save-details```

## Expected behavior
The vllm server can receive 128 concurrent requests.

## Version info
lighteval version: 0.13.0
litellm version: 1.80.8


hey ! thanks for the repport. I won't have time to look into it right now but if there is a hard coded limit it's going to be [here](https://github.com/huggingface/lighteval/blob/main/src/lighteval/models/endpoints/litellm_model.py)

Thanks for the replay! It seems there is no had coded limit in this file so I suspect the limit is in low level repo. I have raised an issue in `litellm` BerriAi/litellm#17688, let's keep tracking :)
Repository: huggingface/lighteval
Issue: [BUG] Accelerate backend with dp fails to load cache
Root cause: Validation Gap

## Describe the bug
In SampleCache, `get_cache_path` doesn't consider the rank or apply a lock when saving/loading parquet files. This causes parquet caches to be potentially corrupted when multiple ranks are launched with accelerate backend in calls to `cache_samples`.

This causes the `get_samples_from_cache` function calls to fail, because the parquet load call in this block fails (`cache_management.py`: line 283):
```python
        for task_id in task_ids:
            if task_id.sampling_method != sampling_method:
                continue
            cache_file = self.get_cache_path(task_id)
            try:
                dataset = load_dataset("parquet", data_files=str(cache_file), split="train")
                dataset_df = dataset.to_pandas().set_index("sample_id")
                task_datasets[task_id] = dataset_df
            except Exception as e:
                logger.warning(f"Error loading prediction cache for {str(task_id)}: {e}")

```

## To Reproduce
I have a bit of a complicated setup so I can't provide a simple repro directly here. I can try to produce one if it's absolute necessary.

## Expected behavior
Only rank 0 should cache the samples and cache with multiple ranks should work.

## Version
I'm using this branch: https://github.com/huggingface/lighteval/pull/1083, but the cache portion of code is identical.

This is how I'm launching:
```bash
accelerate launch --multi_gpu --num_processes=${SLURM_NTASKS} -m \
    lighteval accelerate "model_name=${FINETUNE_OUT_DIR},dtype=bfloat16,batch_size=${BSZ}" \
    ${TASK_NAMES} --custom-tasks src/evaluate/custom_eval.py \
    --output-dir ${OUTPUT_PATH}
```

This is not very well documented, but this is what I gathered should be the correct way to launch the eval script. Please let me know if this is incorrect

hey @Aceticia ! Thanks for raising this issue. I think it's a matte of using a checking the rank of the process and write only if first process. We wont have time to focus in this until next year so if you want to take it up we would be happy to review :)
Repository: huggingface/lighteval
Issue: [BUG] Code execution stuck without any information
Root cause: Resource Mishandling

## Describe the bug
When I execute my evaluation script, the terminal shows the code execution stuck without any information. I just wait for a long time, nothing happend.

<img width="640" height="124" alt="Image" src="https://github.com/user-attachments/assets/ae8f5b7c-0e82-4944-b07f-fec11a1e9ff4" />


## Expected behavior
Loading task done and start evaluation.

## Version info
My environment simply installed with "pip install lighteval[vllm]", lighteval==0.13.0, which is the latest released version. 


solved
Repository: huggingface/lighteval
Issue: [BUG] return _winapi.DuplicateHandle
Root cause: Environment Incompatibility

## Describe the bug
A clear and concise description of what the bug is.

Every question, it returns the following:
```bash
  File "<string>", line 1, in <module>
    from multiprocessing.spawn import spawn_main; spawn_main(parent_pid=17480, pipe_handle=1604)
                                                  ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\chigo\AppData\Local\Programs\Python\Python313\Lib\multiprocessing\spawn.py", line 113, in spawn_main
    new_handle = reduction.duplicate(pipe_handle,
                                     source_process=source_process)
  File "C:\Users\chigo\AppData\Local\Programs\Python\Python313\Lib\multiprocessing\reduction.py", line 79, in duplicate
    return _winapi.DuplicateHandle(
```

## To Reproduce
Please provide all the steps needed to reproduce the behavior, or provide a minimal working example if needed. We will ignore issues missing this section.

```bash
set OPENAI_BASE_URL=http://localhost:11434/v1
set OPENAI_API_KEY=apikey
lighteval eval --timeout 600 --max-connections 1 --max-tasks 1 openai/gpt-oss:20b gsm8k
```


## Expected behavior
A clear and concise description of what you expected to happen.

It should run without printing the warning every question.

## Version info
Please provide your operating system, lighteval version or commit if you installed from main, and pip/conda environment if your problem concerns dependencies.

Windows 10
commit 6496d623f0b2905ce3118ac13179c3a665194759
uv 0.8.13
Repository: huggingface/lighteval
Issue: [FT] Support for retriever-augmented and latent-memory models.
Root cause: Unimplemented Feature Gap

## Issue encountered
LightEval currently evaluates models as `input -> output` text generators. However, an increasing class of retrieval-augmented models perform retrieval and reasoning within a _latent space_ or via tightly-coupled retriever-generator systems, which makes standard evaluation partially supported. 

### What kind of models can we support? 
1. **Compression-native / latent RAG** (e.g. Apple's CLaRa): documents are compressed into learned latent representations; retrieval and reasoning happen in continuous latent space. 
2. **Joint retriever-generator models** (e.g. RETRO, ATLAS): retrieval behavior materially affects generation but is not visible in standard evaluations. 
3. **Latent memory systems** (e.g. DSI): documents are stored and accessed implicitly via model parameters rather than text chunks. 

In these cases, comparison against classic RAG baselines becomes difficult. 

LightEval can already evaluate outputs from such systems, but it lacks a way to model these RAG systems as so, rather than as plain language models. 

## Solution/Feature
I don't have a fixed solution yet, rather I'm opening up an open discussion on the feasibility of this feature, for which I will take full initiative for, if it's viable. 

Upon searching the codebase, I found few files like `pipeline.py` or some driver code that supports classes like `LightevalModel`. Maybe we can create some form of model adapter that wraps a RAG system, that way we can use the existing benchmarks available within LightEval with these new systems. This is ofc theoretical thinking, but I would love your thoughts on this. 

i can drop a more detailed analysis / approach on if this is a good feature to have within lighteval. 

cc: @NathanHB 

hey @akshathmangudi !

> Upon searching the codebase, I found few files like pipeline.py or some driver code that supports classes like LightevalModel. Maybe we can create some form of model adapter that wraps a RAG system, that way we can use the existing benchmarks available within LightEval with these new systems. This is ofc theoretical thinking, but I would love your thoughts on this.

Yes that's how we would do it. Creating a new Model class from `LightevalModel` that wraps the RAG and still calls `greed_until`  function.
Would be happy to review if you want to take this up !

yes ofc, i will draw up a more detailed approach and make an initial commit highlighting the same!
Repository: huggingface/lighteval
Issue: [FT] Support setting the multiple random seeds and decoupling it from model deployment?
Root cause: Unimplemented Feature Gap

## Issue encountered
When I want to run the results of multiple seeds for a task, the existing solution only allows specifying the seed during the initialization of the Pipeline (when initializing VLLMModelConfig). However, for evaluating multiple seeds, it is often only necessary to specify the seed during inference. Currently, this solution requires redeployment every time a seed is specified, and the model deployment of VLLM takes a lot of extra time.

## Solution/Feature
One solution is to decouple the setting of the seed from model deployment, allowing for a single deployment to perform multiple evaluations with multiple seeds and calculate the average. This is similar to a multi-task approach.

## Possible alternatives
No other solutions have been thought of.
Repository: huggingface/lighteval
Issue: [BUG]: lighteval.utils import is_autogptq_available not working
Root cause: Validation Gap

Hi,
It seems like lighteval.utils import is_autogptq_available differs the behavior from  transformers.utils import is_auto_gptq_available
even tho when `auto-gptq` is installed.
```
from transformers.utils import is_auto_gptq_available
from lighteval.utils import is_autogptq_available

print(is_auto_gptq_available())
print(is_autogptq_available())

import importlib
print(importlib.util.find_spec("auto-gptq"))
! pip list | grep "auto-gptq"

# Output
True
False
None
auto-gptq                 0.4.2
```


Hi!
Thanks for the report!
Note: `ŧransformers` uses the same check as we do ([transformers](https://github.com/huggingface/transformers/blob/f6261d7d81edd036fc53bfede65fe91f01a661aa/src/transformers/utils/import_utils.py#L41) vs [lighteval](https://github.com/huggingface/lighteval/blob/b1aa62620882c4045786762ba29aeb8316c44f8d/src/lighteval/utils.py#L180)) to test if a package is imported or not. 
It seems like there is a typo on our side, and we should check if `importlib.util.find_spec("auto_gptq")`, not `auto-gptq`.

Can you try [this PR](https://github.com/huggingface/lighteval/pull/116/files) and tell us if it works for you?

will do!
Repository: huggingface/lighteval
Issue: Add EQ Bench
Root cause: Unimplemented Feature Gap

EQ Bench is a popular benchmark for chat models that aims to measure the emotional intelligence of LLMs. It is popular because it has high correlation with human preference evals like the LMSYS Chatbot Arena, but can be run at a fraction of the cost.

Leaderboard + code + paper: https://eqbench.com/about.html

This one is also model as a judge, so will need #75 first
Repository: huggingface/lighteval
Issue: Add dtype management in inference endpoints
Root cause: Unimplemented Feature Gap

For dtype = float32/bfloat16/float16, we need to change the image creation to 
```
                image = {
                    "health_route": "/health",
                    "env": {
                        # Documentaiton: https://huggingface.co/docs/text-generation-inference/en/basic_tutorials/launcher
                        "MAX_BATCH_PREFILL_TOKENS": "2048",
                        "MAX_INPUT_LENGTH": "2047",
                        "MAX_TOTAL_TOKENS": "2048",
                        "MODEL_ID": "/repository",
                    },
                    "url": "ghcr.io/huggingface/text-generation-inference:1.1.0",
                }
                if config.model_dtype is not None:
                    image["env"]["DTYPE"] = str(config.model_dtype) 
```

For quantization, it's `--quantize bitsandbytes` variations.

Full options are [here](https://huggingface.co/docs/text-generation-inference/en/basic_tutorials/launcher).

Closed by #124
Repository: huggingface/lighteval
Issue: Add BBH subset back!
Root cause: Unimplemented Feature Gap

The BBH subsets (lighteval and harness versions) were removed in this merge commit 
https://github.com/huggingface/lighteval/pull/7/commits/c136ad59fc74bb3eee6546dcf0802eb8c2f3bcbe.
They need to be added back to the table, and the test set need to use BBH again (need to uncomment the call to BBH)
Repository: huggingface/lighteval
Issue: human eval run
Root cause: Unimplemented Feature Gap

It seems that you should support humaneval but the metric for it, "code_humaneval" (which I guess should be pass@k), is not available anywhere. How can I run humaneval?

Thanks!

I think we had `code_humaneval` removed, because it relied on too much custom code compared to others base metrics. I can look into adding it back, though for code evaluations I would actually recommend using [this suite](https://github.com/bigcode-project/bigcode-evaluation-harness).

thanks! 
yeah I'm familiar with this great evaluation harness
I just tried to reproduce the results that someone published with this harness so I wondered why I can't even run it :)

Hm where did you find these results? Maybe the person created a custom eval file?

https://huggingface.co/HuggingFaceH4/starchat2-15b-v0.1#performance

Oh thank you! 
@lewtun maybe?
Could be linked to #63 if you had started taking a look :)

Closing this issue for inactivity