zhiminy/EvalEng huggingface dataset:
Repository: huggingface/lighteval
Issue: perplexity vs logprob
Root cause: Documentation Deficiency

I have 2 scripts for evaluating LLM quality
1. Perplexity based, where the accuracy increases if the golden answer of multiple choice has the lowest perplexity
2. log probs based, where the accuracy increases if the golden answer of multiple choice has the highest log prob scores

How is it possible that the accuracies are different ?


The code for logprob lighteval 

```python

from lighteval.metrics.metrics import Metrics
from lighteval.tasks.lighteval_task import LightevalTaskConfig
from lighteval.tasks.requests import Doc


def prompt_fn_sentence_errors(line, task_name: str = None):
    return Doc(
        task_name=task_name,
        query=f"",
        choices=[
            line["text"],
            line["false_variation1"],
            line["false_variation2"],
            line["false_variation3"],
        ],
        gold_index=0,
    )

TASKS_TABLE = []

for subset in ["legacy", "gernews_2025"]:
    task1 = LightevalTaskConfig(
        name=f"gerbench:sentence_errors_{subset}",
        prompt_function=prompt_fn_sentence_errors,
        suite=["community"],
        hf_repo="ellamind/gerbench_german_sentences_with_errors",
        hf_subset=subset,
        hf_avail_splits=["train"],
        evaluation_splits=["train"],
        metric=[
            Metrics.loglikelihood_acc_norm,
            Metrics.loglikelihood_f1,
            Metrics.mcc,
            Metrics.mrr,
        ],
        version=0,
    )
    TASKS_TABLE.append(task1)

```


The data has some small errors like double space or changed character, for example switching from the german "Hallo" to the english "Hello"
I would like to iterate over the dataset step by step and compare at which samples the accuracy between the approaches is different. Is there an easy way to do this with lighteval ?


For example:
With the lowest perplexity version the model reaches around 80% accuracy, with the lighteval implementation only 65%.

> Sensitivity to word order: Perplexity is more sensitive to the overall word order and flow of the sentence, while log probabilities focus on individual token probabilities. This means that perplexity might favor answer options that are grammatically correct and semantically coherent, even if they don't have the highest individual token probabilities


Is that already the correct explanation ?
Repository: huggingface/lighteval
Issue: [EVAL] Add TUMLU benchmark
Root cause: Unimplemented Feature Gap

Hello! 
We just released the benchmark for Turkic languages. Does it make sense if I add it to lighteval?

## Evaluation short description
- Why is this evaluation interesting?
 First native-language MMLU benchmark for low-resource Turkic languages.

- How is it used in the community?
Just released, MC high-school exam questions

## Evaluation metadata
Provide all available
- Paper url: https://arxiv.org/abs/2502.11020
- Github url: https://github.com/ceferisbarov/TUMLU
- Dataset url:


cc @hynky1999 could interest you I feel!

Is the dataset already on Hugging Face?

@clefourrier Not really (in gated repos), but everything is in github already. 

Gated sounds fine, can you share the path?

Hi, I think it would be very nice addition, we already have TurkishMMLU (which I think is is also part of your dataset right ?)
- See https://github.com/huggingface/lighteval/blob/main/src/lighteval/tasks/multilingual/tasks.py#L2133

To add it we would need following:
1. Have translation literals for the languages you want to add: (https://github.com/huggingface/lighteval/blob/main/src/lighteval/tasks/multilingual/tasks.py#L2133)
2. Add the dataset to hub
3. Replace the TurkishMMLU with your dataset

Do you think you could do that? cc @gaydmi 

@gaydmi Thank you for bringing this up!

@hynky1999 I have a question. Our dataset can be split into subsets in three ways: (a) make each language a subset, (b) make each subject a subset, (c) make each language-subject combination a subset. Which one would you suggest? I could not find any similar examples in the repo.


@hynky1999 Hi, yes, working on it!
@ceferisbarov I personally think option (c) is the best, so we could just add new languages with their tasks.
Like in here: https://github.com/huggingface/lighteval/blob/main/src/lighteval/tasks/multilingual/tasks.py#L2617

I would say ideally use subset for languages and then add column to identify the actuall task subset. You can then use hf_filter arg on task

Both options sound good to me. I have added the dataset to Hugging Face:

https://huggingface.co/datasets/jafarisbarov/TUMLU-mini

@gaydmi let me know if I can help in any other way.

Awesome, cc @gaydmi happy to review the PR once ready
Repository: huggingface/lighteval
Issue: [CORE] Make parsing of model args and generation parameters robust
Root cause: Validation Gap

In #578 we found it was necessary to parse and slice out `generation_parameters` from `model_args` in order to avoid passing the argument twice to `VLLMConfig`. To move fast, I adopted a regex, but ideally we'd want to make this more generic & robust by either improving the string2dict parsing logic for `model_args` so that `generation_parameters` is correctly parsed.

It may also make sense to use something like `ast` or `pydantic` to enforce the desired types in the dict.

fixed in pr #609 @lewtun if you want to take a look !
Repository: huggingface/lighteval
Issue: [BUG] ValueError: Error from parse_expr with transformed code: '６'
Root cause: Validation Gap

when I use `lighteval vllm ` to evaluate Math-500，get the error like this

![Image](https://github.com/user-attachments/assets/a879d1de-928a-443f-be20-f0cb0784f748)

The script I'm using is provided below:

```
export VLLM_WORKER_MULTIPROC_METHOD=spawn
MODEL=Qwen_model/Qwen2.5-Math-1.5B-Instruct
MODEL_ARGS="pretrained=$MODEL,dtype=bfloat16,max_model_length=4096,gpu_memory_utilisation=0.8,tensor_parallel_size=2"
OUTPUT_DIR=data/evals/$MODEL
# AIME 2024
TASK=aime24
lighteval vllm $MODEL_ARGS "custom|$TASK|0|0" \
    --custom-tasks src/open_r1/evaluate.py \
    --use-chat-template \
    --output-dir data/evals/$TASK

# # MATH-500
TASK=math_500
lighteval vllm $MODEL_ARGS "custom|$TASK|0|0" \
    --custom-tasks src/open_r1/evaluate.py \
    --use-chat-template \
    --output-dir data/evals/$TASK

# GPQA Diamond
TASK=gpqa:diamond
lighteval vllm $MODEL_ARGS "custom|$TASK|0|0" \
    --custom-tasks src/open_r1/evaluate.py \
    --use-chat-template \
    --output-dir data/evals/$TASK
```
Thanks for looking into this issue.
Repository: huggingface/lighteval
Issue: Use Vllm as model to generate faster
Root cause: Unimplemented Feature Gap

Hello, 
Is it possible to support vllm to make it work with lighteval? 


Hi! This is going to be out of scope at the moment - we'll keep the issue for future iterations, but we want to focus on supporting HF libs for now

Given the increasing use of generative evals, I think there's a strong case to be made for including a vLLM backend because it currently takes >24h to evaluate a 70B model on MATH (5k prompts) on a single node of 8xH100s with the default `generate()` method from `transformers`. 

I expect vLLM would be at least 2-3x faster which is quite nice when evaluating large models.

Hi! This has been done for a couple of months now :)
Repository: huggingface/lighteval
Issue: Sth wrong in the parser for `generation_parameters` in `main_sglang.py`
Root cause: Unimplemented Feature Gap

## Describe the bug
The parser for the `generation_parameters` is not implemented in `main_sglang.py`.

## To Reproduce
The similar code in `main_vllm.py`: (Line 134-144)
```
    if model_args.endswith(".yaml"):
        with open(model_args, "r") as f:
            config = yaml.safe_load(f)["model"]
        model_args = config["base_params"]["model_args"]
        metric_options = config.get("metric_options", {})
        generation_parameters = GenerationParameters.from_dict(config)
    else:
        generation_parameters = GenerationParameters.from_model_args(model_args)
        # We slice out generation_parameters from model_args to avoid double-counting in the VLLMModelConfig
        model_args = re.sub(r"generation_parameters=\{.*?\},?", "", model_args).strip(",")
        metric_options = {}
```
In `main_sglang.py`(Line 133-139):
```
    if model_args.endswith(".yaml"):
        with open(model_args, "r") as f:
            config = yaml.safe_load(f)["model"]
        model_args = config["base_params"]["model_args"]
        generation_parameters = GenerationParameters.from_dict(config)
    else:
        generation_parameters = GenerationParameters()
```

## Expected behavior
The handling of `generation_parameters` in the code of the two inference engines should be the same. Currently, there is no way to input generation_parameters in sglang.

## Version info
Ubuntu 2402 python3.10 lighteval ed08481


It's easy to fix. Please reply to me if I could help.
Repository: huggingface/lighteval
Issue: [FT] Assistant Response Prefilling
Root cause: Unimplemented Feature Gap

## Solution/Feature
Context:
[docs.anthropic.com/en/docs/build-with-claude/prompt-engineering/prefill-claudes-response](https://docs.anthropic.com/en/docs/build-with-claude/prompt-engineering/prefill-claudes-response)
[anthropic.com/news/claude-2-1-prompting](https://www.anthropic.com/news/claude-2-1-prompting)

I would love to be able to manually inject assistant responses during eval to perform some ablations. Ideally, I would be able to modify the messages like such: 
```
messages = [
    {
        "role": "user",
        "content": prompt,
    },
        "role": "assistant",
        "content": 'According to ',
    }
]
```
And it should use this to generate up through assistant's response but not ending it, so the model continues.

## Possible alternatives
Adopting a custom chat template could work, but it would be nice to see this feature integrated as I anticipate a lot of researchers will have need for easy access to this.
Repository: huggingface/lighteval
Issue: [BUG] Unable to run OALL V2 tasks
Root cause: Documentation Deficiency

## Describe the bug
I am unable to run OALL V2 tasks via lighteval. When I try running the tasks, it says the OALL tasks are invalid 

## To Reproduce
git clone the repo or pip install lighteval and run it via 
lighteval accelerate "pretrained=<model_name>" examples/tasks/OALL_v2_tasks.txt


## Expected behavior
Tasks would run from the OALL_V2_tasks.txt

## Version info
Tried both with 0.7.0 and by git cloning and installing from latest master


hi ! you need to specify the custom task file like so:

```
lighteval vllm "${MODEL_ARGS}"  examples/tasks/OALL_v2_tasks.txt  --custom-tasks community_tasks/arabic_evals.py 
```
Repository: huggingface/lighteval
Issue: [BUG] Output max_length
Root cause: Configuration Error

Hi,

How can I adjust the max_length for generation? I'm using `lighteval vllm` for evaluation, but the answers are all cut short, leading to this warning and evaluation error

```
[ WARNING]: We did not manage to extract a prediction in the correct format.
```

This is the command I'm using
```
lighteval vllm pretrained={args.model_name},dtype=float16,max_model_length=32768,gpu_memory_utilisation=0.8"custom|{args.task_name}|{args.num_shots}|1" \
        --custom-tasks ./evaluate.py \
        --use-chat-template \
        --system-prompt="{SYSTEM_PROMPT}" \
        --output-dir="{OUTPUT_DIR}" 
```

## Version info
0.6.0.dev0


I also encountered the same problem. Have you solved it?

![Image](https://github.com/user-attachments/assets/10a0f41c-5a7b-4b84-8439-cb17df50a690)

Sadly no. I guess it is related to some kind of generation max length

i got the same problem, here is my cmd:
**lighteval accelerate "model_path" "community|my_test|0|0" --output-dir ./eval_results --custom-tasks ./community_tasks/my_test.py --save-details --use-chat-template**

i comment out this line: https://github.com/huggingface/lighteval/blob/main/src/lighteval/models/transformers/transformers_model.py#L851
and eval works.

i am not quite sure why the context is 'padding="max_length"' by the tokenizer, then the output token number is calculated and set to 1 at line:
https://github.com/huggingface/lighteval/blob/main/src/lighteval/models/transformers/transformers_model.py#L877


```pip install vllm>=0.7.1``` solved the problem

> `pip install vllm>=0.7.1` solved the problem

Sorry but I dont think so, I am using vllm 0.7.3 but that still happened

I forgot to mention. You have to download the specific lighteval as well. ```lighteval==0.6.0.dev0``` works for me

> I forgot to mention. You have to download the specific lighteval as well. `lighteval==0.6.0.dev0` works for me

I had installed lighteval of version 0.6.0.dev0  but this problem did not been solved...
Repository: huggingface/lighteval
Issue: [FT] The use of "--save-details" setting for displaying the generated contents and answers
Root cause: Documentation Deficiency

## Issue encountered
If I want to analyze the generated contents and answers, I have to open `--save-details` in the command. Unluckily, the saved `parquet` file is very inconvenient to read. After converting the file to `json` format, the file is still very hard to read due to the redundant `input_tokens` and `cont_tokens` terms.

![Image](https://github.com/user-attachments/assets/e717e5e6-50e6-4a40-8d28-332f0f5e60c2)

## Possible alternatives
I wonder if there is a more convenient solutions to achieve this.


hi ! yes you are right some info are a bit redundant, we weer thinking of slimming down the content of the logged details, but for now we use gradio spaces to load up the parquet files and choose what field to explore.
You can see [here](https://huggingface.co/spaces/SaylorTwift/OpenEvalsModelDetails) a gradio space built for that

> hi ! yes you are right some info are a bit redundant, we weer thinking of slimming down the content of the logged details, but for now we use gradio spaces to load up the parquet files and choose what field to explore. You can see [here](https://huggingface.co/spaces/SaylorTwift/OpenEvalsModelDetails) a gradio space built for that

Thanks for your effort!
Repository: huggingface/lighteval
Issue: [EVAL]Which version of LightEval evaluates AIME 24, AIME 25, LiveCodeBench, and Distill R1 model?
GitHub labels: new-task

## Evaluation short description
- Why is this evaluation interesting?
- How used is it in the community?

## Evaluation metadata
Provide all available
- Paper url:
- Github url:
- Dataset url: 


hi ! version `0.8.0` would be the one to use (pip install lighteval) should install it :)
Repository: huggingface/lighteval
Issue: [EVAL] SimpleQA
Root cause: Unimplemented Feature Gap

## Evaluation short description
SimpleQA is a new benchmark from OpenAI that aims to measure factuality / hallucinations in LLMs. It is also a nice benchmark to measure calibration in LLMs.

## Evaluation metadata
Provide all available
- Paper url: https://arxiv.org/abs/2411.04368
- Github url: https://github.com/openai/simple-evals
- Dataset url: https://openaipublic.blob.core.windows.net/simple-evals/simple_qa_test_set.csv


added in #680
Repository: huggingface/lighteval
Issue: Import issue when using nanotron
Root cause: External Dependency Breakage

When downloading `lighteval` and running `pip install -e .[nanotron]` and then `python src/main_brrr.py` we get an import error.

```
Traceback (most recent call last):
  File "/fsx/nathan_habib/lighteval/src/main_brrr.py", line 9, in <module>
    from brrr.s3_checkpoints import fs_copy
ImportError: cannot import name 'fs_copy' from 'brrr.s3_checkpoints' (/fsx/lighteval/miniconda3/envs/lighteval_dev_2/lib/python3.11/site-packages/brrr/s3_checkpoints/__init__.py)
```
Repository: huggingface/lighteval
Issue: [EVAL] Big-Bench Extra Hard (BBEH)
Root cause: Unimplemented Feature Gap

## Evaluation short description
Google has releases BBEH as a way to compensate for the saturation of BBH in the latest generation of LLMs. Overall looks like a good benchmark to probe reasoning capabilities.

## Evaluation metadata
Provide all available
- Paper url: https://arxiv.org/pdf/2502.19187
- Github url: https://github.com/google-deepmind/bbeh
- Dataset url:


I'd like to implement this benchmark, if it's still up! Also, I found this unofficial hub upload of the dataset: https://huggingface.co/datasets/BBEH/bbeh . Since there's no official upload can we use this one, or would it be better to create our own upload similar to the original BBH: https://huggingface.co/datasets/lighteval/bbh ?

@NathanHB is this still relevant? I would be happy to work on this and add it to lighteval

hey ! yes it is :) We would love to have it in lighteval especially inspect-ai compatible.
Reach out if you need any help on this ?
Repository: huggingface/lighteval
Issue: [BUG] AttributeError: 'torch_dtype' Not Found in VLLMModelConfig When dtype Not Specified
Root cause: Validation Gap

When using `vllm` without specifying a `dtype`, I encountered the following error:
```
AttributeError: 'VLLMModelConfig' object has no attribute 'torch_dtype'
```

I was running open-r1 MATH500 test:
```bash
MODEL_ARGS="pretrained=$MODEL,max_model_length=32768,gpu_memory_utilization=0.8,data_parallel_size=$NUM_GPUS,generation_parameters={max_new_tokens:32768,temperature:0.7,top_p:0.95}"

OUTPUT_DIR=data/evals/$MODEL

lighteval vllm $MODEL_ARGS "custom|math_500|0|0" \
    --custom-tasks src/open_r1/evaluate_math.py \
    --use-chat-template \
    --output-dir $OUTPUT_DIR \
    --save-details 
```

This error originates from this code section:
[utils.py#L55](https://github.com/huggingface/lighteval/blob/ed084813e0bd12d82a06d9f913291fdbee774905/src/lighteval/models/utils.py#L55)
```python
if config is not None:
    return config.torch_dtype
```

### Proposed Solution
To improve robustness, check if `config.torch_dtype` exists before accessing it / Update the documentation to ensure `dtype` must be passed to avoid this error.

hey ! thanks for the feedback, when trying to replicate with latest version of lighteval, i found that vllm did not accept `None` as default. I fixed by using `bfloat16` as default dtype if not submitted by user. #638
Repository: huggingface/lighteval
Issue: [FT] Support local datasets
Root cause: Unimplemented Feature Gap

## Issue encountered
Unable to load a local dataset when creating a custom task.

Currently, as part of the `Create new task` workflow, the class `LightevalTaskConfig` only supports hugging face datasets via the parameter `hf_repo`. This does not allow us to test local datasets.

## Solution/Feature
Under the hood, `load_dataset` from the `datasets` library is called that actually supports local datasets via the parameters `data_files` and `data_dir`. These parameters are not exposed thereby preventing us to load local datasets.

Please let me know if there is another way to achieve this if I've missed.


hi ! There is no way atm to do this, we will not add it for now as it not prio for us but you can for sure open a PR and we will be happy to review :)

@NathanHB I'm interested in helping with implementing this. Should I open a new issue with this or reopen this?

hey ! A PR is about to be merged in #1083  :)
Repository: huggingface/lighteval
Issue: finetasks not working
Root cause: Documentation Deficiency

Hi! I can't get finetasks to work with lighteval. 
The error each time is that the code cannot get task info from examples/tasks/fine_tasks/mcf/ 

`lighteval accelerate     "pretrained=HuggingFaceTB/SmolLM2-1.7B,dtype=bfloat16"     "examples/tasks/fine_tasks/mcf/fr"     --custom-tasks lighteval.tasks.multilingual.tasks`

The code error
`ValueError: Cannot get task info from examples/tasks/fine_tasks/mcf/fr. correct format is suite|task|few_shot|truncate_few_shots`

[log-error.txt](https://github.com/user-attachments/files/19129264/log-error.txt)

you are not using the right format for the task, you need to choose the task you want and run: `suite|task|few_shot|truncate_few_shots` for example: `extended|ifeval|0|0`
Repository: huggingface/lighteval
Issue: [FT] endpoint calling API in parallel is in conflict with rate limiting
Root cause: Configuration Error

I'm developing a custom task ( based on https://huggingface.co/datasets/monsoon-nlp/genetic-counselor-multiple-choice ) - this has ~200 multiple choice questions and some of the questions are long (lots of tokens). Like other multiple choice tasks, it's evaluated by looking at probabilities of the next token being A, B, C, or D.

When running `lighteval endpoint openai gpt-4o` or similar, I get a few results followed by dozens of `openai.RateLimitError`s, which means the task is never evaluated. Depending on the model, I run into the 30,000 token / minute, or 500 request / minute barrier.

I tried editing `dataset = GenerativeTaskDataset(requests=requests, num_dataset_splits=self.DATASET_SPLITS)` to split into more parts (?) but this didn't change outcome.
I see there is `API_RETRY_SLEEP` but this doesn't affect how many are requested initially by `__call_api_parallel`
I would want something like `--override-batch-size`, or something token-based, to pace the requests.

hi ! you have to change `CONCURENT_CALLS` to lower the number of parallel calls :)
Repository: huggingface/lighteval
Issue: Add single `mmlu` config for `lighteval` suite
Root cause: Unimplemented Feature Gap

Currently it seems that to run MMLU with the `lighteval` suite, one needs to specify all the subsets individually as is done for leaderboard task set [here](https://github.com/huggingface/lighteval/blob/6a3e3b9245db9a596d636d71424aa0f2e4358ff6/tasks_examples/open_llm_leaderboard_tasks.txt#L7-L64).

Is it possible to group these together so that one can just run something like this:

```shell
accelerate launch --multi_gpu --num_processes=8 run_evals_accelerate.py \
    --tasks="lighteval|mmlu|5|0" \
    --model_args "pretrained=Qwen/Qwen1.5-0.5B-Chat" \
    --output_dir "./scratch/evals/" --override_batch_size 1
```


Or do you recommend using one of the other suites like `helm` or `original` for this task?

Atm, it's not possible; however, if you run a task with many subsets (using a config file), you should get a display of the average at the task level in the score table. 

If you want to get results comparable to the Open LLM Leaderboard, you'll need to use `lighteval` (you can take a look at the differences between the 3 versions [here](https://huggingface.co/blog/evaluating-mmlu-leaderboard)).

+1. It would be handy to have `lighteval|mmlu_pro`