zhiminy/EvalEng huggingface dataset:
Repository: huggingface/lighteval
Issue: [BUG] latest release not working and causes unexpected keyword argument
Root cause: External Dependency Breakage

## Describe the bug
I am trying to use vLLM to run `aime25` evaluations, but I’m encountering an issue. Even after removing that, all the vLLM-related parameters still have the same problem: they aren’t being mapped or parsed properly through the model’s arguments.

```
TypeError: VLLMModelConfig.__init__() got an unexpected keyword argument 'gpu_memory_utilization'
```

## To Reproduce
Please provide all the steps needed to reproduce the behavior, or provide a minimal working example if needed. We will ignore issues missing this section.

```sh
pip install lighteval[vllm]
pip install lighteval[math]
pip install more_itertools

MODEL=deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B
OUTPUT_DIR=data/evals/DeepSeek-R1-Distill-Qwen-1.5B
# AIME 2024
TASK=aime25
lighteval vllm \
    "pretrained=$MODEL,dtype=bfloat16,max_model_length=32768,gpu_memory_utilization=0.8,generation_parameters={max_new_tokens:32768,temperature:0.6,top_p:0.95}" \
    "custom|$TASK|0|0" \
    --use-chat-template \
    --output-dir $OUTPUT_DIR
```


## Expected behavior
A clear and concise description of what you expected to happen.

It should serve the model on the vllm with the requested parameters and evalaute.

## Version info
Please provide your operating system, lighteval version or commit if you installed from main, and pip/conda environment if your problem concerns dependencies.

```
lighteval==0.7.0
vllm==0.6.3.post1
```

same question

hi ! wasn't able to replicate, I ran with the latest version of lighteval and it works, can you try using version `0.8` ?

```
lighteval vllm "pretrained=deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B,dtype=bfloat16,max_model_length=32768,gpu_memory_utilizati
on=0.8,generation_parameters={max_new_tokens:32768,temperature:0.6,top_p:0.95}" "lighteval|aime25|0|0" --use-chat-template
```
Repository: huggingface/lighteval
Issue: [FT] Support more prequantized models
Root cause: Unimplemented Feature Gap

[repro](https://colab.research.google.com/drive/1J21jE5Bnee9ShrbItHYu7XoxFePH7MEY?usp=sharing)

## Issue encountered
lighteval doesn't support prequantized models using quantization methods like awq 

## Solution/Feature
I've dealt with this by commenting out (https://github.com/huggingface/lighteval/blob/main/src/lighteval/models/transformers/transformers_model.py#L461): 
```
        model = AutoModelForCausalLM.from_pretrained(
            config.pretrained,
            revision=config.revision + (f"/{config.subfolder}" if config.subfolder is not None else ""),
            max_memory=max_memory,
            device_map=device_map,
            torch_dtype=torch_dtype,
            trust_remote_code=config.trust_remote_code,
            cache_dir=env_config.cache_dir,
            offload_folder=env_config.cache_dir,
            token=env_config.token,
            # quantization_config=config.quantization_config, # this line
        )

``` 
when using prequantized model.
Repository: huggingface/lighteval
Issue: Enable majority voting for GSM8k / MATH
Root cause: Unimplemented Feature Gap

Many papers nowadays report the `maj@k` metric for math benchmarks like GMS8k and MATH, where the model generates `k` candidates to a problem and the most common answer is chosen as the final solution (see source [paper](https://arxiv.org/abs/2203.11171) for details).

It would be nice to support `maj@k` as a metric for these benchmark, potentially also including the ability to have CoT prompts as is also common practice.

CoT prompts are on the roadmap :)

It's been done and merged in #158
Repository: huggingface/lighteval
Issue: [BUG] core dump when using endpoint litellm
Root cause: Configuration Error

## Describe the bug
When executing litellm endpoint tasks in a directory on the distributed file system, such as glusterfs, it causes a core dump.

Maybe the root cause is that in “lighteval/models/litellm_model.py“, the litellm cache type used is "disk" without specifying a directory. In disk cache mode, litellm uses ./.litellm_cache as the default directory. When this directory is located on the glusterfs distributed file system, there may be conflicts with SQLite, which is the underlying storage engine of diskcache.

## To Reproduce
lighteval endpoint litellm ...

## Expected behavior


## Version info
commit b9a2e56
Repository: huggingface/lighteval
Issue: [FT] Add task leaderboard_math_hard
Root cause: Unimplemented Feature Gap

## Issue encountered
There is no task "leaderboard math hard"...

## Solution/Feature

https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/leaderboard#math-hard

@hynky1999 @NathanHB 

## Possible alternatives


hi ! we did not need to add math-hard to lighteval, but if you want you can create a custom task t obe able to eval models on it.
Repository: huggingface/lighteval
Issue: Add HumanEval and HumanEval+
Root cause: Unimplemented Feature Gap

The HumanEval and HumanEval+ benchmarks are staples for benchmarking code capabilities of base LLMs. It would be nice to include them in `lighteval` so one doesn't have to switch to another framework like BigCode's

References:

* HumanEval: https://github.com/openai/human-eval
* HumanEval+: https://arxiv.org/abs/2305.01210
* Implementation: https://github.com/evalplus/evalplus?tab=readme-ov-file
* BigCode eval harness: https://github.com/bigcode-project/bigcode-evaluation-harness/tree/main

+1, would be nice to have

+ 1

+1!
Repository: huggingface/lighteval
Issue: [BUG] OSError: [Errno 22] Invalid argument
Root cause: Validation Gap

## Describe the bug
'--save-details' try to save details in 'C:/Users/Admin/Desktop/Python/0.LLMs/lighteval/results/details/HuggingFaceTB-SmolLM2-135M-Instruct-cache/2025-03-21T03-10-00.577563/details_leaderboard|hellaswag|0_202
5-03-21T03-10-00.577563.parquet' causing the error 'OSError: [Errno 22] Invalid argument'

This kind of error usually arises from issues such as Incorrect File Path. Certain characters in the path, like |, may be invalid for filenames in Windows. You'll need to remove or replace them with acceptable characters.

## To Reproduce
Just use '--save-details' in a Windows enviroment.

## Expected behavior
'--save-details' option save details.

## Version info
Windows 10, lighteval 0.7.0.


hi ! We do not support windows in lighteval for now, but you can open a PR to change the file naming. closing for now

I think this error also shows up in a Linux system. 

there is a PR fixing this, we are going to merge it soon !
Repository: huggingface/lighteval
Issue: [FT] Build in a way to specify specific IDs/Lines in Dataset to use as few-shot examples in the same split
Root cause: Unimplemented Feature Gap

## Solution/Feature
When creating a custom task, the few-shot samples can be specified via a split from the dataset. However, it would be convenient if there were a way to make it so I can specify samples from the split that will be used as the examples, similar to what LM Evaluation Harness does here in its yaml config:

```
test_split: train
fewshot_split: train
fewshot_config:
  sampler: id_sampler
  sampler_config:
    id_list: ["2022_21", "2022_88", "2022_143"]
    id_column: id
    exclude_from_task: true

``` 

Something along the lines of:
```
@dataclass
class LightevalTaskConfig:
    """Stored configuration of a given [`LightevalTask`].

    Arguments:
    ...
        hf_subset (str): Subset used for the current task, will be default if none is selected.
        hf_avail_splits (list[str]): All the available splits in the evaluation dataset
        evaluation_splits (list[str]): List of the splits actually used for this evaluation
        few_shots_split (str): Name of the split from which to sample few-shot examples
        few_shots_id_column (str): Column name in the dataset to use for the few_shot_id_list
        few_shots_id_list (list[str]): List of unique identifiers corresponding to the samples in the dataset, based on matching these IDs with the values in few_shots_id_column.
        few_shots_select (str): Method with which to sample few-shot examples
    ...
    """
```

The new arguments would be `few_shots_id_column` and `few_shots_id_list`, and wrote a rough description of what they would look like.
Repository: huggingface/lighteval
Issue: v 0.8.0 pip

Hello i have seen that there is already a tag for 0.8.0 but no release yet. Do you have any idea when we can expect it? 


Most likely next week, cc @NathanHB ?

Okay, thank you. Also i believe your documentation has a mistake, as it references 0.9.0 as the latest stable version, and v.0.8.0 is not even out yet

![Image](https://github.com/user-attachments/assets/75dc7394-fd25-4c07-8591-fb6939c9607b)

hey ! the package is out on pip.
thanks for reporting the issue with the doc, not sure why it does that i will take a look !

No problem @NathanHB. Actually there is one more thing is spotted:
For example model config is supposedly not available in the build versions like v.0.7.0 and is only accessible in the "main" version which is not true.
https://huggingface.co/docs/lighteval/v0.7.0/en/package_reference/model_config
You also don't mention TransformersModelConfig in the documentation, which i am now using with the v.0.8.1 you released today.
I hope i am not being a drag to you right now haha


Feel free to open PRs to correct the points you spotted! As the original ask has been managed, closing this issue :)
Repository: huggingface/lighteval
Issue: [BUG] When using accelerator, current model_args parser won't correctly parse dict-like arguments like generation_parameters
Root cause: Validation Gap

## Describe the bug
Current parser split the arguments with ",", which will cut the dict-like arguments into pieces and causing TypeError.
For example, here is the model_args:
```shell
MODEL_ARGS="pretrained=$MODEL,dtype=bfloat16,generation_parameters={do_sample:True,max_new_tokens:32768,temperature:0.6,top_p:0.95}"
```
When I print the parsed arguments, I got something like this:
```python
{
    ...other args...

    "generation_parameters": "{do_sample:True",
    "max_new_tokens:32768": True,
    "temperature:0.6": True,
    "top_p:0.95}": True
}
```
Clearly the later parts of the generation parameters are parsed as flags.

## To Reproduce
```shell
NUM_GPUS=1
MODEL=/path/to/model
OUTPUT_DIR=/path/to/output
MODEL_ARGS="pretrained=$MODEL,dtype=bfloat16,generation_parameters={do_sample:True,max_new_tokens:32768,temperature:0.6,top_p:0.95}"

lighteval accelerate $MODEL_ARGS "any_task" \
    --use-chat-template \
    --override-batch-size 1 \
    --output-dir $OUTPUT_DIR \
    --save-details
```

## Expected behavior
The parsed args should look like this:
```python
{
    ...other args...

    "generation_parameters":  {
        "do_sample": True,
        "max_new_tokens": 32768,
        "temperature": 0.6,
        "top_p": 0.95
    }
}
```

## Version info
lighteval==0.8.0


+1

Hi ! Thanks for the issue, this should be fixed now :)
Don't hesitate to reopen if you see anything other issue

small nit in your command, `generation_parameters={do_sample:True,max_new_tokens:32768,temperature:0.6,top_p:0.95}"`

`True` should be `true` because it's parsed as json
Repository: huggingface/lighteval
Issue: [BUG]
Root cause: Configuration Error

## Describe the bug
when I run:
```
CUDA_VISIBLE_DEVICES=1,2 accelerate launch --multi_gpu --num_processes=2 -m \
    lighteval accelerate \
    "pretrained=meta-llama/Llama-3.1-8B-Instruct" \
    "lighteval|gsm8k|5|0" \
    --override-batch-size 1
```
I get:
      "qem": 0.6899166034874905,
      "qem_stderr": 0.012740305717376268

when I run:
```
CUDA_VISIBLE_DEVICES=1,2 accelerate launch --multi_gpu --num_processes=2 -m \
     lighteval accelerate \
     "pretrained=meta-llama/Llama-3.1-8B-Instruct" \
     "leaderboard|gsm8k|5|0" \
     --override-batch-size 1
```
I get:
      "qem": 0.001516300227445034,
      "qem_stderr": 0.001071779348549261

which seems low, is this expected?

## Version info
lighteval 0.8.1


I looked into this further and it seems that the main difference is in src/lighteval/tasks/default_tasks.py. 
For, leaderboard:
stop_sequence=["Question=", "Question", "="]
For, lighteval:
stop_sequence=["Question="]

Hi! In both cases, you should also be using --use_chat_template as you evaluate chat models.

Thanks! Adding `--use-chat-template` made a big difference. 

running:
```
CUDA_VISIBLE_DEVICES=2,3 accelerate launch --multi_gpu --num_processes=2 -m \
     lighteval accelerate \
     "pretrained=meta-llama/Llama-3.1-8B-Instruct" \
     "leaderboard|gsm8k|5|0" \
     --override-batch-size 1 \
     --use-chat-template
```
I get:
      "qem": 0.7725549658832449,
      "qem_stderr": 0.011546363312548087
Repository: huggingface/lighteval
Issue: [BUG] Very slow livecodebench scoring
Root cause: Algorithmic Error

## Describe the bug
Much slower scoring compared to livecodebench repo.

## To Reproduce
Compare to LCB repo, scoring takes significantly longer and seems single threaded

## Expected behavior
Match LCB scoring time with same number of threads set. I have tried setting num_process:

```python
def codegen_metric(predictions: list[str], formatted_doc: Doc, **kwargs) -> float:
    """Estimates the Pass@1 metric for the code generation task.
    Extract the code from each prediction, Runs it for each sample and generations,
    and computes the Pass@1 over the outputs.
    """
    # Extract generated code snippets
    generated_code_snippets = [[extract_code(pred) for pred in predictions]]  # noqa: F841
    evaluation_sample = {  # noqa: F841
        "inputs": formatted_doc.specific["inputs"],
        "outputs": formatted_doc.specific["outputs"],
        "fn_name": formatted_doc.specific["fn_name"],
    }
    # This is a list of lists because
    evaluation_sample = [{"input_output": json.dumps(evaluation_sample)}]

    metrics, _ = codegen_metrics(
        evaluation_sample,
        generated_code_snippets,
        k_list=[1],  # Only run for Pass@1
        num_process_evaluate=64,
    )
    return metrics["pass@1"]
```

with no improvement in scoring time

## Version info
Please provide your operating system, lighteval version or commit if you installed from main, and pip/conda environment if your problem concerns dependencies.

latest master
```
conda create -n eval python=3.11
pip install vllm==0.7.2
pip install git+https://github.com/huggingface/lighteval.git#egg=lighteval[extended_tasks] math-verify==0.5.2
```

I have tried metric_name="codegen_pass@1", it runs fast. But when I tried metric_name="codegen_pass@1:16", maybe bug? The warning is `Sequence group 13_parallel_sample_10 is preempted by PreemptionMode.RECOMPUTE mode because there is not enough KV cache space. This can affect the end-to-end performance. Increase gpu_memory_utilization or tensor_parallel_size to provide more KV cache memory. total_num_cumulative_preemption=51 (scheduler.py:1754)`.

Maybe we should split 16 generations into more batch?

This is due to parallelism at a sample level it seems. It still scores all the questions sequentially.
Repository: huggingface/lighteval
Issue: [FT] Support pass@k and average pass@1
Root cause: Documentation Deficiency

## Issue encountered
Will lighteval support pass@k or pass@1 averaged over k samples in the future? I think this feature is quite handy when evaluating reasoning models.

Did you check the current pass@k implementation? What instead would you like to see?

Sorry I just saw #647. Sorry for missing the latest PR.

Hm, #647 is specifically for math - pass@k is already there :)
Repository: huggingface/lighteval
Issue: main_accelerate.py  failed to pass generation config
Root cause: Interface Contract Mismatch

## Describe the bug
`main_accelerate.py`  failed to pass generation config

## To Reproduce
```shell
$MODEL = "SmallDoge/Doge-160M"
$MODEL_ARGS="pretrained=$MODEL,trust_remote_code=True,dtype=bfloat16,max_length=2048,generation_parameters={max_new_tokens:2048,temperature:0.6,top_p:0.95}"
$OUTPUT_DIR = "./lighteval_results/$MODEL"
lighteval accelerate $MODEL_ARGS `
    "lighteval|piqa|0|0" `
    --custom-tasks evaluation/base/tasks.py `
    --override-batch-size 1 `
    --output-dir $OUTPUT_DIR
```

[2025-04-05 09:48:24,300] [    INFO]: PyTorch version 2.4.1+cu124 available. (config.py:54)
{max_new_tokens
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ A:\Anaconda\envs\lighteval\lib\site-packages\lighteval\main_accelerate.py:197 in accelerate      │
│                                                                                                  │
│   194 │   │   model_args_dict["use_chat_template"] = use_chat_template                           │
│   195 │   │   model_args_dict["compile"] = bool(model_args_dict["compile"]) if "compile" in mo   │
│   196 │   │   print(model_args_dict['generation_parameters'])                                    │
│ ❱ 197 │   │   model_config = TransformersModelConfig(**model_args_dict)                          │
│   198 │                                                                                          │
│   199 │   pipeline = Pipeline(                                                                   │
│   200 │   │   tasks=tasks,                                                                       │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
TypeError: TransformersModelConfig.__init__() got an unexpected keyword argument 'temperature'

## Expected behavior
I wish to submit a PR to fix this bug

## Version info
The latest version
Repository: huggingface/lighteval
Issue: [BUG] Transformers model padding should be to "longest"
Root cause: Configuration Error

## Describe the bug
transformers model padding is not efficient, 
https://github.com/huggingface/lighteval/blob/75f44298a3d77da3e301ddd24be4fbca0266be11/src/lighteval/models/transformers/transformers_model.py#L863

As the comments say, "we pad to the longest sequence", it should be `padding="longest"`

## To Reproduce
`lighteval accelerate \
    "pretrained=gpt2" \
    "leaderboard|truthfulqa:mc|0|0"`

## Expected behavior
The initial sequence length does not have to be that long (max length)

## Version info
0.8.0


Hey ! Being fixed in #669
Repository: huggingface/lighteval
Issue: [BUG] can't pickle mappingproxy objects
Root cause: Interface Contract Mismatch

## Describe the bug
Facing this issue: https://github.com/pydantic/pydantic/issues/2192 when running lighteval in databricks

## To Reproduce
Running any of the examples of lighteval in a databricks cluster will result in the above issue.

## Expected behavior
Evaluation should execute.



The error happens when importing the class `InferenceProvidersModelConfig`. The `generation_parameters` attribute should be a `pydantic` `Field()` since the class is a `pydantic` model and not the `field` from `dataclasses`.

https://github.com/huggingface/lighteval/blob/cc95ff274718186f587500556d7001645a273ce8/src/lighteval/models/endpoints/inference_providers_model.py#L71 

Hey ! Thanks for the report, this is fixed in #678
Repository: huggingface/lighteval
Issue: Loading local data for custom tasks
Root cause: Unimplemented Feature Gap

Hi team, I am running some eval with generate custom data locally. Is it possible to dynamically add custom tasks with a local file directory? Ideally I would like to be able to do something like:
`load_datasets("parquet", data_files="path_to_my_data")`
I do not believe `data_files` is an argument to `LightevalTaskConfig` ? 
Thank you!


Hi ! No it's not possible atm but you can definetely add it and we would be happy to review and merge it :)

I resolved it. will post PR eod

Hi @NathanHB. I'm also interested in this feature. Were you happy with the changes originally proposed in #687? If so, happy to resubmit those in a new PR with style checks. Thank you!

hey @davebiagioni ! Yes absolutely, would be happy to review and merge this.

I believe simply adding a `data_files` in the lighteval task config that defaults to None but loads files when defined should work.

Ping if you need any help on this :)

@NathanHB here you go, happy to iterate as needed. thanks!! #1083 

thanks ! added a review :)
Repository: huggingface/lighteval
Issue: [FT] Improve Documentation and Examples
Root cause: Documentation Deficiency

## Issue encountered
We need better documentation for locally trained LLMs. I am experimenting with finetunes and LoRA adapters but I am having a bad time ironing out the required values for the yaml config.

## Solution/Feature
Provide working examples for adapter configurations.

## Possible alternatives
Make the library more coherent? --override-batch-size no longer works but it is still available under `lighteval accelerate`. 

Hey ! Thanks for raising this issue, the `--overide-batch-size` issue should be fixed.

For adding more doc for adapter models can you provide me with an example model (and base model) on the hub you would like to see work ?

+1 to this. The examples for model_config from the github examples folder are not working

@NathanHB it seems like none of these [examples](https://github.com/huggingface/lighteval/blob/main/examples/model_configs/transformers_model.yaml) are following the correct yaml structure. 

I see a different structure in the [docs](https://huggingface.co/docs/lighteval/en/use-vllm-as-backend):

```python
model: # Model specific parameters
  base_params:
    model_args: "pretrained=HuggingFaceTB/SmolLM-1.7B,revision=main,dtype=bfloat16" # Model args that you would pass in the command line
  generation: # Generation specific parameters
    temperature: 0.3
    repetition_penalty: 1.0
    frequency_penalty: 0.0
    presence_penalty: 0.0
    seed: 42
    top_k: 0
    min_p: 0.0
    top_p: 0.9
```



> Hey ! Thanks for raising this issue, the `--overide-batch-size` issue should be fixed.
> 
> For adding more doc for adapter models can you provide me with an example model (and base model) on the hub you would like to see work ?

Thank you for the prompt reply and I'm sorry this got buried in the emails. 

I am using a custom model that I finetuned. Uhh, but you could test it out with any PEFT model. I might release the model in the coming weeks - if I do I will post it here for good measure.

@sciarrilli yes because there is a missmatch between main and latest release, you would need to install from main to be able to use the examples

@ashim-mahara unfirtunatly i don't have the bandwidth right now.
To make it wor youwould need to install from main and edit [this](https://github.com/huggingface/lighteval/blob/71fec806b3045ee5d47a605288e3aa33733938bb/src/lighteval/models/transformers/adapter_model.py#L52) and [here](https://github.com/huggingface/lighteval/blob/71fec806b3045ee5d47a605288e3aa33733938bb/src/lighteval/models/transformers/delta_model.py#L47) too part of the code:

If you open a PR will be happy to review :)