zhiminy/EvalEng huggingface dataset:
Repository: huggingface/lighteval
Issue: [FT] `StrEnum` for `suites` to intuitively document options
Root cause: Unimplemented Feature Gap

## Issue encountered

As of `lighteval==0.10.0`, the `suites` list passed into `LightevalTaskConfig` is just free strings.

A free string leaves infinite options, whereas in practice only about 5 choices are used.

## Solution/Feature

A `StrEnum` be made to make `lighteval`'s code intuitive:

- Has comments/descriptions illuminating each option
- Makes it easy for devs to see all options and quickly pick the right one

```python
class Suites(StrEnum):
    # Core evaluations maintained by the LightEval team
    LIGHTEVAL = "lighteval"
    # Third party evaluations
    COMMUNITY = "community"
```

## Possible alternatives

Importable constants like the below (less preferable since it leads to the proliferation of many imports):

```python
# Core evaluations maintained by the LightEval team
LIGHTEVAL = "lighteval"
# Third party evaluations
COMMUNITY = "community"
```


great suggestion, we will implement when we have the bandwidth
Repository: huggingface/lighteval
Issue: [FT] more docstrings and typing in `Doc`

As of `lighteval==0.10.0`, reading `Doc` I have some requests:

- Document what should one set `choices`/`gold_index` when it's an open answer question
- Default `original_query` to `None` as opposed to empty string, which is misleading
- Fix typing of `specific` to include `| None`: `specific: dict = None` --> `specific: dict | None = None`
- Document if `specific` should be JSON serializable

thanks for the feedback, we are doing a big refacto right now, making the code cleaner and removing bloat. we will add all your suggestions once that is merged :)

Thank you, sounds great! Looking forward to it.

I suggest adding `mypy` to give automated feedback on type hints, here's a subset of the config I like to use:

```toml
[tool.mypy]
check_untyped_defs = true
enable_error_code = [
    "ignore-without-code",
    "mutable-override",
    "redundant-cast",
    "redundant-expr",
    "redundant-self",
    "truthy-bool",
    "truthy-iterable",
    "unimported-reveal",
    "unreachable",
    "unused-awaitable",
    "unused-ignore",
]
show_error_codes = true
warn_redundant_casts = true
warn_unreachable = true
warn_unused_configs = true
warn_unused_ignores = true
```
Repository: huggingface/lighteval
Issue: [FT] supporting train-time vs test-time metrics
Root cause: Unimplemented Feature Gap

## Issue encountered

I have reward functions where:

- At train time: stricter behavior, aka it's harder to get "correct" (1.0 reward)
- At test time: relaxed reward behavior

This is analogous to:

- Train time: high school students only being allowed non-graphing calculators during tests
- Test time: workforce lets employees use computers at will

From https://huggingface.co/docs/lighteval/en/quicktour#basic-usage:

> The syntax for the task specification might be a bit hard to grasp at first. The format is as follows:
> ```none
> {suite}|{task}|{num_few_shot}|{0 for strict `num_few_shots`, or 1 to allow a truncation if context size is too small}
> ```

This syntax well-supports few-shot behaviors, but does not support train-time vs test-time differences.

## Solution/Feature

I have a few ideas:

- Generalizing `num_few_shot` (third entry) to be arbitrary metadata
- Making a new CLI arg such as `--loose` or `--strict`

## Possible alternatives

Duplicating metrics by name, so one goes from `reward_function_x` to `reward_function_x_train` + `reward_function_x_test`. This feels hacky though, it's not using the framework to solve problems for me.
Repository: huggingface/lighteval
Issue: [FT] integrate typo checker to check typos like `refenrence`

## Issue encountered

https://github.com/huggingface/lighteval/blob/v0.10.0/src/lighteval/main_endpoint.py#L355C54-L355C64 has a typo `refenrence`.

## Solution/Feature

Can `lighteval` add tooling to `pre-commit` such as https://github.com/codespell-project/codespell or https://github.com/crate-ci/typos to automatically find/fix typos.

## Possible alternatives

None

hey ! thanks for the sugestion, it's a good idea ! We will add it asap :)
Repository: huggingface/lighteval
Issue: Large memory usage on MATH
Root cause: Configuration Error

Is the MATH benchmark expected to run for anything beyond `batch_size=1`?

Running the following command for a small model gives OOM on a single node of H100s which is a bit surprising to me:

```shell
accelerate launch --multi_gpu --num_processes=8 run_evals_accelerate.py \
    --tasks="lighteval|math:algebra|5|0" \
    --output_dir "./scratch/evals" \
    --model_args "pretrained=Qwen/Qwen1.5-0.5B" \
    --override_batch_size 2
```

Strangely enough, bumping up the batch size for Mistral 7B is fine:

```shell
accelerate launch --multi_gpu --num_processes=8 run_evals_accelerate.py \
    --tasks="lighteval|math:algebra|5|0" \
    --output_dir "./scratch/evals" \
    --model_args "pretrained=mistralai/Mistral-7B-v0.1" \
    --override_batch_size 2
```

Perhaps there's some sort of unbounded generation occurring which is causing the memory to explode for certain models like Qwen?

Hi,
Thanks for the issue!

I can confirm that the generation size is unbounded, which you can see in the task description
```
{"name":"math:algebra","suite":["lighteval","math"],"prompt_function":"math","hf_repo":"lighteval\/MATH","hf_subset":"algebra","hf_avail_splits":["train","test","validation"],"evaluation_splits":["test"],"few_shots_split":null,"few_shots_select":null,"generation_size":null,"metric":["quasi_exact_match_math"],"stop_sequence":["\n"],"output_regex":null,"frozen":false}
```
When `generation_size` is null, there is no bound expect the model's max context length (should be around 8K for both these models though).

I'll check if the paper defines a maximum expected generation size, else will fix the bound to the maximum answer size + 10% maybe?

> I'll check if the paper defines a maximum expected generation size, else will fix the bound to the maximum answer size + 10% maybe?

Yes, alternatively we could set the max gen size to something like 1024 or 2048 tokens since if a model cannot answer in that span then it is likely incorrect. You can see [here](https://github.com/hendrycks/math/blob/357963a7f5501a6c1708cf3f3fb0cdf525642761/modeling/eval_math_gpt.py#L333) that the authors chose 1024 tokens for models that aren't `gpt2-xl`, so 2048 seems like a safe bet

Sounds perfect, will use this rn!
Repository: huggingface/lighteval
Issue: [BUG] Forced to hit `You cannot select the number of dataset splits` with `litellm`
Root cause: Validation Gap

## Describe the bug

When using `LiteLLMClient` we are forced to hit a warning:

```none
[2025-06-06 18:21:13,616] [ WARNING]: You cannot select the number of dataset splits for a generative evaluation at the moment. Automatically inferring. (data.py:237)
```

## To Reproduce

1. From https://github.com/huggingface/lighteval/blob/v0.10.0/src/lighteval/models/abstract_model.py#L58, `DATASET_SPLITS = 4`.
2. From https://github.com/huggingface/lighteval/blob/v0.10.0/src/lighteval/models/litellm_model.py#L229, we pass that to `num_dataset_splits`.
3. Then from https://github.com/huggingface/lighteval/blob/v0.10.0/src/lighteval/data.py#L83-L85, we are forced to hit the above warning.

## Expected behavior

Not being forced to hit a warning.

## Version info

`lighteval==0.10.0`


Not something we plan to address for now as it's non blocking
Repository: huggingface/lighteval
Issue: [BUG] `test` split forced to hit `Careful` warning
Root cause: Validation Gap

## Describe the bug

Using `hf_avail_splits=["test"]` with `LightevalTaskConfig` you are forced to hit:

```none
[2025-06-06 18:21:10,501] [ WARNING]: Careful, the task community|foo is using evaluation data to build the few shot examples. (lighteval_task.py:260)
```

## To Reproduce

Since `"test"` isn't a value in `allowed_splits` at https://github.com/huggingface/lighteval/blob/v0.10.0/src/lighteval/tasks/lighteval_task.py#L251.

## Expected behavior

Not forced to hit a warning with `hf_avail_splits=["test"]`, since `test` is a commonly used split.

## Version info

`lighteval==0.10.0`
Repository: huggingface/lighteval
Issue: [FT] `MetricUseCase.FORMATTING` for reasoning models
Root cause: Unimplemented Feature Gap

## Issue encountered

With reasoning models such as DeepSeek R1, these models have a format reward applied to reasoning traces. From https://arxiv.org/pdf/2501.12948:

> **Format rewards**: In addition to the accuracy reward model, we employ a format reward model that enforces the model to put its thinking process between between ‘< think >’ and ‘</ think >’ tags.

(note I put space around `<>` so GitHub Flavored Markdown doesn't truncate the XML).

## Solution/Feature

A new category for `MetricUseCase` called `FORMATTING`, for these types of evaluations.

## Possible alternatives

There is an argument this format reward should be treated as an accuracy (`MetricUseCase.ACCURACY`), because format reward (when given `1.0` for matching, `0.0` for not matching), can be interpreted as a percentage of correctly formatted responses.

hey ! i'm not sure I understand the issue / solution proposed. could you give an example ?

Yeah basically DeepSeek R1 (and many others now) have a two-part reward:

1. Format reward: checking XML-like syntax in the response
2. Accuracy reward: extract and grade the answer

So for item 1's "format reward" good examples can be:

```python
# Ideal
"<think>thought</think><answer>answer</answer>"
# Minor whitespace added
"<think>\nthought\n</think><answer>\nanswer\n</answer>"
```

Here's the regex we use as our format reward: https://github.com/Future-House/ether0/blob/c8cc676354e926b50ad206a606e04489bc9c95e3/src/ether0/model_prompts.py#L18

And examples with bad formatting are basically anything else:

```python
# Reversed XML
<answer>answer</answer><think>thought</think>
# Duplicate thought
<think>thought</think><think>thought</think><answer>answer</answer>
```
Repository: huggingface/lighteval
Issue: [FT] showing count in Markdown summary table
Root cause: Unimplemented Feature Gap

## Issue encountered

I am seeing this output from `lighteval endpoint litellm` with `lighteval==0.10.0`:

```none
|          Task          |Version|    Metric     |Value|   |Stderr|
|------------------------|------:|---------------|----:|---|-----:|
|all                     |       |ether0_accuracy|    0|±  |     0|
|community:ether0:loose:0|      0|ether0_accuracy|    0|±  |     0|
```

At a glance, it's unclear if I have misconfigured `LightevalTaskConfig.hf_filter`, or if accuracy truly was 0%.

(Note: in actuality, there were 10 questions, all incorrect.)

## Solution/Feature

Showing a count of questions run, to clarify the size of the evaluation.

## Possible alternatives

None
Repository: huggingface/lighteval
Issue: [BUG] `Registry.task_registry` not reimporting the custom task module
Root cause: Validation Gap

## Describe the bug

If you've already imported a custom task module, then invoking `lighteval.main_tasks.list`'s call to `Registry.task_registry` will reimport the module: https://github.com/huggingface/lighteval/blob/v0.10.0/src/lighteval/tasks/registry.py#L132 --> https://github.com/huggingface/lighteval/blob/v0.10.0/src/lighteval/tasks/registry.py#L260

The issue here is then you register the metrics again in `Metrics` enum, which leads to a `TypeError` crash:

```none
TypeError: 'your_metric' already in use as property(<Metrics.your_metric: SampleLevelMetric(...)
```

## To Reproduce

Here is my `tasks.py`:

```python
import statistics

from aenum import extend_enum
from lighteval.metrics.metrics import (
    MetricCategory,
    Metrics,
    MetricUseCase,
    SampleLevelMetric,
)


extend_enum(
    Metrics,
    "your_metric",
    SampleLevelMetric(
        metric_name="your_metric",
        higher_is_better=True,
        category=MetricCategory.GENERATIVE,
        use_case=MetricUseCase.ACCURACY,
        sample_level_fn=lambda *_, **__: 0.0,
        corpus_level_fn=statistics.mean,
    ),
)
```

```python
from lighteval.main_tasks import list as lighteval_list

import package.tasks


def test_task_list() -> None:
   lighteval_list(custom_tasks=package.tasks.__file__)
```

This will throw:

```none
E           TypeError: 'your_metric' already in use as property(<Metrics.your_metric: SampleLevelMetric(metric_name='your_metric', higher_is_better=True, category=<MetricCategory.GENERATIVE: '3'>, use_case=<MetricUseCase.ACCURACY: '1'>, sample_level_fn=<function <lambda> at 0x15ad42840>, corpus_level_fn=<function mean at 0x15ad432e0>)>)
```

## Expected behavior

LightEval to not re-import modules already imported.

## Version info

`lighteval==0.10.0`

In the meantime, this can be resolved via a `patch`:

```python
from unittest.mock import patch

from lighteval.main_tasks import list as lighteval_list

import package.tasks


with patch(
    "lighteval.tasks.registry.create_custom_tasks_module",
    side_effect=[package.tasks],
):
    lighteval_list(custom_tasks=package.tasks.__file__)
```

thanks for the issue ! haven't had the time to look at PRs yet but if you oppened one with a fix i will take a look :)
Repository: huggingface/lighteval
Issue: [FT] Allow configuration of `litellm` cache location
Root cause: Architectural Constraint

## Issue encountered

In `lighteval==0.10.0`, the creation of `litellm`'s `Cache`: https://github.com/huggingface/lighteval/blob/v0.10.0/src/lighteval/models/litellm_model.py#L59

Does not allow for configuration of the cache location.

## Solution/Feature

Allow configuration of the cache location via an environment variable like so:

```python
LIGHTEVAL_LITELLM_CACHE_LOCATION = os.environ.get("LIGHTEVAL_LITELLM_CACHE_LOCATION")

litellm.cache = Cache(type="disk", disk_cache_dir=LIGHTEVAL_LITELLM_CACHE_LOCATION)
```

## Possible alternatives

N/a


Great idea ! Though I would prefer to allow the cache location to be submitted via the model config, feel free to open a PR :)
Repository: huggingface/lighteval
Issue: [FT] VL model support for vllm/sglang backend?
Root cause: Unimplemented Feature Gap

## Issue encountered
need to load VL model with VLLM/SGLANG backend

## Solution/Feature
i see there is VLMTransformersModel support for accelerate, any plans to support VLLM/SGLANG?



Hi ! Yes there are, though not very high prio right now we would be happy to help !
We are going to merge a big refacto soon allowing for more flexibility in models types, stay tuned for this :)

> Hi ! Yes there are, though not very high prio right now we would be happy to help ! We are going to merge a big refacto soon allowing for more flexibility in models types, stay tuned for this :)

any ETA yet?
Repository: huggingface/lighteval
Issue: [Question] generation max_new_tokens or generation_size? which parameter takes effect?
Root cause: Documentation Deficiency

Recent reasoning models (deepseek-ai/DeepSeek-R1-0528-Qwen3-8, for example) have a maximum new token length of 64k or longer.
However, the default generation size for AIME24, AIME25, GPQA Diamond, or Live Code Bench is 32k.
Will the max_new_tokens setting in the generation parameters overwrite the configuration in the task definition?

Hi ! yes it will, the value in task definition is for when you do not specify anything in the generation config. However, you can set them both to None and then the model will just generate until its max length
Repository: huggingface/lighteval
Issue: Mamba OOM deep dive
Root cause: Resource Mishandling

https://huggingface.slack.com/archives/C05B0D42GHE/p1750438139961689?thread_ts=1747664911.185729&cid=C05B0D42GHE

Can repro with 
`accelerate launch lighteval accelerate "model_name=tiiuae/Falcon-H1-3B-Base,dtype=bfloat16,trust_remote_code=True,batch_size=1"  "lighteval|ruler_32768|0|0 --output-dir=~/tmp/_logs --max-samples 10`

@eliebak @NouamaneTazi 

This error seems unrelated to lighteval, as far as I can tell.

In `FalconH1Mixer` (the mamba mixer equivalent, to compute the SSM transformation), in `torch_forward`, we OOM on the  [`G_intermediate` computation](https://github.com/huggingface/transformers/blob/07aab1af1ed80d252d7be9661e2d6ee11f7ed8e8/src/transformers/models/falcon_h1/modeling_falcon_h1.py#L940) 
`G_intermediate = C[:, :, :, None, :, :] * B[:, :, None, :, :, :]`. 

B and C (of shape torch.Size([1, 266, 128, 32, 256]) ) have been cast to float32 ([here](https://github.com/huggingface/transformers/blob/07aab1af1ed80d252d7be9661e2d6ee11f7ed8e8/src/transformers/models/falcon_h1/modeling_falcon_h1.py#L916)). So the matrix multiplication tries to get a result of size torch.Size([1, 266, 128, 128, 32, 256]) , so a matrix of (I think) 36B numbers (35 701 915 648), which at 4 bytes per number (since we're in float32) takes ~140GB (142 807 662 592 bytes), way bigger than our 80G mem per GPU.
Repository: huggingface/lighteval
Issue: Max token issue on RULER
Root cause: Validation Gap

https://huggingface.slack.com/archives/C05B0D42GHE/p1750437950262289?thread_ts=1750436546.960999&cid=C05B0D42GHE

Command: `lighteval vllm model_name=/fsx/nouamane/projects/lighteval/elie-lc-prolong-64k-dclm-mix-seq64k-rope10M-20000,dtype=bfloat16,trust_remote_code=False,add_special_tokens=False,max_model_length=131072,max_num_batched_tokens=131072,use_dual_chunk_attention=False,generation_parameters={temperature:0},data_parallel_size=8,enforce_eager=False 'lighteval|ruler_131072|0|0'`

Can repro with `lighteval vllm   "model_name=/fsx/nouamane/projects/lighteval/elie-lc-prolong-64k-dclm-mix-seq64k-rope10M-20000,dtype=bfloat16,trust_remote_code=False,add_special_tokens=False,max_model_length=131072,max_num_batched_tokens=131072,generation_parameters={temperature:0},data_parallel_size=2,use_dual_chunk_attention=False,enforce_eager=False" "lighteval|ruler_131072|0|0"`

Can detect the wrong item by raising an exception instead of logging in `_eventual_warn_about_too_long_sequence` in transformers, `transformers/tokenization_utils_base.py`

Item 481 at 'lighteval|ruler_131072:fwe|0|0`, details [here](https://huggingface.slack.com/archives/C05B0D42GHE/p1750691646084039?thread_ts=1750436546.960999&cid=C05B0D42GHE)

@NathanHB co assigning you the issue because you've got the code to regen but feel free to tell me where to get it if you need
Repository: huggingface/lighteval
Issue: Documenting how to launch multilingual tasks
Root cause: Documentation Deficiency

Atm, need to use custom tasks to launch them, must be documented

Came here to open this issue. It took me some time to figure out that I need to a pass a custom task file and not a folder:
```
lighteval accelerate "model_name=gpt2" "community|tumlu:azerbaijani|0|0" --custom-tasks .\community_tasks\turkic_evals.py
```

It would be nice if there was an example like this in README:

+1
Repository: huggingface/lighteval
Issue: Remove extended tasks system and move everything to a "custom task" folder - add the extended task system as custom task
Root cause: Unimplemented Feature Gap

https://huggingface.slack.com/archives/C05B0D42GHE/p1750675378956839?thread_ts=1749899149.610739&cid=C05B0D42GHE
Repository: huggingface/lighteval
Issue: [FT]  Tokenizing the data right before launching the evaluation job and saving it to the cache
Root cause: Unimplemented Feature Gap

This would reduce overhead and be less error-prone

Some refs:
- https://huggingface.co/docs/huggingface_hub/guides/manage-cache
- https://huggingface.co/docs/datasets/en/cache
- https://github.com/huggingface/datasets/blob/161f99d94a1daf8380eabdb826048a0652510ee6/src/datasets/packaged_modules/cache/cache.py
Repository: huggingface/lighteval
Issue: [BUG] System Prompt option now missing?
Root cause: Configuration Error

## Describe the bug
I was using an older version of LightEval packaged with OpenR1 repo, it had a `--system-prompt "xxx"` arg for lighteval.

That seems to be gone. I'm told that it is now a model arg, though I dont see that in the docs for vllm, but also if any comma is in the system prompt, it will break everything.

## To Reproduce
use

```
        lighteval vllm "$MODEL_ARGS" "$benchmark_task" \
            --system-prompt "$system_prompt" \
            --use-chat-template \
            --output-dir "$output_dir" 2>&1 | tee -a "$log_file"
```

mmm it seems this also no longer works?

Usage: lighteval vllm [OPTIONS] MODEL_ARGS TASKS
Try 'lighteval vllm --help' for help.
╭─ Error ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ No such option: --use-chat-template

Yes the system prompt now moved to the model's arg to facilitate the logging and make the codebase clearer. I was using yaml config files for model and did not think of that edge case, thanks for the issue I will think of of fix asap !
In the meantime, using a yaml config file will work., also for the doc you need to check for the main branhc has we haven't done a release yet

https://huggingface.co/docs/lighteval/main/en/package_reference/models

I can't seem to get it to work even when using yaml config file. Here's my yaml file:

model_parameters:
    model_name: "Qwen/Qwen2.5-3B"
    dtype: "bfloat16"
    data_parallel_size: 4
    gpu_memory_utilization: 0.8
    max_model_length: 32768
    use_chat_template: True
    system_prompt: 'You are a helpful AI Assistant, designed to provided well-reasoned and detailed responses. You FIRST think about the reasoning process step by step and then provide the user with the answer. Please enclose your final answer in the box: \boxed{Your Answer}. Please stop generation immediately after outputing the box.'
    add_special_tokens: True
    generation_parameters:
      temperature: 0.9
      top_k: 50
      min_p: 0.0
      top_p: 1.0
      max_new_tokens: 3076


Error: ValidationError: 1 validation error for VLLMModelConfig
system_prompt
  Extra inputs are not permitted [type=extra_forbidden, input_value='You are a helpful AI Ass...fter outputing the box.', 
input_type=str]

are you using the latest main commit ?
Repository: huggingface/lighteval
Issue: [FT] Why not use math_verify as a direct dependency?
Root cause: Architectural Constraint

Hi team,

I noticed that the mathematical parsing and verification functions in lighteval appear to be a vendored (copied) version of the [Math-Verify](https://github.com/huggingface/Math-Verify) package.

Manual syncs, like the one in PR #535, seem to confirm this.

To simplify maintenance and ensure we always have the latest bug fixes and improvements, would it be possible to add math_verify as a direct project dependency instead of maintaining a local copy?


Hi, to give a bit of background, the math-verify started as a lighteval PR to improve the handling of math evaluations in lighteval, but as a time went we decided to out-source it to separate package as it can be used not just for evals but for verification as well.

There are small nuances between the two version -> The lighteval version hasn't been updated for a while, while the math-verify still lacks the "multilingual" support. Thus in order to be full dependency we would have to resolve the discrepancies. I do plan to do it but haven't found time for it yet, but it would be ideal to have it just pure dependency as it's painful to keep the versions in line

Thanks for the detailed background, that clarifies things perfectly.
Repository: huggingface/lighteval
Issue: [BUG] Cannot import name taskinfo_selector from lighteval.tasks.registry
Root cause: Validation Gap

## Describe the bug

It seems that some parts of the codebase are outdated given recent commits (which I assume were refactors). This includes importing the `taskinfo_selector` when calling `lighteval tasks inspect`

## To Reproduce

Run `lighteval tasks inspect` to any of the community tasks


## Version info
Commit: 88bd36ae001ddd85100fc41263ddb2af438e173c
Python: 3.12


Hi! Yes i missed this function. I opened a PR to fix it, thanks for the report !

this should be fixed in #887