zhiminy/EvalEng huggingface dataset:
Repository: huggingface/lighteval
Issue: [BUG] Cache management failed for tasks using MetricGrouping
Root cause: Validation Gap

## Describe the bug
SampleCache:_get_task_hash uses LightevalTaskConfig:__str__ to compute task hash, but LightevalTaskConfig:__str__
doesn't handle tasks using MetricGrouping correctly, for MetricGrouping field corpus_level_fn is a dict, so another level of loop is needed.

## To Reproduce
Cache doesn't work for "leaderboard|truthfulqa:mc|0"

## Expected behavior
Cache should work

## Version info
0.11.0


Hm this one could be fixed fast by adding an `__str__` method to the MetricGrouping I think - do you think you'd have the time for a PR?

Sure, let me have a try
Repository: huggingface/lighteval
Issue: [BUG] LightevalTaskConfig doesn't support comparison, but Cache management uses sorted on it
Root cause: Validation Gap

## Describe the bug
Specify two tasks as below:
helm|boolq|0
helm|boolq:contrastset|0
The current logic will assume "helm|boolq" is a group and will add "helm|boolq:contrastset" again, then "helm|boolq:contrastset" will appear twice, later at SampleCache:_get_task_hash sorted is used to sort these two, and
an error "<" is not supported on LightevalTaskConfig is raised

## To Reproduce
Specify two tasks as below in yaml:
helm|boolq|0
helm|boolq:contrastset|0

## Expected behavior
Run should succeed

## Version info
0.11.0


Oh cool edge case, good catch! Here we would actually rename `boolq` to `boolq:default` so that a task does not also carry a group name.

How about **sorted**(self.registry.task_to_configs[f"{task_suite}|{task_name}"]), currently comparison operator is not defined for LightevalTaskConfig, so this place will fail if more than one config is met, any thought?
Repository: huggingface/lighteval
Issue: [BUG] Cache management failed for tasks using sample_params to update function attrs
Root cause: Algorithmic Error

## Describe the bug
Metric:__call__ used "&".join(f"{k}={v}" for k, v in sample_params.items()) to update metric name, but the value might be function, which will change for each run.

## To Reproduce
Task that use sample_params, such as:
leaderboard:gsm8k:0

    metrics=[
        Metrics.exact_match(sample_params={"normalize_gold": gsm8k_normalizer, "normalize_pred": gsm8k_normalizer})
    ],

## Expected behavior
Cache should work

## Version info
main branch


hey! thanks for the issue. We won't have time to focus on this for now as we are integrating inspect-ai for cache!
Repository: huggingface/lighteval
Issue: How to evaluate MMLU-Pro
Root cause: Documentation Deficiency

Hi,

Thank you for the wonderful work!

I just want to ask how to perform the evaluation on MMLU-Pro, as I don't see any related code besides the README.

There is code related MMLU-Pro in Pr 835, but it was merged to branch model_release, not sure why.

https://github.com/huggingface/lighteval/pull/835

https://github.com/huggingface/lighteval/tree/model_release/src/lighteval/tasks/extended

hey ! we are adding mmlu_pro in the main tasks here #1031
Repository: huggingface/lighteval
Issue: [BUG] Python API doesn't seem to work with lighteval|gsm8k|<few-shot-number>
Root cause: Configuration Error

## Describe the bug
Lighteval's python API doesn't seem to work with lighteval|gsm8k|<few-shot-number>. It looks like LightEval is looking for the GSM8K dataset under `lighteval/openai/gsm8k` instead of using the Hugging Face dataset loader.

## To Reproduce
Run the following code and you should get the following error: 
FileNotFoundError: Couldn't find any data file at <your-working-path>/lighteval/openai/gsm8k

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from lighteval.logging.evaluation_tracker import EvaluationTracker
from lighteval.models.transformers.transformers_model import (
    TransformersModel,
    TransformersModelConfig,
)
from lighteval.pipeline import ParallelismManager, Pipeline, PipelineParameters


def load_model():
    base_model = "Qwen/Qwen2.5-7B"
    tokenizer = AutoTokenizer.from_pretrained(base_model, trust_remote_code=True)
    tokenizer.padding_side = "left"
    
    if tokenizer.pad_token_id is None:
        tokenizer.pad_token_id = tokenizer.eos_token_id

    model = AutoModelForCausalLM.from_pretrained(
        base_model,
        device_map="auto",
        torch_dtype=torch.bfloat16,
        trust_remote_code=True,
    )

    return model, tokenizer


def main():
    model, tokenizer = load_model()

    # Setup lighteval
    evaluation_tracker = EvaluationTracker(
        output_dir="./eval_results",
        save_details=True,
        push_to_hub=False,
    )

    pipeline_params = PipelineParameters(
        launcher_type=ParallelismManager.NONE,
        custom_tasks_directory=None,
        max_samples=10,
    )

    config = TransformersModelConfig(
        model_name="Qwen/Qwen2.5-7B",
        batch_size=4,
    )

    lighteval_model = TransformersModel.from_model(model, config)

    # This triggers the FileNotFoundError
    pipeline = Pipeline(
        model=lighteval_model,
        pipeline_parameters=pipeline_params,
        evaluation_tracker=evaluation_tracker,
        tasks="lighteval|gsm8k|0",
    )

    pipeline.evaluate()
    pipeline.save_and_push_results()
    pipeline.show_results()


if __name__ == "__main__":
    main()
```

## Expected behavior
You should see this error:
FileNotFoundError: Couldn't find any data file at <your-working-path>/lighteval/openai/gsm8k

## Version info
- OS: Linux (x86_64, kernel 5.14.0-503.19.1.el9_5.x86_64)
- LightEval version: 0.11.0
- Python version: 3.13 (conda environment)
- Conda: yes (Anaconda/Miniconda-based setup)
- Install method: pip install lighteval
- Location: site-packages within conda env

hey ! this will be fixed in latest release (today). but it's already fixed on main if you need.
Repository: huggingface/lighteval
Issue: [BUG] fail to execute task harness|bbh:movie_recommendation|0
Root cause: Validation Gap

## Describe the bug
LightevalTask:_get_docs_from_split expects self.formatter to return a doc, but default_prompts.py:bbh_movie_recommendation return [] for a certain line, which caused
error in line "doc.id = str(ix)"

## To Reproduce
Run task:
harness|bbh:movie_recommendation|0

## Expected behavior
should succeed.

## Version info
0.11.0


facing same error, let me know if you have fixed it.

The change below can bypass this issue, which just ignore that doc.

src/lighteval/tasks/lighteval_task.py‎
 
                doc = self.formatter(item, self.name)
                if not doc:
                    continue
                doc.id = str(ix)

Hey ! Thanks for the issue. Can you open a PR and we will merge it asap ?

Close as already fixed in “Fixing mixeval”
Repository: huggingface/lighteval
Issue: [EVAL] Add kyrgyzLLM benchmark
Root cause: Unimplemented Feature Gap

Hi, 

We just open-sourced the Kyrgyz LLM Evaluation Dataset. 

## Evaluation short description
- Why is this evaluation interesting?

KyrgyzLLM-Bench is the first comprehensive benchmark suite for deep language understanding in Kyrgyz. It is interesting because it provides broad, culturally grounded coverage by combining native benchmarks (such as KyrgyzMMLU and KyrgyzRC) with carefully translated and post-edited international benchmarks (such as HellaSwag, WinoGrande, BoolQ, GSM8K, and TruthfulQA).

- How used is it in the community?
As the benchmark was released recently, its adoption by the community is just beginning. It is significant because it's the first comprehensive benchmark suite for deep language understanding, specifically in the Kyrgyz language, providing a new and essential tool for researchers and developers.

## Evaluation metadata

- Paper url: https://ieeexplore.ieee.org/document/11206960
- Github url:  https://github.com/golden-ratio/kyrgyzLLM_bench
- Dataset url: https://huggingface.co/collections/TTimur/kyrgyzllm-bench
- OpenLLM Leaderboard: https://huggingface.co/spaces/TTimur/OpenLLMKyrgyzLeaderboard_v0.1

Thanks you!

Hey ! great work on the benchmark we would love to have it in lighteval and share it with the community :)
Feel free to open a PR and we will review asap
Repository: huggingface/lighteval
Issue: [BUG] [eval] --max-samples 10 and then without, you get the cached data from the previous run.
Root cause: Validation Gap

## Describe the bug
A clear and concise description of what the bug is.

## To Reproduce
Please provide all the steps needed to reproduce the behavior, or provide a minimal working example if needed. We will ignore issues missing this section.

## Expected behavior
A clear and concise description of what you expected to happen.

## Version info
Please provide your operating system, lighteval version or commit if you installed from main, and pip/conda environment if your problem concerns dependencies.
Repository: huggingface/lighteval
Issue: Why custom task cannot success ?  ValueError(f"Cannot find task {expanded_task} in task list or in custom task registry")
Root cause: Documentation Deficiency

<html>
<body>
<hr>
<h3>🧩 Title</h3>
<p><strong>Why custom task cannot success ?  ValueError(f"Cannot find task {expanded_task} in task list or in custom task registry")</strong></p>
<hr>
<h3>🐛 Description</h3>
<p>Hi, I followed the <strong>custom task tutorial</strong> and wrote my own evaluation script.<br>
However, every time I run it, I get the following error:</p>
<pre><code>ValueError: Cannot find task my_custom_task in task list or in custom task registry
</code></pre>
<p>I can confirm that:</p>
<ul>
<li>
<p>my custom task script exists and is accessible;</p>
</li>
<li>
<p>I also set <code inline="">LOCAL_DATASET_PATH</code> to load my dataset from local files;</p>
</li>
<li>
<p>but the task still cannot be found or executed.</p>
</li>
</ul>
<hr>
<h3>💡 What I Did</h3>
<ol>
<li>
<p>Created a new custom task file, e.g.</p>
<pre><code>my_custom_task.py
</code></pre>
<img width="932" height="952" alt="Image" src="https://github.com/user-attachments/assets/aaa3b77a-326b-402c-8130-ca6f57f1968b" />
</li>
<li>
<p>Tried to run:</p>
<pre>

<img width="1537" height="433" alt="Image" src="https://github.com/user-attachments/assets/c4910b34-78cc-42f8-a12a-0018620e55e9" />

</pre>
</li>
<li>
<p>I also set:</p>
<pre><code class="language-bash">export LOCAL_DATASET_PATH=/path/to/my/local/dataset
</code></pre>
<p>because I’m working in an offline environment.</p>
</li>
</ol>
<h3>🪪 Actual Result</h3>
<pre><code>ValueError: Cannot find task my_custom_task in task list or in custom task registry
</code></pre>
<hr>

<hr>
<p>Could you please clarify how a <strong>custom task script</strong> is supposed to be loaded and registered correctly — especially when using local datasets in an offline environment?</p>
<p>Thanks!</p>
<hr>
</body>
</html>

hey @stringofn ! Thanks for the issue.

it seems like your task is named hellaswag, but in the error message it's names `my_custom_task`. Are you sure you try to run hellaswag ?


> hey [@stringofn](https://github.com/stringofn) ! Thanks for the issue.
> 
> it seems like your task is named hellaswag, but in the error message it's names `my_custom_task`. Are you sure you try to run hellaswag ?

Hi @NathanHB ! I solved it , thank you !

great !
Repository: huggingface/lighteval
Issue: Wrong attribute in AvgAtN
Root cause: Algorithmic Error

https://github.com/huggingface/lighteval/blob/566a7be3cb742de6854b832c3ff3fbddd1406316/src/lighteval/metrics/metrics_sample.py#L1202

This attribute should be `self.n` and not `self.k`.


I found the same. 

also here as well ? https://github.com/huggingface/lighteval/blob/566a7be3cb742de6854b832c3ff3fbddd1406316/src/lighteval/metrics/metrics_sample.py#L1238


hey ! thanks for the catch, will merge the fix asap :)

Thanks, this PR https://github.com/huggingface/lighteval/pull/1049 adresses the issue
Repository: huggingface/lighteval
Issue: StarCoder2 3B SFT models give CUDA OOM on IFEval
Root cause: External Dependency Breakage

For some peculiar reason, I am getting CUDA OOM when evaluating an SFT of `bigcode/starcoder2-3b` on `ifeval`. Note this doesn't happen with the 15b models which suggests either a bug on the `transformers` side, but I'm opening the issue first here in case you know if `lighteval` handles these models differently or whether it's a user error on my part :)

Here's the command to reproduce:

```
accelerate launch --multi_gpu --num_processes=8 run_evals_accelerate.py \
    --model_args "pretrained=HuggingFaceH4/starcoder2-3b-sft-v00-deploy" \
    --use_chat_template \
    --tasks "extended|ifeval|0|0" \
    --custom_tasks "extended_tasks/ifeval/main.py" \
    --override_batch_size 1 \
    --output_dir "./scratch/evals"
```

Note you'll need `transformers` from `main` and the version I'm using is:

```
pip install 'transformers @ git+https://github.com/huggingface/transformers.git@0290ec19c901adc0f1230ebdccad11c40af026f5'
```

I'm also using `lighteval` commit `df21407d9f714bde9ecfb4dd8283afdc2150eec3`

I've inspected the inputs / outputs and everything looks good until I hit one sample that seems to blow up the memory.

Edit: the OOM issue is also present in `gsm8k`

Hi ! Thanks for the details, I was able to reproduce the issue and look into it. 
Seems like it is indeed a bug in `transformers` (when using model parallel with 8GPUs it OOMs as well). `lighteval` does not handle these model differently, it could have been the dtype which is not set in your case but setting it to `float16` does not solve the issue.

Interestingly, removing `--use_chat_template` makes it work for `--max_samples 1`.

What is particularly interesting is that samples should be sorted by context size, so hitting a sample where it failed (but not at the start) would be a matter of the generation being too big. 
There could be something in the chat template which encourages the model to continue writing, what's the chat template for this one?

This particular model uses ChatML and appears to follow the prompts OK when I look at the intermediate outputs. I'll see if I can repro the issue with pure `transformers` code and open an issue there
Repository: huggingface/lighteval
Issue: [FT] Prompt Level Caching
Root cause: Unimplemented Feature Gap

## Issue Encountered

Currently, the caching mechanism stores the entire evaluation split but does not consider sampling parameters (e.g., temperature). These parameters significantly influence model behavior, meaning that cached responses may not align with new configurations if the sampling settings change.

In particular, when modifying LiteLLM sampling parameters, cache hits can occur that produce inconsistent or incoherent results relative to the new settings.

## Proposed Solution / Feature

Instead of the current caching approach, a **request-based caching mechanism** could be implemented using a library like [`diskcache`](https://grantjenks.com/docs/diskcache). This method would cache based on the full request payload, ensuring that changes to sampling parameters (or any other request field) generate distinct cache keys.

Below is an illustrative example using an OpenAI-style request:

```python
import hashlib, json, requests, diskcache

# Prepare request parameters
request_params = {
    "url": f"{self.config.base_url}/chat/completions",
    "headers": {
        "Authorization": f"Bearer {self.config.api_key}",
        "Content-Type": "application/json",
    },
    "json": {
        "model": self.config.model_name,
        "messages": [{"role": "user", "content": doc.query}],
        "n": doc.num_samples,
        "max_tokens": self.config.max_tokens,
        "temperature": self.config.temperature,
        "top_p": self.config.top_p,
        "min_p": self.config.min_p,
        "seed": self.config.seed,
        **self.config.extra_body,
    },
    "timeout": self.config.timeout,
}

# Cache lookup and update
with diskcache.Cache(self.config.cache_dir or "/tmp/vllm_cache") as cache:
    key = hashlib.sha256(json.dumps(request_params, sort_keys=True).encode()).hexdigest()
    if key not in cache:
        cache[key] = {
            "response": requests.post(**request_params).json(),
            "request": request_params,
        }
    response = cache[key]["response"]
```

This approach allows multiple concurrent processes or threads to safely access the same cache, enabling consistent reuse across evaluations on a shared filesystem.

Adopting diskcache would also reduce maintenance complexity by replacing the custom caching logic with a robust, well-tested framework.

If this proposal is accepted, I’d be happy to open a PR implementing it.

Hey ! we would be happy to review a PR using disk cache. Though this mechanism would need to also be used for other backends.

That's great, I will give it a shot then.
Repository: huggingface/lighteval
Issue: MajAtN.compute calls compute_score with swapped arguments
Root cause: Algorithmic Error

## Description
In `lighteval/src/lighteval/metrics/metrics_sample.py` line 1258, the `MajAtN.compute` method calls:

```python
return self.compute_score(new_model_response, new_doc)
````

However, `compute_score` (e.g., `MultilingualExtractiveMatchMetric.compute`) expects the signature:

```python
def compute(self, doc: Doc, model_response: ModelResponse) -> float
```

As a result, the arguments are swapped, which cause runtime errors.

## To Reproduce

Steps to reproduce the behavior:

1. Install `lighteval` version 0.12.2.
2. Define a metric using `MajAtN` with a subclass of `MultilingualExtractiveMatchMetric`.
3. Call `compute` on a sample `Doc` and `ModelResponse`.
4. Observe a `TypeError` or incorrect metric values due to the argument order mismatch.

## Expected Behavior

`MajAtN.compute` should pass arguments in the correct order:

```python
return self.compute_score(new_doc, new_model_response)
```

so that `compute_score(doc, model_response)` executes correctly.

```




hey ! thanks for the bug report, i won't have the time right now but it seems like a simple fix and would be happy to review and merge your PR.

Sure, just created a PR.

merged :) Thanks for your help !!
Repository: huggingface/lighteval
Issue: [EVAL] Long Horizon Execution
Root cause: Unimplemented Feature Gap

## Evaluation short description
- Long context
- used by Jan moodel https://huggingface.co/janhq/Jan-v2-VL-low
- 
## Evaluation metadata
Provide all available
- Paper url: https://arxiv.org/abs/2509.09677
- Github url: https://github.com/long-horizon-execution/measuring-execution/
- Dataset url: https://huggingface.co/datasets/arvindh75/Long-Horizon-Execution


hi @NathanHB! i was browsing through the repo's open issues and stumbled across this issue. can i get some more information on the issue? 

upon reading the links you sent me, i assume that you want the paper to be implemented? 

i could work on this after i get some information. 

hey @akshathmangudi ! Yep exactly, there is a nice codebase but it would be extremly useful if you added it :)

The idea is to use inspect-ai to implement the task, no need to other backends for now !

You can look at `mmlu-pro` on how to implement a task with inspect-ai in lighteval

i see! 

if it's fine, i can work on this and drop an implementation approach in a few hours and tag you so you can review it! 

sounds good !

## Ideation

So upon loading the dataset, I noticed that there are 50,000 entries per example, so for an initial effort, it might be better to truncate the dataset to the first 100 or 1000 examples and scale-up. 

Next, we will start with the single-turn approach and implement multi-turn later in the picture, as these approaches help in calculating the cumulative responses. 

We will then construct a dictionary with `input` and `value` columns from the dataset and format the prompt with the dictionary and key sequence. We will follow the repo's prompt structure and ask for cumulative sums in `<answer>` tags.

We will use `inspect-ai` for the evaluation framework, which involves loading the dataset, sending prompts to model, scoring responses and managing the evaluation loop. 

@NathanHB 

Author of that evaluation here, @viciousAegis and I are quite happy to help if there are any questions :)
Repository: huggingface/lighteval
Issue: Improving input token throughput?
Root cause: Others

so, i ran some benchmark with VLLM (I actually modified some parts of the code so it would work with vllm on tpu), and the logs show something like this:
```
Processed prompts:  16%|▏| 1456/9263 [37:28<11:01:04,  5.08s/it, est. speed input: 86.99 toks/s, outp                                                                                                     ProceProcessed prompts:  64%|████████████████████████████████████████████████████████                               

| 5971/9263 [2:16:18<1:17:49,  1.42s/it, est. speed input: 72.62 toks/s, output: 2683.22 toks/s]
```
the input tok/s is much worse compared to the output tok/s, do you think this could be a bottleneck?

hey ! so yes this is probably the bottleneck. What task are you using ?
Pretty sure it's a vLLM issue though, as we feed the engine with all the inputs in one time and let the engine process them.

this one: https://github.com/carlesoctav/lighteval/blob/1b6b5ccff13365ef2d5b28ad6f027cb0ca80c9d9/eval.py
i also made some change since vllm-tpu is using vllm 0.11 
https://github.com/carlesoctav/lighteval/blob/1b6b5ccff13365ef2d5b28ad6f027cb0ca80c9d9/src/lighteval/models/vllm/vllm_model.py#L464

alright ! And do you need any help on this ? I dont think the bottleneck is caused by lighteval but if you think otherwise we would love to work towards making it better 

nah, all good. im opening this issue probbaly to aks for a comparison of vllm input throughput on GPU.
Repository: huggingface/lighteval
Issue: Deploying evaluation for finetuned model as AWS SM pipeline step
Root cause: Documentation Deficiency

Hello,

I was wondering if there is possibility to incorporate lighteval into SM Pipelines. We are having custom fine-tuning pipeline that uses HuggingFace estimator to fine-tune the model in TrainingStep. Now, we would like to evaluate the fine-tuned model using your library, however I have seen that there is only option to either pass endpoint name with some additional config, or you can only evaluate base models from the HF Hub.

Therefore I have two questions:
1. Is there a way to evaluate fine-tuned model without first deploying it? I assume no, but still worth asking I guess ;)
2. If no, do you maybe know if there is a way to deploy the model inside the pipeline? I have seen recently the blogpost(https://www.philschmid.de/sagemaker-evaluate-llm-lighteval), however I would like to use this method in the pipeline itself. Is there a way to do it in such way? I have only seen its possible through Lambda step, however I would love to double check if there is any recommended way from your side to tackle that issue.

Thanks,
Piotr

Hi!

Thanks for your interest!

It is at the moment possible to evaluate models in 3 ways:
- by deploying an inference endpoint with your model 
- locally using transformers/accelerate, if your model is `AutoModelForCausalLM.from_pretrained` compatible
- locally using nanotron (likely not your use case)

So it is possible to evaluate models without first deploying them as long as they are transformers compatible.

I've never used SM Pipelines so I'm unsure about the rest of your question, feel free to add more information about your use case.

Best,
Clémentine

Closing for inactivity