zhiminy/EvalEng huggingface dataset:
Repository: huggingface/lighteval
Issue: [BUG] sync `LightevalTaskConfig` docstring with types/defaults
Root cause: Documentation Deficiency

## Describe the bug

The docstring in `LightevalTaskConfig` ([link](https://github.com/huggingface/lighteval/blob/v0.9.2/src/lighteval/tasks/lighteval_task.py#L68-L89)) lags the type hints:
- `evaluation_splits` says `list[str]` but it's actually `ListLike` so it can be `tuple`. Also, the default of `validation` isn't documented
    - Imo a default of `validation` should not be there, as most people just have `train`/`test`
- `few_shots_split` says `str`, but it's actually optional with a default of `None`

There's more cases (e.g. `generation_size`) but I didn't state them here.

## To Reproduce

N/a

## Expected behavior

Up-to-date docstrings. For typed code, it's a best practice to not to restate types in the docstring for this very reason.

## Version info

```none
lighteval==0.9.2
```

Hi! Good catch, feel free to open a PR for this :)
Repository: huggingface/lighteval
Issue: [FT] add `py.typed` so `lighteval` can work with type checkers
GitHub labels: documentation

good first issue

help wanted

feature

## Issue encountered

Seen with `lighteval==0.9.2` and `mypy==0.15.0`:

```none
a.py:1:1: error: Skipping analyzing "lighteval.metrics.utils.metric_utils": module is installed, but missing library stubs or py.typed marker  [import-untyped]
    from lighteval.metrics.utils.metric_utils import SampleLevelMetric
    ^
```

## Solution/Feature

Add a `py.typed` so type checkers can inspect this package's code.

## Possible alternatives

N/a
Repository: huggingface/lighteval
Issue: [FT] `lighteval file` eval backend to work with stored JSONL/CSV files
Root cause: Unimplemented Feature Gap

## Issue encountered

I have stored LLM completions that I can export to a JSONL or CSV file, and would like to posthumously evaluate them on a few metrics.

## Solution/Feature

Something like `lighteval file` to evaluate completions stored locally.

## Possible alternatives

N/a

hi! We have something similar already, if you check for example main_vllm, you have a `load_response_from_details_dat_if` option. this uses details already generated by lighteval but it can be a base for your usecase.
Repository: huggingface/lighteval
Issue: [BUG] Make Probability metric not trigger Loglikehood requests twice when used alongside Accuracy Metrics
Root cause: Algorithmic Error

## Describe the bug
When Probability metric is used alongside Accuracy metric, we evaluate 2xN_SAMPLES requests. We don't need to do this as both use the same logprobabilities for evaluation.

## To Reproduce
Use this as custom tasks
```
from lighteval.metrics.metrics import Metrics
from lighteval.tasks.lighteval_task import LightevalTaskConfig
from lighteval.tasks.default_prompts import LETTER_INDICES
from lighteval.metrics.normalizations import LogProbCharNorm, LogProbPMINorm, LogProbTokenNorm
from lighteval.tasks.templates.utils.formulation import CFFormulation, HybridFormulation, MCFFormulation
from lighteval.tasks.templates.multichoice import get_mcq_prompt_function
from lighteval.tasks.templates.hellaswag import get_hellaswag_prompt_function
from lighteval.tasks.templates.qa import get_qa_prompt_function
from lighteval.metrics.dynamic_metrics import loglikelihood_acc_metric, probability_metric
from lighteval.tasks.multilingual.utils.task_utils import get_metrics_for_formulation
from lighteval.tasks.multilingual.tasks import xcsqa_tasks
from lighteval.utils.language import Language
from lighteval.metrics.metrics import Metrics

hellaswag_task = [
    LightevalTaskConfig(
        name=f"hellaswag_cf",
        suite=["lighteval"],
        prompt_function=get_hellaswag_prompt_function(
            language=Language.ENGLISH,
            adapter=lambda line: {
                # We don't use activity_label as they are not available
                "ctx_a": line["ctx_a"],
                "ctx_b": line["ctx_b"],
                "continuations": line["endings"],
                "gold_idx": int(line["label"]),
            },
            formulation=CFFormulation(),
        ),
        hf_repo="Rowan/hellaswag",
        hf_subset="default",
        evaluation_splits=["validation"],
        hf_avail_splits=["train", "validation"],
        metric=get_metrics_for_formulation(
            CFFormulation(),
            [
                loglikelihood_acc_metric(normalization=LogProbTokenNorm()),
                probability_metric(normalization=None),
            ],
        ),
    )
]
```

## Expected behavior
Only N_SAMPLES loglikelihood requests are called.

## Version info
commit: 1607dc1014006e122f962d7d0cf7b6bab2483333
Repository: huggingface/lighteval
Issue: [FT]  Custom details and results saving path
Root cause: Unimplemented Feature Gap

 i have a local ckpt which local path are /base_path/name/step any way to save them in output_folder/results/name/step instead of output_folder/results/name_step ? (for instance for now it's saving `lighteval/outputs-lc/results/_fsx_elie_bakouch_smollm3_training_1004-nn_nanotron_nanotron-to-hf_hf_baseline-NO-swa_355000` )
Repository: huggingface/lighteval
Issue: [BUG] incorrect type hints such as `callable`
Root cause: Validation Gap

## Describe the bug

Reading the code, it's clear this code base can really benefit from type checking, running vanilla `mypy` reveals over 1.2k type errors.

For example, `callable` is not a valid type hint: https://github.com/huggingface/lighteval/blob/v0.9.2/src/lighteval/metrics/utils/metric_utils.py#L62-L63

## To Reproduce

```bash
uv sync --extra dev
uv pip install mypy==0.15.0
mypy src
```

## Expected behavior

Type hints to be correct.

## Version info

`lighteval==0.9.2`
Repository: huggingface/lighteval
Issue: [FT] Python API docs using small model that can run on Mac
Root cause: Documentation Deficiency

## Issue encountered

https://github.com/huggingface/lighteval/blob/v0.9.2/docs/source/using-the-python-api.mdx uses `HuggingFaceH4/zephyr-7b-beta` which is too big for a developer machine.

## Solution/Feature

Using a model that is much smaller in docs.

## Possible alternatives

N/a

Hi! Feel free to use a model from the SmolLM family for your use case! 
https://huggingface.co/collections/HuggingFaceTB/smollm2-6723884218bcda64b34d7db9

@NathanHB we could add a comment in the doc to suggest a smollm model for devs with less than 16G of RAM, wdyt?

To share, `HuggingFaceTB/SmolLM2-135M-Instruct` with the task `"lighteval|triviaqa|0|0"` and `PipelineParameters(launcher_type=ParallelismManager.ACCELERATE, max_samples=5)` still crashed my Mac with M3 Pro with 36 GB RAM.

Hmmmmmmm this is not normal at all - can you share precisely what you launched?

and the logs?

(For ref to know how much mem you should need for which model size: https://github.com/huggingface/evaluation-guidebook/blob/main/contents/troubleshooting/troubleshooting-inference.md#estimating-memory-requirements)

Yeah it's with this code snippet (and I updated the task):

```python
from datetime import timedelta

from accelerate import Accelerator, InitProcessGroupKwargs
from lighteval.logging.evaluation_tracker import EvaluationTracker
from lighteval.models.transformers.transformers_model import TransformersModelConfig
from lighteval.pipeline import ParallelismManager, Pipeline, PipelineParameters


if __name__ == "__main__":
    accelerator = Accelerator(
        kwargs_handlers=[InitProcessGroupKwargs(timeout=timedelta(seconds=3000))]
    )

    pipeline = Pipeline(
        tasks="original|mmlu:abstract_algebra|5|1",
        pipeline_parameters=PipelineParameters(
            launcher_type=ParallelismManager.ACCELERATE, max_samples=2
        ),
        evaluation_tracker=EvaluationTracker(output_dir="./results"),
        model_config=TransformersModelConfig(
            model_name="HuggingFaceTB/SmolLM2-135M-Instruct", use_chat_template=True
        ),
    )

    pipeline.evaluate()
    pipeline.save_and_push_results()
    pipeline.show_results()
```

My Mac just crashes during `pipeline.evaluate` with Python 3.12 and:

```none
accelerate==1.4.0
lighteval==0.9.2
transformers==4.51.3
```

Could you share the full logs you get?

Yeah here they are, then my Mac freezes and becomes non-responsive:

```none
[2025-05-21 10:20:19,128] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to mps (auto detect)
W0521 10:20:19.278000 10400 torch/distributed/elastic/multiprocessing/redirects.py:29] NOTE: Redirects are currently not supported in Windows or MacOs.
INFO 05-21 10:20:20 [__init__.py:239] Automatically detected platform cpu.
--max_samples WAS SET. THESE NUMBERS ARE ONLY PARTIAL AND SHOULD NOT BE USED FOR COMPARISON UNLESS YOU KNOW WHAT YOU ARE DOING.
If you want to use extended_tasks, make sure you installed their dependencies using `pip install -e .[extended_tasks]`.

0it [00:00, ?it/s]
```
Repository: huggingface/lighteval
Issue: [FT] Support computing/displaying stddev in the final table from the variation in sample scores (when using sampling metrics) instead of a bootstrap
Root cause: Unimplemented Feature Gap

## Issue encountered

I was hoping to run an evaluation with N=5 replicates, as well as an get easy way to get error bars.

## Solution/Feature

Some parameter in `PipelineParameters`.

## Possible alternatives

It's possible that `PipelineParameters.max_samples` does this, but it's unclear to me.

When you say replicates, do you mean sampling different parameters? 
If yes, you need to define your own custom task (and possibly metric if you don't want to use a pass at or maj at metric).

However, you can also get error bars using the bootstrapped stderr

`max_samples` allows you to run the evaluation on a limited amount of different samples (for example 5) to test that the pipeline is working well for you before starting a full run. 

> When you say replicates, do you mean sampling different parameters?

I mean something like prompting a QA task to the language model N times, evaluating the responses for accuracy, and delivering mean/std dev on the responses.

> `max_samples` allows you to run the evaluation on a limited amount of different samples (for example 5) to test that the pipeline is working well for you before starting a full run.

Ah then this is not a num replicates concept, thanks for clarifying.

---

To give a concrete example, perhaps something like prompting an LLM 10 times in parallel: "What is 2 + 2?"

And calculating mean and standard deviation on the answer containing "4".

Ok, so you indeed want a sampling metric (= prompting the model with a number of samples to generate, then computing a score on the different versions of each sample, then aggregating) - it would not exactly be trivial to add the stddev as such in the bootstrapping section (where we compute the value for the stderr) but you could probably extract the score from the details generated.
Repository: huggingface/lighteval
Issue: [FT]  Add tests for nanotron
Root cause: Unimplemented Feature Gap

## Issue encountered
add end to end tests for nanotron model


Hi @NathanHB , I would like to take this up.


hey @NathanHB, 

was wondering if i can get some input on how to work on this. would love some pointers to get started on :) 

hey ! This would not be the best issue to get started as we are planning on moving to `inspect-ai` and might therefore deprecate this :)
Repository: huggingface/lighteval
Issue: [BUG] Is AIME24 broken?
Root cause: Algorithmic Error

Hi folks,

I've been trying to reproduce DeepSeek-R1-Distill-Llama-8B evals, but seems like there is something wrong going on with AIME24 (MATH-500 and GPQA:Diamond are looking fine):

```bash
|          Task          |Version|        Metric        |Value |   |Stderr|
|------------------------|------:|----------------------|-----:|---|-----:|
|all                     |       |math_pass@1:1_samples |0.6763|±  |0.0534|
|                        |       |math_pass@1:4_samples |0.6782|±  |0.0388|
|                        |       |math_pass@1:8_samples |0.4375|±  |0.0636|
|                        |       |math_pass@1:16_samples|0.2188|±  |0.0318|
|                        |       |math_pass@1:32_samples|0.1094|±  |0.0159|
|                        |       |math_pass@1:64_samples|0.0547|±  |0.0080|
|                        |       |gpqa_pass@1:1_samples |0.5051|±  |0.0356|
|                        |       |gpqa_pass@1:4_samples |0.4710|±  |0.0273|
|                        |       |gpqa_pass@1:8_samples |0.4836|±  |0.0264|
|lighteval:aime24:0      |      2|math_pass@1:1_samples |0.4667|±  |0.0926|
|                        |       |math_pass@1:4_samples |0.4583|±  |0.0668|
|                        |       |math_pass@1:8_samples |0.4375|±  |0.0636|
|                        |       |math_pass@1:16_samples|0.2188|±  |0.0318|
|                        |       |math_pass@1:32_samples|0.1094|±  |0.0159|
|                        |       |math_pass@1:64_samples|0.0547|±  |0.0080|  <-- 0.0547 vs 0.439 (Open-R1 README)
|lighteval:gpqa:diamond:0|      1|gpqa_pass@1:1_samples |0.5051|±  |0.0356|
|                        |       |gpqa_pass@1:4_samples |0.4710|±  |0.0273|
|                        |       |gpqa_pass@1:8_samples |0.4836|±  |0.0264|
|lighteval:math_500:0    |      2|math_pass@1:1_samples |0.8860|±  |0.0142|
|                        |       |math_pass@1:4_samples |0.8980|±  |0.0108|            
```

i.e. I am seeing `math_pass@1:64_samples` 5.47 versus the expected 43.9 (expected based on https://github.com/huggingface/open-r1?tab=readme-ov-file#reproducing-deepseeks-evaluation-results).

(cross-posting this from Open-R1 https://github.com/huggingface/open-r1/issues/655 since I am not sure who the owner is for reasoning evals)

cc @lewtun just in case

I meet the same problem.
When I use lighteval==0.8.1, the test accuracy of AIME2024 on DeepSeek-R1-Distill-Qwen-1.5b is 26%. However, when I upgrade the lighteval to 0.10.0, the AIME2024 on DeepSeek-R1-Distill-Qwen-1.5 b is approximately 36%.  I haven't changed anything else.

What's more, the acc of AMC2023 reported in official docs is about 62.9%. But when using lighteval==0.10.0, the result is as follows:
|     Task     |Version|     Metric     |Value|   |Stderr|
|--------------|------:|----------------|----:|---|-----:|
|all           |       |extractive_match| 0.75|±  |0.0693|
|custom:amc23:0|      1|extractive_match| 0.75|±  |0.0693|

I feel confused about why this happened.
Repository: huggingface/lighteval
Issue: [BUG] Python API documentation
Root cause: Documentation Deficiency

## Describe the bug
The Python API documentation example contains invalid parameters that cause runtime errors. The PipelineParameters constructor doesn't accept env_config or override_batch_size parameters.

https://huggingface.co/docs/lighteval/main/en/using-the-python-api

## To Reproduce
1. Copy the example code from the Python API documentation
2. Run the script
3. Get AttributeError for non-existent parameters

```
pipeline_params = PipelineParameters(
    launcher_type=ParallelismManager.ACCELERATE,
    env_config=EnvConfig(cache_dir="tmp/"),  # ❌ This parameter doesn't exist
    custom_task_directory=None,
    override_batch_size=1,  # ❌ This parameter doesn't exist
    max_samples=10
)
```

## Expected behavior
The documentation example should use valid PipelineParameters constructor arguments that actually work.

## Version info
- cloned main branch repo (codebase up to date)
- Lighteval Version: 0.9.1.dev0

## Suggested Fix
Update the documentation example to use correct parameter names:
```
pipeline_params = PipelineParameters(
    launcher_type=ParallelismManager.ACCELERATE,
    custom_tasks_directory=None, 
    max_samples=10
)
```
Repository: huggingface/lighteval
Issue: [BUG] OpenAIClient fails when using newer GPT models
Root cause: Validation Gap

## Describe the bug
The OpenAI model client has two issues:

1. Tokenizer initialization failure: The client fails to initialize when using newer OpenAI models not supported by tiktoken (e.g., o1-preview, o1-mini, gpt-4.1 variants). The current fallback tries using AutoTokenizer, but these models aren't in the huggingface models list (OpenAI models).

<img width="905" alt="Image" src="https://github.com/user-attachments/assets/16f74f9f-8cfd-4f93-bdba-a37a046c4530" />

 
2. Unused parameter: The OpenAIClient.__init__() method signature requires an env_config parameter, but this parameter is not passed when you use the Pipeline Class to create a run.

<img width="806" alt="Image" src="https://github.com/user-attachments/assets/1ba6af76-3674-42cf-ad14-4285cfe98528" />

## To Reproduce
Basically run the python api code in the lighteval documentation here. https://huggingface.co/docs/lighteval/main/en/using-the-python-api and use the OpenAIModelConfig.

```
import lighteval
from lighteval.logging.evaluation_tracker import EvaluationTracker
from lighteval.models.endpoints.claude_model import ClaudeModelConfig
from lighteval.models.endpoints.openai_model import OpenAIModelConfig
from lighteval.pipeline import ParallelismManager, Pipeline, PipelineParameters
from lighteval.utils.imports import is_accelerate_available

if is_accelerate_available():
    from datetime import timedelta
    from accelerate import Accelerator, InitProcessGroupKwargs
    accelerator = Accelerator(kwargs_handlers=[InitProcessGroupKwargs(timeout=timedelta(seconds=3000))])
else:
    accelerator = None

def main():
    evaluation_tracker = EvaluationTracker(
        output_dir="./results",
        save_details=True,
        push_to_hub=True,
        hub_results_org="xxx",
    )

    pipeline_params = PipelineParameters(
        launcher_type=ParallelismManager.ACCELERATE,
        custom_tasks_directory=None,
        max_samples=1,
   )
    model_config = OpenAIModelConfig(
            model="gpt-4.1",
            api_key="your_api",
    )

    task = "lighteval|gpqa|0|0"

    pipeline = Pipeline(
        tasks=task,
        pipeline_parameters=pipeline_params,
        evaluation_tracker=evaluation_tracker,
        model_config=model_config,   
    )

    pipeline.evaluate()
    pipeline.save_and_push_results()
    pipeline.show_results()

if __name__ == "__main__":
    main()
```

## Expected behavior
1. Robust tokenizer fallback: The client should successfully initialize with any OpenAI model using a proper fallback chain:
- Try tiktoken with the actual model
- Fall back to tiktoken with "gpt-4o" encoding (because gpt4.1, o3, o1 all use the gpt-4o tokenizer)
2. OpenAIClient does not need to take in a env_config (so we remove it).

## Version info
- cloned main branch repo (codebase up to date)
- Lighteval Version: 0.9.1.dev0

Hi ! thanks for the fix :) OpenAi client is deprecated and we advise to use litellm 

yes, that is actually better. thanks!
Repository: huggingface/lighteval
Issue: [FT] Store system prompt in results
Root cause: Unimplemented Feature Gap

## Issue encountered

Currently, when one provides a system prompt at runtime, the system prompt is only visible in the `full_prompt` column of the details. It would be great if this could be propagated to the results file as a `config_general.system_prompt` field.

## Solution/Feature

Propagate the system prompt at runtime to a `config_general.system_prompt` field in the results file, and leave as an empty string by default

## Possible alternatives

Currently I workaround this issue as follows:

```
jq --arg prompt "$SYSTEM_PROMPT" '.config_general.system_prompt = $prompt' "$filepath" > "${filepath}.tmp" && mv "${filepath}.tmp" "$filepath"
```

The refacto PR #709 Just made this the case by moving the --system-prompt to the model config, whichi is then logged in the results file
Repository: huggingface/lighteval
Issue: [BUG] 1
GitHub labels: bug

## Describe the bug
A clear and concise description of what the bug is.

## To Reproduce
Please provide all the steps needed to reproduce the behavior, or provide a minimal working example if needed. We will ignore issues missing this section.

## Expected behavior
A clear and concise description of what you expected to happen.

## Version info
Please provide your operating system, lighteval version or commit if you installed from main, and pip/conda environment if your problem concerns dependencies.
Repository: huggingface/lighteval
Issue: Add AGIEval
Root cause: Unimplemented Feature Gap

AGIEval is a popular set of benchmarks that was popularised by Teknium/Nous in models like OpenHermes. It would be nice to include in `lighteval` so we can compare internally how our models stack up on this benchmark :)

* Ref paper: https://arxiv.org/abs/2304.06364
* Ref code: https://github.com/dmahan93/lm-evaluation-harness/tree/add-agieval

Ref command from [AutoEval](https://github.com/mlabonne/llm-autoeval/blob/11a1b95388f3696b2783cc5151cb3ab1b2d784c8/runpod.sh#L39-L46):

```shell
    benchmark="agieval"
    python main.py \
        --model hf-causal \
        --model_args pretrained=$MODEL_ID,trust_remote_code=$TRUST_REMOTE_CODE \
        --tasks agieval_aqua_rat,agieval_logiqa_en,agieval_lsat_ar,agieval_lsat_lr,agieval_lsat_rc,agieval_sat_en,agieval_sat_en_without_passage,agieval_sat_math \
        --device cuda:$cuda_devices \
        --batch_size auto \
        --output_path ./${benchmark}.json
```

Would you need AGIEval or BBH first?

> Would you need AGIEval or BBH first?

Maybe we can do BBH first since you already have made a big dent in it in #7 ?
Repository: huggingface/lighteval
Issue: [BUG] Faithfulness metric fails because the SummaCZS model is instantiated but never assigned
Root cause: Validation Gap

## Describe the bug
A clear and concise description of what the bug is.

Faithfulness metric fails because the SummaCZS model is instantiated but never assigned

https://github.com/huggingface/lighteval/blob/53df85985d3239b59c02922e83c0c099fd0b0ddf/src/lighteval/metrics/metrics_sample.py#L698-L699


## To Reproduce
Please provide all the steps needed to reproduce the behavior, or provide a minimal working example if needed. We will ignore issues missing this section.

```
from lighteval.tasks.requests import Doc
from lighteval.metrics.metrics_sample import Faithfulness

document = "Scientists are studying Mars to learn about the Red Planet and find landing sites for future  missions. One possible site, known as Arcadia Planitia, is covered in strange sinuous features. The shapes could be signs that the area is actually made of glaciers, which are large masses of slow-moving ice. Arcadia Planitia is in Mars’ northern lowlands."

summary = "There are strange shape patterns on Arcadia Planitia. The shapes could indicate the area might be made of glaciers. This makes Arcadia Planitia ideal for future missions."

doc = Doc(query="", choices=[], gold_index=0, specific={"text": document})
Faithfulness().compute(formatted_doc=doc, predictions=[summary])

File lighteval/metrics/metrics_sample.py:706, in Faithfulness.compute(self, predictions, formatted_doc, **kwargs)
    704 if self.normalize_pred:
    705     prediction = self.normalize_pred(prediction)
--> 706 return self.summac.score_one(inp, prediction)["score"]

AttributeError: 'NoneType' object has no attribute 'score_one'
```

## Expected behavior
A clear and concise description of what you expected to happen.

`Faithfulness.compute(predictions, formatted_doc) ` to return a `dict[str, float]` of the faithfulness scores

## Version info
Please provide your operating system, lighteval version or commit if you installed from main, and pip/conda environment if your problem concerns dependencies.

- Operating System: macOS Monterey Version 12.4
- Python Version: Python 3.11.1
- lighteval version: lighteval==0.10.0


I opened a PR to fix this issue: https://github.com/huggingface/lighteval/pull/795