zhiminy/EvalEng huggingface dataset:
Repository: huggingface/lighteval
Issue: [BUG] Issue with LightevalTaskConfig.stop_sequence Attribute When Unset
Root cause: Validation Gap

## Describe the bug
When running `aimo_evals.py` from community tasks, an error occurs indicating that the Tokenizer does not accept `None` as input. This issue stems from how `stop_sequence` is handled in `LightevalTaskConfig`. Since most tasks in `lighteval` explicitly define `stop_sequence`, this problem may have gone unnoticed.

https://github.com/huggingface/lighteval/blob/0977fde60ba3564cf34c53606a53962d81eef7b0/src/lighteval/tasks/lighteval_task.py#L133

## Proposed Solution
Set the default value for stop_sequence to an empty list (`[]`) instead of `None`. Testing shows that this change resolves the issue.

``` python
self.stop_sequence = tuple(self.stop_sequence) if self.stop_sequence is not None else tuple()
```

## To Reproduce
``` sh
lighteval accelerate \
    "pretrained=gpt2" \
    "community|aimo_progress_prize_1|0|0" \
    --custom-tasks "./community_tasks/aimo_evals.py"
```

## Expected behavior
The test should run successfully without any errors.

## Version info
This issue occurs in a version installed directly from the `main` branch using the following command:
``` sh
pip install -e ".[dev]"
```
Repository: huggingface/lighteval
Issue: [FT] Rerun evaluations with new metrics based on completions saved in details file
Root cause: Unimplemented Feature Gap

## Issue encountered
Rerunning an evaluation with a new metric requires rerunning the entire inference currently, which can be very costly.

## Solution/Feature
It would be great, if we could specify a details file containing the predictions and use that to compute more metrics on.


@clefourrier @NathanHB I am happy to implement this. Do you have suggestions for how to best solve this?

It would be great ! I think the best way would be to recreate the `sample_id_to_response` from the details file and run the metric on these.

from the pipeline.py file:

```
sample_id_to_responses = self._run_model()
self._compute_metrics(sample_id_to_responses)
```

you would need to inspect what is in `sample_id_to_response` and try to make it from the details file.

Great, will try that, thanks Nathan!
Repository: huggingface/lighteval
Issue: [BUG] Issue with the scorer Attribute Initialization in ROUGE
Root cause: Validation Gap

## Describe the bug
The `ROUGE` class does not initialize the `scorer` attribute in its constructor. However, the compute method attempts to use it directly, which causes an error under all circumstances.

https://github.com/huggingface/lighteval/blob/a1c610daab3f796a273ab56f78b5f5fe9614b8aa/src/lighteval/metrics/metrics_sample.py#L442-L501

## Proposed Solution
Add an initialization for `self.scorer` as `None` in the `__init__` method.

## To Reproduce
Run any test involving ROUGE, such as the following:
``` sh
lighteval accelerate \
    "pretrained=gpt2" \
    "helm|summarization:xsum-sampled|0|0"
```

## Expected behavior
The test should execute without issues.

## Version info
The issue was encountered with the version installed directly from the `main` branch.
Repository: huggingface/lighteval
Issue: [FT] Enhancing CorpusLevelTranslationMetric with Asian Language Support
Root cause: Unimplemented Feature Gap

## Issue encountered
While working on several Japanese benchmark tasks, I observed that standard BLEU, CHRF, and TER metrics are suboptimal for Asian languages.
To address this, I propose adding a parameter to CorpusLevelTranslationMetric that allows integration with tokenizers tailored for Asian languages.

## Solution/Feature
`SacreBLEU` already includes tokenizers designed for Asian languages, which lack space-separated words. By modifying the implementation slightly, we can extend CorpusLevelTranslationMetric to better handle these languages.

https://github.com/mjpost/sacrebleu/blob/0f351010b8b641aaa59fe75b98d7cc522bf221eb/sacrebleu/metrics/bleu.py#L110-L208

## Possible alternatives
A clear and concise description of any alternative solutions or features you've considered.
Repository: huggingface/lighteval
Issue: [BUG] By default pip install lighteval is installing the cpu only torch version , its killing dependencies.
Root cause: Architectural Constraint

## Describe the bug
"pip install lighteval"

installs just cpu version of torch.

## To Reproduce
pip install lighteval

pip list | grep torch

## Expected behavior
pip install lighteval should check gpu config and install the torch accordingly.

## Version info
lighteval =v0.7, issue happens on both conda/pip irrespective of version
Repository: huggingface/lighteval
Issue: [BUG] Dataset 'lighteval/MATH-Hard' doesn't exist on the Hub
Root cause: External Dependency Breakage

## Describe the bug
When I'm using the openllm leaderboard evaluation from llm_eval_harness, this pops up as an error: **Dataset 'lighteval/MATH-Hard' doesn't exist on the Hub.**

 The URL [https://huggingface.co/datasets/lighteval/MATH-Hard](https://huggingface.co/datasets/lighteval/MATH-Hard) is empty now. 


## To Reproduce
Go to [https://huggingface.co/datasets/lighteval/MATH-Hard](https://huggingface.co/datasets/lighteval/MATH-Hard)

## Expected behavior
At least show something


Same problem here.

I meet the same error too.

Did anyone find a temporary solution for this with an exactly similar dataset? I reckon this dataset would probably be uploaded to HF multiple times.

Same problem with **'lighteval/MATH'** : "DatasetNotFoundError: Dataset 'lighteval/MATH' doesn't exist on the Hub or cannot be accessed."

> Did anyone find a temporary solution for this with an exactly similar dataset? I reckon this dataset would probably be uploaded to HF multiple times.

Can use this checkpoint : https://huggingface.co/datasets/hendrycks/competition_math

We have created a drop-in replacement for the original dataset here: [DigitalLearningGmbH/MATH-lighteval](https://huggingface.co/datasets/DigitalLearningGmbH/MATH-lighteval).

It contains appropriate builder configs s.t. it can be directly used with https://github.com/huggingface/lighteval/blob/main/examples/nanotron/custom_evaluation_tasks.py#L252 (you just need to adjust `hf_repo="lighteval/MATH"` to `hf_repo="DigitalLearningGmbH/MATH-lighteval"`. Hope this helps someone.


Hi! The original dataset has been taken down by a DMCA notice (more info [here](https://huggingface.co/datasets/hendrycks/competition_math/discussions/5)) so we had to remove access to it from the hub - we're very sorry and hope this will be solved soon, it's super inconvenient for all of us. 

Closing the issue in the meantime as there is nothing we can do on our side.
Repository: huggingface/lighteval
Issue: [FT] Enable lazy model initialization
Root cause: Unimplemented Feature Gap

## Issue encountered
Evaluating large models (> 30B parameters) is hard, especially with limited hardware. Especially when there are many metrics to be evaluated, it can significantly increase the time needed for the strong machine to be running. For example when I want to evaluate a 70B model on a large dataset and then compute many LLM-judge metrics, it can occupy a 4xA100 machine for days, incurring significant cost. The GPUs are only actually active during the first few hours for inference. Afterwards they are just sitting idle. 

## Solution/Feature
Therefore, ideally, we would like to just run inference with one metric and save the results to the details files. Then in a second step we load the responses from the details files and just run the metrics. For that we can use a significantly smaller machine. Loading from the details files is being added in PR #488. However, to evaluate the metrics, we still need to load the entire model into memory, defeating the purpose. Loading the model only right before running it would alleviate this issue. 

## Possible alternatives
Alternatively, we could mock the model.
Repository: huggingface/lighteval
Issue: 3 [short] comments (Batch size, launching, installation)
Root cause: Documentation Deficiency

Hello! I've been experimenting with `lighteval` and wanted to share 3 things:
1. I haven't found any information anywhere about the batch size. In all the examples, `--override_batch_size 1` is used, I think it would be great to add a description of its function (In [`run_evals_accelerate.py`](https://github.com/huggingface/lighteval/blob/92e9b5055c8adddca2f070c6acbe3c0c15ecaf66/run_evals_accelerate.py#L76) for example) and mention that otherwise the batch size is calculated automatically.
2. When trying to execute `python -m accelerate launch --multi_gpu ...` as indicated in the README, I get the following error: `/usr/bin/python: No module named accelerate.__main__; 'accelerate' is a package and cannot be directly executed`. I have no problem running `accelerate launch --multi_gpu ...`.
3. In my case using zsh, when executing `pip install -e .[accelerate,quantization,adapters]` I get the following error: `no matches found: .[accelerate,quantization,adapters]`. It is fixed by introducing ' ' in the package+dependencies (`pip install -e '.[accelerate,quantization,adapters]'`).

Toni

Interesting, thanks a lot for this feedback (and for testing our lib :partying_face: ) !
I'll add fixes for all of these in the doc

@lewtun ended up doing a super cool update of our README following his experiments, and I feel like he answered all your issues - wydt?

Now everything is clearer! Thanks!

Toni
Repository: huggingface/lighteval
Issue: Add `ligheval --list-tasks`
Root cause: Unimplemented Feature Gap

This command should list all the available tasks (in extended tasks and the task table)

Hello,

I would like to work on this. Do you need special formatting for the list? 

Hi, thanks for your interest!
I think providing the tasks grouped by suite would be nice, for example
```
- lighteval: all the tasks available, sorted alphabetically
- community: all the tasks available, sorted alphabetically
...
```

We have a lot of tasks so it would be better to have them in a list in that format, this makes reading easier as you can `ligheval --list-tasks | grep mmlu` for example.

```
- lighteval:
    - task 1
    - task 2
- community:
    - task 1 
    - task 2
```

I understand. I'm working on it now.
Repository: huggingface/lighteval
Issue: <think> tags for thinking models
Root cause: Unimplemented Feature Gap

Thinking models like DeepSeek-R1 emit thinking tags in the output. Is there a way to filter these out easily? Currently they make it directly into the output and so mess up metrics.

Nope, not at the moment! We used to have regex parsers but they were underused so we removed them.

I had the same question and got something working. For now it's more of a hack, but hopefully this is a starting point to get it working generally. Here is my [branch](https://github.com/mapmeld/lighteval-tasks/tree/attempt2) and [demo notebook](https://colab.research.google.com/drive/1e4rZoEHWe3kVPDsdSGcUMhcVAAXLVOkD?usp=sharing). 

Notes:
- could add a flag (similar to `use_chat_template`). I saw some code uses [add_reasoning_prompt](https://huggingface.co/search/full-text?q=add_reasoning_prompt)
- `<think>` is already in tokenizer_config.json's `chat_template` field, so maybe instead of hardcoding, possible to detect common tokens there, and then insert it in the chat template again?
- I needed to change my answer options from `["A", "B"...]` to `["The answer is A", ...]`
- 2048 tokens seems like the right size
- didn't test few-shot
- this isn't maximally efficient because it works on one doc at a time, but I don't know how well large batches would fit

Here's the key section in prompt_manager:
```python
        elif use_chat_template:
            chat_preview = self.model.tokenizer.apply_chat_template(
                output, tokenize=False, add_generation_prompt=True
            )
            tokenized = self.model.tokenizer(chat_preview, return_tensors="pt").to(self.model.device)
            prepared_batch = Batch(
                input_ids=tokenized["input_ids"],
                input_mask=tokenized["attention_mask"],
                input_lengths=[len(tokenized["input_ids"][0])],
                truncated=[False],
                padded=[False],
            )
            response = self.model._generate(
                batch=prepared_batch,
                max_new_tokens=2048,
                stop_tokens=["</think>"],
            )
            all_start = chat_preview + response[0].result[0] + "</think>"
            return all_start, num_effective_fewshots
```

Hi! This is now fixed on main after [this PR](https://github.com/huggingface/lighteval/pull/882). The details also contain both the original prediction and the post processed one.
Repository: huggingface/lighteval
Issue: Token indices sequence length is longer than the specified maximum sequence length for this model
Root cause: Documentation Deficiency

Hi, guys

I am evaluating `meta-llama/Llama-3.2-3B` on Thai language finetasks.
Here I got this warning during the run.

```
Token indices sequence length is longer than this model's specified maximum sequence length (1947 > 1024). Running this sequence through the model will result in indexing errors
```

This is kind of surprising because If I am not mistaken, llama 3.2 3B's input content length is 131,072 not 1024.
So, I am not sure why I am getting this warning. 

Any ideas?

Here is my command 

```
lighteval accelerate \
        "pretrained=${model},dtype=bfloat16,max_length=1024" \
        "examples/tasks/fine_tasks/mcf/th.txt" \
        --custom-tasks "src/lighteval/tasks/multilingual/tasks.py"
        --no-use-chat-template
```


It's because of the max_length you're specifying in the above command, the `,max_length=1024` above :)

Thanks for the reply

Hmm, I understand that this warning is about the input sequence is longer than the model's input context window. But the max_length, according to the document website, 

> max_length (Optional[int]): The maximum length of the generated output.

is the max number of tokens to generate which I think the meaning is not the same thing.

Am I misunderstand something?

Hi!
It's not the maximum number of tokens to generate, it's the maximum possible length you allow the model to have in prompt + generation (you could see it as a set context length).
I think the parameter you're looking for is just above, [max_gen_toks](https://github.com/huggingface/lighteval/blob/4f381b352c0e467b5870a97d41cb66b487a2c503/src/lighteval/models/transformers/transformers_model.py#L110)

Hmm

Then maybe the documents is not up to date

![image](https://github.com/user-attachments/assets/70c39366-392a-4bc3-a857-6adc9bf59e62)

Can you confirm this?

I would say unclear rather than not up to date, as this as always been the expected mechanism ^^
Would `max_length: maximum allowed context size for the model (including prompt and generation)` be clearer?

I think I get your point now. Thank you so much!!!

Hi,

After some experiments
I tried setting `max_legnth` to 2048, 10000, or even omitting this arg or using `max_new_toks=1024` instead but I still got the same warning.

I don't know if this is the correct behavior.

Same warning with 1024 each time?

We set the max length for the model [here](https://github.com/huggingface/lighteval/blob/4f381b352c0e467b5870a97d41cb66b487a2c503/src/lighteval/models/transformers/transformers_model.py#L543) - in short, if the max_length is provided that's what we use, if not we look at different model and tokenizer parameters to try to find it (it's not homogeneous, where this parameter is changed a lot through the history of transformers), and if we can't find it we use 2048 

> Same warning with 1024 each time?

Not the same every time. The part in parenthesis `(number > max_length)` changes, especially the `number`. 
I think I understand this now. I should omit this arg.
Repository: huggingface/lighteval
Issue: lighteval with llama3.2  [RuntimeError: No executable batch size found, reached zero.]
Root cause: Resource Mishandling


Hi

I tried to use lighteval with `meta-llama/Llama-3.2-1B`.
I want to run the evaluation for Thai finetasks. But I got this error instead `RuntimeError: No executable batch size found, reached zero.`
Any ideas? I have a full log with me if you want.
Here is my command 

```
echo "Running lighteval for model: $model"
    lighteval accelerate \
    "pretrained=${model},dtype=bfloat16" \
    "examples/tasks/fine_tasks/mcf/th.txt" \
    --custom-tasks "src/lighteval/tasks/multilingual/tasks.py" \
    --no-use-chat-template
```

I  am using `lighteval 0.6.0.dev0 ` and `torch 2.2.2+cu121`
I didn't use pip install install lighteval but clone the repo instead and `pip install -e . [dev]`


Hi!
Yes, the full log would be useful :)

[log_llama_3_2_03.txt](https://github.com/user-attachments/files/18584813/log_llama_3_2_03.txt)

Please scroll down to the bottom of this file

Hi, any update, sir?

@clefourrier 

I also encountered this issue while benchmarking the DeepSeek-Distilled model on the Math500 dataset. I build lighteval from source. And I'm running a 7B model on an RTX 4090 (24GB). @clefourrier @Nevermetyou65 Do you have any ideas on how to solve this? I’d really appreciate any helpful insights. Thanks!

@Louym 
I am still waiting for the ans. But my work around is to run lighteval with llama3.2 separately for each dataset.

I want to do evaluation on Thai funetasks which is composed of multiple dataset of Thai languages. When I used the command in my post I got the error. So, I just changed the data arg and run lighteval for each dataset in Thai fine tasks 

Thank you for your response! I ran a single dataset using:
lighteval accelerate "pretrained=$MODEL_NAME" "$task"
@Nevermetyou65 May I ask what device you are using? Could it be an OOM issue? Also, @clefourrier, is there any argument to set the batch size? And is there any built setting to reproduce the results from the DeepSeek paper?


@Louym I used GPU A100 with 40GB RAM.

For the batch size adjustment, i think you can check the name of arg using `lighteval --help` or `lighteval accelerate --help`. Try one of these. I can not remember exact command.

@Louym 

here is my command

```
lighteval accelerate \
"pretrained=meta-llama/Llama-3.2-3B,dtype=bfloat16,model_parallel=True" \
"lighteval|belebele_tha_Thai_mcf|5|1" \
--custom-tasks "src/lighteval/tasks/multilingual/tasks.py" \
--dataset-loading-processes 8 \
--cache-dir "./le_cache" \
--no-use-chat-template \
--override-batch-size 8
```

Thank you so much and I will have a try.

It turns out that switching to a GPU with larger memory resolved the issue. Thank you so much for your help! @Nevermetyou65
Repository: huggingface/lighteval
Issue: updated pypi package with torch>=2.0,<3.0
Root cause: Configuration Error

I noticed that the pyproject.toml in pypi for 0.7.0 is set to 
```
 "torch>=2.0,<2.5"
```
The github version fixes this issue.
https://github.com/huggingface/lighteval/blob/515bd01db0d8cf2c0f7362d5413911c3eee8782a/pyproject.toml#L60

Can you update the pypi package? Causes a number of issues with vllm dependencies especially since vllm (currently) requires torch 2.5.1

Thanks!
Repository: huggingface/lighteval
Issue: [BUG] The default validation model cannot be loaded
Root cause: Configuration Error

## Describe the bug

```shell
lighteval accelerate "pretrained=$MODEL,max_length=2048,trust_remote_code=True"  \
    "lighteval|triviaqa|5|0"  \
    --override-batch-size 16 \
    --output-dir $OUTPUT_DIR
```

got

```
[2025-01-31 15:01:07,551] [    INFO]: PyTorch version 2.4.1+cu124 available. (config.py:54)
...
│ ╭───────────────────────── locals ──────────────────────────╮    │
│ │         cache_dir = None                                  │    │
│ │       cached_path = <object object at 0x000001D221EA32C0> │    │
│ │ deprecated_kwargs = {}                                    │    │
│ │          filename = 'model.safetensors'                   │    │
│ │ has_file_in_cache = False                                 │    │
│ │  local_files_only = False                                 │    │
│ │      path_or_repo = 'Elron/bleurt-tiny-512'               │    │
│ │           proxies = None                                  │    │
│ │         repo_type = None                                  │    │
│ │          response = <Response [401]>                      │    │
│ │          revision = 'main'                                │    │
│ │             token = None                                  │    │
│ │    use_auth_token = None                                  │    │
│ ╰───────────────────────────────────────────────────────────╯    │
╰──────────────────────────────────────────────────────────────────╯
OSError: Elron/bleurt-tiny-512 is not a local folder or a valid     
repository name on 'https://hf.co'.
```

## To Reproduce

```shell
export HF_ENDPOINT=https://hf-mirror.com
lighteval accelerate "pretrained=JingzeShi/Doge-20M,max_length=2048,trust_remote_code=True"  \
    "lighteval|triviaqa|5|0"  \
    --override-batch-size 16 \
    --output-dir lighteval_results
```


## Expected behavior
Fix this error...

## Version info

OS: Ubuntu 24.04.1 LTS (x86_64)
GCC version: (Ubuntu 13.2.0-23ubuntu4) 13.2.0
Clang version: Could not collect
CMake version: version 3.31.1
Libc version: glibc-2.39
Is CUDA available: True
CUDA runtime version: 12.6.85
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 4090


I've try to submit a PR to fix this bug.

This problem also occurs in my code. How can I solve it?

@BobTsang1995 I solved this problem by logging back into `huggingface-cli`.

Hi @BobTsang1995, Did you solve this problem? If not I would like to reopen this issue as it does not seem to use the previously cached data correctly.

@BobTsang1995 How do you solve this problem? my environment can not get the huggingface directly.
Repository: huggingface/lighteval
Issue: Add CLI to evaluate models, list supported tasks etc
Root cause: Unimplemented Feature Gap

If would be nice if one could launch evaluation jobs directly from the command line with something like:

```bash
lighteval --tasks="lighteval|hellaswag|5|1" --output_dir "/scratch/evals" --model_args "pretrained=gpt2
```

By default we could look for the default `accelerate` config, but allow users to override this if needed with

```bash
lighteval --accelerate_config=path/to/accelerate/config --tasks="lighteval|hellaswag|5|1" --output_dir "/scratch/evals" --model_args "pretrained=gpt2
```

It would also be nice if the CLI would produce a list of supported tasks with something like

```bash
lighteval --list-tasks
```

We now have a better CLI and can show supported task. #228 will soon allow to display samples too, so I'm closing :)