zhiminy/EvalEng huggingface dataset:
Repository: huggingface/lighteval
Issue: Handling last token in tokenized_continuation
Root cause: Validation Gap

Hi,

In:

https://github.com/huggingface/lighteval/blob/a98210fd3a2d1e8bface1c32b72ebd5017173a4c/src/lighteval/models/base_model.py#L842

if single_token:
            inputs = [request.tokenized_context for request in batch]
        else:
            inputs = [
                request.tokenized_context + request.tokenized_continuation[:-1] for request in batch
            ]  # The last token (an eos) doesn't need to be given to the model

However, I cannot find where we strictly set "tokenized_continuation" to be ended with <eos>.

If this is true, then the eval results are not correct, especially for short tokenized_continuation



Hi, thanks for the issue!

We use [this function](https://github.com/huggingface/lighteval/blob/a98210fd3a2d1e8bface1c32b72ebd5017173a4c/src/lighteval/models/abstract_model.py#L136) to tokenize the different representations, which will add bos/eos automatically if `add_special_tokens` is True. 

However, [for the base model](https://github.com/huggingface/lighteval/blob/a98210fd3a2d1e8bface1c32b72ebd5017173a4c/src/lighteval/models/base_model.py#L79C43-L79C61), I just checked and the default is False if this parameter is not explicitely given in the model config, so I believe you are right! 

cc @NathanHB for a double check

It seems you are right yes. I will double check on my side. Thanks for the catch !

> Hi, thanks for the issue!
> 
> We use [this function](https://github.com/huggingface/lighteval/blob/a98210fd3a2d1e8bface1c32b72ebd5017173a4c/src/lighteval/models/abstract_model.py#L136) to tokenize the different representations, which will add bos/eos automatically if `add_special_tokens` is True.
> 
> However, [for the base model](https://github.com/huggingface/lighteval/blob/a98210fd3a2d1e8bface1c32b72ebd5017173a4c/src/lighteval/models/base_model.py#L79C43-L79C61), I just checked and the default is False if this parameter is not explicitely given in the model config, so I believe you are right!
> 
> cc @NathanHB for a double check

So when using BaseModel() to load a hf format model, do i need to delete `[:-1]` in `request.tokenized_context + request.tokenized_continuation[:-1] for request in batch `?

I also discover this problem because the last token of the original choice is missing in prepared_batch fed into the model. But when I deleted `[:-1]` and run exact the same evaluation I didn't get expected results. The accuracy curve remains at random guess instead of an upward trend as before.

Is there anything else should I modify? Thank you!
Repository: huggingface/lighteval
Issue: Shape mismatches in MMLU
Root cause: Algorithmic Error

I've setup accelerate and running multi-gpus with

`accelerate launch run_evals_accelerate.py --tasks="leaderboard|mmlu:abstract_algebra|0|0" --output_dir "/weka/home-marcob/lighteval/scores" --model_args "pretrained=gpt2"`

I get:

> Cannot apply desired operation due to shape mismatches. All shapes across devices must be valid.
> [rank3]: Operation: accelerate.utils.operations.gather`
> [rank3]: Input shapes:
> [rank3]:   - Process 0: [1024]
> [rank3]:   - Process 1: [2048]
> [rank3]:   - Process 2: [2048]
> [rank3]:   - Process 3: [2048]
> [rank3]:   - Process 4: [2048]
> [rank3]:   - Process 5: [2048]
> [rank3]:   - Process 6: [2048]
> [rank3]:   - Process 7: [2048]

Note that with the same identical setup I can run if I use 5-shots, i.e. the following doesn't throw an error

`accelerate launch run_evals_accelerate.py --tasks="leaderboard|mmlu:abstract_algebra|5|0" --output_dir "/weka/home-marcob/lighteval/scores" --model_args "pretrained=gpt2"`

The same error happens systematically when running the custom MMLU eval from finewebedu (https://huggingface.co/datasets/HuggingFaceFW/fineweb/blob/main/lighteval_tasks.py#L12).

Also note that without configuring accelerate to explicitly verify tensor shapes no error is thrown and the process just hangs indefinitely

Thanks for reporting!
We normally have a pad and gather to prevent this (which pads too short splits before gathering if needed). We'll investigate what happens here, as it should apply.

Hi! I can't reproduce, can you try to reinstall from main and tell me if you're still getting an error?
It would also be nice to provide your detailed accelerate config 

Closing for inactivity
Repository: huggingface/lighteval
Issue: Is there any available example based on lighteval/run_evals_nanotron.py ?
Root cause: Documentation Deficiency

Firstly, thanks for providing this framework. I am following the Fineweb pipeline, using Nanotron for training and Lighteval for evaluation, but met some problems.

**1. Following the [tiny_llama demo from nanotron](https://github.com/huggingface/nanotron/tree/main#training-a-tiny-llama-model), I got one of the checkpoints like this:**

![image](https://github.com/huggingface/lighteval/assets/120651629/f90d93c1-9b33-4b0a-8080-97609c3c950b)

**2. Following [run_evals_nanotron.py](https://github.com/huggingface/lighteval/blob/main/run_evals_nanotron.py), I launch the task with:**
``torchrun --nproc-per-node 1 lighteval/run_evals_nanotron.py --checkpoint-config-path nanotron/examples/config_tiny_llama.yaml --lighteval-override lighteval/examples/nanotron/lighteval_config_override_template.yaml``

**Then I got the error:**

![image](https://github.com/huggingface/lighteval/assets/120651629/b42dadde-159d-447d-ba2b-aa4665bb2c50)


**It seems that there's something to do with the format of .yaml of both lighteval and nanotron-model. Can you provide an example just like you do with accelerate? Thank you.**

I solved this error by modifying [lighteval_config_override_template.yaml](https://github.com/huggingface/lighteval/blob/main/examples/nanotron/lighteval_config_override_template.yaml), deleting the first line and commenting the ``recompute_granularity: null``. But I wonder why this works, Is this a bug?
Repository: huggingface/lighteval
Issue: What is `qem` for gsm8k evaluation?

As titled.

Thank you!

Hi! Quasi exact match!
(Fraction of instances where the normalized prediction matches the normalized gold (normalization done on whitespace, articles, capitalization, ...). )

Thank you very much for your prompt reply!

Hi,

A quick question, what is the difference between `exact_match` and quasi exact match`? Does that mean `qem` did another step of normalization?

Thanks!

Hi!
You'll find all the metrics explanations available in the readme [here](https://github.com/huggingface/lighteval?tab=readme-ov-file#metrics-for-generative-tasks). Indeed, `qem` applies post processing to the results, where the exact match does not.

Thank you!
Repository: huggingface/lighteval
Issue: [FT] Caching
Root cause: Unimplemented Feature Gap

See [slack](https://huggingface.slack.com/archives/C055130UYQG/p1721995050911309)

Cache each task which finished to 1) avoid re-running it 2) restart evals which failed 3) avoid re-calling model as a judge when re-running 

Hi~ really need this feature!
One question: When I use the same tasks to evaluate the models(with same architecture but are from different runs), will the evaluation sample (docs and requests) be cached?

Good to know there's interest! We have not decided yet, but it's likely we'll have a check on the model commit to make sure we only cache relevant samples for each model

That's Cool! thanks for your explanation!
Repository: huggingface/lighteval
Issue: [FT] To keep `target_for_fewshot_sorting` for another purpose
Root cause: Unimplemented Feature Gap

Hi there!

In the code it's stated that `target_for_fewshot_sorting` might be removed in the future, maybe because `line["__few_shots"]` is introduced.

https://github.com/huggingface/lighteval/blob/506abda3ec1e05e033997c4dbf7537db4285432d/src/lighteval/tasks/requests.py#L172

I suggest keeping `target_for_fewshot_sorting`, but for another purpose. To let the user determine the criteria by which fewshot examples are sorted and picked in fewshot manager.

My concrete example is in fewshot structured generation in which model is to generate function call json representations. I want to pick the examples, in a `balanced` setting, according to the name of the functions but currently they're picked according to the whole representation of the function call which doesn't make sense as all of them are different.
Repository: huggingface/lighteval
Issue: [BUG] Building from source does not package the needed jsonl associated files
Root cause: Configuration Error

## Describe the bug
`pip install .` does not include the jsonl files, leading to `FileNotFoundError: [Errno 2] No such file or directory: '/home/...miniconda3/envs/tmp/lib/python3.12/site-packages/lighteval/metrics/judge_prompts.jsonl'`

## To Reproduce
`pip install .` (but `pip install -e .` will work)


cc @NathanHB to fix before the next release

Hi @NathanHB, I checked out this issue. You just need to add "include src/lighteval/metrics/*.jsonl" in the MANIFEST.in file and it works!

<img width="442" alt="Screenshot 2024-07-30 at 10 37 17 PM" src="https://github.com/user-attachments/assets/b3c69abf-c3e4-4d94-af30-4801614429b4">


fixed by #264
Repository: huggingface/lighteval
Issue: [DOC] Document the custom model config files


 Could you please point to the files/ folder you'd like to document? I'd love to open a PR to fix this.

cc: @clefourrier 

I don't think we have a lot of documentation on our model creation files (the yaml files you can use to load models) (though it's been improved since the creation of this issue) - you should look for the ModelConfig example files ([here](https://github.com/huggingface/lighteval/tree/main/examples/model_configs)) and all the Config classes at the top of each model file in [this folder](https://github.com/huggingface/lighteval/tree/main/src/lighteval/models).
It would make sense to open 2 PRs: one to update the docstrings of all config files, and one to add a "Create a model" guide page in the doc :)
Repository: huggingface/lighteval
Issue: Forcing evaluation dtype doesn't work
Root cause: Algorithmic Error

I tried to force the evaluation to be in a different data type than specified in the model config by passing `model_args="...,dtype=<dtype>"`, however, nothing happened and the model was evaluated with the dtype from the model config.

I found that the culprit is this function:
https://github.com/huggingface/lighteval/blob/6041e36ebab3a89b29808b9d940b3ecc206badef/src/lighteval/models/utils.py#L32
and you can see here that if a model config exists then the dtype specified in the `model_args` is ignored.

Hi ! Do you have reproductions steps for us to be able to fix the issue ? I proposed a fixed in #263
Repository: huggingface/lighteval
Issue: [FT] Docs for the configs
GitHub labels: documentation

feature

prio

## Issue encountered
Seems like a lot of features are supported but none of the examples show how best to use them.

## Solution/Feature
Even just one sample config file that details all the various features available for use - quantization, env_config, etc, would be greatly helpful. 

## Posssible alternatives
A more comprehensive document explaining each feature in detail. 



Yep, we clearly need to add this, thanks for bringing it up! 

Can take an initial stab at it since I've gone through the repo recently. Do we want to have a config or have proper docs? What is your recommendation?

We have config examples [here](https://github.com/huggingface/lighteval/tree/main/examples/model_configs) and you could link them in the doc, and add a small paragraph of explanation in the readme
Repository: huggingface/lighteval
Issue: [BUG] Cannot run IFEval due to `AttributeError: 'str' object has no attribute 'category'`
Root cause: Validation Gap

## Describe the bug
On commit `6041e36ebab3a89b29808b9d940b3ecc206badef` it seems that IFEval cannot be run anymore. Using the example in the README

```shell
lighteval accelerate \
    --model_args "pretrained=HuggingFaceH4/zephyr-7b-beta" \
    --use_chat_template \
    --tasks "extended|ifeval|0|0" \
    --output_dir "./evals"
```

gives

```
Traceback (most recent call last):
  File "/fsx/lewis/miniconda3/envs/lighteval/bin/lighteval", line 8, in <module>
    sys.exit(cli_evaluate())
  File "/fsx/lewis/git/hf/lighteval/src/lighteval/__main__.py", line 58, in cli_evaluate
    main_accelerate(args)
  File "/fsx/lewis/git/hf/lighteval/src/lighteval/logging/hierarchical_logger.py", line 175, in wrapper
    return fn(*args, **kwargs)
  File "/fsx/lewis/git/hf/lighteval/src/lighteval/main_accelerate.py", line 90, in main
    task_dict = Registry(cache_dir=env_config.cache_dir).get_task_dict(
  File "/fsx/lewis/git/hf/lighteval/src/lighteval/tasks/registry.py", line 152, in get_task_dict
    tasks_dict[task_name] = task_class(custom_tasks_module=custom_tasks_module)
  File "/fsx/lewis/git/hf/lighteval/src/lighteval/tasks/registry.py", line 267, in __init__
    super().__init__(name, cfg, cache_dir=cache_dir, custom_tasks_module=custom_tasks_module)
  File "/fsx/lewis/git/hf/lighteval/src/lighteval/tasks/lighteval_task.py", line 187, in __init__
    ignored = [metric for metric in self.metrics if metric.category == MetricCategory.IGNORED]
  File "/fsx/lewis/git/hf/lighteval/src/lighteval/tasks/lighteval_task.py", line 187, in <listcomp>
    ignored = [metric for metric in self.metrics if metric.category == MetricCategory.IGNORED]
AttributeError: 'str' object has no attribute 'category'
```

## To Reproduce
See above.

## Expected behavior
IFEval runs without error.

## Version info
Please provide your operating system, lighteval version or commit if you installed from main, and pip/conda environment if your problem concerns dependencies.
Repository: huggingface/lighteval
Issue: ValueError: Cannot find tasks lighteval
Root cause: Documentation Deficiency

This command based on the ReadMe
```
accelerate launch --num_processes=0 -m \
    lighteval accelerate \
    --model_args="pretrained=gpt2" \
    --tasks "lighteval|truthfulqa:mc|0|0" \
    --override_batch_size 1 \
    --output_dir="./evals/"
```
yields in colab this Error:
```
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/content/lighteval/src/lighteval/__main__.py", line 93, in <module>
    cli_evaluate()
  File "/content/lighteval/src/lighteval/__main__.py", line 58, in cli_evaluate
    main_accelerate(args)
  File "/content/lighteval/src/lighteval/logging/hierarchical_logger.py", line 175, in wrapper
    return fn(*args, **kwargs)
  File "/content/lighteval/src/lighteval/main_accelerate.py", line 90, in main
    task_dict = Registry(cache_dir=env_config.cache_dir).get_task_dict(
  File "/content/lighteval/src/lighteval/tasks/registry.py", line 151, in get_task_dict
    task_class = self.get_task_class(task_name, custom_tasks_registry=custom_tasks_registry)
  File "/content/lighteval/src/lighteval/tasks/registry.py", line 107, in get_task_class
    raise ValueError(
ValueError: Cannot find tasks lighteval|truthfulqa:mc in task list or in custom task registry (None)
```

Also the link to the [metadata table](https://github.com/huggingface/lighteval/blob/main/src/lighteval/tasks/tasks_table.jsonl) can not be found.

Hi!
Please follow our bug report template next time :)

- What is the version of lighteval you are currently using?
- What is the hardware you are running this on?

Quite sure it's also linked to [this issue](https://github.com/huggingface/lighteval/issues/243), but if you have the problem for the task table you must be using an old version of lighteval.

This task does not exists in the lighteval suite, you need to do use `leaderboard|truthfulqa:mc|0|0` 

the metatdata table has been detracted in recent version of lighteval, we recomend upgrading your version :)

@NathanHB Which version do you mean? The metatdata table and the command are currently written in the ReadMe.

oh then you have nothing to do, simply re-run your command using `leaderboard|truthfulqa:mc|0|0` and it should work. The metadata table should not appear in the README it's an error from our part, we will update asap.
Repository: huggingface/lighteval
Issue: [BUG] judge_prompts.jsonl not found
Root cause: Environment Incompatibility

Hi, I have installed the current version of lighteval following the instructions in the repository, but cannot run any evals due to the following error:
`'/home/dkuznedelev/miniconda3/envs/pysparse/lib/python3.10/site-packages/lighteval/metrics/judge_prompts.jsonl'`
Seems like for some reason `judge_prompts.jsonl` is not present in the conda environment. How can one fix this issue?

**Run command**
```bash
MODEL_NAME=meta-llama/Meta-Llama-3.1-8B
accelerate launch --multi_gpu --num_processes=2 --main_process_port=29501 -m \
    lighteval accelerate \
    --model_args "pretrained=$MODEL_NAME" \
    --tasks "examples/tasks/open_llm_leaderboard_tasks.txt" \
    --override_batch_size 1 \
    --output_dir="./evals/open_llm_leaderboard/Meta-Llama-3.1-8B"
```

**System specs**
```
Python 3.10.14
torch                     2.2.2
accelerate                0.28.0
transformers              4.43.1
lighteval                 0.4.0.dev0
```

Hi! This is a duplicate of https://github.com/huggingface/lighteval/issues/243, so I'm closing your issue. 
TLDR: The quickest fix is to do an editable install from source, waiting for our clean fix it in the next release, sorry about the bug!

I resolved the issue via manually putting `judge_prompts.jsonl` from repository into conda env.
Repository: huggingface/lighteval
Issue: [EVAL] Update IFEval dataset
Root cause: Unimplemented Feature Gap

## Evaluation short description
Let's now use this implementation from the authors (ported by @lewtun ): https://huggingface.co/datasets/google/IFEval



I also updated the one in the Eleuther harness for consistency: https://github.com/EleutherAI/lm-evaluation-harness/pull/2218
Repository: huggingface/lighteval
Issue: Data Loading Issues in Offline Mode
Root cause: Documentation Deficiency

Hello,

I am currently trying to run an evaluation in offline mode, but I'm encountering issues due to the lack of internet access, which prevents me from using Hugging Face repositories. I've attempted a few solutions, but none have worked so far. I'm specifically struggling with data loading and need to specify the data path manually instead of relying on the Hugging Face repo.

Could someone guide me on how to properly load the data by specifying a local path?

Your assistance would be greatly appreciated.

Thank you.

Hi ! If you downloaded the dataset before then you should be able to-redownload in offline mode [see here](https://huggingface.co/docs/datasets/en/loading#offline)
What dataset are you trying to load ?

problem solved , Thank you! 

@shahad2099 Can you mention in the comments how you solved the problem?
Repository: huggingface/lighteval
Issue: [FT] Open ai endpoint
Root cause: Documentation Deficiency

## Evaluation Short Description

I am attempting to evaluate a model similar to ChatGPT using the OpenAI interface. Is there any way to accomplish this?

### Why is this evaluation interesting?

This evaluation is interesting because it would allow for the assessment of models like ChatGPT or custom models using the OpenAI interface.

### How used is it in the community?

I believe many people will find this useful, as it can facilitate the evaluation of various models within the OpenAI ecosystem.

## Evaluation Metadata

Provide all available resources:
- [OpenAI Python](https://github.com/openai/openai-python)


Hi! We now support Open AI models [here](https://github.com/huggingface/lighteval/blob/main/src/lighteval/models/openai_model.py)!
Repository: huggingface/lighteval
Issue: [BUG] Zero accuracy in Hellaswag for Llama-2-7b (using 8bit quantization)
Root cause: Validation Gap

command:

accelerate launch run_evals_accelerate.py --model_args="Llama-2-7b-chat-hf-8bit,quantization_config="load_in_8bit=True"" --tasks "helm|hellaswag|1|0" -- --output_dir ./evalscratch

Result is 0% correct
Llama-3 works fine as does MMLU for Llama-2.

Is there any way to log the individual outputs?

I found by inspecting the parquet output that gold and prediction differ in a leading space:
[' A'] ['C']
[' B'] ['C']
[' C'] ['C']
[' A'] ['A']
[' B'] ['B']
[' B'] ['B']
[' A'] ['B']
[' D'] ['D']
[' A'] ['B']
[' B'] ['C']

is the space in gold normal?


Hi! 
This is due to tokenization issues iirc.
A simple fix would be to change the task to be `Answer: ` , then gold: `A` instead of `Answer:`, ` A` @NathanHB wdyt?
Repository: huggingface/lighteval
Issue: [BUG] Question on batch preparation in MMLU evaluation
Root cause: Algorithmic Error

The bug I met is similar to #203. I'm trying to reproduce the evaluation results of [ablation model trained on FineWeb](https://huggingface.co/HuggingFaceFW/ablation-model-fineweb-v1/tree/main), using LightEval of commit_id=a98210fd3a2d1e8bface1c32b72ebd5017173a4c.

The MMLU result of step-5000/10000/15000/19000/24000 (namely, 5 ckpts from the first 50b consumed tokens) are as below: 
![img_v3_02ed_4a3641b1-a270-4aed-84cc-9b47ce4447eg](https://github.com/user-attachments/assets/f793206e-7b69-4d00-a895-ae73fc471b4b)

I don't know what causes this gap, when debugging I discover that:
![image](https://github.com/user-attachments/assets/fdce0a09-5ecd-46be-8284-00dd5436bc7d)

The last token of the prepared_batch is missing. Does this mean the evaluation results of [fineweb blogpost](https://huggingface.co/spaces/HuggingFaceFW/blogpost-fineweb-v1#/) is inaccurate?

But when I delete `[:-1]` in  https://github.com/huggingface/lighteval/blob/aaa8bbf705b6f090fb07ad36503f39b5e922a6df/src/lighteval/models/base_model.py#L851

The evaluation results became totally random guess for all ckpts. I suppose there are more lines to modify, or something else caused the gaps in my reproduction results.



I've tried adding the param `add_special_tokens=True` in config file but the last token is still missing

Using the latest repo (commit_id = 7261d80d5679cd91c5c20cf2a7823f092ff66251), I evaluated the same 5 ckpts again (red line in figure). The results are still below the official results.
![plot_mmlu_acc_norm](https://github.com/user-attachments/assets/c2f3b6c6-efc4-42aa-addb-be64476da7af)

And when examining the prepared_batch, the last token still seemed to be missing.
![image](https://github.com/user-attachments/assets/f9a521db-b071-4460-aba9-d71779ddeda9)

My command:
```
accelerate launch --num_processes=1 -m \
    lighteval accelerate \
    --model_args="pretrained=/mnt/data/user/tc_agi/caijie/fineweb_models/ablation-model-fineweb-v1_5000,trust_remote_code=True" \
    --override_batch_size 128 \
    --custom_tasks "/data/fineweb-pipeline/lighteval-main/lighteval_tasks.py" \
    --output_dir "/data/fineweb-pipeline/lighteval-main/evals/" \
    --tasks "custom|mmlu:abstract_algebra|0|1"
```



Thanks for the report, we'll investigate!
cc @hynky1999 and @guipenedo for the fineweb aspect
Repository: huggingface/lighteval
Issue: [FT] Task groupings as separate tasks
Root cause: Unimplemented Feature Gap

## Issue encountered
When one wants to launch all MMLU tasks from cli, he needs to specify all subsets to get the correct average. This is very user unfriendly and I had people asking me how to run all MMLU tasks because of this exact issue

## Solution/Feature
- For every available tasks that contain subsets (:subset), we should automatically create a task group containing all task subsets.
- Not sure if it works already, but one should be able to run task_groups and normal tasks in single run. These primitives should be transparent for the CLI user.