zhiminy/EvalEng huggingface dataset:
Repository: huggingface/lighteval
Issue: Add Code-Centric Interface to LightEval for Enhanced Usability
Root cause: Unimplemented Feature Gap

Enhancing the functionality of LightEval to better accommodate coding workflows is paramount. The current approach relies heavily on command-line interaction (CLI), but a more code-centric interface would greatly benefit users.

Consider the following refinement:

```python
# Install LightEval package
pip install lighteval

from lighteval import Evaluator, EvaluatorArguments

def configure_dataset():
    # Define dataset formatting and evaluation parameters here

# Initialize evaluator for custom dataset evaluations
evaluator = Evaluator(
    model=model,
    eval_dataset=dataset,
    metric="loglikelihood_acc",
    dataset_text_field=configure_dataset,
    args=EvaluatorArguments(
        # Specify additional arguments for evaluation configuration
        # e.g., batch size, evaluation steps, etc.
        # Example:
        batch_size=32,
        num_workers=4,
        ...
    ),
)

# Initiate the evaluation process
evaluator.evaluate()

# Display results and publish statistics to the Hugging Face Hub
evaluator.show_results()
evaluator.push_results()
```

This revised approach emphasizes a more structured and Pythonic usage of LightEval, with clear functions to define dataset formatting and evaluation specifics. Additionally, it leverages the `EvaluatorArguments` class to encapsulate additional evaluation configurations like batch size and number of workers. The usage of `Evaluator` and related methods is aligned with conventional Python programming paradigms, enhancing usability and integration within code-centric workflows.


if this is a feature you guys believe would be beneficial, I am eager to contribute to its development and enhancement.

@clefourrier @NathanHB 

That sounds very cool, and I think we'll want a feature like this, yes!

Let us discuss it a bit internally and we'll come back to you?

@clefourrier Certainly!

There are a few developers who are interested in something similar within the communities I am part of. I also have a very basic implementation of it working.

I look forward to hearing from you and the team, and I would also love to contribute.

hey @clefourrier just wanted to know if there are any updates with respect to the status of this issue and is this something you guys would be interested in adding?

Hi! Sorry we've been a bit under the water atm!
Yes it's still something we'd love to add/have the community add - we first wanted to merge what we are doing with the CLI (having an actual `ligtheval` CLI to call instead of using the scripts, draft is [here](https://github.com/huggingface/lighteval/pull/152)), then adapt the function calls to allow both CLI and code centric calls as above.

To get to the above, we would need to use the new model configuration files, maybe add a task configuration file too, and then create an `Evaluator` class which would wrap up our `main_accelerate` code. 

If you're really eager to start on it, you can give it a go rn and we'll iterate on the fly, wdyt?
Repository: huggingface/lighteval
Issue: `LatexTableWriter` created but never used.
Root cause: Unimplemented Feature Gap

In this line a latex writer is created but never used later. Some functionality seems to be missing.

https://github.com/huggingface/lighteval/blob/996b50427b136c347029f6ed1379a05250029c16/src/lighteval/evaluator.py#L143

Yep, we need to either remove it or add the option to use the latex export at save time, thanks for noticing (it was mostly a debug/internal option)
Repository: huggingface/lighteval
Issue: How to run 30b plus model with lighteval when accelerate launch failed? OOM
Root cause: Configuration Error

CUDA Memory OOM when I launch an evaluation for 30b model using lighteval.


Whats the correct config for it?

Hi! 
Could you specify the precision and hardware you are using?

8A100 for 70b model. 


Is there any examples for evaluating a max model for open_llm_leaderboard_tasks in a fast way?

Hi! 
- In which precision are you running the 70B model? 
- Are your A100 80G or 40G?

It's not going to need the same model parallelism/data parallelism ratio depending on it.

Hi, closing this issue for inactivity, but feel free to re-open if you need help.
Repository: huggingface/lighteval
Issue: Allow passing a `GenerationConfig` for generative evals
Root cause: Unimplemented Feature Gap

For some benchmarks, it can be preferable to generate responses via sampling with non-zero temperature. Although this affect reproducibility, it could be useful for power users who are constructing an internal benchmark with `lighteval`

One way to do this would be by allowing one to pass a `generation_config` field in the task metadata which is ultimately parsed by `transformers.GenerationConfig` ([docs](https://huggingface.co/docs/transformers/v4.37.2/en/main_classes/text_generation#transformers.GenerationConfig)) before being passed to the `generate()` method

@lewtun can you check #428 to tell me if there are other params you'd want to see?
Repository: huggingface/lighteval
Issue: Feature: Checkpointing on task level.
Root cause: Unimplemented Feature Gap

I would like to request / suggest the following new feature:

## Background

When I use cheap Azure low-priority instances or AWS spot instances they might be preempted. If this happens the evaluation must restart from the beginning.

## New Feature

It would be cool to write a "checkpoint" for every task. So if multiple tasks are evaluated like with 
[open_llm_leaderboard_tasks](https://github.com/huggingface/lighteval/blob/main/examples/tasks/open_llm_leaderboard_tasks.txt) then it can load tasks that already have been evaluated...
And we do not have to restart at the beginning.

Hi!
This would be very hard to do, as, for efficiency purposes, we do inference for all requests of the same types in batch, and then only do metrics computations - such a system would require us to rewrite the entirety of the code base while losing overall speed performance.
I don't think we will consider it - I suggest you launch evaluations on one task at a time if you have such needs.

Ok. So lets close this again?
Repository: huggingface/lighteval
Issue: Expose a few model predictions / gold answers in the logs
Root cause: Unimplemented Feature Gap

For generative benchmarks like MATH / GSM8k / IFEval, it would be great to have some visibility in the logs on how the prompts are formatted, what the generations look like, what the gold answer is etc.

Currently, the best approach I've found is to first run the benchmark with `--max_samples` and then manually inspect the details Parquet file. However this is rather cumbersome, especially when launching many evals in parallel :)

Perhaps we can store the first N examples in the logs?

aren't the details also stored in json format ? that would make it easier for you to inspect them.
otherwise, good idea to log out the first element of each task.
Repository: huggingface/lighteval
Issue: `Could not initialize the JudgeOpenAI model` and `openi` import error 
Root cause: Validation Gap

FYI I'm seeing a lot of these "errors" in my logs when running lighteval

```
Could not initialize the JudgeOpenAI model:
[Errno 2] No such file or directory: 'src/lighteval/tasks/extended/mt_bench/judge_prompts.jsonl'
```

This doesn't cause an exception, but maybe good to clean up from the logs?

I also learned that you need the `openai` dep even if you don't run mt_bench due to an import error from this line: https://github.com/huggingface/lighteval/blob/fc428f600c40b214b4d784540d1a0db5ed193d30/src/lighteval/metrics/llm_as_judge.py#L30

I would recommend either promoting the `openai` dep to a core dep or having a checker like `is_openai_available()` then only importing the method if it is

thanks for the catch ! Did not pay attention to those on my local install, I will solve this asap :)

Fixed by #223.
Repository: huggingface/lighteval
Issue: Add Sympy equivalence for MATH / GSM8K?
Root cause: Unimplemented Feature Gap

In the [Minerva](https://arxiv.org/pdf/2206.14858.pdf) and LLeMMa papers, `sympy` is used to ensure equivalence of predicted / gold answers, e.g. ensuring $1/ \sqrt{3}$ and $\sqrt{3}/3$ are treated the same. From the Minerva paper:

> After applying this normalization function, we checked whether the formatted target and prediction strings are SymPy-equivalent. SymPy equivalence is determined by parsing the answers via sympy.parsing.latex.parse_latex and then checking whether substracting the two resulting SymPy objects and applying sympy.simplify gives zero. We set a timeout of 5s when calling sympy.simplify, and labeled strings as nonequivalent if this timeout was exceeded.
For MATH problems, SymPy equivalence improved overall accuracy by around 1%. See Table 6 for the accuracies in MATH with only exact string match vs. SymPy equivalence.

Although the difference between Minerva & OpenAI models was only 1%, would it make sense to add `sympy` to the MATH metric for both correctness and potentially uncovering larger variation among open models?

Hi!
Yes, I think that's a very good idea.

cc @NathanHB since you've been working on this a bit?
Repository: huggingface/lighteval
Issue: Version of a task should be configurable.
Root cause: Unimplemented Feature Gap

The `LightevalTask` class has a VERSION. This is not configurable and fixed to 0. See:

https://github.com/huggingface/lighteval/blob/11b48333b46ecd464cc3979de66038c87717e8d6/src/lighteval/tasks/lighteval_task.py#L166

I think in the results table the version number is also displayed. See:

|                         Task                         |Version| Metric |Value |   |Stderr|
|------------------------------------------------------|------:|--------|-----:|---|-----:|
|all                                                   |       |acc     |0.8737|±  |0.0094|
|                                                      |       |acc_norm|0.8737|±  |0.0094|
|community:german_rag_eval:_average:0                  |       |acc     |0.8737|±  |0.0094|
|                                                      |       |acc_norm|0.8737|±  |0.0094|
|community:german_rag_eval:choose_context_by_question:0|      0|acc     |0.7290|±  |0.0141|
|                                                      |       |acc_norm|0.7290|±  |0.0141|
|community:german_rag_eval:choose_question_by_context:0|      0|acc     |0.8490|±  |0.0113|
|                                                      |       |acc_norm|0.8490|±  |0.0113|
|community:german_rag_eval:context_question_match:0    |      0|acc     |0.9770|±  |0.0047|
|                                                      |       |acc_norm|0.9770|±  |0.0047|
|community:german_rag_eval:question_answer_match:0     |      0|acc     |0.9400|±  |0.0075|
|                                                      |       |acc_norm|0.9400|±  |0.0075|

If I slightly change the task or the prompt (e.g. #171 ) - it would be nice to increase the version number. Should this be changed / added?

Would also appreciate this feature (and obviously someone thought of this already, given the existence of VERSION). 👍 

Hi! Yep, we'll need to add the mechanism again - we had removed it to simplify the code, and because we did not want to fix any task version while still being in alpha.

@clefourrier do you want me to suggest a PR or do you want to implement it?

If you have the time to open a PR I'd be grateful! 
You'll need to add a default version argument to the TaskConfig, and edit the task_table.jsonl to set "version" to 0 there for all tasks, and edit your own to pass it to 1.

> If you have the time to open a PR I'd be grateful! You'll need to add a default version argument to the TaskConfig, and edit the task_table.jsonl to set "version" to 0 there for all tasks, and edit your own to pass it to 1.

I started here: #181

This is done with PR #181. Closing...
Repository: huggingface/lighteval
Issue: Performance compared to lm-evaluation-harness
Root cause: Algorithmic Error

Hi,

Thanks for sharing this package, it has lots of cool features!

I saw that arc-challenge was taking about twice longer that what I have with harness, I ran the following commands with lighteval:

```bash
# ran with 4A100 GPUs -> 611s
time  accelerate launch --multi_gpu --num_processes=4 lighteval/run_evals_accelerate.py --model_args="pretrained=meta-llama/Meta-Llama-3-8B" --tasks "leaderboard|arc:challenge|25|0" --output_dir "arc_challenge" --override batch_size 8
```
and the following command from harness (using big-refactor branch):
```bash
# ran with 4A100 GPUs -> 319s and big-refactor branch
time accelerate launch -m lm_eval \
    --model hf \
    --model_args pretrained=meta-llama/Meta-Llama-3-8B,dtype="bfloat16" \
    --tasks arc_challenge \
    --batch_size 8 \
    --num_fewshot=25
```

Of-course, many things could cause this but I wanted to know if you have faced something similar or benchmarked light-eval compared to Harness? 

If not, would you have a suggestion to get similar performance? (it seems bf16 are used by default so it should not be the culprit)

Hi! Is your configuration for accelerate the same? But otherwise I'll take a look, thanks for opening this issue

Yes, I used the default one, thanks.

When you say the default one, what do you mean precisely? (Could you give the output of your configuration?)
Because when launching lighteval, you specifically select the number of processes to use, and not when launching the lm_eval one.

If the model is small enough to fit 2 times on a GPU, you could be doing DP8 with lm_eval, and only DP4 for lighteval (which would also explain the difference in speed)

I was using DDP with 4 GPUs in the two cases (if I got everything right 😅).

This is the output acceleration config I got with lighteval:
```bash
The following values were not passed to `accelerate launch` and had defaults used instead:
	`--num_machines` was set to a value of `1`
	`--mixed_precision` was set to a value of `'no'`
	`--dynamo_backend` was set to a value of `'no'`
```
and the one I got with lm_eval:
```bash
	`--num_processes` was set to a value of `4`
		More than one GPU was found, enabling multi-GPU training.
		If this was unintended please pass in `--num_processes=1`.
	`--num_machines` was set to a value of `1`
	`--mixed_precision` was set to a value of `'no'`
	`--dynamo_backend` was set to a value of `'no'`
```

One thing that I am now wondering that could cause this gap is whether lighteval uses bf16 by default (which could cause a large gap). I will rerun by setting it explicitly and let you know.

Thanks a lot!

I reran setting bf16 explicitly and it took 11 min with the following command

```bash
time accelerate launch --multi_gpu --num_processes=4 lighteval/run_evals_accelerate.py     --model_args="pretrained=meta-llama/Meta-Llama-3-8B,dtype="bfloat16""     --tasks "leaderboard|arc:challenge|25|0"     --output_dir "arc_challenge2" --override_batch_size 8
```
and it took longer than lm_eval so bf16 does not seem to be the culprit.
Repository: huggingface/lighteval
Issue: Zero scores on cnn-dm benchmark from HELM
Root cause: Configuration Error

When running the evaluation of Sheared-LLaMA-1.3B and original LLaMA-7B on helm|summarization:cnn-dm, I get zero scores:

```bash
accelerate launch --multi_gpu --num_processes=3 run_evals_accelerate.py --model_args "pr
etrained=princeton-nlp/Sheared-LLaMA-1.3B,model_parallel=True" --task "helm|summarization:cnn-dm|0|0" --override_batch_size 1 --out
put_dir "./evals/"
```

Output:

```bash
|           Task            |Version|         Metric          |Value|   |Stderr|
|---------------------------|------:|-------------------------|----:|---|-----:|
|all                        |       |rouge1                   |    0|±  |     0|
|                           |       |rouge2                   |    0|±  |     0|
|                           |       |rougeL                   |    0|±  |     0|
|                           |       |summac                   |    0|±  |     0|
|                           |       |summarization_coverage   |    0|±  |     0|
|                           |       |summarization_density    |    0|±  |     0|
|                           |       |summarization_compression|    0|±  |     0|
|helm:summarization:cnn-dm:0|      0|rouge1                   |    0|±  |     0|
|                           |       |rouge2                   |    0|±  |     0|
|                           |       |rougeL                   |    0|±  |     0|
|                           |       |summac                   |    0|±  |     0|
|                           |       |summarization_coverage   |    0|±  |     0|
|                           |       |summarization_density    |    0|±  |     0|
|                           |       |summarization_compression|    0|±  |     0|
```

Seems to have something to do with task_prompt_formatting
Change the prompt as follows:
```shell
instruction="### Instruction: Summarize the following passage in 3 sentences.\n", 
query=f"### Instruction: Summarize the following passage in 3 sentences.\n### Passage: {line['article']}\n### Summary: ",
```
And this issue can be fixed


Thanks so much for debugging, would you be OK with opening a PR to share this fix with the community?

Not sure if it has something to do with the evaluated model itself. When I use another finetuned model, the original code seems to be fine.
Maybe adjusting the task_prompt_formatting ourselves according to requirements is needed.
Repository: huggingface/lighteval
Issue: Issue when saving results with fsspec==2023.12.1
Root cause: External Dependency Breakage

Dear lighteval team,

I had this version fsspec==2023.12.1, and I had issues once the evaluation is done is failing. I am using now fsspec==2024.3.1 and it works fine. Although I did pip install -e . before running the eval_accelerate.
It would be grade if you add this to the setup requirements.
Thanks

Hi @Hamza-Alobeidli , what do you use fsspec for? We should not be needing it for lighteval.

Hi @Hamza-Alobeidli, just did a fresh install of lighteval to reproduce and it seems to install the 2024 version by default. closing this issue as we cannot seem to replicate. Don't hesitate to re-open if needed :)
Repository: huggingface/lighteval
Issue: `apply_target_perplexity_metric` pops only the first response
Root cause: Algorithmic Error

Hi there. 

`apply_target_perplexity_metric` seems to pop only the first related response not all of them.

https://github.com/huggingface/lighteval/blob/ad42e43bcc3bd50fdba68936999bf553bf53b9e4/src/lighteval/metrics/__init__.py#L31-L36

While its related responses are as much as the number of gold choices.
Repository: huggingface/lighteval
Issue: An apparent bug in drop's dealing with multi-span answer
Root cause: Algorithmic Error

Hi there! 🤗

It seems that `drop_metrics` selects only the first span when an answer is of type multi-span:

https://github.com/huggingface/lighteval/blob/ad42e43bcc3bd50fdba68936999bf553bf53b9e4/src/lighteval/metrics/harness_compatibility/drop.py#L149-L153

Maybe we could remove the first `if` statement and change the second one accordingly to fix it.


I think we had to use this system to be compatible with the results of the Eleuther AI Harness - could you check if they have updated the mechanism so we see if we update on our side too? :)

It seems that it's correct [there](https://github.com/EleutherAI/lm-evaluation-harness/blob/70e1de09c2d11e95b54d37d1c2145a0b23f57635/lm_eval/tasks/drop/utils.py#L69-L74):
```python
    for gold_answer in golds:
        exact_match, f1_score = get_metrics(preds, gold_answer)
        if gold_answer[0].strip():
            max_em = max(max_em, exact_match)
            max_f1 = max(max_f1, f1_score)
    return {"em": max_em, "f1": max_f1}
```

Yes, you are right, I checked the downstream logic and it seems like they don't take the first item anywhere - I think it's something we do by assuming the length is always one, but we would have to check this and add an assert.
If you want to do edits and open a PR, feel free to :)
Repository: huggingface/lighteval
Issue: Expose `stop_sequence` at command line
Root cause: Unimplemented Feature Gap

Models like `llama-3` use a chat template where the expected stop sequence is `
` instead of the common EOS token used in other models. This means that generative benchmarks like `ifeval` continue generating past the EOS token and give incorrect results.

One way to handle this would be to either include this special token as a default in the generative benchmarks or alternatively expose `--stop_sequence` as an argument in the main script that users can control.

Hi @lewtun , just to clarify, is `
` a different stop token for the chat template specifically?
And is the model using [
](https://huggingface.co/meta-llama/Meta-Llama-3-8B/blob/62bd457b6fe961a42a631306577e622c83876cb6/tokenizer_config.json#L2054) for the rest?

Other question, do you think fixing #16 would be enough to solve this too?
Repository: huggingface/lighteval
Issue: The helm|piqa task is generative but has generation_size=-1.
Root cause: Configuration Error

The `helm|piqa` task listed in `tasks_table.jsonl` here: https://github.com/huggingface/lighteval/blob/a98210fd3a2d1e8bface1c32b72ebd5017173a4c/src/lighteval/tasks/tasks_table.jsonl#L797C1-L797C472.

has `"generation_size":-1` even though `"metric":["exact_match"...` which are mutually exclusive.

For example, this command fails for me --
```
accelerate launch --multi_gpu --num_processes=8 run_evals_accelerate.py \
    --model_args "pretrained=gpt2" \
    --tasks "helm|piqa|0|1" \
    --override_batch_size 1 \
    --output_dir="./evals/"
```

with error:
```
ValueError: `max_new_tokens` must be greater than 0, but is -1.
```

Thanks.

Hi! This sounds like an error on our side! If you have the time, could you take a look at the helm code base to see which generation size should be used?

I also have this problem

Hi! I opened a PR which should fix this issue, thanks for reporting and for your patience!
Repository: huggingface/lighteval
Issue: Different results for gsm8k via lighteval compared to internal pipeline
Root cause: Algorithmic Error

Hi,

I get a zero score with lighteval for gsm8k for one of the models and for the same model, the results are a bit better, at least in 20s. Any idea why this could be happening? And how is the lm_eval gsm8k different than lighteval gsm8k?

Thanks

Hi!
There are many ways to implement GSM8K - the kind of difference you refer to could be a difference in end of sentence token management on in answer extraction. It's also possible that our parsing is stricter than the one from the harness.

I hope this helps!

Same problem here. Also got a zero acc on both leaderboard|gsm8k|0|0 and lighteval|gsm8k|0|0

same +1 

Same

newest version of gsm8k in lighteval should work better, also if the model does not perform, add some fewshots it really helps for gsm8k