zhiminy/EvalEng huggingface dataset:
Repository: huggingface/lighteval
Issue: [FT] Any Pypi package for this toolkit?
Root cause: Documentation Deficiency


Hi ! Yes you can install using: `pip install lighteval`, to add some needed extras, you can
`pip install lighteval[extra]`

| extra name   | description                                                               |
|--------------|---------------------------------------------------------------------------|
| accelerate   | To use accelerate for model and data parallelism with transformers models |
| tgi          | To use Text Generation Inference API to evaluate your model               |
| nanotron     | To evaluate nanotron models                                               |
| quantization | To evaluate quantized models                                              |
| adapters     | To evaluate adapters models (delta and peft)                              |
| tensorboardX | To upload your results to tensorboard                                     |
| vllm         | To use vllm as backend for inference                                      |


> Hi ! Yes you can install using: `pip install lighteval`, to add some needed extras, you can `pip install lighteval[extra]`
> 
> extra name	description
> accelerate	To use accelerate for model and data parallelism with transformers models
> tgi	To use Text Generation Inference API to evaluate your model
> nanotron	To evaluate nanotron models
> quantization	To evaluate quantized models
> adapters	To evaluate adapters models (delta and peft)
> tensorboardX	To upload your results to tensorboard
> vllm	To use vllm as backend for inference

Thanks! Would you put this information in the readme by any chance?

Good idea, opened #307
Repository: huggingface/lighteval
Issue: [FT] Remove obsolete config properties (frozen, output_regex)
Root cause: Unimplemented Feature Gap

## Request
Removed obselete config properties:
- frozen
- output_regex

For more context see:
https://github.com/huggingface/lighteval/pull/289
```
the output_regex is legacy from the harness, we needed it for bigbench tasks but this is now handled by the metrics direclty. Frozen was to make sure the task did not change over time but we are now using task versioning for this.
```


Agree on both. 
Note: `frozen` was initially for tasks that we had thoroughly debugged and that were included in the test suite, hence that were not expected to change definition as they were robust. So it's also an indicator of higher quality.
Repository: huggingface/lighteval
Issue:  [FT] Provide an interface for easier edit of parametrizable metrics
Root cause: Documentation Deficiency

## Issue encountered
When using a metric which is parametrizable, you need to create a new custom metric to edit default parameters - for some metrics like the BertScorer it's harder to do, see #310 point 3

## Solution/Feature
We need to provide an interface or examples on this


Thank you! Dropping in related ideas here as they come up:
Are we open to passing in `bert_scorer` into BertScore initialization?
reference: https://github.com/huggingface/lighteval/blob/7295c78fcb63604cc18c43dd62349a6120ac8600/src/lighteval/metrics/metrics_sample.py#L539
Repository: huggingface/lighteval
Issue: [FT] pass trust_remote_code as flag for loading datasets with custom code
Root cause: Unimplemented Feature Gap

## Issue encountered
When trying to run an evaluation while loading a dataset with custom code, there doesn't seem to be an easy way to set the option `trust_remote_code=True`.

The command line will say this:
```
The repository for swiss_leading_decision_summarization_eval contains custom code which must be executed to correctly load the dataset. You can inspect the repository content at https://hf.co/datasets/swiss_leading_decision_summarization_eval.
You can avoid this prompt in future by passing the argument `trust_remote_code=True`.

Do you wish to run the custom code? [y/N] 
```

If the user can't type in `y` quick enough (not within 10 ish seconds), there is an error:
```
  File "/Users/chuandu/Documents/workspace/legal_llm_evaluation/llm_eval_env/lib/python3.10/site-packages/datasets/load.py", line 131, in resolve_trust_remote_code
    raise ValueError(
ValueError: The repository for swiss_leading_decision_summarization_eval contains custom code which must be executed to correctly load the dataset. You can inspect the repository content at https://hf.co/datasets/swiss_leading_decision_summarization_eval.
Please pass the argument `trust_remote_code=True` to allow custom code to be run.
```

## Solution/Feature
Currently we can pass `trust_remote_code=True` for the model, but a separate variable is needed for loading datasets with custom code `--model_args "pretrained=Qwen/Qwen2-0.5B-Instruct,trust_remote_code=True"`

A similar way to pass this to the lighteval CLI would be great.



The parameter you are looking for (in custom tasks where you define your own `LightevalTaskConfig`) is [here](https://github.com/huggingface/lighteval/blob/ca4139c2a2c0cfcc6798c288ba49f3690600c74c/src/lighteval/tasks/lighteval_task.py#L102).
Sorry I missed the issue - does that answer it for you?
Repository: huggingface/lighteval
Issue: Bias metric(s)
Root cause: Documentation Deficiency

In the README ([section](https://github.com/huggingface/lighteval#metrics-for-generative-tasks)), the `bias` metric from HELM is listed, but I could not find this one in the codebase. I assume it is not implemented, and was wondering if there are plans to add this? Or maybe I am just not looking in the right place :see_no_evil:

Anyway, thanks for open-sourcing this, its cool

Hi!
Good catch! We used to have it, but I removed it before the release, as we tried to have a smooth and homogeneous code base metrics wise, and it was actually quite messy.
To be more specific, we want to have one general standard version of any given metric, and use customizable standards (= one exact match, one quasi exact match, but you can change normalisation if needed) - this one was not standard and not generalizable.

I could go in the history of the internal version and give you the files plus some tips on how to use them if you'd like however :)

Hi @clefourrier ! thanks for your reply
I see, that makes sense. I brought it up out of curiousity cause the bias topic is interesting to me at the moment, but I can move forward with the HELM version for now. Thanks for the offer though, I might come back on it in the future.

Closing the issue as my question is answered :) if you have time it might be nice to remove the metric from the README though

Will do, thanks!
Repository: huggingface/lighteval
Issue: [FT] Enable batched dataset_filter
Root cause: Architectural Constraint

## Issue encountered
When defining a custom `dataset_filter` in custom `LightevalTaskConfig` ([code](https://github.com/huggingface/lighteval/blob/7295c78fcb63604cc18c43dd62349a6120ac8600/src/lighteval/tasks/lighteval_task.py#L99)), I wanted to specify a `hf_filter` which filters the dataset by language. 

It seems that by default, we do not process the examples in batches:
- entrypoint: https://github.com/huggingface/lighteval/blob/7295c78fcb63604cc18c43dd62349a6120ac8600/src/lighteval/utils/utils.py#L239
- batch is False by default [here](https://github.com/huggingface/datasets/blob/13f18e3339e8e4c8d6205e9d0a93f41b9ed39f37/src/datasets/dataset_dict.py#L1944)

More specifically, I tried to implement a vectorized filtering function, which did not work unless `batched=True`, however, it seems difficult to control this value.

My initial language filter was 
```
def create_language_filter(target_language):
    def language_filter(examples):
            return [language == target_language for language in examples['language']]
    return language_filter
```
if  `dataset=dataset.filter(dataset_filter, batched=False)`, the dataset is actually not filtered by language during testing. When I ran `dataset=dataset.filter(dataset_filter, batched=True)`, the filtering was successful.

Testing code is below. Maybe this is not representative of how the lighteval task runs?
```
        dataset_filter = create_language_filter(language)
        dataset=dataset.filter(dataset_filter, batched=True) # switch between False and True
        for i, sample in enumerate(islice(dataset, 5)):
            print(f"\nSample {i + 1}:")
            print(f"Language: {sample['language']}")
            print(f"Text: {sample['text'][:100]}...")
```

Therefore, I modified the function as follows, but the evaluation could be slower due to single example processing during filtering?
```
def create_language_filter(target_language):
    def language_filter(examples):
        if isinstance(examples['language'], list):
            return [language == target_language for language in examples['language']]
        else:
            return examples['language'] == target_language
    return language_filter
```

## Solution/Feature
I am wondering if there is interest in:
1. exposing this parameter to be easily configurable
2. set batched to be default True
3. whether there is another way to run the filtering such that this isn't an issue?

Thank you!
Repository: huggingface/lighteval
Issue: [EVAL] Add RewardBench
Root cause: Unimplemented Feature Gap

## Evaluation short description
- Why is this evaluation interesting?

RewardBench is perhaps the _only_ evaluation suite to provide broad coverage of the strengths and weaknesses of reward models across domains like reasoning, chat, and safety. It would be great to have this in `lighteval` so one can have a unified evaluation suite and not have to hop across different repos and dependencies.

- How used is it in the community?

RewardBench is the de facto leaderboard for comparing reward models and is widely used by the post-training subset of the community for advanced methods like RL, rejection sampling, and others. 

## Evaluation metadata
Provide all available
- Paper url: https://arxiv.org/abs/2403.13787
- Github url: https://github.com/allenai/reward-bench
- Dataset url: https://huggingface.co/datasets/allenai/reward-bench

**Note:** this eval is not a typical LLM eval since it relies on sequence classification. I'm not sure if that is out of scope for `lighteval`

It's in scope for the generative models, not as much for the classifier ones (as we would need to add a whole new pipeline for loading and running them) - do you think having it only for gen models would already be interesting enough?

> It's in scope for the generative models, not as much for the classifier ones (as we would need to add a whole new pipeline for loading and running them) - do you think having it only for gen models would already be interesting enough?

OK I think that would be too restrictive since most reward models today are currently trained as sequence classifiers. Feel free to close the issue if you want :)
Repository: huggingface/lighteval
Issue: [EVAL] Add ArenaHardAuto
Root cause: Unimplemented Feature Gap

## Evaluation short description
- Why is this evaluation interesting?

Many benchmarks are getting saturated by new models. LMSYS has crowd-sourced a variety of hard prompts from the community and this provides a strong correlation with Elo scores.

- How used is it in the community?

Recent papers are starting to report ArenaHard as a core metric to measure the improvements from new post-training methods. It is also becoming a new alternative to MT-Bench due to it's difficulty and real-world source of prompts.

## Evaluation metadata
Provide all available
- Paper url:  https://arxiv.org/abs/2406.11939
- Github url: https://github.com/lm-sys/arena-hard-auto
- Dataset url: https://github.com/lm-sys/arena-hard-auto/blob/main/data/arena-hard-v0.1/question.jsonl
Repository: huggingface/lighteval
Issue: [EVAL] Add MixEval
Root cause: Unimplemented Feature Gap

## Evaluation short description

- Why is this evaluation interesting?

MixEval is a cheap and fast to run benchmark that provides strong correlation with the Elo scores on LMSYS's chatbot arena. This is useful for model developers to gauge how well their model is likely to do in real-world environments that go beyond academic tasks (I know Elo scores / arenas have their weaknesses, but having a proxy for this is still useful!)

- How used is it in the community?

Used by the post-training community to provide a proxy for human evaluation in the chatbot arena (typically reported in papers).

## Evaluation metadata
Provide all available
- Paper url: https://arxiv.org/abs/2406.06565
- Github url: https://github.com/Psycoy/MixEval/
- Dataset url: https://huggingface.co/datasets/MixEval/MixEval


Done ! In PR #337
Repository: huggingface/lighteval
Issue: [BUG] VLLM backend uses sampling with t=1.0 even when num_samples==1
Root cause: Algorithmic Error

## Describe the bug
The VLLM backend uses sampling with`t=1.0` even when `num_samples==1`

When `num_samples==1` the transformers backend switches to `do_sample=False`:
https://github.com/huggingface/lighteval/blob/9cd26bc702ed16260b04d6167115ea4a79ff59fe/src/lighteval/models/base_model.py#L613-L624
The VLLM backend does not do this:
https://github.com/huggingface/lighteval/blob/9cd26bc702ed16260b04d6167115ea4a79ff59fe/src/lighteval/models/vllm_model.py#L278-L294



## To Reproduce
```shell
lighteval accelerate --model_args=vllm,pretrained=Qwen/Qwen2.5-Math-1.5B-instruct,dtype=float16,use_chat_template=True --tasks=lighteval|math_cot:algebra|0|0 --output_dir=./evals/math_cot
```

## Expected behavior
The vllm backend should match the behavior of the transformers backend.
Repository: huggingface/lighteval
Issue: [BUG] Incorrect link in the docu
GitHub labels: bug

In:
https://github.com/huggingface/lighteval/wiki/Adding-a-Custom-Task

replace:
https://github.com/huggingface/lighteval/wiki/adding_new_metric

by:
https://github.com/huggingface/lighteval/wiki/Adding-a-New-Metric

Hi ! Thanks for the heads up, it's fixed
Repository: huggingface/lighteval
Issue: Custom task execution
Root cause: Documentation Deficiency

I've implemented a custom task and also changed BaseModelConfig argument:
trust_remote_code: bool = True

but when executing:
lighteval accelerate --model_args "pretrained=gpt2" --tasks "community|{custom_task}" --custom_tasks "community_tasks/kinit_evals.py" --output_dir "./evals"

I got:
  File "C:\Users\tomas.javurek\AppData\Local\Programs\Python\Python312\Lib\site-packages\datasets\load.py", line 131, in resolve_trust_remote_coe
    raise ValueError(
ValueError: The repository for kinit_evals contains custom code which must be executed to correctly load the dataset. You can inspect the reposi.
Please pass the argument `trust_remote_code=True` to allow custom code to be run.

I am not sure where to pass it.

Hi! It's in your custom task definition, not for the model. Here the error is when loading the data from your task.
You'll also need to make sure that you're using the latest version of lighteval.

Note: please follow the issue template next time (provide the full stack trace, full command, custom code, etc) so we have an easier time debugging!
Repository: huggingface/lighteval
Issue: [BUG] batch_size = auto:1 issue
Root cause: Algorithmic Error

## Disclaimer 
It's not a Lighteval bug but an lm-evaluation-harness one. I'm just letting you know so you can investigate the batch size issue

## Describe the bug
I got the bug with setting the `batch_size = auto`. It originally occurred with the `glaiveai/Reflection-Llama-3.1-70B` evaluation, but it has repeated itself now with `rombodawg/Rombos-LLM-V2.5-Qwen-32b` (I ran a manual evaluation for this model to check for an another bug)

The error occurs when `batch_size = auto:1` is passed to the evaluation process, so it hangs the evaluation process. The longest I detected was two days with no "vital signs" in the log file. According to `htop` and `nvidia-smi` the process was indeed running. 

This issue can be fixed by removing the `--batch_size auto` flag from the slurm file to set "batch_size=1" for the whole evaluation process.

## To Reproduce
1. I use [lm-evaluation-harness](https://github.com/huggingface/lm-evaluation-harness/tree/adding_all_changess), with `adding_all_changess` branch. My accelerate setup is this:
```
accelerate launch --multi_gpu --num_processes 4 \
    -m lm_eval \
    --model_args pretrained=${model},revision=${revision},trust_remote_code=False,dtype=bfloat16,parallelize=True \
    --tasks leaderboard \
    --batch_size auto \
    --output_path eval_results \
    --log_samples \
    --hf_hub_log_args hub_results_org=open-llm-leaderboard,hub_repo_name=${model////__}-details,push_results_to_hub=True,push_samples_to_hub=True,public_repo=True
```

## Expected behavior
I expect that `batch_size=auto` will not block the evaluation process when it is automatically set to 1

p.s. Wow, what a cool automatic bug report template!
p.p.s. Please, ping me if something is unclear or I need to change the type of this issue. I decided to describe it as a bug
Repository: huggingface/lighteval
Issue: [FT] Single token completion loglikelihood auto-detection
Root cause: Unimplemented Feature Gap

## Issue encountered
- If all choices for loglikehood task are exactly one token, one can only run single pass to compute their logprobs. This is the case of MCF formulation (A/B/C) of tasks, which is the most used. However currently if one wants to leverage this fast evaluation, he needs to use special metric variant (metric_single_token). This is not only annoying to maintain, but many users don't know about this and don't benefit from potential speed-up.

## Solution/Feature
We could detect the single token case automatically during loglikehood requests computation.
1. Group loglikehood requests by context
2. From each group select those that have exactly one token
3. Run the single_token requests from each group using the single-token workflow.

## Benefits
Huge speed-up all mcq tasks. Easier to maintain models and metrics (no need to create and handle single token variants of metrics)
Repository: huggingface/lighteval
Issue: [FT] More general approach than `output_regex` to model answer extraction
Root cause: Unimplemented Feature Gap

Hi there!

Currently, we  can specify an `output_regex` in the task config to extract the model answer from its generated content. This config option is used in `apply_generative_metrics`.

We could replace `output_regex` with an answer extractor object that performs arbitary operations to extract the answer. For example, that of [MMLU-Pro](https://arxiv.org/pdf/2406.01574) is as follows:

![image](https://github.com/user-attachments/assets/10e67a90-1166-400e-ad9b-c34af83b1370)
Repository: huggingface/lighteval
Issue: [FT] Pipeline does not fully handle `trust_remote_code` to load dataset
Root cause: Configuration Error

## Issue encountered
`trust_remote_code` is now mandatory to load dataset specifically with a script.

## Solution/Feature
Add `trust_remote_code` as an input to `create_custom_tasks_module()` then in `get_custom_tasks()` and in `Pipeline.pipeline_parameters` (`lighteval.tasks.registry`).

```
class Pipeline:
    def _init_tasks_and_requests(self, tasks):
        with htrack_block("Tasks loading"):
            with local_ranks_zero_first() if self.launcher_type == ParallelismManager.NANOTRON else nullcontext():
                # If some tasks are provided as task groups, we load them separately
                custom_tasks = self.pipeline_parameters.custom_tasks_directory
                tasks_groups_dict = None
                if custom_tasks:
                    _, tasks_groups_dict = get_custom_tasks(custom_tasks, trust_remote_code=self.pipeline_parameters.trust_remote_code)
                if tasks_groups_dict and tasks in tasks_groups_dict:
                    tasks = tasks_groups_dict[tasks]

                ...

```

Maybe a more general fix could be to pass a generic **kwargs as an additional input to `get_custom_tasks()`.



Hi! I'm quite sure we fixed this since, can you check if that's working for you from main?
Repository: huggingface/lighteval
Issue: [FT] Using lighteval to evaluate a model on a single sample, how?
Root cause: Documentation Deficiency

Thank you the team for the great work. I have a question. Can you please help me to use lighteval to evaluate a model on a single sample? 

For example, if I have an input from mmlu I, my model generates output O, how can I use lighteval to evaluate O with using the Acc metric?

Thanks!

Hi ! Do you mean using something like `--max_samples 1` ?

Hello @NathanHB, thank you for your fast reply. I want to customize the model prompt during evaluation, so then I need to evaluate per sample. Could you guide me how I can achieve this? Thanks

You could set the batch size to one to make sure that each sample is seen individually

> seen

Hello @clefourrier @NathanHB, thanks for your replies. Do you support customizing the model prompt for every sample? Let's say my model is M and I have two questions in MMLU Q1 and Q2. 

For Q1, I want to prompt I1 + Q1 -> M
For Q2, I want to prompt I2 + Q2 -> M

Does lighteval have any python interface supporting the above example?

Greatly thanks

You could use a custom task and make the prompt conditional on a sample index for example :) 

I see, thanks!
Repository: huggingface/lighteval
Issue: [EVAL]: Add more African Benchmarks
Root cause: Unimplemented Feature Gap

## Evaluation short description
- Why is this evaluation interesting?
This focuses on 16 African languages, evaluated on three knowledge QA and reasoning tasks such as AfriMMLU, AfriMGSM and AfriXNLI, human translated from MMLU, MGSM and XNLI respectively. 

- How used is it in the community?

## Evaluation metadata: IrokoBench
Provide all available
- Paper url: https://arxiv.org/abs/2406.03368
- Github url: 
- Dataset url: https://huggingface.co/collections/masakhane/irokobench-665a21b6d4714ed3f81af3b1 

## Evaluation metadata: Uhura
Provide all available
- Paper url: 
- Github url: 
- Dataset url: https://huggingface.co/datasets/ebayes/uhura-arc-easy-clean ,  https://huggingface.co/datasets/ebayes/uhura-truthfulqa-clean 

## Evaluation metadata: SIB-200
Provide all available
- Paper url: https://aclanthology.org/2024.eacl-long.14/ 
- Github url: https://github.com/dadelani/sib-200 
- Dataset url: https://huggingface.co/datasets/Davlan/sib200


@NathanHB

Hi @dadelani , thanks for your issue!

This sounds like a great idea, do you want to add them yourself to the library and open a PR? We've got a guide [here](https://github.com/huggingface/lighteval/wiki/Adding-a-Custom-Task) for adding tasks and one [here](https://github.com/huggingface/lighteval/wiki/Contributing-to-multilingual-evaluations) specific to multilingual evaluations :)

Hi,
Iroko bench is already implemented

See: the [tasks.py](https://github.com/huggingface/lighteval/blob/main/src/lighteval/tasks/multilingual/tasks.py#L239)
Currently only the swahili subset is available, because we don't have correct translations for anchor words, that are required during the evaluation (e.g. answer/question).

If you are a native speaker in any of the languages we would love your help in improving that!
See the [guide](https://github.com/huggingface/lighteval/wiki/Contributing-to-multilingual-evaluations#contributing-a-small-translation) how to add a translations: 

Great, thank you @clefourrier and @hynky1999 . 

For the anchor words, maybe you can use google translate of the keywords. I can provide for Yoruba below:

Question: Ìbéèrè
Answer: Ìdáhùn

Great ! do you think you can open a PR for the evals and the keywords ? We will not use google translate as it can be unreliable.

As Clementine said, the documentation to add a translation is here:
https://github.com/huggingface/lighteval/wiki/Contributing-to-multilingual-evaluations