zhiminy/EvalEng huggingface dataset:
Repository: huggingface/lighteval
Issue: [BUG] Wrong environment variable used for cache directory
Root cause: Configuration Error

## Describe the bug
The env config is using the wrong environment variable to get the cache directory. This leads to duplicate model/asset downloads.

## To Reproduce
The [cache_dir](https://github.com/huggingface/lighteval/blob/main/src/lighteval/utils/utils.py#L195) value uses `HF_HOME` environment variable

## Expected behavior
Uses [HF_HUB_CACHE](https://huggingface.co/docs/huggingface_hub/main/en/package_reference/environment_variables#hfhubcache) environment variable

## Version info
github `main`


Good catch! Let's update this!
Repository: huggingface/lighteval
Issue: [FT] Evaluation using a multi-document RAG based on statistical tools and LLM as judge
Root cause: Unimplemented Feature Gap

## Issue encountered
It would be good to have a system for evaluating both the relevance of the RAG and its use by the LLM in producing the response. My first intuition would be a multi-stage system with, on the one hand, an evaluation in the manner of Sentence Transformers' information-retrieval evaluator (@tomaarsen), and on the other hand, an evaluation with certain metrics of the model's ability to interpret the documents from the context provided by the RAG system (mutli-documents) in order to test :
- Answer Relevancy
- Faithfulness
- Contextual Precision
- Contextual Recall
- Contextual Relevancy
- Tool Correctness
- Hallucination

## Solution/Feature
For the moment, it seems to me that the simplest way to implement this system is to combine two tools, sentence-transformers and DeepEval, and then aggregate the results in the form of a synthetic indicator, but this method requires a multitude of libraries and a connection between the database necessary for the sentence-transformers and DeepEval evaluators.

On the information retrieval (IR) side of things, we have a solid and extensive leaderboard called BEIR (Benchmark for Information Retrieval), which is included in MTEB (Massive Text Embedding Benchmark) under the Retrieval tab: https://huggingface.co/spaces/mteb/leaderboard. This will soon be extended to MMTEB with more languages, etc.

However, as you're noticing, this is only one part of the puzzle for RAG applications. I'm not very familiar with the various RAG evaluation tools out there, so I can't provide much feedback on that part I'm afraid.

- Tom Aarsen

Hi ! Thanks for the issue. I'm not very familiar either with RAG evaluation and adding it to lighteval would reqquire some effort. If you are familiar with it, we would love it if you could open a PR with a Proof of Concept !

To adding RAG would require adding a RAG evaluation function in the `base_model.py` model, as well as providing the documents in the requests. Do you have a particular benchmark to add in mind ?
Repository: huggingface/lighteval
Issue: [FT] Is it possible to save the predictions to prevent rerunning expensive inference
Root cause: Documentation Deficiency

## Issue encountered
When evaluating large models, significant costs and delays can occur for inference, especially on larger datasets. Possibly I want to re-evaluate my predictions using different metrics. 

## Solution/Feature
I want the predictions to be saved in an inspectable cache which can be used when the evaluation is run again.


Hi, thanks for the issue! 

If you use the different saving parameters (as indicated in the [doc](https://github.com/huggingface/lighteval/wiki/Saving-and-reading-results)), your predictions (results and/or details) are saved and can be used for reinspection later on. The quickest way to get what you need is therefore using the details file to recompute the metrics on them by hand. 

Since not all metrics use the same generation methods, we have not prioritized a cache atm (to prevent risks such as running a greedy eval, then a sampling one, and accidentally using the same results for metric computations), but we'll add your suggestion to our todo!

Great, thanks so much!
Repository: huggingface/lighteval
Issue: [FT] Add Gemba MQM Translation Metric
Root cause: Unimplemented Feature Gap

## Issue encountered
The metrics only include rather outdated translation metrics. 

## Solution/Feature
[Gemba MQM](https://aclanthology.org/2023.wmt-1.64.pdf) seems to be a current metric. Adding it would make translation evaluation better.



Looks nice ! This is using external APIs and does not seem to have a PyPI package so we would need to implement it in Lighteval. This not not high priority but if you need it feel free to open a PR and we can help you set it up :)

Great, thanks! 
Yes, I see two avenues:
1. Fork their repo, publish a pip package and integrate it like that.
2. Just copy their prompts and post processing functions.

IMO option 1 is cleaner and also allows other people to use the metric more easily. 

@chuandudx Would you be interested in taking this?

This is done: I forked the repo (https://github.com/joelniklaus/gemba) and deployed it to pypi (https://pypi.org/project/gemba/). I also included it in PR #389. We can also think about pulling some metrics from this PR to the metrics_sample so everyone can use them afterwards.
Repository: huggingface/lighteval
Issue: Zero scores on SIQA benchmark from HELM with microsoft/phi-1_5 model
Root cause: Algorithmic Error

When running the evaluation of Phi1.5 on SIQA benchmark from HELM, I get zero scores
```bash
 accelerate launch --multi_gpu --num_processes=8 run_evals_accelerate.py --model_args "pretrained=microsoft/phi-1_5,trust_remote_code=True"  --tasks "helm|siqa|0|0"  --override_batch_size 1 --save_details --output_dir=phi
```
Output
```bash
|   Task    |Version|Metric|Value|   |Stderr|
|-----------|------:|------|----:|---|-----:|
|helm:siqa:0|      0|em    |    0|±  |     0|
|           |       |qem   |    0|±  |     0|
|           |       |pem   |    0|±  |     0|
|           |       |pqem  |    0|±  |     0|
|all        |      0|em    |    0|±  |     0|
|           |       |qem   |    0|±  |     0|
|           |       |pem   |    0|±  |     0|
|           |       |pqem  |    0|±  |     0|
```

Just checked and it's actually not good in zero shot, it's generating "\n" systematically
Repository: huggingface/lighteval
Issue: [FT] Support llama.cpp inference
Root cause: Unimplemented Feature Gap

## Issue encountered
Currently, inference of open models on my Mac device is quite slow since vllm does not support mps.

## Solution/Feature
Llama.cpp does support mps and would significantly speed up local evaluation of open models.

## Posssible alternatives
Allowing the use of the mps device in other ways of loading models would also work.


Hi! Feel free to open a PR for this if you need it fast as our roadmap for EOY is full :)

Sounds good. Might do at some point, for now it is not a priority for me.

would be an awesome feature IMO! cc @gary149
Repository: huggingface/lighteval
Issue: [FT] The word "pretrained" is required in model_args but not in model_config_path
Root cause: Architectural Constraint

## Issue encountered
We need to pass "pretrained:MODEL_ID" in --model_args, but this is not necessary in --model_config_path? Can we remove it?

## Solution/Feature
It should be handled identically in both cases: without the word "pretrained"?

As discussed with @clefourrier.


More generally, should homogeneize all configs
Repository: huggingface/lighteval
Issue: [BUG] Bootstrapping compute's stderr is very slow
Root cause: Algorithmic Error

## Describe the bug
Bootstrapping compute's stderr. is very slow (in this instance taking 25min to complete). Is there a way to speed this up?

<img width="1137" alt="Screenshot 2024-11-27 at 15 26 43" src="https://github.com/user-attachments/assets/152a4072-f9bb-4502-adca-db1245e8cde7">

## To Reproduce
Check out the branch [add_swiss_legal_evals](https://github.com/JoelNiklaus/lighteval/tree/add_swiss_legal_evals) and run the following command:
```
python -m lighteval accelerate \
  --model_args openai,model=gpt-4o-mini \
  --tasks community|sdst-text_level:de-fr|3|0 \
  --custom_tasks lighteval/community_tasks/swiss_legal_evals.py \
  --output_dir outputs \
  --override_batch_size 1 \
  --save_details 
```

## Expected behavior
It should complete much faster, maybe using multiprocessing to speed it up. Could we look up the already computed metrics per sample instead of recomputing the metrics for each sample? Especially for expensive metrics this makes a huge difference.

## Version info
Happening on both a MacBook Pro 16-inch, Nov 2023 and on a Ubuntu 22 server.


Hi ! I agree that it should not be that slow. Not sure why we recompute on every sample, we should not need to since they are already computed, @clefourrier any idea why ? 🤔 

Because some part of the computation happens at the aggregated sample level. Basically, each metric is computed in 2 steps:
- sample level
- aggregation (usually, it's a mean over all the corpus).

For some metrics however, the sample level step is nothing, and everything is computed at the corpus level (typically the case for corpus BLEU).

When bootstrapping, we re-sample from our population of samples, and recompute the aggregation over the new population (and do this 1K times iirc). If the most expensive part of the computation is the aggregation step, then it becomes costly.

Oh yeah the corpus level can only be recomputed I forgot, thanks !
Repository: huggingface/lighteval
Issue: [FT] Add System Prompt field in LightevalTaskConfig that can be used by model clients
Root cause: Unimplemented Feature Gap

## Issue encountered
My models currently don't follow the template I give. I want to give a system prompt that nudges the models to provide output the way I want it.

## Solution/Feature
We could add a new field in LightevalTaskConfig that can be consumed by model clients like LiteLLMClient to send to the API.

## Possible alternatives
Alternatively, we could use the generation_grammar, but I think this may be a bit overkill and may be more difficult to implement for different API providers. 


@clefourrier Happy to open a PR for this. Please let me know if you see a different solution.

Hi! I'm adding auto-scaling for inference endpoints atm, and will edit the generation mechanisms after that, possibly to get a single homogeneous system - if you need it super fast feel free to add it before, but I'll probably edit the system next week :)

Ok, I see. Thanks for letting me know. I will use a hack in the meantime in that case.

Just briefly asking to double check: This edit to the generation mechanism will also affect the openai/litellm model loaders?

Yes the aim is to have one better centralised system 

Need to add to #428 

Hi @JoelNiklaus , from what I can see, it's already available for all models with the latest CLI refacto: there's a system prompt param, which is then used when creating the requests from the samples. Do you need stg else wrt this?

I saw that, thanks. However, to me it is still not clear how to use the system prompt in the model_loader (e.g., litellm). The system prompt is not part of GreedyUntilRequest. Also it does not come through with the model_config.

Because you should not use it in the model_loader ^^
The system prompt is added to the request by the PromptManager, which creates the request from the sample, number of few shots, and system prompt if needed. (It's in ligtheval/tasks/prompt_manager)

The system prompt will appear in the details if you want to check this

Haha ok. Sorry if a stupid question, but how can I access the system prompt before I make a call to the API?
https://github.com/huggingface/lighteval/blob/main/src/lighteval/models/openai_model.py#L82

As discussed on Slack the PromptManager might need to be adapted to make passing through the system prompt possible.

As of now there is no way to only grab the system prompt. One solution would be to add a `system_prompt` field to the requests when creating them in the `PromptManager` so that you can access them in `litellm_model`

Sounds good. I enabled it in PR #385.
Repository: huggingface/lighteval
Issue: [FT] Numpy 2.0 support
Root cause: External Dependency Breakage

## Issue encountered
lighteval does not support numpy 2.0

https://huggingface.slack.com/archives/C055130UYQG/p1733147324129099?thread_ts=1733143922.334659&cid=C055130UYQG


`ml-dtypes>=0.5` (https://github.com/jax-ml/ml_dtypes/blob/v0.5.0/pyproject.toml#L24) requires `numpy>=2` for Python 3.13.

This means `lighteval` is not resolvable in certain environments now due to requiring `numpy` v1.

Gently pinging @clefourrier @NathanHB again as this is an important dependency issue to resolve IMO

Will do it this morning after I wrap up caching, thanks a lot for the ping

It's been done in #913
Repository: huggingface/lighteval
Issue: [FT] Adding caching for each dataset run
Root cause: Unimplemented Feature Gap

## Issue encountered
When running large evals with many dataset configurations it is very painful to rerun everything in case something fails. 

## Solution/Feature
It would be great if intermediate results could be cached, for example the computed metrics of each dataset. 



I am just looking out to contribute in HF repo. Just trying to understand code base and see if I can add this feature. 

@punitvara We would want results to be saved after each batch run, and to be reused if an evaluation is launched with the exact same parameter configuration.
TBH, it's not a trivial PR to work on - if you're unfamiliar with lighteval, I would suggest working on #324, #325, or maybe #355 to get to know the code base first
Repository: huggingface/lighteval
Issue: [DOC] Add missing models to docs
GitHub labels: documentation

Add missing models to docs: `docs/source/package_reference/models.mdx`
- AdapterModel
- NanotronLightevalModel
- VLLMModel

Currently they raise import errors.

Discussed in:
- https://github.com/huggingface/lighteval/pull/413#issuecomment-2513735282
Repository: huggingface/lighteval
Issue: [BUG] Issue with Serverless HuggingFace Inference
Root cause: Validation Gap

## Describe the bug

Running Serverless HF Inference as instructed in [Documentation](https://github.com/huggingface/lighteval/wiki/Evaluate-the-model-on-a-server-or-container#text-generation-inference-tgi) fails:

```
File "/home/tobias/src/tamingLLMs/tamingllms/.venv/lib/python3.12/site-packages/lighteval/models/tgi_model.py", line 70, in __init__
    model_precision = self.model_info["model_dtype"]
                      ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
KeyError: 'model_dtype'
```

Next, if you manually set

```python
model_precision = "float16"
```
in `lighteval/models/tgi_model.py", line 70`

You get a second error:

`ValueError: batch_size should be a positive integer value, but got batch_size=-1`

which then gets resolved if you pass `--override_batch_size 1` to lighteval bash command.


## To Reproduce
```
lighteval accelerate  --model_config_path="endpoint_model.yaml" --tasks "leaderboard|mmlu:econometrics|0|0" --output_dir="./evals/"
```

endpoint_model.yaml:
`
model:
  type: "tgi"
  instance:
    inference_server_address: "https://api-inference.huggingface.co/models/Qwen/Qwen2.5-Math-1.5B-Instruct"
    inference_server_auth: "<API-KEY>"
    model_id: null
`



## Expected behavior

Run benchmark using Serverless HF Inference

## Version info

python = "^3.11"
lighteval = {extras = ["accelerate"], version = "^0.6.2"}

## Other Suggestions

The user keeps getting ‘model is currently loading’

One recommendation would be to leverage `wait_for_model` param to avoid breaking the command line call if the model is still loading.



Hi! Can you try with the code of #445 ?
Repository: huggingface/lighteval
Issue: [FT] Run on Server using Python API
Root cause: Documentation Deficiency

## Solution/Feature

Ability to [Run on Server](https://github.com/huggingface/lighteval/wiki/Evaluate-the-model-on-a-server-or-container) using [Python API](https://github.com/huggingface/lighteval/wiki/Using-the-Python-API)


Hi! Can you describe a bit more what you would like to see? You can run on server using a model config from a model path, no?

Hi, thanks for the follow up.

Ate you saying we can simply pass a tgi config yaml file to `model_config`
param of Pipeline class like below?

Could you please provide a minimally reproducible example since I belive
this scenario is not covered in the docs. Thank you!

```
pipeline = Pipeline(
        tasks=task,
        pipeline_parameters=pipeline_params,
        evaluation_tracker=evaluation_tracker,
        model_config=config.yaml,
        custom_task_directory=None, # if using a custom task
    )
```

On Mon, Dec 9, 2024, 5:03 AM Clémentine Fourrier ***@***.***>
wrote:

> Hi! Can you describe a bit more what you would like to see? You can run on
> server using a model config from a model path, no?
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/lighteval/issues/424#issuecomment-2527203461>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ADTMY3OTCFXSVVB2T5UHEED2EVFFRAVCNFSM6AAAAABTDUBRXKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKMRXGIYDGNBWGE>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>


You can look at how it's done in the main_endpoint.py file :)
Repository: huggingface/lighteval
Issue: [FT] Enable the evaluation of any function
Root cause: Unimplemented Feature Gap

## Issue encountered
I need to evaluate google translate and other methods that do not fit in any of the evaluation modes supported currently. 

## Solution/Feature
I am not sure what the best way to do this is, but I see the following general solution: 
We add one ModelConfig, where the user can specify a function that is called for a batch of examples. As long as the user provides the output correctly formatted, they can evaluate any system they like. It does not need to fit into the model_configs supported so far.
Happy to propose a PR for this.
Repository: huggingface/lighteval
Issue: [FT] Fail faster when passing unsupported metrics to InferenceEndpointModel
Root cause: Validation Gap

## Issue encountered
When passing an unsupported metric (e.g. single-token metric) to `InferenceEndpointModel`, an error is raised from `Pipeline.evaluate`:
```
ValueError: Endpoint models can't use single token metrics. Change the metric to the standard version
```
This happens once the endpoint has been created and the model has been created and loaded.

I wonder if the error could be raised earlier for optimization reasons: the process fails faster and resources are not wasted.

## Solution/Feature
Raise the error before the endpoint has been created.

## Possible alternatives
Leave it as it is.


Hm, good idea for the checks, especially for costly metrics. We would also want the LLM as judge to stop if the open ai key is not provided for example.

Also linked to https://github.com/huggingface/lighteval/issues/355
Repository: huggingface/lighteval
Issue: [FT] Align parameter names in config files and config classes
Root cause: Configuration Error

## Issue encountered
Currently, the example config files have a parameter named `dtype`.

When parsing the config file into the config class, this parameter is renamed to `model_dtype`.


## Solution/Feature
Align the name to `model_dtype` in both config files and config classes.

### Context:
In the config files
- it already exists a `model` parameter called `model_name`
- there are also `instance` parameters called `instance_type` and `instance_size`

This was discussed in:
- https://github.com/huggingface/lighteval/pull/434?new_mergebox=true#issue-2732500452

## Possible alternatives
Leave it as it is.


That works for me if you want to homogeneize the code across the models for this :)
Repository: huggingface/lighteval
Issue: [BUG] how to eval large scale model use 1dp+8pp?
GitHub labels: bug

## Describe the bug
I tired to eval a large scale model use1dp+8pp with accelerate. I use the command like the following:
```
accelerate launch --multi_gpu --num_processes=1 run_evals_accelerate.py \
    --model_args="pretrained=<path to model on the hub>" \
    --model_parallel \
    --tasks <task parameters> \
    --output_dir output_dir
```
The error is ```ValueError: You need to use at least 2 processes to use --multi_gpu```

How to solve this problem?

## Version info
lighteval-0.3.0


Hi ! to use DP = 1 you need to run wihtout accelerate. In your case the command would be:

```
lighteval accelerate "pretrained=gpt2,model_parallel=True" "leaderboard|truthfulqa:mc|0|0"
```

The command you described above is deprecated, Where did you find it ?

https://github.com/huggingface/lighteval/blob/v0.3.0/README.md#evaluating-a-large-model-with-pipeline-parallelism

I use an old version. Does it support DP = 1 + PP=K?

I updated lighteval to V0.6.0 and tried again.
I used the following command:
```
export CUDA_VISIBLE_DEVICES=0,1,2,3 
lighteval accelerate "pretrained=<model_path>,model_parallel=True" "leaderboard|truthfulqa:mc|0|0"
``` 
It failed to load the model with MP, and only GPU 0 was occupied. 
```
[2024-12-25 03:29:43,045] [    INFO]: Model parallel was set to False, max memory set to None and device map to None (base_model.py:389)
```
I found 'LOCAL_WORLD_SIZE' and 'WORLD_SIZE' was none.
How to solve this problem?

Hi ! You are right, I opened a PR you can checkout it should solve the issue.
Repository: huggingface/lighteval
Issue: [EVAL] Add JGLUE Test
Root cause: Unimplemented Feature Gap

JGLUE is a widely used test set in the Japanese LLM research community, consisting of five sub-tests (with MARC-ja removed due to a request from Amazon):

- MARC-ja
- JSTS
- JNLI
- JSQuAD
- JCommonsenseQA

Most of these sub-tests have English counterparts, and all of these are available for use under the `CC-BY-SA-4.0` license.

JGLUE does not have an official dataset mirror on Hugging Face, and some of the tests lack community mirrors as well. I am currently processing and uploading the four sub-tests that remain accessible.

Due to performance issues with `llm-jp-eval` on my device, I am working on integrating the test sets used by `llm-jp-eval` into `lighteval`. If successful, this integration could greatly improve the evaluation of Japanese LLMs.

## Evaluation metadata
Provide all available
- Paper url:
- Github url: https://github.com/yahoojapan/JGLUE
- Dataset url:


Hi! This sounds good! You might encounter specific issues related to tokenization, so it would be good to double check this when it's added :)

Great ! Don't hesitate to refer to the documentation to add your custom tasks.
https://huggingface.co/docs/lighteval/adding-a-custom-task
Repository: huggingface/lighteval
Issue: [BUG] Issue with CACHE_DIR Default Value in Accelerate Pipeline
Root cause: Configuration Error

## Describe the bug
The default value for CACHE_DIR in the pipeline creates several issues. On many systems, such as Ubuntu distributions, users typically lack permissions for the root directory (`/`), which prevents the pipeline from running. On Windows, it creates a `scratch` folder on the C drive, which is not ideal.

https://github.com/huggingface/lighteval/blob/0977fde60ba3564cf34c53606a53962d81eef7b0/src/lighteval/main_accelerate.py#L34

## Proposed Solution
Remove the default value for CACHE_DIR and instead return None. This will allow the pipeline to follow the default logic of the transformers library.

``` python
CACHE_DIR: str = os.getenv("HF_HOME")
```

## To Reproduce
Run the example provided in the README on any Linux distribution:

``` sh
HF_HOME= lighteval accelerate \
    "pretrained=gpt2" \
    "leaderboard|truthfulqa:mc|0|0"
```

## Expected behavior
The running should execute without errors.

## Version info
The issue was encountered on the `main` branch installed with `pip install -e .[dev]`.