zhiminy/EvalEng huggingface dataset:
Repository: huggingface/lighteval
Issue: [FT] Faster generation with TransformersModel by using less padding
Root cause: Algorithmic Error

## Issue encountered

I noticed that the `greedy_until` function in `TransformersModel` uses excessive padding. In my case, I have a test set where my largest input has 27k tokens but most of the inputs are under 8k tokens. The current implementation uses `max_context_continuation_size_allowed` as the `max_length` in the tokenizer, which corresponds to the number of tokens for the largest samples in the **entire dataset** plus the maximum number of output tokens. This unnecessarily increases the evaluation time. 

## Solution/Feature
Instead of using `max_context_continuation_size_allowed` when tokenizing the batch contexts, it would be better to use something like this (untested):

```python
largest_sample_in_batch = len(batch[0].tokenized_context) 
max_generation_size = batch[0].generation_size if batch[0].generation_size else self.max_length - largest_sample_in_batch
max_length = min(largest_sample_in_batch + max_generation_size, self.max_length)

tokenized = self.tokenizer(
   ...
    max_length=max_length   # Only this needs to change
   ...
).to(self.device)
```

The calculations are essentially the same as the ones being done already in the code, only that we don't look at the first sample in the entire dataset but the first sample in the batch for determining the `max_length`.

If you think this makes sense, I could open a pull request.


Hi ! We closed a PR #663  that fixes padding for transformers and opened one for continuous batching #714
Repository: huggingface/lighteval
Issue: evaluation error: TypeError: EngineArgs.__init__() got an unexpected keyword argument 'worker_use_ray'
Root cause: External Dependency Breakage

The evaluation script using only 1 GPU works out fine. However, when employing data data_parallel_size >1. an error shows up.

**TypeError: EngineArgs.__init__() got an unexpected keyword argument 'worker_use_ray'**


```

RayTaskError(TypeError): [36mray::run_inference_one_model()[39m (pid=718457, ip=10.137.8.13)
  File "/home/huayuc/miniconda3/envs/openr1/lib/python3.11/site-packages/lighteval/models/vllm/vllm_model.py", line 336, in run_inference_one_model
    llm = LLM(**model_args)
          ^^^^^^^^^^^^^^^^^
  File "/home/huayuc/miniconda3/envs/openr1/lib/python3.11/site-packages/vllm/utils.py", line 1039, in inner
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/huayuc/miniconda3/envs/openr1/lib/python3.11/site-packages/vllm/entrypoints/llm.py", line 210, in __init__
    engine_args = EngineArgs(
                  ^^^^^^^^^^^
**TypeError: EngineArgs.__init__() got an unexpected keyword argument 'worker_use_ray'**
(pid=718453) INFO 02-03 16:25:25 __init__.py:183] Automatically detected platform cuda. [repeated 3x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/user-guides/configure-logging.html#log-deduplication for more options.)
(pid=718453) WARNING 02-03 16:25:25 cuda.py:333] Detected different devices in the system:  [repeated 3x across cluster]
(pid=718453) WARNING 02-03 16:25:25 cuda.py:333] NVIDIA A100-SXM4-80GB [repeated 12x across cluster]
(pid=718453) WARNING 02-03 16:25:25 cuda.py:333] NVIDIA DGX Display [repeated 3x across cluster]
(pid=718453) WARNING 02-03 16:25:25 cuda.py:333] Please make sure to set `CUDA_DEVICE_ORDER=PCI_BUS_ID` to avoid unexpected behavior. [repeated 3x across cluster]
```

My system setting:
`vllm==0.7.1.  torch==2.5.1.  lighteval == 0.6.0.dev0
`


Command

```
MODEL=deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B
NUM_GPUS=1
MODEL_ARGS="pretrained=$MODEL,dtype=float16,data_parallel_size=$NUM_GPUS,max_model_length=32768,gpu_memory_utilisation=0.8"
TASK=math_500
OUTPUT_DIR=data/evals/$MODEL

CUDA_VISIBLE_DEVICES=0,1,2,3 lighteval vllm $MODEL_ARGS "custom|$TASK|0|0" \
    --custom-tasks src/open_r1/evaluate.py \
    --use-chat-template \
    --system-prompt="Please reason step by step, and put your final answer within \boxed{}." \
    --output-dir $OUTPUT_DIR 
```

Any idea why this happens? Thank you!

It seems `vllm` deprecated `worker_use_ray` in favour of `distributed_executor_backend` in [this PR](https://github.com/vllm-project/vllm/pull/12383/files#diff-ea8b8ff63961713ccb62d78e53e96404b587b7828cb9fee08a9e5576bf563673L395). I'll fix this if @hynky1999 doesn't beat me to it :)
Repository: huggingface/lighteval
Issue: More flexibility in parameters for OpenAI / LiteLLM
Root cause: Architectural Constraint

1. Why not expose entire LiteLLM config as parameter or something when running evals?
2. I'm actually looking to run [open-r1 evals](https://github.com/huggingface/open-r1) with any api provider. This don't seem possible with current. The OpenAIClient class should also allow using any base_url. Parameter like temperature should be available to modify directly.

![Image](https://github.com/user-attachments/assets/93552172-92bb-4a08-93b4-dcfdb2bfdc45)

+1 also it would be great to have examples of usage with openai endpoints and LiteLLM.

and [Inference Providers on the Hub](https://huggingface.co/blog/inference-providers)

I saw @NathanHB and @gary149 just [committed some improvements](https://github.com/huggingface/lighteval/commit/d6de1fe221172c79b49f4d08d5009e03ec65982b) so that various configuration options can be submitted via a YAML file (gj!), which is workable, but I think that proper flags or a way to submit via command line would still be useful for building eval pipelines (vs having to add YAML file generation into my scripts, which is a big bloat/PITA).

hi ! we added some more options for inference providers openai and litellm, it should also be much easier to add more options now as we switched to pydantic (simply modify the `model_config` in the right model file.
Repository: huggingface/lighteval
Issue: couldn't find it in the cached files and it looks like Elron/bleurt-tiny-512, how to set the model path?
Root cause: Configuration Error

How to set the eval model path?
## Eval
when I use the script to eval model  with MATH-500

`NUM_GPUS=8 # Set to 8 for 32B and 70B models
MODEL=Deepseek_R1_distill/Qwen2.5-32B-Open-R1-Distill/
MODEL_ARGS="pretrained=$MODEL,dtype=bfloat16,max_model_length=32768,gpu_memory_utilisation=0.8,tensor_parallel_size=$NUM_GPUS"
OUTPUT_DIR=data/evals/Qwen2.5-32B-Open-R1-Distill

lighteval vllm $MODEL_ARGS "custom|math_500|0|0" \
    --custom-tasks src/open_r1/evaluate.py \
    --use-chat-template \
    --output-dir $OUTPUT_DIR
`


##  Error
Error: We couldn't connect to 'https://huggingface.co' to load this file, couldn't find it in the cached files and it looks like Elron/bleurt-tiny-512 is not the path to a directory containing a file named 
config.json.

Where to set the eval model path in the script?

You shouldn't need the model, it should be lazy loaded now https://github.com/huggingface/lighteval/pull/536
Repository: huggingface/lighteval
Issue: [BUG] tiktoken is not optional dependency
Root cause: Validation Gap

## Describe the bug
When installing from current branch, I am getting error that the tiktoken was not found

## To Reproduce
Install from source and run any eval
```
2025-02-08 20:46:12.232490: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1739047572.447031    1808 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1739047572.507275    1808 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-02-08 20:46:12.954291: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
Traceback (most recent call last):
  File "/content/./Math-Verify/evaluate_model.py", line 5, in <module>
    from lighteval.pipeline import ParallelismManager, Pipeline, PipelineParameters
  File "/usr/local/lib/python3.11/dist-packages/lighteval/pipeline.py", line 39, in <module>
    from lighteval.models.model_loader import TransformersModel, load_model
  File "/usr/local/lib/python3.11/dist-packages/lighteval/models/model_loader.py", line 32, in <module>
    from lighteval.models.endpoints.openai_model import OpenAIClient, OpenAIModelConfig
  File "/usr/local/lib/python3.11/dist-packages/lighteval/models/endpoints/openai_model.py", line 56, in <module>
    import tiktoken
ModuleNotFoundError: No module named 'tiktoken'
```

## Expected behavior
Work without having tiktoken installed:


tiktoken is optional but not if you have openai installed, as you can see [here](https://github.com/huggingface/lighteval/blob/35d73d5b36a22c4fae64e66c4bfe1e785c2c8b45/src/lighteval/models/endpoints/openai_model.py#L57)

I see, closing
Repository: huggingface/lighteval
Issue: how can i use this "community|alghafa:meta_ar_dialects " as a task
Root cause: Documentation Deficiency


Hello, I am getting this error after running this code 
`qwen_model` = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B")

pipeline = Pipeline(
    tasks=domain_tasks,
    pipeline_parameters=pipeline_params,
    evaluation_tracker=evaluation_tracker,
    model=qwen_model
)

pipeline.evaluate()

qwen_results = `pipeline.get_results()`
ValueError: Cannot find tasks community|alghafa:meta_ar_dialects in task list or in custom task registry)
How can i solve this please help me out.
Thanks, Manas. 

hi ! you are using the wrong task name, closing this for now feel free to reopen if you have more issues.
Repository: huggingface/lighteval
Issue: [BUG] Nanotron runner imports non-existant
Root cause: Validation Gap

## Describe the bug
Attempting to run a nanotron eval (`lighteval nantron`) will result in an error since `hierarchical_logger` does not exist. See https://github.com/huggingface/lighteval/blob/main/src/lighteval/main_nanotron.py#L55 

## To Reproduce
Run 
```
lighteval nanotron --checkpoint-config-path some-nanotron-path --lighteval-config-path some-lighteval-path
```

## Expected behavior
Nanotron eval works

## Version info
Reproduced on fac17bb634e5c3411cbabbc655b3ead86fd0fc0a

Follow up -- if you remove `hierarchical_logger`, https://github.com/huggingface/lighteval/blob/main/src/lighteval/pipeline.py#L73 errors. Looks like it was renamed to `lighteval.models.nanotron.nanotron_model`

Have this issue too. Can confirm that the fixes provided by @jquesnelle work. 1) remove the htrack_block import and code block in main_nanotron.py and 2) rename the import in pipeline.py.

great ! thanks for the fix, can you open a PR so that we can review and merge ? :)
Repository: huggingface/lighteval
Issue: [BUG] Use sampling for greedy mode
Root cause: Validation Gap

## Describe the bug
A clear and concise description of what the bug is.

https://github.com/huggingface/lighteval/blob/d6de1fe221172c79b49f4d08d5009e03ec65982b/src/lighteval/models/vllm/vllm_model.py#L281

The `generate` is True if this call does not give the generate parameter. Then the `_generate` will use the default temperature=1.0 for decoding.

## To Reproduce
Please provide all the steps needed to reproduce the behavior, or provide a minimal working example if needed. We will ignore issues missing this section.

## Expected behavior
A clear and concise description of what you expected to happen.

## Version info
Please provide your operating system, lighteval version or commit if you installed from main, and pip/conda environment if your problem concerns dependencies.


hi ! what's the expected behaviour and what is your issue here ?
Repository: huggingface/lighteval
Issue: Append revision to filepath in `--output_dir`?
Root cause: Architectural Constraint

Currently, `lighteval` stores results/details in a path that is determined by the model name, e.g.

```shell
scratch/evals
├── details
│   └── Qwen
│       └── Qwen1.5-0.5B-Chat
│           ├── 2024-02-26T15-36-31.681219
│           │   └── details_lighteval|truthfulqa:mc|0_2024-02-26T15-36-31.681219.parquet
│           └── results_2024-02-26T15-36-31.681219.json
└── results
    └── Qwen
        └── Qwen1.5-0.5B-Chat
            └── results_2024-02-26T15-36-31.681219.json
```

However, I am quite often evaluating models with different _revisions_ and the current save logic groups these all together in the same subfolder which makes it hard to determine which result corresponds to which run. 

Would it make sense to append the model revision parameter to the filepaths, e.g. something like this for the `main` revision (or whatever is passed to the `revision` arg in the script):

```shell
scratch/evals
├── details
│   └── Qwen
│       └── Qwen1.5-0.5B-Chat
│           └── main
│               ├── 2024-02-26T15-36-31.681219
│               │   └── details_lighteval|truthfulqa:mc|0_2024-02-26T15-36-31.681219.parquet
│               └── results_2024-02-26T15-36-31.681219.json
└── results
    └── Qwen
        └── Qwen1.5-0.5B-Chat
            └── main
                └── results_2024-02-26T15-36-31.681219.json
```

My current workaround is to manually specify the model path in `--output_dir={ORG}/{MODEL_ID}/{REVISION}` and then glob the files. This is fine, but a bit clunky because one ends up with a long nested path like `{ORG}/{MODEL_ID}/{REVISION}/results/{ORG}/{MODEL_ID}`


Would `{ORG}/{MODEL_ID}_{REVISION}` work for you?
I think it would allow you to get the info you need, without creating too many nested levels for users for which it would be irrelevant.
If that would, I can add it to our logging.

> Would `{ORG}/{MODEL_ID}_{REVISION}` work for you? I think it would allow you to get the info you need, without creating too many nested levels for users for which it would be irrelevant. If that would, I can add it to our logging.

That would be perfect! Note that using the revision (not necessarily the SHA) would be ideal so that one can distinguish e.g. branches or tags like `vX.Y` from each other. I realise this is quite a niche ask, so don't worry if it's too annoying to include

Actually after thinking about this a bit more, a somewhat better approach IMO would be to let the user specify the filepath in `--output_dir`, after which we dump the details/results there. This way people like me can nest the results as desired and we don't need to hardcode more logic :)

We expect the results to follow a specific path when pushed to the hub, which is one of the reasons why we have this nested architecture. At which level would you want to have control over the path?

The revision information could also appear in the .json output in "config_general".
Repository: huggingface/lighteval
Issue: Lighteval endpoint openai
Root cause: Documentation Deficiency

## Issue encountered
Hi HF Team, I want to use NVIDIA api to evalute deepseek R1.

The api call format is like

client = OpenAI(
    base_url="https://integrate.api.nvidia.com/v1",
    api_key="NVIDIA_API_KEY"
)

Can I Use this format to use lighteval endpoint openai? I am not sure how to pass the api_key and base_url into the endpoint

Thanks




Hi ! you can instead use `lighteval litellm`, documentation is [here](https://huggingface.co/docs/lighteval/main/en/use-litellm-as-backend)
Repository: huggingface/lighteval
Issue: Weird cuda OutOfMemoryError error
Root cause: Resource Mishandling

Hi

I need some helps.

I am trying to evaluate my pre-trained model with Thai fine tasks. Here is my command

```
export CUDA_VISIBLE_DEVICES="0,1"
echo "Running lighteval for model: meta-llama/Llama-3.2-3B"
lighteval accelerate \
"pretrained=meta-llama/Llama-3.2-3B,dtype=bfloat16,model_parallel=True" \
"examples/tasks/fine_tasks/mcf/th.txt" \
--custom-tasks "src/lighteval/tasks/multilingual/tasks.py" \
--dataset-loading-processes 8 \
--cache-dir "./le_cache" \
--no-use-chat-template \
--override-batch-size 4
```

When I ran this command I got error 
```
OutOfMemoryError: CUDA out of memory. Tried to allocate 6.02 GiB. GPU 0 has a total capacity of 39.59 GiB of which 5.52 GiB is free. 
Process 35878 has 674.00 MiB memory in use. Process 32242 has 33.41 GiB memory in use. Of the allocated memory 28.39 GiB is allocated by 
PyTorch, and 3.70 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting 
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  
(https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
```
Which is kind of strange to me. My batch size is very small and here is my GPU machines spec

<img width="1373" alt="Image" src="https://github.com/user-attachments/assets/ca151aa8-8d68-478a-ad69-bafc0d326c0e" />

The only thing I can guess is that I have 7 dataset to evaluate but I still got no idea
```
# mcf.th.txt
# General Knowledge (GK)
lighteval|meta_mmlu_tha_mcf|5|1
lighteval|m3exams_tha_mcf|5|1

# Reading Comprehension (RC)
lighteval|belebele_tha_Thai_mcf|5|1
lighteval|thaiqa_tha|5|1
lighteval|xquad_tha|5|1

# Natural Language Understanding (NLU)
lighteval|community_hellaswag_tha_mcf|5|1
lighteval|xnli2.0_tha_mcf|5|1
```

Any Ideas?



I use lighteval 0.6.0.dev0 and torch 2.2.2+cu121. I clone this repo and `pip install -e .[dev]`

hi ! maybe the prompt are just to big to fit on the GPUs.
for model parallel we recomend using vllm, see [here](https://huggingface.co/docs/lighteval/use-vllm-as-backend) for the docs
Repository: huggingface/lighteval
Issue: Issues with torch version other than 2.2.2
Root cause: Environment Incompatibility

I got an error using lighteval with torch version 2.3.0, 2.3.1, 2.4.0, 2.5.1. 
Basically, I can only use lighteval with torch 2.2.2.
The error I got is 

```
RuntimeError: r.nvmlDeviceGetNvLinkRemoteDeviceType_ INTERNAL ASSERT FAILED at "../c10/cuda/driver_api.cpp":27, please report a bug to 
PyTorch. Can't find nvmlDeviceGetNvLinkRemoteDeviceType: /lib/x86_64-linux-gnu/libnvidia-ml.so.1: undefined symbol: 
nvmlDeviceGetNvLinkRemoteDeviceType
```

Here is my GPU machines

<img width="1375" alt="Image" src="https://github.com/user-attachments/assets/f6c5badf-7503-4d2a-92e8-06dc6a71a37d" />



hi ! tis is an issue with your cuda version and not lighteval, closing the issue for now
Repository: huggingface/lighteval
Issue: [BUG] LiteLLM backend fails with ifeval since prompt passes is str not List
Root cause: Validation Gap

In certain tests like with ifeval, the prompt passed to the LiteLLM backend is a str and will fail:
```
{'model': 'openai/testmodel', 'messages': "Generate a summary of the following passage in all capital letters:\n\nWorld War II or the Second World War, often abbreviated as WWII or WW2, was a global conflict that lasted from 1939 to 1945. The vast majority of the world's countries, including all of the great powers, fought as part of two opposing military alliances: the Allies and the Axis. Many participants threw their economic, industrial, and scientific capabilities behind this total war, blurring the distinction between civilian and military resources. Aircraft played a major role, enabling the strategic bombing of population centres and the delivery of the only two nuclear weapons ever used in war. World War II was by far the deadliest conflict in history, resulting in an estimated 70 to 85 million fatalities, mostly among civilians. Tens of millions died due to genocides (including the Holocaust), starvation, massacres, and disease. In the wake of Axis defeat, Germany, Austria and Japan were occupied, and war crimes tribunals were conducted against German and Japanese leaders.\n\n", 'logprobs': False, 'base_url': 'http://localhost:8000/v1', 'n': 1, 'caching': True, 'api_key': 'x', 'temperature': 0.5, 'top_p': 0.9, 'seed': 0, 'repetition_penalty': 1.0, 'frequency_penalty': 0.0, 'max_completion_tokens': 1280}
  0%|                                                                                                                                               | 0/541 [00:00<?, ?it/s]
[2025-02-16 00:25:39,181] [ WARNING]: Error in API call: Invalid message type: <class 'str'>. Expected dict or Pydantic model., waiting 3 seconds before retry 1/5 (litellm_model.py:194)
```

LiteLLM expects to see a chat completion formatted List: https://docs.litellm.ai/docs/completion/input#input-params-1

Fix is relatively simple and will submit a followup patch that's confirmed to work.

Hi ! Thanks for raising this issue, you actually need to pass `--use-chat-template` as argument when using litellm models
Repository: huggingface/lighteval
Issue: [FT] LiteLLM concurrency parameters hard-coded
Root cause: Architectural Constraint

I noticed that for LiteLLM the concurrency is hard-coded for an arbitrary number (20). Well, a lot of the request behavior is hard coded, but the concurrency is the one that is probably least likely to work well as a one size fits all.

If everything is going through the YAML file, maybe it can be expanded, although the YAML file is another ball of wax (makes automated runs much harder when you can't just pass in parameters via a call)

I think, this should fix it right? github.com/huggingface/lighteval/pull/953
Repository: huggingface/lighteval
Issue: [BUG] vllm seed is always 1234
Root cause: Algorithmic Error

## Describe the bug
vllm seed is always 1234
https://github.com/huggingface/lighteval/blob/d6de1fe221172c79b49f4d08d5009e03ec65982b/src/lighteval/models/vllm/vllm_model.py#L87
https://github.com/huggingface/lighteval/blob/d6de1fe221172c79b49f4d08d5009e03ec65982b/src/lighteval/models/vllm/vllm_model.py#L185


Hey ! fixed in #585
Repository: huggingface/lighteval
Issue: [FT] Propagate batch size control for vLLM backend
Root cause: Architectural Constraint

## Issue encountered

With vLLM backend, currently there's no way for us to control the batch size defined in [here](https://github.com/huggingface/lighteval/blob/main/src/lighteval/main_vllm.py#L125) and the vLLM model [config](https://github.com/huggingface/lighteval/blob/main/src/lighteval/models/vllm/vllm_model.py#L77) does not have ways to determine a specific batch size. However, we can control the maximum number of sequences (batch size) in vLLM directly from examples such as [this](https://github.com/vllm-project/vllm/blob/30172b4947c52890b808c6da3a6c7580f55cbb74/examples/offline_inference/neuron.py#L18).

## Solution/Feature
- Propagate the `max_num_seqs` parameter into the initialization of the vLLM model.

## Possible alternatives
- Other alternatives are to implement batching ourselves, which is an overkill since the vLLM backend already supports that.


I'm happy to take this on, but I figured it warrants a discussion since vLLM leverages continuous batching, which behaves differently than the common understanding of a "fixed" batch size in the LightEval world.

hey ! it would be usefull to expose the `max_num_seqs` argument. you can add it to the generation config and propagate it to `LLM` :)

Hi @NathanHB -- I just pushed a PR out #588 and please review it when you get the chance!