zhiminy/EvalEng huggingface dataset:
Repository: EleutherAI/lm-evaluation-harness
Issue: Low results on BBH
Root cause: Configuration Error

Hi, thanks for your great work!

I have problem when reproducing the results of Qwen-2.5-7b-Instruct on BBH, it is much lower than the reported number (around 32 compared with 70). Here is the script I run:

lm_eval --model sglang \
    --model_args pretrained="Qwen/Qwen2.5-7B-Instruct",dp_size=8,tp_size=1,dtype=auto \
    --tasks bbh \
    --device cuda \
    --batch_size 8 \
    --apply_chat_template \
    --gen_kwargs max_new_tokens=1024,do_sample=false \
    --log_samples \
    --cache_requests true \
    --trust_remote_code \
    --confirm_run_unsafe_code

Could you please help me figure out the problem? Thank you!

Hi! Do you have a reference for the reported number?

> Hi! Do you have a reference for the reported number?

Hi, I found this table in Qwen3 Technical Report. It seems to be BBH CoT 3-shot. Is it a different setting with your evaluation code? Thank you!

<img width="1107" height="392" alt="Image" src="https://github.com/user-attachments/assets/48f2fe29-e80c-4427-a1d6-99a8bcb748de" />

What output do you get when using the normal huggingface backend rather than sglang?
Repository: EleutherAI/lm-evaluation-harness
Issue: accelerate stuck when aggregating results in multi-GPU evaluation (was working before)
Root cause: External Dependency Breakage

Hello team,

I recently encountered an issue when running lm-evaluation-harness with accelerate on a multi-GPU setup. I am testing models on Darija benchmarks, and I noticed that when I use accelerate, the evaluation hangs at the aggregation stage, with only the main GPU still active.

Here’s the script I’m using:

bash
# Main parameters
TASKS=("darijammlu" "darijahellaswag" "darija_sentiment" "darija_summarization" "darija_transliteration")
SMALL_MODELS=("Qwen/Qwen3-0.6B" "Qwen/Qwen3-1.7B")

# Loop
for model_id in "${SMALL_MODELS[@]}"; do
  for task_id in "${TASKS[@]}"; do
    accelerate launch -m lm_eval \
      --model hf \
      --model_args "pretrained=${model_id},trust_remote_code=True,dtype=auto,local_files_only=True" \
      --tasks "${task_id}" \
      --batch_size auto \
      --log_samples \
      --output_path "./results/${model_id}_${task_id}" \
      --cache_requests true \
      --seed 42 \
      --wandb_args project=test-darija-benchmark-evals,name="${model_id}-${task_id}-eval"
  done
done


What I observed:
- All GPUs run normally during the batch processing phase.

- At the end, only one GPU (the coordinator) remains active for hours without finishing.

- It appears that the aggregation of results across devices is stuck.

- When I run the same task without accelerate, it completes successfully in about 1–20 minutes.

Additional Notes:
- This issue did not occur about a week ago. The same setup worked fine.

- The problem seems to come from the accelerate integration (maybe a new version?).



Please let me know if this is a known issue, or if there’s a workaround/fix. I’m happy to provide logs or test changes if needed.

Thanks for your work!

Mohamed

Hi! I haven't been able to reproduce this using:
`accelerate launch -m lm_eval --model hf -model_args pretrained=Qwen/Qwen3-1.7B -t darijammlu,darijahellaswag,darija_sentiment,darija_summarization,darija_transliteration --batch_size 8 --trust_remote_code --limit 20`

Have you tried using a fresh environment? I would also try running each benchmark individually to isolate which one might be causing the issue.

I have also encountered the same issue and setting the batch size to a specific value (instead of auto) seems to solve it.
Repository: EleutherAI/lm-evaluation-harness
Issue: lm_eval inference is slow
Root cause: Algorithmic Error

Why is the CPU high load and the GPU low load when I test mmlu-pro, but the inference is indeed running on the GPU?

lm_eval \
  --model hf \
  --model_args "pretrained=/home/ubuntu/project_3_LLM_benchmarks_data/model/deepseek-ai/DeepSeek-R1-Distill-Qwen-14B,trust_remote_code=true,parallelize=True,dtype=float16" \
  --tasks mmlu_pro \
  --batch_size 16 \
  --num_fewshot 5 \
  --gen_kwargs '{"temperature":0,"top_p":1,"do_sample":false}'

<img width="1661" height="380" alt="Image" src="https://github.com/user-attachments/assets/177b576b-cf60-4376-8d1a-d4b1d9934281" />

<img width="1196" height="359" alt="Image" src="https://github.com/user-attachments/assets/6a10e005-af6e-4f2c-920d-a3041cd7dfe1" />

<img width="1059" height="502" alt="Image" src="https://github.com/user-attachments/assets/59f4d432-3ded-4149-8a6f-24dda8c5f812" />

<img width="986" height="81" alt="Image" src="https://github.com/user-attachments/assets/81463223-10fd-4ae8-8872-0d7ef7ef712a" />

Hello, did you figure out what the issue was? I have a somewhat similiar issue.
Repository: EleutherAI/lm-evaluation-harness
Issue: Extremely Slow Startup Speed When Using vLLM
Root cause: Configuration Error

I've encountered a significant performance issue when using vLLM as the backend for `lm-evaluation-harness`.

After executing the `lm_eval` startup script, there is a very long delay of about **10 minutes** before the vLLM model actually starts to load. During this time, there is no apparent activity or log output, which makes it seem like the process is hanging.


Which versions of LM Eval, VLLM, transformers and torch are running in your environment? Additionally what model and task(s) were you attempting to load?

Hi, we use vllm=0.9.2, transformers=4.53.1, torch=2.7.0, lm_eval=0.4.9; we evaluate on llama-3.2-1b using standard benchmarks like gsm8k, mmlu, etc.
Repository: EleutherAI/lm-evaluation-harness
Issue: `repetition_penalty` is incorrectly set  when using `vllm`
Root cause: Configuration Error

Hi! Thanks for your great work and contributions!

I'm currently investigating the performance gap problem between `hf` and `vllm` backends, and I found a potential bug about `vllm` backend.

Specially, according to the official [document](https://docs.vllm.ai/en/stable/api/vllm/config.html#vllm.config.ModelConfig.generation_config), the `generation_config` should be automatically loaded from the `huggingface` and the according parameters should be set to the `SamplingParams` when using `vllm` backend. However, when using `Qwen/Qwen2.5-Coder-7B-Instruct` model and choose `vllm` backend, the `repetition_penalty` is not correctly set to `1.1`(please refer to the official [generation_config](https://huggingface.co/Qwen/Qwen2.5-Coder-7B-Instruct/blob/main/generation_config.json)). In that case, `vllm` will skip all the `penalties` when processing the logits. In contrast, all the according parameters in `generation_config` is correctly set when using `hf` backend and all the `penalties` will be applied.

To fix the problem, I manually set `"repetition_penalty": 1.1` in the `gen_kwargs` and the performance almost matches with `hf`:

| gsm8k_cot_128_0shot       | backend    | flexible-extract |
|---------------------------|------------|------------------|
| Qwen2.5-Coder-7B-Instruct | hf         | 51.56            |
| Qwen2.5-Coder-7B-Instruct | vllm       | 34.38            |
| Qwen2.5-Coder-7B-Instruct | vllm_fixed | 50.78            |

For you reference, the command I run is:
```bash
lm_eval --model $BACKEND \
    --model_args pretrained=Qwen2.5-Coder-7B-Instruct \
    --tasks gsm8k_cot \
    --batch_size auto \\
    --limit 128 \
    --gen_kwargs '{"max_gen_toks": 2048, "repetition_penalty": 1.1, "until": ["
", "
", "
"]}'
    --num_fewshot 0 \
```

and my experiment env is:
```bash
torch==2.7.0
transformers==4.52.4
lm_eval==0.4.9
vllm==0.9.2
```

Thanks for figuring this out! Can you open a PR fixing the issue?

@StellaAthena Can you help to review the PR #3160? Thanks!
Repository: EleutherAI/lm-evaluation-harness
Issue: How to config the doc_to_text when the number of choices is variable
Root cause: Documentation Deficiency

How to config the doc_to_text when the number of choices is variable？

Hi! you can use a `!function ` [constructor](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/docs/new_task_guide.md#using-python-functions-for-prompts):
```yaml
doc_to_text: !function utils.custom_fn
```
Repository: EleutherAI/lm-evaluation-harness
Issue: remove literal list parsing "[...]" form doc_to_target
Root cause: Architectural Constraint

Currently we check if the target string can be parsed as a list and parse it if so:
https://github.com/EleutherAI/lm-evaluation-harness/blob/8c05cfe04fafcdd41dd64019f2b3797ef54dcd81/lm_eval/api/task.py#L1332-L1338

However this should be handled at the dataset level as it creates ambiguity, where the target _string_ might be list-like. Tasks that need list targets should explicitly return lists from their doc_to_target implementation rather than relying on string parsing. 

This also causes `bigbench_list_functions_generate_until` to produce incorrect results, as the task expects literal string representations of lists (e.g., "[1, 2, 3]") to be evaluated as strings.

Will need to test, which tasks rely on this condition before removing.
Repository: EleutherAI/lm-evaluation-harness
Issue: Datasets with loading scripts - no longer supported
Root cause: External Dependency Breakage

As of `datasets==4.0`, loading scripts and `trust_remote_code` are no longer supported. Started this thread to track tasks which still rely on the scripts so that the datasets can either be updated upstream in HF or in the task configs.

Just add a comment, if anyone comes across any.

afrobench's `masakhapos`: `masakhane/masakhapos`

Gone through all the `a***` groups. Found remote code, missing datasets, an incorrect name, and some datasets marked as remote code required but have been fixed on the hub to work without code. I may have missed some, but this is what I found so far.

`aclue`: `tyouisen/aclue` (remote code)

`aexams`: `Hennara/aexams` (remote code)

`afrobench`:
- `adr`: `masakhane/diacritics-restoration` (missing)
- `afriqa`: `masakhane/afriqa-gold-passages` (remote code)
- ~~`afrisenti`: `masakhane/afrisenti`~~ (fixed on hub, task needs updating)
- `flores`: `facebook/flores` (remote code)
- ~~`mafand`: `masakhane/mafand`~~ (fixed on hub, task needs updating)
- `masakhaner`: `masakhane/masakhaner-x` (remote code)
- `masakhapos`: `masakhane/masakhapos` (remote code)
- ~~`nollysenti`: `Davlan/nollysenti`~~ (fixed on hub, task needs updating)
- ~~`ntrex`: `masakhane/ntrex_african`~~ (fixed on hub, task needs updating)
- ~~`salt`: `Sunbird/salt`~~ (fixed on hub, task needs updating)
- ~~`xlsum`: `csebuetnlp/xlsum`~~ (fixed on hub, task needs updating)

`anli`: `anli` (should be `facebook/anli`)

`arab_culture_completion`: `boda/arabic_cluture` (missing)

`asdiv`: `EleutherAI/asdiv` (remote code)

`super-glue`: added full path in #3169 

`mgsm`: `juletxara/mgsm` also has a similar issue!

mmlu has same issue

Hey all, is there any current plan to support datasets 4.0 and grater beyond what is discussed in this issue? If we don't run the tests outline above, can the pin at least be removed and moved to some kind of raised error on the remaining tests which have issues?

[longbench v1](https://huggingface.co/datasets/zai-org/LongBench) `zai-org/LongBench` still uses remote code, would not load without trust_remote_code=True

This problem also occurs with [race](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/race/README.md).

updated the race dataset to parquet. btw also discovered the `datasets-cli` utility, which I didn't even know existed but apparently been around for awhile. `datasets-cli convert_to_parquet EleutherAI/race` was pretty quick

Same issue with crows_pairs

also same issue with hendrycks ethics 

> also same issue with hendrycks ethics

converted hendrycks_ethics to parquet on [HF](https://huggingface.co/datasets/EleutherAI/hendrycks_ethics/discussions/2), but @baberabb has to merge it
Repository: EleutherAI/lm-evaluation-harness
Issue: Qwen3 models take ages to evaluate on minerva_math. Is there a way to speed it up?
Root cause: Configuration Error

Hey I am trying evaluate the Qwen3-8b model  (multiple pruned and quantized versions) on minverva_math, but it takes around 2-4days. Is this normal? Do i have to put of cot thinking? I already tried passing --system_instruction "/nothink", but that didn't change anything. Thank you so much for your help!


Running generate_until requests:  27%|██▋       | 1332/5000 [13:41:52<57:34:56, 56.51s/it]Both `max_new_tokens` (=2048) and `max_length`(=965) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

Running generate_until requests:  27%|██▋       | 1333/5000 [13:43:14<65:27:52, 64.27s/it]Both `max_new_tokens` (=2048) and `max_length`(=965) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)


Are you making sure you're evaluating on your GPU rather than CPU, and which GPU are you equipped with?

> Are you making sure you're evaluating on your GPU rather than CPU, and which GPU are you equipped with?

Yes I am using an H100 GPU with 80GB ram. I feel like it could be because of the chain-of-though reasoning?

Oh, I just re-read and depending on your quantization it might actually run slower. In my experience 8bit quants can actually run slower than 16bit or 4bit quants. Can you try running a vanilla Qwen3 model?

Hi,
I have the same problem with the evaluation task. At first, it was normal and each iteration took 1 or 2 seconds, but suddenly it became too slow. (like 60 to 100 sec per iter). Could you let me know if you managed to resolve it?

Did anyone manage to speed this up?
Repository: EleutherAI/lm-evaluation-harness
Issue: How do I use prompts in Hugging Face conversational format with doc_to_text?
Root cause: Documentation Deficiency

## Short Problem Description [question]

When defining a task using **doc_to_text**, I can't seem to use a conversational prompt, as it is parsed to str format and not parsed back when it reaches `apply_chat_template()`. Is there any way to do this that I'm missing?


## Full Problem Description

When for example doing `doc_to_text: "{{ text }}"` in a .yaml task definition file, if my text is in the format

`[{"role": "system", "content": "XXX"}, {"role": "user", "content": "YYY"}, {"role": "assistant", "content": "ZZZ"}]`

it gets converted into str format, instead of keeping the conversational format. I believe this is intended behaviour of **doc_to_text**, but if that's the case I can't understand how to use prompts in conversational format.

## Why is this relevant

Conversational format abstract the specific model tags, which I need to do in my work as I intend to use multiple models with different formats. Therefore, manually parsing them in a `utils.py` would be sub-optimal and something to avoid, if possible. 


## Workaround I found

I don't think this is the correct way of implementing it, but we can re-parse from str to conversational prompt just before `chat_templated` is assigned here (this what I do currently)

https://github.com/EleutherAI/lm-evaluation-harness/blob/091aaf6f31f3e87e81135a5d204c30707711d94a/lm_eval/models/huggingface.py#L1474-L1486

Just some background: The way the LM eval harness works is that it attempts to be compatible with a variety of models. In terms of HF models that means supporting models which both do and don't have chat templates, but for some backends the models don't even have a concept of a "chat template" period. This means that in general we have to store requests in a string format.

This doesn't stop you from using the message list itself to parse the strings in `doc_to_text`. Let's say your dataset has a `messages` field which contains the list of messages, and let's assume the second to last message is always the `user` message you want to parse as the prompt and the last message is the `assistant` message you always want to use as a target. You can do something like:
```yaml
doc_to_text: '{{messages[-2]["content"]}}'
doc_to_target: '{{messages[-1]["content"]}}'
```
Then by launching the task with `--apply_chat_template` it will first parse the messages from the dataset and then grab the parsed text+target pair and put them back into the conversational format if the model supports templating.

This can also be extended to multiple conversation turns by exploiting how fewshot messages are handled with `--fewshot_as_multiturn` but I'll leave that to another response only if you're interested.

Unfortunately this disregards the system prompt as there is no `doc_to_system`... which might not be what you're looking for, however some modern LMs expect to use the system prompt they were trained with and might act unexpectedly when forced to use a different one.

Thank you for your answer, that perfectly responds to my inquiry.
Repository: EleutherAI/lm-evaluation-harness
Issue: trust_remote_code: true
Root cause: Configuration Error

since `datasets==4.0` this is unsupported, but the .yaml files here still have it. This causes lots of warnings when evaluating.

we can remove the lines with

```
#!/bin/bash
find lm-evaluation-harness/lm_eval/tasks -name "*.yaml" -exec grep -l "trust_remote_code: true" {} \; | while read file; do
    if [ $(awk '/^dataset_kwargs:/,/^[^ ]/{if(/^  /) count++} END{print count+0}' "$file") -eq 1 ]; then
        sed -i '/^dataset_kwargs:$/,/^  trust_remote_code: true$/d' "$file"
    else
        sed -i '/^  trust_remote_code: true$/d' "$file"
    fi
done
```



Task mmlu won't work?

Many tasks which originally required remote code but have previously needed it in older dataset versions still contain that line in their config files. MMLU for example no longer has a builder script, making the "trust remote code" directive redundant despite the warnings, but other tasks aren't so lucky.

We do plan on removing those lines, but once we identify all tasks that need to be updated on HF's end.

You can help by adding tasks which don't work at all under `datasets>=4.0.0` to [this issue thread](https://github.com/EleutherAI/lm-evaluation-harness/issues/3171). Try not to add a task which has already been mentioned in the thread, and please don't include datasets which do work but give warnings as we want to focus on fixing or finding alternate repos for datasets that *actually* require remote code.
Repository: EleutherAI/lm-evaluation-harness
Issue: Whats the difference between acc and exact_match on multiple_choice tasks?

Hi~
I wonder whats the difference between acc and exact_match on multiple_choice tasks? Since it seems like that on perplexity-based method, acc should equal to exatch_match?

This depends on the task, but some tasks come in two flavors: logprob/perplexity based and generation based. acc uses the former over all possible choices, while exact_match uses the latter as it is checking the generated response produces the expected target.

If you'd like a more concrete response for a specific task just let me know which one and I can give you specifics!
Repository: EleutherAI/lm-evaluation-harness
Issue: Support Fill In The Middle Evaluations
Root cause: Unimplemented Feature Gap

I would like to lm-eval to run fim evaluations for example on qwen-2.5-coder:

To run [safim](https://github.com/gonglinyuan/safim)
and [santacoder](https://github.com/bigcode-project/bigcode-evaluation-harness/blob/main/bigcode_eval/tasks/santacoder_fim.py)
and [Humaneval-FIM](https://github.com/openai/human-eval-infilling)


<img width="543" height="494" alt="Image" src="https://github.com/user-attachments/assets/d071c0f6-ad5d-437f-b10a-474c01d26b23" />

<img width="2774" height="1035" alt="Image" src="https://github.com/user-attachments/assets/f3fce385-ed1b-4e10-824d-31f7987c860e" />
https://qwenlm.github.io/blog/qwen2.5-coder-family/



The problem with FIM tasks is actually getting the model to fill in the middle because decoder only LLMs (so pretty much all recent LMs) only work from left to right, and typically aren't trained for FIM tasks so lack special FIM tokens to specify prefix, suffix and middle. Different models might prefer different prompting schemes, and controlling for that will be difficult.

This is still worth looking into, however, but we'll have to do research on if there are established conventions and I'm not sure how high of a priority this is.

i know for example qwen-2.5-coder, the fim tokens are part of the tokenizer.  The fim template however is not included in the model configs

https://huggingface.co/Qwen/Qwen2.5-Coder-32B-Instruct/raw/main/tokenizer.json

```
{
      "id": 151659,
      "content": "
",
      "lstrip": false,
      "normalized": false,
      "rstrip": false,
      "single_word": false,
      "special": false
    },
    {
      "id": 151660,
      "content": "
",
      "lstrip": false,
      "normalized": false,
      "rstrip": false,
      "single_word": false,
      "special": false
    },
    {
      "id": 151661,
      "content": "
",
      "lstrip": false,
      "normalized": false,
      "rstrip": false,
      "single_word": false,
      "special": false
    },
    {
      "id": 151662,
      "content": "
",
      "lstrip": false,
      "normalized": false,
      "rstrip": false,
      "single_word": false,
      "special": false
    },
Repository: EleutherAI/lm-evaluation-harness
Issue: Is lls same as log likelihood/log probability for multiple choice tasks?
Root cause: Documentation Deficiency

Hi team,

Is lls ([here](https://github.com/EleutherAI/lm-evaluation-harness/blob/4f8195f18fbc1b6d212314509d7525e1178e036c/lm_eval/api/task.py#L1580)) the log likelihoods/log probabilities for multiple choice tasks? 

**Context**: I am trying to save NLLs (negative log likelihood) for multiple choice tasks such as `hellaswag`, but they look way off (in 100s) which I believe doesn't make sense.

To provide more context, something like the below code snippet in my local codebase right after [this line](https://github.com/EleutherAI/lm-evaluation-harness/blob/4f8195f18fbc1b6d212314509d7525e1178e036c/lm_eval/api/task.py#L1659)

```
if "nll_choice" in use_metric:
    if self.multiple_target:
        raise ValueError(
            "nll_choice is not implemented for multiple_target"
        )
    else:
        result_dict["nll_choice"] = -lls[gold]
```

After some inspection, I believe [this](https://github.com/EleutherAI/lm-evaluation-harness/blob/4f8195f18fbc1b6d212314509d7525e1178e036c/lm_eval/api/task.py#L1637) is actually the log likelihood/log probabilities as these get passed to `brier_score` which accepts only probabilities. 

I thought maybe `lls` is just log values and I might have to use `prob_norm`, but I was confused with [this](https://github.com/EleutherAI/lm-evaluation-harness/blob/4f8195f18fbc1b6d212314509d7525e1178e036c/lm_eval/api/task.py#L1599) particular `argmax`!

Can someone explain what `lls` is in context of multiple choices and how to proceed with storing NLL values.

Thanks in advance.

As far as I can tell `lls` are the logprobs from `F.log_softmax` in HFLM models or from the API of other backends which support returning logprobs.

`prob_norm` are the *softmax* scores, not the log likelihood/logprobs. `brier_score` iirc accepts probabilities (i.e. softmax scores) not log probabilities.
Repository: EleutherAI/lm-evaluation-harness
Issue: test llm on storycloze report error
Root cause: Configuration Error

command:
lm_eval --model hf --tasks storycloze_2018 --model_args pretrained=/mnt/g/gpt2

what I do:
install lm_eval into pythone3 env. other datasets is working well. I download csv of storycloze, change datasets.load_datasets path as csv of dir. other hand ,I change stroycloze datasets path in yaml.

report error:
ValueError: BuilderConfig 'story_cloze' not found. Available: ['default']