zhiminy/EvalEng huggingface dataset:
Repository: EleutherAI/lm-evaluation-harness
Issue: An error occurred: 'choices' (in openai chat completion)
Root cause: Interface Contract Mismatch

I encountered an error in the file openai_completion.py while using an OpenAI inference-compatible model. The error message is:

"An error occurred: 'choices'"

After printing the response, I received the following output:
{'status': 'FAILED', 'error': 'err.supplier_error', 'supplierError': 'Start of structure or map found where not expected'} 

This suggests that the API response does not match the expected format. What could be the possible cause of this issue, and how can I resolve it?

Have you found a cause or workaround for this issue? I same the issue.
Repository: EleutherAI/lm-evaluation-harness
Issue: modelscope installed will lead some problems
Root cause: Configuration Error

ran eval code on alibaba cloud

```
lm_eval --model vllm \
    --model_args "pretrained=Qwen2.5-3B,tensor_parallel_size=1,dtype=auto,gpu_memory_utilization=0.9,data_parallel_size=1" \
    --tasks gsm8k \
    --batch_size auto \
    --device cuda:0 \
```

```export HF_ENDPOINT=https://hf-mirror.com

pip uninstall modelscope -y # it will lead to modelscope to download the dataset, but since they have different name in modelscope and huggingface, this will fail
```
>or: modelscope.hub.errors.RequestError: Url = https://www.modelscope.cn/api/v1/datasets/EleutherAI/hendrycks_math, Request id=a81d68cd-35f2-4c2d-ab08-92aa27e913ac Code = 10020101002 Message = 不存在的数据集(means dataset not exist),                Please specify correct dataset_name and namespace.
```
export VLLM_USE_MODELSCOPE=0 # if set to True vllm will load modelscope and fail, this will lead to vllm mot loaded and no exception reported
mey
```
>or: NameError: name 'LLM' is not defined when using vllm in lm_eval

Hi! you can change the dataset name in the task configs. for example for gsm8k [here](https://github.com/EleutherAI/lm-evaluation-harness/blob/af2d2f3e79140ae5b6833ce1046f1519dc08b9df/lm_eval/tasks/gsm8k/gsm8k.yaml#L4)
Repository: EleutherAI/lm-evaluation-harness
Issue: HOW TO ADD NEW TASK?
Root cause: Documentation Deficiency

I want to add mmlu-redux, but I have not more retail tutorial. I learn a course from new_task_guid. such as 1.fill the dataset_path and split in yaml, 2.fill     a process_docs in utils. but report error. if somebody have a better guide and send it , best wishes. 

Same Question.
Repository: EleutherAI/lm-evaluation-harness
Issue: Embedding checkpoint size mismatch when using peft on DeepSeek-R1-Distill-Qwen-1.5B.
Root cause: Validation Gap

## Summary
I encountered a size mismatch problem like in https://github.com/EleutherAI/lm-evaluation-harness/issues/1707#issue-2244839671 when evaluating peft model (DeepSeek-R1-Distill-Qwen-1.5B), but the root cause is due to the inherent mismatch with model's `vocab_size` and tokenzier's `vocab_size`, and not that I manually modified the tokenizer. 

I'm not sure if other models have such a mismatched feature.
## Problem
when using `lm-eval` on `DeepSeek-R1-Distill-Qwen-1.5B` trained with lora checkpoint:
```bash
lm_eval --model hf \
--model_args pretrained=$base_path,peft="trained_models/codealpaca_lora_20250227_221049_r128/checkpoint-60000" \
--tasks $eval_tasks \
--batch_size $batch_size \
--confirm_run_unsafe_code  \
--output_path "eval_results/DeepSeek-R1-Distill-Qwen-1.5B" 
```

I got the following error:

```bash
RuntimeError: Error(s) in loading state_dict for PeftModelForCausalLM:
size mismatch for base_model.model.lm_head.modules_to_save.default.weight: copying a param with shape torch.Size([151936, 1536]) from checkpoint, the shape in current model is torch.Size([151665, 1536]).
```

## Root cause:
- In `DeepSeek-R1-Distill-Qwen-1.5B`, the model's `vocab_size` and the tokenizer's `vocab_size` is different. The former is **151643**, the latter is **151936**.
  ```python
  t1 = AutoTokenizer.from_pretrained("./model_ckpt/DeepSeek-R1-Distill-Qwen-1.5B")
  t1.vocab_size
  >>>151643
  ```
  ```python
  model.lm_head.weight.shape
  >>>torch.Size([151936, 1536])
  ```

- But in `https://github.com/EleutherAI/lm-evaluation-harness/blob/a87fe425ec55d90083510fc8b2a07596b76e57b3/lm_eval/models/huggingface.py#L647`:
  ```python
      if self._model.config.vocab_size != len(self.tokenizer):
          # resize model for LoRAs with added tokens
          eval_logger.info(
              f"Model config indicates vocab_size='{self._model.config.vocab_size}', but found tokenizer with vocab size '{len(self.tokenizer)}'. Resizing model embedding layer..."
          )
          self._model.resize_token_embeddings(len(self.tokenizer))
  ```
it will resize the embedding according to the tokenizer's vocab_size leading the mismatch error.

## How to fix?
- Simply comment this `self._model.resize_token_embeddings(len(self.tokenizer))` will be useful.
- But I'm not sure if other models have this mismatch problem (inherent mismatch with model's vocab_size and tokenzier's vocab_size). So we need a more elegant solution.
Repository: EleutherAI/lm-evaluation-harness
Issue: Smooth landing errors during post processing
Root cause: Validation Gap

I just spent 15h running Llama 1B eval on math500 on a H100 and the code broke because antlr4-python3-runtime isn't installed. I wonder if there's a way to smooth land these issues.

Hi! was it a custom task? I don't think we have a config for math500 yet. Generally the best practice is to add an import [check](https://github.com/EleutherAI/lm-evaluation-harness/blob/a87fe425ec55d90083510fc8b2a07596b76e57b3/lm_eval/tasks/minerva_math/utils.py#L19) for tasks which require custom code, so any errors occur at import time.

And 15h is way to long for 1B on H100 for 500 samples. Is your task using many repeats?

Hi! I was running your llama3 PR :) 

It had 64 repeats but I tried only 10.

oh forgot about that. 😅 I should merge them already

Can you please make the metrics parsing fail silently? 

I'm losing so much of inference compute when it fails.

> Can you please make the metrics parsing fail silently?
> 
> I'm losing so much of inference compute when it fails.

This makes a lot of sense, @baberabb can we do this? Alternatively it may be feasible to ameliorate this issue with better caching.
Repository: EleutherAI/lm-evaluation-harness
Issue: API Model: Custom handling of refused prompt
Root cause: Unimplemented Feature Gap

# Problem description

I am using the API models to benchmark an OpenAI model.

When running inference, some prompts are refused by the model, due to possible different reasons: violence/sexuality/etc

Right now, the behaviour is:
- Try sending prompt: controlled by the `max_retries` parameter
- When all retries fail, send an error

The code that performs this is here: https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/models/api_models.py#L439

# Desired modification

I would like to be able to customize the way the inference behaves when facing those kind of `refused` prompt. For example, I would like to be able to continue doing inference on the next prompt, instead of raising an error.
This could be done for example by defining a `refused_prompt_json_answer`, that would be stored as prompt answer for all refused prompts

# Possible problems

- How to use a `refused` prompt output when computing the scores ? 

For example, on a multiple choice question, if I'm sending 5 different prompts, with results:
- 4 correct answers
- 1 refused prompt

What is my final score ?
- if I just ignore the refused prompt, 100% (4/4)
- if I consider the refused prompt as wrong answer, 80% (4/5)

Maybe this behaviour should also be customizable

It would be simple to add option to continue other samples in case of error, but the metric calculation would be tricky as you identified.  Not sure what the best way would be tbh, if any one else has any ideas

> It would be simple to add option to continue other samples in case of error, but the metric calculation would be tricky as you identified. Not sure what the best way would be tbh, if any one else has any ideas

I see 2 paths:
1. Ignore the refused prompt in final score computation. This could be done whatever the model/whatever the benchmark
2. Set a custom score for the refused prompt in final score computation. The correct custom score probably depends on the benchmarking task, so it might be good to link the custom score to the benchmarking task
Repository: EleutherAI/lm-evaluation-harness
Issue: HF `batch_size=auto` unreliable
Root cause: Algorithmic Error

Hi, can someone explain why in tasks with output_type: generate_until the batch_size is always 1? Even if we change the batch_size, it will always remain 1. Is it because of how these tasks are evaluated compared to multiple-choice tasks? I would appreciate it if someone could explain it to me. 

Hi! No, it should batch appropriately. what command are you using?

This is the command I am using  `lm_eval --model hf --model_args pretrained=microsoft/Phi-3.5-mini-instruct,dtype="bfloat16",device="cuda",parallelize=False  --tasks gsm8k  --batch_size "auto" --limit 100  --output_path /home/alireza/D1/benchmarking_results/ --log_samples  --gen_kwargs do
_sample=True,temperature=0.9` after a couple of experiments, I observed that using "auto" for detecting largest batch_size that can be fit into the memory does not work properly for generate_until tasks. I tested with both translation tasks and gsm8k, in both cases "auto" detected 1 as largest batch_size. but when I defined larger batch_size, it worked. Moreover, when tested with multiple choice tasks, the "auto" works perfectly. 

I met the same issue
Repository: EleutherAI/lm-evaluation-harness
Issue: verbosity or verbostiy?

Isn't verbostiy here a typo?

https://github.com/EleutherAI/lm-evaluation-harness/blob/dce4bd35addefc1df70f739cfafbdd367dbbe045/lm_eval/evaluator.py#L73

thanks for raising the issue!
Repository: EleutherAI/lm-evaluation-harness
Issue: cluade sonnet 3.5 and 3.7 humaneval 0%
Root cause: Configuration Error


The cluade sonnet 3.5 and 3.7 models were humaneval evaluated and the results are shown below. 
They both have a value of 0 as shown below. Can we trust these results? 

| Tasks |Version| Filter |n-shot|Metric| |Value| |Stderr|
|---------|------:|-----------|-----:|------|---|----:|---|-----:|
|humaneval| 1|create_test| 0|pass@1| | 0|± | 0|

lm_eval \
    --model anthropic-chat-completions \
    --tasks humaneval  \
    --model_args model=claude-3-5-sonnet-20241022,tokenized_requests=False \
    --apply_chat_template \
    --output_path output  
    --confirm_run_unsafe_code 
    
Is there an issue with the execution options?

Hi! i have `humaneval_instruct` in #2650, which works better for chat models. Getting perfect score for `claude-3-7-sonnet-20250219` with `--limit 10`

The issue is that humaneval prompts the model with a partial code snippet and expects the model to complete it. The input+output is then concatenated and executed to check if code is correct.

However instruct models have a tendency to add a lot of noise to the outputs The instruct variant uses a gen_prefix so that the model only continues with the code provided.


The first one seems to have been caused by an issue with stop_sequence.
'stop_reason': 'stop_sequence', 'stop_sequence': '\ndef'

However, after the fix, it was still 0%, so I added the humaneval_instruct you told me. As a result, the accuracy increased.

claude-3-7-sonnet-20250219
|      Tasks       |Version|  Filter   |n-shot|Metric|   |Value|   |Stderr|
|------------------|------:|-----------|-----:|------|---|----:|---|-----:|
|humaneval_instruct|      1|create_test|     0|pass@1|   |    1|±  |     0|

claude-3-5-sonnet-20241022
|      Tasks       |Version|  Filter   |n-shot|Metric|   |Value|   |Stderr|
|------------------|------:|-----------|-----:|------|---|----:|---|-----:|
|humaneval_instruct|      1|create_test|     0|pass@1|   |  0.6|±  |0.1633|

Thank you!
Repository: EleutherAI/lm-evaluation-harness
Issue: Add AIME 2024 and LiveCodeBenchmark to the gold standard evaluation harness
Root cause: Unimplemented Feature Gap

First, thank you for creating and maintaining lm-evaluation-harness, which has become the gold standard for open-source LLM benchmarking. Its comprehensive coverage and reliability make it invaluable to the AI community.
I'd like to request adding support for two important tasks that would further strengthen this excellent platform:
(1) AIME 2024 
(2) LiveCodeBenchmark

yes！

+1 Has anyone just created a custom task that they would like to share?

Hi! @Allen-labs @imagoodman-aa @radna0 I'll be happy to work on AIME and will submit a PR soon.

+1
Repository: EleutherAI/lm-evaluation-harness
Issue: Layer-by-layer inference evaluation of large models.
Root cause: Documentation Deficiency

I'm testing a relatively large model, LLama2-70B. I currently have a 24GB GPU, which cannot fully load the model. I would like to know how to load the model into the GPU layer by layer for computation in order to use lm_eval to evaluate its performance.









It's going to be really slow, but you should look up the CPU offload documentation in the `transformers` library. You should be able to use that by passing the necessary args through the lm_eval command line call.
Repository: EleutherAI/lm-evaluation-harness
Issue: MMLU COT Giving less accuracy
Root cause: Configuration Error

I was trying to check for MMLU COT 0 shot configuration. But since there is no configuration available for that I went ahead with mmlu_flan_cot_zeroshot. But this is giving really less accuracy ranging between 28-33%. But the data published by HuggingFace is  73.0% for llama 3.18B. I tried to update the prompts still no improvements. Can anyone please help with the MMLU COT configuration.

Same issue. Achieved 0.4651 EM with mmlu_flan_cot_zeroshot on Llama 3.1 8B instruct. Not sure why.
Repository: EleutherAI/lm-evaluation-harness
Issue: Deviation in HumanEval Benchmark Results
Root cause: Configuration Error

I did some experiments with the newly added HumanEval benchmark. I used Qwen2.5-7B-Instruct and Llama-3.1-8B-Instruct models. In both cases, the resulting pass@1 scores deviates from what is reported in the technical papers. The image bellow is taken from the Qwen2.5 technical paper. 

<img width="418" alt="Image" src="https://github.com/user-attachments/assets/d6d926d8-07b5-4d2e-9219-848f7d350098" /> 

This is the code I used: `m_eval --model hf --model_args pretrained=Qwen/Qwen2.5-7B-Instruct,dtype="bfloat16",device="cuda",parallelize=False --tasks humaneval  --batch_size 1 --confirm_run_unsafe_code` 

![Image](https://github.com/user-attachments/assets/c97e26a9-f807-48e5-82ef-e6b85fb069ef)
![Image](https://github.com/user-attachments/assets/eed50ef4-2296-4ad4-bd26-d1a9d361e87d) 

I really appreciate if someone guide me what I am doing wrong or how I can replicate the numbers?


The sample question, i evaluate Llama-3.1-8B, and the results are as followed:

![Image](https://github.com/user-attachments/assets/b8775088-8336-48f1-bb10-10e80c3d7876)

+1

Hi @ds-anik @NieShenRuc. Sorry this slipped my notice. 

For instruct models, it generally better to use the `humaneval_instruct` variant (the prompt is based on what they used for the llama evals). 

Also see #2954 for additional context.

Hi,
I tried newest stable version `0.4.9.1`. I got a slightly better score 66.4%,  but still far from the reported 72.6%.

I use `humaneval_instruct` and `--apply_chat_template`:
```bash
# lm_eval version `0.4.9.1`, commit d021bf846218f1bb3bdc0603864789329476d464
lm_eval --model hf --model_args pretrained=meta-llama/Llama-3.1-8B-Instruct,dtype="bfloat16",device="cuda",parallelize=False --tasks humaneval_instruct  --batch_size 1 --confirm_run_unsafe_code --apply_chat_template
```

```bash
2025-11-17:19:13:03 INFO     [loggers.evaluation_tracker:280] Output path not provided, skipping saving results aggregated
hf (pretrained=meta-llama/Llama-3.1-8B-Instruct,dtype=bfloat16,device=cuda,parallelize=False), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: 1
|      Tasks       |Version|  Filter   |n-shot|Metric|   |Value |   |Stderr|
|------------------|------:|-----------|-----:|------|---|-----:|---|-----:|
|humaneval_instruct|      4|create_test|     0|pass@1|   |0.6646|±  | 0.037|
```
Repository: EleutherAI/lm-evaluation-harness
Issue: Evaluating Pretrained LM always need few-shot example?
Root cause: Documentation Deficiency

Hi, I was new to the library. 
I am wondering where I can modify (or see the default) prompt for evaluating LM on, for example, Hellaswag.
Also, should I always use few-shot settings to let the model answer with the correct format if my model is a pretrained on without instruction-tuning?
e.g., Gemma-2-2b


I am also curious about if the model generates all the possible choices within the max_token_len, which of them would be parsed to be the final answer? (my experience on pretrained LM suggests that this would be a very common scenario)

Thanks in advance.
Repository: EleutherAI/lm-evaluation-harness
Issue: Multi-NPU evaluation supported?

I'd like to know if multi-NPU evaluation is currently supported. I looked at this PR [https://github.com/EleutherAI/lm-evaluation-harness/pull/1886](url)  and it seems there are only examples for single NPU. Have I missed any progress?
@haileyschoelkopf @ji-huazhong 

You could refer to the following example:
```
lm_eval --model hf  --model_args pretrained=Qwen1.5-0.5B,parallelize=True,device='npu:7' --tasks arithmetic --batch_size 64
```
cc @yuhkalhic 

Yes, we support multi-NPU via accelerate. Please try the accelerate example in the readme but with NPU as the device type and let us know if you run into issues.
Repository: EleutherAI/lm-evaluation-harness
Issue: Add ability to specify custom local completions headers
Root cause: Unimplemented Feature Gap

I started thinking about this after testing a model hosted via Kserve inference service, which supports OpenAI APIs for certain serving runtimes. My setup uses an ingress gateway (no DNS) and, as a result, I need to supply the "Host" header in addition to the base URL. I'm not sure it's worth subclassing the LocalCompletions class in this case.

I would appreciate the community's thoughts on having the ability to specify custom headers on the command line. There may be certain implications of this that I haven't considered :)
Repository: EleutherAI/lm-evaluation-harness
Issue: Edit existing MMLU dataset and send it to evaluate
Root cause: Documentation Deficiency

I am trying to update the existing MMLU dataset and remove some prompts from it and then send it to the evaluate function. I have tried many ways but for some reason it doesn't work. Either it ends up running again on the original dataset or would ask me to go through modify some classes. Does anyone have a step by step idea on how I can achieve this?

Hi! Can you provide a bit more detail on how you want to modify? Also have you taken a look at this [doc](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/docs/new_task_guide.md)

closing the issue now but feel free to open
Repository: EleutherAI/lm-evaluation-harness
Issue: Why not apply `num_return_sequences` (or `n` for vllm) for each generation instead of `repeats`?
Root cause: Architectural Constraint

Hi, 

I noticed for each generation, even if multiple sequences required (e.g. self-consistency), the result always grab the first output instead of taking advantage of the number of return sequences argument in generation https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/models/vllm_causallms.py#L429
Instead, the doc points me to use `repeats` for self-consistency, which is much less efficient than setting `num_return_sequences`. if i understand correctly, the two setting should derive equivalent output given the same initial random seed. 
Please correct me if I'm wrong or is there any design decision involved. 

Thanks,
Repository: EleutherAI/lm-evaluation-harness
Issue: Generation cut off for Qwen with max_length=16384
Root cause: Configuration Error

Hi! I have some troubles evaluating Qwen-2.5 models on GSM8K. I have set max_length=16384 in my generation arguments but I still find answers to be cut off.

My arguments are as below:

```
lm_eval --model vllm \
        --model_args pretrained=Qwen/Qwen2.5-7B-Instruct,tensor_parallel_size=4,gpu_memory_utilization=0.8,data_parallel_size=1,max_length=16384 \
        --tasks gsm8k_qwen_zeroshot \
        --device cuda \
        --output_path $HOME/qwen2.5-7b-instruct/eval_res/zeroshot \
        --log_samples \
        --batch_size auto
```

And the task yaml is as here:

```
dataset_name: main
dataset_path: gsm8k
test_split: test
doc_to_target: '{{answer if answer is defined else target}}'
doc_to_text: "
system\nYou are a helpful assistant.
\n
user\n{{question}}
\n
assistant\n"


filter_list:
- filter:
  - function: regex
    regex_pattern: "#### (\\-?[0-9\\.\\,]+)"
  - function: take_first
  name: strict-match
- filter:
  - function: regex
    group_select: -1
    regex_pattern: (-?[$0-9.,]{2,})|(-?[0-9]+)
  - function: take_first
  name: flexible-extract
generation_kwargs:
  do_sample: false
  until:
  # - 'Q:'
  - "
"
  - "
"
tag:
- chain_of_thought
metadata:
  version: 3.0
metric_list:
- aggregation: mean
  higher_is_better: true
  ignore_case: true
  ignore_punctuation: false
  metric: exact_match
  regexes_to_ignore:
  - ','
  - \$
  - '(?s).*#### '
  - \.$
num_fewshot: 0
# output_type: generate_until
repeats: 1
task: gsm8k_qwen_zeroshot
target_delimiter: ""
```

Not sure if I missed anything, so would be great if I can get some help! Thanks!

check max_gen_toks(default:256)
Repository: EleutherAI/lm-evaluation-harness
Issue: How to load local XWinograd dataset
Root cause: Documentation Deficiency

Hello, recently I try to evaluate some hf model on xwinograd benchmark and I download the model and dataset to the local directory. Then I follow  #2491  to change the dataset_path field of "lm-evaluation-harness/lm_eval/tasks/xwinograd/xwinograd_common_yaml" into my local path of XWinograd, but I encountered the following error:
![Image](https://github.com/user-attachments/assets/ca435ed9-b939-409d-9156-efd8cda852d1)
I'm sure the xwinograd data files are not corrupted, and I believe I've downloaded the right repo [Muennighoff/xwinograd](https://huggingface.co/datasets/Muennighoff/xwinograd). Could you please tell me how to solve the problem? Thanks!

Hello!

I think you're confused wrt that issue. The statement there is to set the dataset path to be the location that you downloaded the data to. It looks like you've set it to be the directory that the yaml file is in instead.

> Hello!
> 
> I think you're confused wrt that issue. The statement there is to set the dataset path to be the location that you downloaded the data to. It looks like you've set it to be the directory that the yaml file is in instead.

@StellaAthena Oh, sorry for demonstrating the wrong picture by mistake. Exactly my current version gives the right path to the downloaded data now, but when I try to evaluate the task 'xwinograd_en' I encountered the following error:
![Image](https://github.com/user-attachments/assets/bb16991d-738a-49a2-b7c2-b7af48762c7b)
It seems the error is similar to the issue #2743 , so is there any good resolution to this kind of problem?
 

Hmmm. @baberabb xid you have a chance to look into this further? I know you weren't able to replicate it in a notebook but it seems to be effecting multiple datasets. Maybe it's something that only crops up when running offline?

@StellaAthena I think I've solved my problem following #2821, thanks for your help!