zhiminy/EvalEng huggingface dataset:
Repository: EleutherAI/lm-evaluation-harness
Issue: promptsource:* does not work
Root cause: Documentation Deficiency

I would like to run lm_eval with promptsource. According to the [docs](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/docs/new_task_guide.md#importing-a-prompt-from-promptsource), I should be able to run eval with all the promptsource templates using `use_prompt: "promptsource:*"`. However, I get the following error when I try it:

```bash
ValueError: * not in prompt list ['GPT-3 Style', 'I think they mean', 'Who or what is/are', 'by p they mean', 'does p stand for', 'does the pronoun refer to', 'in other words', 'p is/are r', 'replaced with', 'the pronoun refers to']
```

The command I run:
```bash
lm_eval  --model hf --model_args pretrained=meta-llama/Llama-3.2-1B --include_path configs/eval --tasks my_eval
```
 and the config I use config:
```yaml
group: my_eval
use_prompt: "promptsource:*"
output_type: multiple_choice
task:
  - wsc
```

Is this no longer supported?

Hi! The docs seem to be wrong, as it doesn't seem possible to specify `*` as a catch-all here. Rather you'll have to manually specify each prompt. The current way to create slight variants of the same task is a bit cumbersome (but working on fixing that in a [refactor](https://github.com/EleutherAI/lm-evaluation-harness/tree/smolrefact)! ), but you can import a config with `include`  and overload fields, for example in the `lm_eval/tasks/super_glue/wsc` folder:
```
tag: my_eval
include: default.yaml
task: wsc_gpt
use_prompt: "promptsource:GPT-3 Style"
```
here `tag` allows to run all the tagged tasks with `--tasks my_eval`

Thanks for explanation and the work around! Is the catch-all feature going to be included in the refactor as well? 

> Thanks for explanation and the work around! Is the catch-all feature going to be included in the refactor as well?

I'm not sure about prompt source; haven't used it, so will need to take a look (they also don't seem to have any new commits in 2 years), but will be much easier to generally quickly create task variants in a single config.
Repository: EleutherAI/lm-evaluation-harness
Issue: Are all the CUDA packages really necessary if installing lm-eval[api]?
Root cause: Architectural Constraint

I have containerized this tool (Awesome work by the way!) and it has been very helpful.

I noticed, when installing `lm-eval[api]` the container ends up being about 8GB in size whether or not I install lm-eval[api] or lm-eval via PIP.

Since I am automating this workflow and deploying/testing via gitlab runners that dont have GPU computer capabilities (Just using the remote VLLM endpoints), it doesnt seem like all of the nvidia packages should be required.

Is there a way to lighten the installation footprint if you know you dont need CUDA?
Repository: EleutherAI/lm-evaluation-harness
Issue: Results not matching papers?
Root cause: Environment Incompatibility

Hello! I am new to the tool, so I am sorry if I am using it wrong.

I am running gpt-oss-120b via 2X H100 GPUs.  It is being served via vLLM and an openai compatible endpoint.

After running an mmlu test, I received these results

|      Groups      |Version|Filter|n-shot|Metric|   |Value |   |Stderr|
|------------------|------:|------|------|------|---|-----:|---|-----:|
|mmlu              |      2|none  |      |acc   |↑  |0.2849|±  |0.0037|
| - humanities     |      2|none  |      |acc   |↑  |0.3541|±  |0.0067|
| - other          |      2|none  |      |acc   |↑  |0.2678|±  |0.0079|
| - social sciences|      2|none  |      |acc   |↑  |0.2395|±  |0.0077|
| - stem           |      2|none  |      |acc   |↑  |0.2426|±  |0.0076|


For this, I ran with a script, but ultimately I run this:
``` 
# Run the eval
lm_eval \
    --model local-completions \
    --confirm_run_unsafe_code \
    --tasks mmlu --batch_size ${LM_EVAL_BATCH_SIZE} \
    --model_args \
    model=${ENDPOINT_MODEL},base_url=${OPENAI_BASE_URL}/v1/completions,num_concurrent=2,max_retries=3,tokenized_requests=False,tokenizer=/model
```


LM_EVAL_BATCH_SIZE is set to 2.  Am I doing anything wrong? I also run humaneval and received a score of 0.22

I think this is an issue with vllm and EAGLE3 speculative decoding. 

After I removed this feature and re-ran the benchmark,  it was as expected!
Repository: EleutherAI/lm-evaluation-harness
Issue: Incorrect log-likelihood token scoring
Root cause: Algorithmic Error

Update: I confused position indexing (next-token distribution)

## Summary

~~[`base._loglikelihood_tokens`](https://github.com/EleutherAI/lm-evaluation-harness/blob/b0b76d87fc84b2bfdfa88b71ce4592f35f707428/lm_eval/base.py#L221) calculates incorrect scores for prompt continuations because of last-token truncation.~~


## Details
~~Given a "continuation" token sequence of length `N`, only the first `N-1` entries get appended to the corresponding context to form a model input. I.e. `model_input = ctx + cont[:-1]` , as opposed to the complete `model_input = ctx + cont`. The relevant lines for last-token-deletion:~~

```python
inp = torch.tensor(
    (context_enc + continuation_enc)[-(self.max_length + 1) :][:-1],
    dtype=torch.long,
).to(self.device)
```
~~Its surrounding source code is found [here](https://github.com/EleutherAI/lm-evaluation-harness/blob/b0b76d87fc84b2bfdfa88b71ce4592f35f707428/lm_eval/base.py#L266).~~

~~On inspection, what ends up happening is that the continuation logits get improperly sliced from the logits on model input, `inp`.~~

```python
inplen  = inp.shape[0]  # NOTE: No batch-dim
...
contlen = len(continuation_enc)
cont_logits = logits[inplen - contlen : inplen]
```
~~Since the actual number of continuation tokens (`contlen`) is greater than the number of continuation tokens fed to the model thru `inp`, we gather continuation logits from the wrong token positions.~~
+
https://github.com/EleutherAI/lm-evaluation-harness/blob/b0b76d87fc84b2bfdfa88b71ce4592f35f707428/lm_eval/base.py#L317

## Who Can Help
cc @leogao2  - What's the reasoning behind last token deletion? What am I missing?

@jon-tow Hello, I also have doubts about why we need to remove the last token. If you could explain it to me, I would be very grateful.
Repository: EleutherAI/lm-evaluation-harness
Issue: [feat] openai-mrcr
Root cause: Unimplemented Feature Gap

[mrcr](https://huggingface.co/datasets/openai/mrcr) is a long-context dataset where the "needle" blends into the dataset, goes from 4k to 1M. One of the long context benchmarks that haven't been saturated, would be good to support. The model has to prepend a string that's graded as an exact match while the rest of the response is graded on a similarity score. Would be good to support
Repository: EleutherAI/lm-evaluation-harness
Issue: [feat] openai-graphwalks
Root cause: Unimplemented Feature Gap

[graphwalks](https://huggingface.co/datasets/openai/graphwalks) is a long-context dataset that evaluates multi-hop long-context reasoning. The model has to do some amount of reasoning about a set of nodes that it sees at inference. Would be useful as another long-context reasoning task I think
Repository: EleutherAI/lm-evaluation-harness
Issue: assist
Root cause: Documentation Deficiency

This is not a issue per se,
I just wondering if I can use reasoning token is correct or not? think_end_token

`!lm_eval --model hf --model_args pretrained=EpistemeAI/metatune-gpt20b-R0,parallelize=True,dtype=bfloat16 --tasks gsm8k_cot,gpqa_diamond_cot_n_shot,mmlu_stem  --num_fewshot 0 --apply_chat_template --fewshot_as_multiturn --gen_kwargs temperature=1,top_p=1,think_end_token,max_new_tokens=1701  --batch_size 1 --limit 30  --device cuda:0  --output_path ./eval_harness/gpt-oss-20b3`

I suggest you check the tokenizer.json file of the model to find the actual think_end_token it uses, and then set it accordingly in the command-line arguments. Alternatively, you could interact with the model and observe its output to find the correct think_end_token.
Repository: EleutherAI/lm-evaluation-harness
Issue: [Bug] Truncation before think_end_token causes incomplete reasoning to be parsed as the final response
Root cause: Validation Gap

Description
There appears to be a logical issue in how truncated outputs are handled when using the think_end_token to separate a model's reasoning from its final response.

The current system relies on the presence of think_end_token to identify and extract the final response (resp). However, if the model's generation is truncated before it emits this token, the parsing logic fails.

Context & Steps to Reproduce
I am running inference tasks using vLLM, even on a setup with 2x L20 GPUs, a batch_size of 1, and a max_gen_tokens set to 16384.

In some cases, the model's generation is cut off mid-reasoning, before it has completed its thought process and generated the think_end_token.

Actual Behavior
When the output is truncated mid-thought:

The think_end_token is absent from the generated text.

The parsing logic, failing to find the separator, incorrectly assumes the entire output is the final response.

As a result, the resp field is populated with the model's incomplete, truncated reasoning.

Expected Behavior
If the model's output is truncated before the think_end_token is generated, this should be treated as a task failure or an incomplete generation.

In this scenario, the resp field should be empty (e.g., "" or None), as the model never successfully completed its reasoning to produce a final answer. The current behavior of returning the truncated thought process as the response is incorrect.

Thank you for maintaining this project. I believe addressing this truncation edge case will significantly improve the robustness of parsing model outputs, especially for complex reasoning tasks.

I'm happy to provide more details or test data if needed.

Facing similar issues here when running Qwen3-32B with vLLM (max_model_len=32k) - would appreciate help!

Update: I resolved this issue by increasing the max_gen_tokens, as it was defaulted to 256. This is not directly related to the issue you are facing, but hopefully it helps!

@StephenXie I'm glad your problem is solved. I re-examined my description and realized it was problematic. I actually set max_gen_tokens to 16384 and I have now corrected my description. For my problem, in the near future I‘ll submit a PR to present my concise solution.

+1
Repository: EleutherAI/lm-evaluation-harness
Issue: Failed to reproduce Qwen3-0.6B-Base performance on GSM8K
Root cause: Configuration Error

`accelerate launch -m lm_eval --model hf --model_args pretrained=Qwen/Qwen3-0.6B-Base,trust_remote_code=true,dtype=bfloat16 --tasks gsm8k --batch_size 4 --num_fewshot 8 --confirm_run_unsafe_code --gen_kwargs "do_sample=False,max_new_tokens=512"`

I have tried this command but failed to produce the reported score 59.59 in the original paper. Anything I have missed?

|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     8|exact_match|↑  |0.4981|±  |0.0138|
|     |       |strict-match    |     8|exact_match|↑  |0.4973|±  |0.0138|

<img width="365" height="312" alt="Image" src="https://github.com/user-attachments/assets/c15a3289-0604-47eb-829f-1d51f78ad157" />


Are all your parameter configurations consistent with those reported in the paper? If the paper does not provide the complete configuration, I suggest you check the logs to see if the generation is being truncated. If so, you may need to adjust the relevant parameters accordingly.
Repository: EleutherAI/lm-evaluation-harness
Issue: [Bug?] GGUF model fails with tasks where the prompt is empty and first token logprob is None
Root cause: Validation Gap

I'm trying to evaluate a GGUF model in Harness on the `xnli_en` task. I'm using a llama.cpp inference server and this is the command I use to launch Harness:

```
python -m lm_eval --tasks xnli_en --num_fewshot 0 --batch_size 1 --model gguf --model_args base_url=http://0.0.0.0:8000
```

And this is the error I get:
```
  File "lm_eval/models/gguf.py", line 28, in get_result
    continuation_logprobs = sum(tokens_logprobs[idx:-1])
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: unsupported operand type(s) for +: 'int' and 'NoneType'
```

It fails in the [`get_result`](https://github.com/EleutherAI/lm-evaluation-harness/blob/03c44adc0586f88bb343a74da1a1c602103536dd/lm_eval/models/gguf.py#L24) function of `gguf.py`. I added some prints to check the values of the relevant variables before the line where the error happens and I get these values:

```
idx=0
context_length=0
offsets=[0, 2, 5, 8, 13, 14, 19, 20, 22, 23, 24, 29, 31, 37, 38, 42, 43, 46, 53, 57, 61, 64, 69, 72, 76, 83, 87, 95, 99, 103, 104]
tokens=['  ', 'And', ' he', ' said', ',', ' Mama', ',', ' I', "'", 'm', ' home', '.,', ' right', '?', ' Yes', ',', ' He', ' called', ' his', ' mom', ' as', ' soon', ' as', ' the', ' school', ' bus', ' dropped', ' him', ' off', '.', '  ']
tokens_logprobs=[None, -6.998805046081543, -5.038498401641846, -2.2241363525390625, -0.8130159378051758, -12.096660614013672, -0.23284895718097687, -1.0226982831954956, -1.8182697296142578, -0.25342831015586853, -5.443194389343262, -7.728053569793701, -8.038619041442871, -0.31676554679870605, -4.495013236999512, -0.6343331336975098, -6.412057399749756, -5.437701225280762, -2.835432767868042, -1.886159062385559, -6.232597351074219, -4.107452392578125, -0.029145970940589905, -4.905628204345703, -5.4738616943359375, -2.2996106147766113, -0.9678385853767395, -0.015012849122285843, -0.19180922210216522, -0.8565495610237122, -1.0037039518356323]
```

Apparently the first token is always empty and the function fails because the first element of `tokens_logprobs` (and `top_logprobs` further down as well) are `None`. The `xnli` tasks in particular have an empty prompt (`doc_to_text: ''`), so I suppose that is why the first token is an empty token with logprob `None`. Apparently this is the expected behaviour of the llama.cpp server (?). The same does not happen in other multiple-choice tasks where the prompt is not empty, e.g. `openbookqa`.

Am I doing something wrong, or should the implementation of `gguf.py` be adapted to skip this None?
Repository: EleutherAI/lm-evaluation-harness
Issue: Request for Corporate CLA option to enable employees to contribute

I reached out to contact@eleuther.ai two weeks ago about this, but have not received a response. Giving this a try instead.

I noticed that EleutherAI's projects use GitHub’s CLA Assistant, which currently presents only an individual Contributor License Agreement (CLA) to would-be contributors. As is typical under corporate law in most jurisdictions, individual developers generally lack authority to bind their employers to legal agreements. Projects that wish to accept contributions from companies therefore generally need to offer a corporate CLA signed by an authorized company representative.

Could you please let me know if there’s a corporate CLA option available, or if one could be established? 

Thanks! 





Hi! Sorry missed this, and thanks for bringing this up. Let me poke around and see what we can do. You wouldn't have an example of a corporate CLA you've come across? Would be helpful as a reference point!

Sure! The [Apache Software Foundation's Corporate CLA](https://www.apache.org/licenses/cla-corporate.pdf) has been used as a model by many other organizations. 

Of course, my "favorite" CLA is no CLA at all, and instead using the Developer Certificate of Origin (DCO). Much less friction! It's good enough for the Linux kernel! 

We have signed an adapted Apache CLA with multiple companies before. If you'd like to send over a signed doc we'd be happy to countersign and return it.
Repository: EleutherAI/lm-evaluation-harness
Issue: [Bug] mgsm task fails with "RuntimeError: Dataset scripts are no longer supported, but found mgsm.py"
Root cause: External Dependency Breakage

Hello,

When trying to run the `mgsm` task (e.g., `mgsm_direct_en` or `mgsm_en_cot`), I encountered the following error:

```
RuntimeError: Dataset scripts are no longer supported, but found mgsm.py
```

### Problem Analysis

I tracked down the issue to the task configuration files in `lm_eval/tasks/mgsm/`. The `.yaml` files in my version point to `dataset_path: juletxara/mgsm`.

This specific dataset repository (`juletxara/mgsm`) appears to be an older version that relies on a root `mgsm.py` script. This script is incompatible with modern versions of the `datasets` library (\>= 2.11.0), which have deprecated and now prohibit loading dataset scripts this way.

### Workaround (For Info)

As a temporary workaround, I found that **`jbross-ibm-research/mgsm`** (which appears to be a script-less mirror) works correctly with the new `datasets` library.

When I manually edited my local `.yaml` files to use `jbross-ibm-research/mgsm`, the task ran successfully.

I am sharing this finding in case it helps the team investigate or find a permanent, official fix for the `juletxara/mgsm` incompatibility.

Thank you\!

(P.S. This is my first time submitting an issue, so please forgive me and let me know if I've missed any information or made any mistakes in formatting.)
Repository: EleutherAI/lm-evaluation-harness
Issue: Plans to add LiveCodeBench
Root cause: Unimplemented Feature Gap

Hi lm-eval team,


I am currently working on adding LiveCodeBench(lcb) to lm-eval, I'd like to know if nobody is working on it right now, so we don't make double work

Main dataset: https://huggingface.co/datasets/livecodebench/code_generation_lite
Papers: https://arxiv.org/pdf/2403.07974

Thanks!
Repository: EleutherAI/lm-evaluation-harness
Issue: How to run accuracy test on Grok-2 through lm_eval tool
Root cause: Documentation Deficiency

HI 

Gork-2 needs a customized tokenizer as the introduction in https://huggingface.co/xai-org/grok-2/blob/main/README.md, so for SGlang, we can do perf benchmark like below command:

python3 -m sglang.bench_serving --dataset-name random --random-input-len 1024 --random-output-len 1024 --num-prompt 64  --random-range-ratio 1.0 --max-concurrency 64 --output-file Grok-2-in1024-out1024-c64.json --port 30001 --tokenizer-path /models/grok-2/tokenizer.tok.json

but for lm_eval, how do we run the accuracy test with the customized tokenizer? 


I tried to run the below command, which is with the tokenizer path, but still failed:
lm_eval --model local-completions --model_args model=/models/grok-2/,base_url=http://localhost:30000/v1/completions,num_concurrent=256,max_retries=10,max_gen_toks=2048,trust_remote_code=True,tokenizer=/models/grok-2/tokenizer.tok.json --tasks gsm8k --batch_size auto --num_fewshot 5 --output_path ./results_accuracy/SGL_Grok_2

my lm_eval version is 0.4.9.1

Thanks
Repository: EleutherAI/lm-evaluation-harness
Issue: TriviaQA: trailing whitespace in predictions lowers EM
Root cause: Validation Gap

Hi, thanks for this great work! In our evaluation of Marin-32B, we found that TriviaQA EM was dropping solely due to trailing whitespace in the model’s predictions. Applying `.strip()` to the outputs raised EM from 0.1142 → 0.7360, without changing any actual answers.

The current implementation appears to remove only leading whitespace. If this adjustment seems reasonable, I can submit a small PR to add the trailing-whitespace normalization.

```json
{
   "doc_id":1,
   "doc":{
      "question":"Which Lloyd Webber musical premiered in the US on 10th December 1993?",
      "question_id":"tc_33",
      "question_source":"http://www.triviacountry.com/",
      "answer":{
         "aliases":[
            "Sunset Blvd",
            "West Sunset Boulevard",
            "Sunset Boulevard",
            "Sunset Bulevard",
            "Sunset Blvd."
         ],
         "normalized_aliases":[
            "sunset boulevard",
            "sunset bulevard",
            "west sunset boulevard",
            "sunset blvd"
         ],
         "matched_wiki_entity_name":"",
         "normalized_matched_wiki_entity_name":"",
         "normalized_value":"sunset boulevard",
         "type":"WikipediaEntity",
         "value":"Sunset Boulevard"
      }
   },
   "target":"['Sunset Blvd', 'West Sunset Boulevard', 'Sunset Boulevard', 'Sunset Bulevard', 'Sunset Blvd.']",
   "filtered_resps":[
      "sunset boulevard "
   ],
   "exact_match":0.0
}
```

Example of what we are seeing where `sunset boulevard ` doesn't get matched with `sunset boulevard`

Hi! Thanks for raising this, and would appreciate a PR! I think stripping all surrounding whitespaces  in the [remove_whitespace](https://github.com/EleutherAI/lm-evaluation-harness/blob/68b03658ace40fa93221518b30096485a2387c58/lm_eval/filters/extraction.py#L107) filter would work better in almost all cases (unless I'm forgetting something obvious).

Perfect, we'll upstream this then! Just wanted to make sure that change wasn't too broad for your purposes!
Repository: EleutherAI/lm-evaluation-harness
Issue: Double check all of the zero/few-shot formats
Root cause: Documentation Deficiency

Once we have everything implemented, we should double check that we actually have the same zero/few shot format as was used for GPT3. The paper is, as expected, terribly vague, so we should try contacting OA for more info. Making sure this is done correctly is critical before proceeding further.
Repository: EleutherAI/lm-evaluation-harness
Issue: Perplexity scores do not match with HF evaluate
Root cause: Algorithmic Error

Hi @StellaAthena 

thanks for you amazing work! I'm trying to add a new perplexity task to lm-evaluation-harness but I found that the perplexity scores do match the ones from HF evaluate (same model and same data). The differences should not be due to the rolling window implementation since it occurs also when input texts are very short.

You can find a minimal example on Colab: https://colab.research.google.com/drive/16Q5t5I4AyVkvLDw7LWhgNR52BQJMgcfs?usp=sharing

- gpt2 model
- two input texts from wikitext
- eval-harness PPL = **674.95**
- HF evaluate PPL = **37.60**

Is there any misunderstanding on my side or is one of the two PPL implementations wrong?

Best,
Malte

Hi @malteos! I've opened a PR to `evaluate` which uses base e for the perplexity calculation instead. That results in the following perplexities, with `add_start_token=False`: 
`'Robert Boulter is an English film , television and theatre actor .',`: 97.61385345458984
`'In 2006 , Boulter starred alongside Whishaw in the play Citizenship'`: 341.8270263671875

These line up with the a manual ppl calculation as follows (the gpt2 implementation shifts the label tokens one over on the inside): 
```
In [20]:
    ...: model = GPT2LMHeadModel.from_pretrained('gpt2')
    ...: tokenizer = GPT2TokenizerFast.from_pretrained('gpt2')
    ...:
    ...: x = "In 2006 , Boulter starred alongside Whishaw in the play Citizenship"
    ...: input_ids = torch.tensor(tokenizer.encode(x))
    ...:
    ...: loss = model(input_ids, labels=input_ids)[0]
    ...: print(np.exp(loss.cpu().detach().numpy()))
```

Setting `add_start_token=True` to allow for ppl calculation including the first word results in [105.20044708251953, 282.4992370605469], which is still divergent, so it looks like these might be different. Hopefully this is a good starting point for debugging (I haven't had a chance to dig into the lm-evaluation-harness implementation yet).

@malteos does the above reply address your concern?

Closing as inactive.