zhiminy/EvalEng huggingface dataset:
Repository: EleutherAI/lm-evaluation-harness
Issue: Enhancement : Allow hf backend to properly load model configurations for smoll3 (and the like) architectures' finetunes
Root cause: Validation Gap

### Issue 

Finetunes from smollm3 (-like) architectures do not have `original_max_position_embeddings` and cannot be loaded in hf or vllm backends easily

### Repro

https://huggingface.co/Tonic/petite-elle-L-aime-3-sft/blob/main/config.json

### Expected behavior

easily load torch finetuned models with hf or vllm backends 

### Actual Behavior

the constructors dont like the config

### Solution 

add some fallbacks to easily load torch finetunes with smollm3-like configurations

So this appears to be a bug with... something? I'm not sure what exactly, but `original_max_position_embeddings` shouldn't be required when using the "yarn" scaling type. Can you try upgrading your `transformers` version, and also can you post the error message here, that will be very helpful.

i'm currently running a fork , but i'll recreate the error and get back to you
Repository: EleutherAI/lm-evaluation-harness
Issue: MNLI task giving (very) different results than the HuggingFace task accuracy metric
Root cause: Validation Gap

Hello, thank you for working on the eval-harness! 

I am trying out the harness in the simplest possible setting: Training a GPT2 model on the MNLI dataset using the HuggingFace trainer, and evaluate the task accuracy for MNLI.

I have subclassed HF's `Trainer` to include eval-harness evaluation in its `evaluate` method, so that I can get regular metrics during training.

```python
class CustomEvalTrainer(Trainer):
    ...

    def evaluate(
        self,
        eval_dataset = None,
        ignore_keys = None,
        metric_key_prefix = "eval",
    ):
        # Run the default LM loss evaluations, this includes HF's own calculation of the MNLI accuracy metric
        results_dict = super().evaluate(eval_dataset, ignore_keys, metric_key_prefix)

        # Additionally perform lm-evaluation-harness evaluations

        # Eval harness requires a saved HF transformers model as input, so we save to file and pass that as input
        eval_model_path = Path(self.args.output_dir) / f"eval_harness_model-{self.state.global_step}"
        self.save_model(eval_model_path)

        eval_harness_args = {
            "model": "gpt2",
            "model_args": f"pretrained={str(eval_model_path)}", # This is the latest trained
            "tasks": ["mnli"],
            "num_fewshot": 0,
            "batch_size": 4,
            "device": "cuda:0",
            "no_cache": False,
            "limit": 100, # Currently set to 100 for quicker evaluation, but the problem persists even when limit=None
        }

        eval_output = evaluator.simple_evaluate(**eval_harness_args)

        # ... log the MNLI results from `eval_output`
```

And I use this CustomEvalTrainer inside [this HuggingFace example script](https://github.com/huggingface/transformers/blob/v4.18.0/examples/pytorch/language-modeling/run_clm.py) to run my training.

The default HF evaluate method also does its own computation of the MNLI accuracy metric, so I have two MNLI accuracy metrics.

The results I obtained are below:

![image](https://user-images.githubusercontent.com/7796965/167310384-88b2e024-7dc1-45f6-905d-43d975b0821d.png)
- `eval_accuracy` is the HF MNLI metric which improves nicely by >4%, `dev.mnli.acc` is the eval-harness MNLI metric which takes a single jump of +2%, then stays still.

Ideally, both of these metrics should be (almost) the same, so I am wondering why my eval-harness metric behaves so strangely.

So my questions are:
1. Can anyone spot any weird issues with the way I'm using the eval-harness? I have a fully-reproducible version of this problem in [this Colab](https://colab.research.google.com/drive/1WzjmKClfrRszu4hYVZ8ZToJchXNUjGyQ?usp=sharing).
2. If not, is it possible that there is a bug in the MNLI task implementation? In particular, this issue https://github.com/EleutherAI/lm-evaluation-harness/issues/248 caught my eye.


Versions:
```
lm-eval==0.2.0
transformers==4.18.0
```

Thank you!
Repository: EleutherAI/lm-evaluation-harness
Issue: How to run lm eval test with Azure openAI endpoint
Root cause: Documentation Deficiency

Hi 

I tried to run lmeval with the below command 

export BASE_URL="https://<sub>.openai.azure.com/"
export API_VERSION="2025-03-01-preview"
export DEPLOYMENT_NAME="gpt-4o"

lm_eval --model local-chat-completions  --tasks mmlu_pro_math  --apply_chat_template --limit 50 --model_args "model=$MODEL_NAME,azure_deployment=$DEPLOYMENT_NAME,api_version=$API_VERSION,base_url=$BASE_URL,api_key=$AUTH_TOKEN,num_concurrent=1,max_retries=3,tokenizer=/data/local_tokenizer,tokenized_requests=False" --confirm_run_unsafe_code --output_path /data/eval_result

Iam getting error as resource not found , I am able to connect to endpoint through curl or python langchan based code
Please send me the correct command to run evaluation
Repository: EleutherAI/lm-evaluation-harness
Issue: How to evaluate Mixture of Experts (MoE) based NeMo models?
Root cause: Unimplemented Feature Gap

Hi team,

I'm working with a Mixture of Experts (MoE) based model trained using NVIDIA NeMo Megatron-GPT, and I'm interested in evaluating it using lm-eval.

So far, I've successfully evaluated standard dense .nemo models by wrapping them in a custom NeMoModel class, but I'm unsure how to proceed with MoE models, which involve dynamic expert routing and additional parallelism.

❓ Questions:
Is there any known support or guidance for evaluating MoE models within lm-eval?

What would be the recommended way to integrate MoE-specific configurations, such as:

Number of experts

Top-k expert selection

Expert parallelism

Are there any existing wrappers or configs used for MoE model inference that I can extend?

Would you accept a PR to integrate support for MoE NeMo models if not already supported?

🧩 Model Context:
Framework: NVIDIA NeMo (Megatron GPT)

Checkpoint: .nemo format

Example config:

I’d appreciate any pointers or existing discussions around this.

Thanks!
Repository: EleutherAI/lm-evaluation-harness
Issue: HFLM Python integration way slower than CLI
Root cause: Algorithmic Error

Hello, I have been currently trying to evaluate some models. When doing it with the following script, it takes about 5 minutes (which is fine):
```
#!/bin/bash
#SBATCH --account bsc03
#SBATCH --qos=acc_debug
#SBATCH --gres=gpu:1
#SBATCH --cpus-per-task=80
#SBATCH --time 00:15:00

export NUMEXPR_MAX_THREADS=128
export CUDA_VISIBLE_DEVICES=0

model="models/Llama-2-7b-hf"
task="arc_easy,arc_challenge,hellaswag,lambada_openai,lambada_standard,openbookqa,piqa,winogrande"
lm_eval --model hf \
        --model_args pretrained=$model,dtype=bfloat16,device_map=auto \
        --task $task \
        --batch_size 64
```

However, when using HFLM class, it takes more than 20, but I am presumably doing the same thing:
```import torch
import transformers
from lm_eval import evaluator
from lm_eval.models.huggingface import HFLM

eval_model = HFLM(
    pretrained="models/Llama-2-7b-hf",
    dtype="bfloat16",
    device_map="auto",
)
results = evaluator.simple_evaluate(
    model=eval_model,
    tasks=["arc_easy","arc_challenge","hellaswag","lambada_openai","lambada_standard","openbookqa","piqa","winogrande"],
    batch_size=64,
)
print(results['results'])
```

I want to use HFLM because I intend to pass an already-loaded model and it's way more convinient than saving the model and calling the CLI script.

Any advice? What am I doing wrong? Is this a bug?

Thanks
Repository: EleutherAI/lm-evaluation-harness
Issue: `flexible-extract` filter in GSM8K evaluation fails to properly normalize numeric answers
Root cause: Validation Gap

  Summary

The flexible-extract filter used for GSM8K evaluation does not correctly normalize extracted answers before comparison. It fails to strip common non-numeric characters, leading to correct model answers being marked as incorrect.

  Steps to Reproduce
1. Run GSM8K evaluation on a model that produces natural language answers.  
2. Observe the results where using "flexible-extract" fails.
3. Inspect the samples.jsonl output file.

  Example of Failure

   * Question: "Rory orders 2 subs for $7.50 each... and she wants to add a $5.00 tip. What will her delivery order cost?"
   * Model Response: "...the total cost of Rory's delivery order is $29.00."
   * Ground Truth Answer: 29
   * `filtered_resps`: ['$29.00']
   * Result: The comparison fails because the extracted string '$29.00' does not equal the target 29.

  This occurs with various characters, including currency symbols ($), commas (,), and trailing periods (.).

  Expected Behavior

  The flexible-extract post-processing logic should strip common non-numeric characters from the extracted
  string to ensure a robust comparison against the numeric ground truth. The evaluation should measure
  mathematical reasoning, not strict adherence to formatting.

  Proposed Solution

  Update the normalization step for flexible-extract to remove characters like $, ,, and . from the
  extracted answer before the final comparison.

Hi! Thanks for reporting this! This sounds very reasonable. If you're interested in submitting a PR for this, will very much appreciate it!

Duplicate of https://github.com/EleutherAI/lm-evaluation-harness/issues/2707.
Repository: EleutherAI/lm-evaluation-harness
Issue: Evaluate Thinking models on MMLU tasks
Root cause: Documentation Deficiency

Hi, apologies if the question is naive. How do we evaluate thinking models like Deepseek or GPT OSS 20B on tasks like MMLU? I see the MMLU task has a generative variant defined. But how do I evaluate reasoning or thinking models? Do we need to write our own postprocessing script to extract the answer post reasoning tags? Thanks

Hi! I think the `mmlu_pro_llama` variant should work on this, but still needs some testing. I recently added `think_end_token: str` to the model args to `vllm` and `hf`, which strips the reasoning traces from the generated context. Should be enough, but might have some edge cases I didn't think of.

ps: use a [subtask](https://github.com/EleutherAI/lm-evaluation-harness/blob/7f04db12d2f8e7a99a0830d99eb78130e1ba2122/lm_eval/tasks/llama3/instruct/mmlu_pro/_mmlu_pro.yaml#L3) with `--limit` for testing.

Thanks :)
Repository: EleutherAI/lm-evaluation-harness
Issue: Plans to add new long-context benchmark like LongBench v2, Babilong, InfiniteBench datasets?
Root cause: Unimplemented Feature Gap

Hi lm-eval team, 
I am wondering if there are plans to add LongBench v2, Babilong, InfiniteBench, and Phonebook datasets to the evaluation tasks? These are useful for long-context LLM evaluation.

- LongBench v2: 2nd-gen long-text benchmark with 20 tasks (8k–2M words), focusing on real-world deep reasoning. https://github.com/THUDM/LongBench
- Babilong: Focuses on ultra-long context reasoning (up to 10M tokens) with 20 tasks, testing fact inference across extended documents.https://github.com/booydar/babilong
- InfiniteBench: A long-context suite (100k+ tokens) with 12 tasks (retrieval, math, code, etc.) for real/scenario-based evaluation.https://github.com/OpenBMB/InfiniteBench

Thanks!

Hi! Thanks for the suggestions! I'd be very interested if anyone wants to implement these, as I'm a bit swamped at the moment. Would welcome PRs!

Willing to take a stab at LongBench v2 if nobody else is working on it!

Edit: LongBench v2 looks like LongBench v1 (there isn't even a separate dataset), now working on BabiLong

Edit2: I got confused; LongBench v2 has a separate dataset, just somewhat buried

@jannalulu  any update? Thanks

Hi lm-eval team, 

I’m very interested in contributing to the EleutherAI lm-evaluation-harness repository.
I saw the issue("good-first-issue") about adding LongBench v2, Babilong, and InfiniteBench benchmarks for long-context evaluation
I would like to work on this, even if no one has been officially assigned. I am willing to implement the evaluation scripts for any of these benchmarks to start with.

I'll start by exploring the datasets and understanding the integration requirements.

Could you please let me know if there are any specific requirements or guidelines I should follow while implementing this?



hi @foreverpiano 

i am a begineer starting with good-first-issues first so can you just guide me like how to go about this issue first if you are comfortable , it will be helpful for me as a begineer as well.

Thankyou 

Hi @TASMAYU, babilong has already been implemented and I'm currently working on longbench v2. Feel free to work on infinitebench!

Hi @baberabb @jannalulu,
I’d like to help by adding InfiniteBench to the evaluation tasks. I see it’s mentioned in this issue and partially covered in #3256, while BabiLong and LongBench v2 have already been merged separately.

Is anyone currently working on InfiniteBench specifically? If not, I’m happy to open a PR that adds InfiniteBench following the patterns of the other long-context benchmarks.

Let me know! :)
Repository: EleutherAI/lm-evaluation-harness
Issue: Feature request: Allow multiple_choice tasks to run in "generate-only" mode for CoT analysis
Root cause: Unimplemented Feature Gap

Hi,

For `multiple_choice` style tasks (e.g., `commonsenseqa`, `arc_challenge`), the harness currently only computes **loglikelihood** without generating full model output. This makes sense for scoring, but sometimes I also want to **inspect the raw generated text / reasoning traces** (e.g., for error analysis or CLEAR framework).

My use case:
- I want to run the exact same multiple-choice datasets, but instead of only scoring with loglikelihood, I want the model to *generate* its answer (possibly with CoT prompts) so I can analyze reasoning traces.
- I don’t need scoring in this mode — just the output text saved via `--log_samples`.

Questions:
1. Is there already a flag or config option to run `multiple_choice` tasks in a "generate-only" mode (similar to `--predict_only`) while still using the same task definitions?
2. If not, would you consider supporting such a mode so we can easily reuse existing MCQ tasks for qualitative analysis?

Thanks!

Hi! this is on my TODO! hopefully should at least have a PR ready in ~2 weeks.
Repository: EleutherAI/lm-evaluation-harness
Issue: Duplicated and incomplete instructions for compiling janitor_util.cpp
Root cause: Documentation Deficiency

Hi! I am big fan of the project. Maybe just a nitpick, but `./lm_eval/decontamination/janitor.py` file contains the following comments:

```
# This is a cpp module. Compile janitor_util.cpp with:
# c++ -O3 -Wall -shared -std=c++11 -fPIC $(python3 -m pybind11 --includes) janitor_util.cpp -o janitor_util$(python3-config --extension-suffix) -undefined dynamic_lookup
```

This only works for MacOS. There are more detailed instructions in `/home/jafar/lm-evaluation-harness/scripts/clean_training_data/README.md` file, but this is what comes up at error trace so it took me a minute to find the correct instructions in README. It is a minor issue, but still worth fixing.
Repository: EleutherAI/lm-evaluation-harness
Issue: IndexError for MBPP dataset when using 4 GPUs
Root cause: Validation Gap

As described in title, we I ran mbpp dataset with the newest lm-eval (0.4.9.1), I encountered:

```
Traceback (most recent call last):
  File "/home/xiangzhong_guest/jyc/heat/eval.py", line 44, in main
    results = simple_evaluate(
              ^^^^^^^^^^^^^^^^
  File "/home/xiangzhong_guest/anaconda3/envs/heat/lib/python3.11/site-packages/lm_eval/utils.py", line 456, in _wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/xiangzhong_guest/anaconda3/envs/heat/lib/python3.11/site-packages/lm_eval/evaluator.py", line 357, in simple_evaluate
    results = evaluate(
              ^^^^^^^^^
  File "/home/xiangzhong_guest/anaconda3/envs/heat/lib/python3.11/site-packages/lm_eval/utils.py", line 456, in _wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/xiangzhong_guest/anaconda3/envs/heat/lib/python3.11/site-packages/lm_eval/evaluator.py", line 600, in evaluate
    task.apply_filters()
  File "/home/xiangzhong_guest/anaconda3/envs/heat/lib/python3.11/site-packages/lm_eval/api/task.py", line 1246, in apply_filters
    f.apply(self._instances)
  File "/home/xiangzhong_guest/anaconda3/envs/heat/lib/python3.11/site-packages/lm_eval/api/filter.py", line 55, in apply
    for inst, resp in zip(instances, resps):
  File "/home/xiangzhong_guest/anaconda3/envs/heat/lib/python3.11/site-packages/lm_eval/filters/selection.py", line 23, in <lambda>
    return map(lambda r: r[0], resps)
```

I think the error is due to a GPU returned an empty list. What should I do if I want to skip an item in the evaluation?
Repository: EleutherAI/lm-evaluation-harness
Issue: code2text benchmark corrupted yaml configs
Root cause: Validation Gap

Hello,

It seems that the configs for the `code2text` tasks are not formatted properly and when I try to run `lm_eval` with:
```bash
--tasks code2text_go,code2text_java,code2text_javascript,code2text_php,code2text_python,code2text_ruby
````
 I get the following error: 
```bash
TypeError: TaskConfig.__init__() got an unexpected keyword argument 'group'.
````

I am submitting a PR with the necessary changes to fix this issue.
Repository: EleutherAI/lm-evaluation-harness
Issue: Use of `__future__` prevents use in python 3.13 and later
Root cause: External Dependency Breakage

While building for [NixOS/nixpkgs](https://github.com/NixOS/nixpkgs), compilation fails on python 3.13 with `error: future-1.0.0 not supported for interpreter python3.13` 

The `future` package is a python 2-to-3 compatibility layer and has been fully absorbed into Python. [What's new in python 3.13](https://docs.python.org/3/whatsnew/3.13.html) states:

>Future statements are no longer triggered by relative imports of the __future__ module, meaning that statements of the form from .__future__ import ... are now simply standard relative imports, with no special features activated. (Contributed by Jeremiah Gabriel Pascual in gh-118216.)

In this case, you're using `from __future__ import annotations` in three places. In all of these cases, you can use [`inspect.get_annotations()`](https://docs.python.org/3/library/inspect.html#inspect.get_annotations) to read the annotations, see https://docs.python.org/3/howto/annotations.html#annotations-best-practices

looks like its fixed? https://github.com/NixOS/nixpkgs/issues/417098

`future` really shouldn't bring in any breaking changes.

> looks like its fixed? [NixOS/nixpkgs#417098](https://github.com/NixOS/nixpkgs/issues/417098)
> 
> `future` really shouldn't bring in any breaking changes.

From the bug:
>  I was able to fix this for my personal use-case

Sadly, not a help when packaging. 

P.S. It's not clear the import of `annotations` from `__future__` does anything in this code.
Repository: EleutherAI/lm-evaluation-harness
Issue: arc_challenge benchmark: acc is "1" but acc_norm is "0"
Root cause: Algorithmic Error

## Situation
I evaluated Qwen/Qwen3-0.6B over the benchmark **arc_challenge** using 
```powershell
lm_eval --model hf     --model_args pretrained=Qwen/Qwen3-0.6B     --tasks arc_challenge     --device cuda:0     --batch_size auto     --log_samples     --output_path ./eval_out   --wandb_args project=Qwen3-0.6B_arc_challenge
``` 
## log samples
Then I checked the log samples, and in question 1 (doc_id 1), I found the acc is "1" while acc_norm is "0". 

<img width="1537" height="632" alt="Image" src="https://github.com/user-attachments/assets/721eff69-e6f2-4bf9-98d7-bf9354a1f77d" />

**Howcome after normalization, it turns out to be incorrect even though this is a multiple choice question??**
Repository: EleutherAI/lm-evaluation-harness
Issue: Gemma3-1b-it's GSM8K reulsts mismatch with offical report 62.8%, mine is at most 22.2%
Root cause: Validation Gap

Thanks for your contributions to the community. When I tried to reproduce the zero-shot gsm8k results reported in Google's new Gemma3-1b-it model, I had low performance compared to the official report one.

hf (pretrained=google/gemma-3-1b-it), gen_kwargs: (None), limit: None, num_fewshot: 0, batch_size: auto
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     0|exact_match|↑  |0.2221|±  |0.0114|
|     |       |strict-match    |     0|exact_match|↑  |0.0250|±  |0.0043|

and the below is the command I am using to run the evaluation:

lm_eval --model hf --model_args pretrained=google/gemma-3-1b-it --tasks gsm8k --device cuda:5 --batch_size auto
Repository: EleutherAI/lm-evaluation-harness
Issue: HaluEval is not present in main
Root cause: Unimplemented Feature Gap

Hi team,

Not sure if there's a reason but I don't see Halueval in the tasks of main branch but the PR has been closed- https://github.com/EleutherAI/lm-evaluation-harness/tree/79d59a3f1e2ec5ce417ef2d4815d2fe0c8b528e8/lm_eval. 

Is there a particular reason why this is not merged? -- cc @pminervini 

PR worked in the past which has been closed - https://github.com/EleutherAI/lm-evaluation-harness/pull/1076
Repository: EleutherAI/lm-evaluation-harness
Issue: lm_eval compatibility with NeMo v2.0 models (config file missing error)
Root cause: Unimplemented Feature Gap


I have been experimenting with **`lm_eval`** using NeMo models. Based on my understanding, there are two different NeMo formats:

* **Version 1.0**

  * Models are saved in the format:

    ```
    Meta-Llama-3-8B-Instruct.nemo
    ```
  * Running `lm_eval` with this format works fine and I don’t face any issues.

* **Version 2.0**

  * The saved model format looks different (see screenshots below).
  * When I try to run `lm_eval` with this format, I get errors saying that the **config file is missing**. i did look inside the folders i could find model.yaml but not model_config.yaml .
  *  Note :  This models works fine with regular nemo inference code.

Screenshots:

*** CLI Code :** 

<img width="848" height="127" alt="Image" src="https://github.com/user-attachments/assets/ec7e11db-0b65-4c28-b196-6dddc91c97ec" />

**nemo 2.0 model format :**

<img width="373" height="161" alt="Image" src="https://github.com/user-attachments/assets/a41d5ddf-dab7-4bec-9300-f6aa0c629780" />

**error with lm_eval wrt nemo 2.0 model**

<img width="1027" height="32" alt="Image" src="https://github.com/user-attachments/assets/c7b8156b-6673-4b01-85fd-9fd93663606a" />

---

### Question

* Is **`lm_eval` currently supported for NeMo version 2.0 models**?
* If yes, how can I properly configure or resolve the missing config error [manually change the name]?
* If not, is there a roadmap or timeline for adding NeMo v2.0 support to `lm_eval`?