zhiminy/EvalEng huggingface dataset:
Repository: EleutherAI/lm-evaluation-harness
Issue: Performance decreasing when passing from 0 to few shots?
Root cause: Algorithmic Error

Hello! In order to evaluate In-Context Learning in some LLMs, I tried the following code to check for the performances of google/gemma-2-9b on mmlu_elementary_mathematics. 

`for n_shots in shots_list:
    print(f"\nTesting with {n_shots} shot(s)...")
    gc.collect()
    torch.cuda.empty_cache()
    print("Memory Cleaned!")
    
    eval_output = lm_eval.simple_evaluate(
        model="hf",
        model_args=f"pretrained={model_id},load_in_4bit=True",
        tasks=task_list,
        num_fewshot=n_shots,
        limit=10,
        batch_size=8,
        log_samples= True,
        
        # CORRECT SETTINGS FOR BASE MODELS
        apply_chat_template=False,
        fewshot_as_multiturn=False
    )

    accuracy = eval_output['results']["mmlu_elementary_mathematics"]['acc,none']
    results_data.append({
        "shots": n_shots,
        "accuracy": accuracy,
        "samples": json.dumps(eval_output['samples'], indent=4)
    })`



Results are as follows:

0-Shots: Accuracy = 0.7
1-Shot: Accuracy = 0.5
2-Shots: Accuracy = 0.6.

Is it possible that accuracy decreases passing from 0 to 1 and then 2 shots? Am I doing something wrong? Do you have any suggestions if I want to demonstrate the increase in accuracy with in-context learning?
Repository: EleutherAI/lm-evaluation-harness
Issue: [feat] absence-bench
Root cause: Unimplemented Feature Gap

AbsenceBench ([paper](https://arxiv.org/abs/2506.11440), [repo](https://github.com/harvey-fin/absence-bench)) is a benchmark that evaluates LLMs' ability to identify missing information from long contexts, using examples from poetry, numerical sequences, and GitHub pull requests. Unlike needle-in-a-haystack tests that ask models to find inserted off-topic information, AbsenceBench prompts models to detect and recall what has been intentionally omitted—a distinct and underexplored capability.

This benchmark would be a valuable addition to lm-eval because it finds a reasoning gap in current LLMs that existing long-context benchmarks don't address.
Repository: EleutherAI/lm-evaluation-harness
Issue: Feature request: stdio-based harness
Root cause: Unimplemented Feature Gap

Would it be possible to add an evaluation mode where the model is represented simply as an arbitrary command that takes input from `stdin` and outputs predictions to `stdout`? Perhaps with some trick for batch processing.

E.g. `python lm-eval --tasks lambada,hellaswag --command console-gpt2 console-arg0 console-arg1` where `console-gpt2` is an executable name.

You mean something like this?
```bash
python main.py --model gpt2  --model_args pretrained=EleutherAI/gpt-neo-2.7B --device 0 --tasks lambada,hellaswag
```

No, I mean something like this:

```
python main.py --model-command totally-not-a-gpt2-executable totally-not-a-gpt2-executable-parameter1 ... --tasks lambada,hellaswag
```
which would then launch a different process (`totally-not-a-gpt2-executable totally-not-a-gpt2-executable-parameter1`) which would provide whatever implementation (e.g. TensorFlow, PyTorch, Theano, raw C code, remote exe launched over `ssh` etc) of whatever language model running that would be answering the task queries via standard input-output.

See the PR for a possible implementation of the idea.
Repository: EleutherAI/lm-evaluation-harness
Issue: max_model_length for gemma3-12b is set to 2048 (_DEFAULT_MAX_LENGTH) with the hf processor
Root cause: Configuration Error

There is this line in the hf processor:  https://github.com/EleutherAI/lm-evaluation-harness/blob/59b3ba263be2c6ac7f60a4b6a1219f8a8acb552b/lm_eval/models/huggingface.py#L468
```        if hasattr(self.tokenizer, "model_max_length"):
            if self.tokenizer.model_max_length == TOKENIZER_INFINITY:
                return self._DEFAULT_MAX_LENGTH
```

I wonder why is this the case? I am evaluating gemma3-12b which has `model_max_length = TOKENIZER_INFINITY  =  1000000000000000019884624838656`
this sets the max_length to only 2048 (_DEFAULT_MAX_LENGTH). Is this intended?
Repository: EleutherAI/lm-evaluation-harness
Issue: [BUG] Cannot pickle 'msgspec.msgpack.Encoder' object when call save_results_samples func
Root cause: Interface Contract Mismatch

# How to reproduce?
Init a model
```py
from lm_eval.api import registry
from lm_eval import evaluator, utils
model = registry.get_model('vllm').create_from_arg_string(
                config.model_args,
                {
                    "batch_size": config.batch_size,
                    "max_batch_size": config.max_batch_size,
                    "device": config.device,
                },
            )
```
Pass the initialized model of the `lm_eval.app.model.LM` class as the `model` parameter (**which is valid according to Line 255**) into the `simple_evaluate` function in `evaluator.py`. 
Then, the `evaluation_tracker` is called in 
https://github.com/EleutherAI/lm-evaluation-harness/blob/69ecd0b929701d346c1119d5cd0563ade4ab3536/lm_eval/evaluator.py#L348-L356

Here comes the BUG, the `model_source` parameter should be `Str` type but receives  `lm_eval.app.model.LM`
https://github.com/EleutherAI/lm-evaluation-harness/blob/69ecd0b929701d346c1119d5cd0563ade4ab3536/lm_eval/loggers/evaluation_tracker.py#L83-L90

leads to a fail when calling the `asdict` function in https://github.com/EleutherAI/lm-evaluation-harness/blob/69ecd0b929701d346c1119d5cd0563ade4ab3536/lm_eval/loggers/evaluation_tracker.py#L223

Hi! Thanks for flagging this. The PR should fix it
Repository: EleutherAI/lm-evaluation-harness
Issue: Forced version of `numexpr` is incompatible with latest NumPy
Root cause: External Dependency Breakage

It is sort of not a problem with this package exactly, but because `lm-eval` forces a specific version of `numexpr`, but neither `numexpr` nor `lm-eval` limit NumPy, `numexpr` fails to install and so does `lm-eval` due to the issue, that was fixed in the later version of `numexpr`: https://github.com/pydata/numexpr/issues/397

Workaround: force install `numpy==1.22.*`
Repository: EleutherAI/lm-evaluation-harness
Issue: Evaluation result of hendrycks_math keeps 0
Root cause: Algorithmic Error

Hi, I've tried several models on hendrycks_math benchmark, but all of them get 0 values

Here is my command: `lm_eval --model vllm     --model_args pretrained=model/Qwen3-4B-Instruct-2507,dtype=auto,data_parallel_size=4     --tasks hendrycks_math500     --batch_size auto`

<img width="1398" height="100" alt="Image" src="https://github.com/user-attachments/assets/bdcc618b-38f7-46db-a74b-83468e7aeee1" />

<img width="1420" height="103" alt="Image" src="https://github.com/user-attachments/assets/31c6788c-2246-45a9-af4b-d6b33d04cdbc" />

<img width="1419" height="102" alt="Image" src="https://github.com/user-attachments/assets/33e8d721-483f-4259-99c4-a0113d7a6965" />

<img width="1403" height="100" alt="Image" src="https://github.com/user-attachments/assets/83d52049-cd71-488a-a3f0-307b0ced8fa8" />

Me too. @jannalulu 

#3477 I fix the problem by replacing the "extract-answer" part and adding normalization for "\xxx{}". Then we can get the correct results:
```
[2025-12-24 03:04:50] INFO evaluation_tracker.py:272: Output path not provided, skipping saving results aggregated
hf (pretrained=Qwen/Qwen2.5-7B), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: 5
|      Tasks      |Version|Filter|n-shot|  Metric   |   |Value|   |Stderr|
|-----------------|------:|------|-----:|-----------|---|----:|---|-----:|
|hendrycks_math500|      1|none  |     0|exact_match|↑  | 0.37|±  |0.0216|
```
I have submitted the PR #3478.

Or you can use `minerva_math500`, which has better prompt and more convincing results .
Repository: EleutherAI/lm-evaluation-harness
Issue: [bug] in hendrycks_math
Root cause: Validation Gap

https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/hendrycks_math/utils.py#L69

this logic is so naive that
model which generate something like this
"Mark your solution with \\boxed "
result in failure.
Repository: EleutherAI/lm-evaluation-harness
Issue: Forced version of `datasets` is incompatible with Python 3.10
Root cause: External Dependency Breakage

See https://github.com/huggingface/datasets/issues/3324

Hey, @lostmsu ! Thanks for raising an issue. We don't force a specific `datasets` version but instead set the minimum version to `2.0.0` [here](https://github.com/EleutherAI/lm-evaluation-harness/blob/2598f990372e17f39ca432b03f0c279f6fa6118b/setup.py#L23). If your environment has `datasets` pre-installed you may need to upgrade it for Python3.10.

I am being bitten by the default pip version resolution here? The version of `datasets` I got was by creating a fresh conda + py3.10 environment and running the setup.py that you linked.
Repository: EleutherAI/lm-evaluation-harness
Issue: Trouble running Decontamination
Root cause: Documentation Deficiency

I’m having trouble getting the decontamination to run, even though I followed all the steps described in the [Deepwiki](https://deepwiki.com/search/how-to-do-decontamination-in-l_70837fc5-6273-4349-84e5-a300bc665294?mode=fast) guide.

I'm working on PiQA benchmark (English version). I wanted to test the flow before applying to my custom training data, so I generated the .bkt.txt files for the training split of PiQA, sorted then compressed them, with the info.json file in the same directory.

added the task attributes to the YAML file,
```
task: piqa
dataset_path: json
# output_type: multiple_choice
dataset_kwargs:
  data_files:
    validation: /mnt/h/Custom_Models/lm-evaluation-harness/scripts/dataset/english_datasets/piqa/piqa_test_labeled.jsonl
output_type: generate_until
training_split: null
validation_split: validation
test_split: null
doc_to_text: "Question: {{goal}} 0.{{sol1}} 1.{{sol2}} Answer with the number corresponding to the correct choice. Answer: "

doc_to_target: label
**should_decontaminate: true
doc_to_decontamination_query: "{{goal}}"**
process_results: !function utils.process_results_piqa
metric_list:
  - metric: exact_match
    aggregation: mean
    higher_is_better: true
    ignore_case: true
    ignore_punctuation: true
    regexes_to_ignore:
      - " "
generation_kwargs:
  until:
    - "Q:"
    - "\n\n"
    - "
"
    - "."
  do_sample: false
  temperature: 0.0
metadata:
  version: 1.0
```

I couldn’t find any documentation or examples showing how the n-grams generator output is supposed to be wired into the evaluation task. I tried setting the LM_EVAL_DECONTAMINATION_PATH environment variable and also experimented with hard coding the path directly in decontaminate.py, but in both cases the outputs look unchanged. I don’t see any metrics with the _decontaminate suffix.

I’m likely missing something obvious here. Any pointers or guidance on how to point out to the generated training ngrams directory?

@ceferisbarov @kiersten-stokes can you help out?
Repository: EleutherAI/lm-evaluation-harness
Issue: Ministral-3 / Mistral3 models fail to load: AutoModelForCausalLM incompatible with Mistral3ForConditionalGeneration
Root cause: Architectural Constraint

## Summary

`lm-evaluation-harness` cannot evaluate **Ministral-3** models (e.g. `mistralai/Ministral-3-3B-Instruct-2512`) even when using **Transformers 5.0.0rc1** and the latest `mistral-common`.

The failure occurs because `lm_eval` always loads Hugging Face models via `AutoModelForCausalLM`, while Ministral-3 models are exposed in Transformers as **`Mistral3ForConditionalGeneration`**, not as causal LM classes.

This leads to a hard failure during model initialization.

---

## Environment

- **lm-evaluation-harness**: current `main`
- **transformers**: `5.0.0rc1`
- **python**: 3.12
- **torch**: CUDA-enabled
- **model**: `mistralai/Ministral-3-3B-Instruct-2512`  
  (same behavior with `*-BF16`)

---

## Reproduction command

```bash
lm_eval \
  --model hf \
  --model_args pretrained=mistralai/Ministral-3-3B-Instruct-2512 \
  --tasks hellaswag \
  --device cuda:0 \
  --batch_size 8
```

---

## Error message

```text
ValueError: Unrecognized configuration class
<class 'transformers.models.mistral3.configuration_mistral3.Mistral3Config'>
for this kind of AutoModel: AutoModelForCausalLM
```

---

## Root cause analysis

- Transformers ≥ 5.x correctly introduces:
  - `Ministral3Config`
  - `Mistral3ForConditionalGeneration`
- `Ministral3Config `is **not mapped** to `AutoModelForCausalLM`
- `lm_eval ` **hard-codes** usage of `AutoModelForCausalLM`
- Therefore, Ministral-3 cannot be loaded at all, even with:
  - `trust_remote_code=True`
  - `backend=causal`
  - BF16 / FP8 variants

This is not a Transformers bug. Loading works when done explicitly:

```python
from transformers import Mistral3ForConditionalGeneration

model = Mistral3ForConditionalGeneration.from_pretrained(
    "mistralai/Ministral-3-3B-Instruct-2512"
)
```
However, `lm_eval `has no code path to use this model class.

---

## Expected behavior

One of the following:
1. `lm_eval `adds explicit support for `Mistral3ForConditionalGeneration`
2. `lm_eval` supports conditional-generation models for log-likelihood–based tasks
3. Documentation clearly states that Ministral-3 is currently unsupported
Repository: EleutherAI/lm-evaluation-harness
Issue: [Bug] SGLang integration error in init
Root cause: Interface Contract Mismatch

I noticed that the SGLang init in https://github.com/EleutherAI/lm-evaluation-harness/blob/5d7dc4c0bbaac6f1186a8086bbb9c1da43be24cf/lm_eval/models/__init__.py#L46 is not correct.

It should be `"sglang": "lm_eval.models.sglang_causallms:SGLangLM"` if we don't modify the class name here:  https://github.com/EleutherAI/lm-evaluation-harness/blob/c9772b90a8ee95b0df1ba76651443a0534c96aad/lm_eval/models/sglang_causallms.py#L34 

@baberabb @zhaochenyang20

It's a quick change. Which one do you prefer? Change the class name or change the init?
Repository: EleutherAI/lm-evaluation-harness
Issue: Replicate open_llm_leaderboard metrics
Root cause: Documentation Deficiency

I am looking to replicate metric values (IFEVAL, BBH, MATH, GPQA, MUSR, MMLU) published on huggingface open_llm leaderboard. Could someone assist me in identifying the correct command to replicate and match the numbers displayed on the leaderboard.
Repository: EleutherAI/lm-evaluation-harness
Issue: How can I use this evaluation in a cluster blocked for hugging face server?
Root cause: Documentation Deficiency

For the model I'm using: --model_args pretrained=/path/to/model

But for the datasets, how can I pass some evaluation datasets to the code?
My cluster blocked access for hugging face server, so I can't use the name of the dataset (example: glue/wnli) to download from hugging face. I need to download the datasets, put in some folder in my cluster and pass the path to the evaluation code.

Can someone help me?
Repository: EleutherAI/lm-evaluation-harness
Issue: Is there some problem with the evaluation code?
Root cause: Validation Gap

I'm a bit confused about the evaluation. I was debugging the code (using the lambada task) and I came across this situation. The input is being cut in the wrong position?

Because in this example the word "drapes" appears as "drap" and the model is completing correctly with "es".
The target is "drapes" and because of the wrong cut, the match is different.

Someone can comment something about it?

Input text (inp):
Yanking back a fistful of the thief's hair and sinking her teeth into an unprotected throat. It was a strange, unsettling vision, but for a moment it almost seemed real. She became aware that Bonnie and Meredith were looking at her. "Well?" she said, feeling slightly uncomfortable. "I could tell you weren't listening," sighed Bonnie It can be anywhere from 4% up to 12%, with the average around 6% or 8%. The broker then turns around and shares his or her proceeds with the selling broker, who is the broker representing the buyer. (Confusing, I know.) In a net listing, however, the owner receives a specified — net — amount from the sale, with the excess going to the broker I wanted to shout myself hoarse at him, but I couldn't do it. I just wanted him gone. When I didn't move, he closed the heavy drapes around the bed, sealing me off in a dark little cage. 23 Antiquary I didn't sleep for hours. I could hear him at his desk, writing away, hidden from me only by the **drap**

Model output (greedy_tokens):
tensor([[259, 299]])
es

Target (cont_toks):
tensor([[95904, 299]])
drapes
Repository: EleutherAI/lm-evaluation-harness
Issue: Clarify Lambada Task
Root cause: Unimplemented Feature Gap

When OpenAI created GPT-2, they also created a custom, non-standard lambada evaluation dataset. OpenAI also changed the metric for evaluation by counting number of times the last BPE token is predicted incorrectly instead of the last word. This produces a huge difference in performance score, totally over 10%. They used this easier version of Lambada for evaluating GPT-2 as well. For more details, see [here](https://github.com/huggingface/transformers/issues/491) and [here](https://github.com/openai/gpt-2/issues/131#issuecomment-497136199).

According to @jon-tow, we implement the openai version, _not_ the standard version. We should implement both, and call them `lambada_standard` and `lambada_openai` respectively. In particular, we should **not implement a task called lambada** because years after the fact this is still causing widespread confusion and we want to force the user to pay attention to it.
Repository: EleutherAI/lm-evaluation-harness
Issue: Upstream `bigscience-workshop/lm-evaluation-harness` core features
Root cause: Unimplemented Feature Gap

Some users have expressed interest in merging some of the core features from the [BigScience `lm-evaluation-harness` + `promptsource`](https://github.com/bigscience-workshop/lm-evaluation-harness) integration into `master`.

The critical features we'd like to include:
- [ ] Model API improvements such as HuggingFace Seq2Seq language model support and general [`accelerate`](https://github.com/huggingface/accelerate) backed parallelism (relevant source [here](https://github.com/bigscience-workshop/lm-evaluation-harness/blob/master/lm_eval/api/model.py)).
- [ ] Robust `promptsource` integration to provide users with flexible prompt templating. `promptsource` is a hard dependency of the BigScience fork, specifically in the core Task class, `PromptSourceTask` (relevant source [here](https://github.com/bigscience-workshop/lm-evaluation-harness/blob/bdd1d3f9e4480b5fe89850b98ac1cc8b8dfab024/lm_eval/api/task.py#L212)). For example, `PromptSourceTask.process_results` uses `promptsource` metric metadata tags to automatically extract metric methods and apply them to necessary args without forcing task implementors to override `process_results` for common metrics.

Would-be-nice-to-port features:
- [ ] Fine-grained logging. The BigScience fork gives users the ability to log detailed model outputs such as per-question/prompt answers, which is useful for analysis and post-evaluating the model on metrics not yet made available in the harness. (Relevant sources [here](https://github.com/bigscience-workshop/lm-evaluation-harness/blob/bdd1d3f9e4480b5fe89850b98ac1cc8b8dfab024/lm_eval/api/task.py#L553) and [here](https://github.com/bigscience-workshop/lm-evaluation-harness/blob/bdd1d3f9e4480b5fe89850b98ac1cc8b8dfab024/lm_eval/api/task.py#L620))
- [ ] Optional (carbon) emissions tracking (relevant source [here](https://github.com/bigscience-workshop/lm-evaluation-harness/blob/bdd1d3f9e4480b5fe89850b98ac1cc8b8dfab024/main.py#L195)) for large model evals.

Assignees can ping me (@jon-tow) for general support and @samsontmr for opinions on the BigScience fork side.
Repository: EleutherAI/lm-evaluation-harness
Issue: Using `lmeval` as a library
Root cause: Unimplemented Feature Gap

I'm using `lmeval` as a library in https://github.com/allenai/catwalk, an evaluation framework that borrows a lot from this one, and adds more stuff on top.

However, there is no recent release of `lmeval`, and if I want to upload mine to pypi, I cannot depend on a branch in GitHub. I need a proper release. Also, as others [have](https://github.com/EleutherAI/lm-evaluation-harness/issues/348) [noted](https://github.com/EleutherAI/lm-evaluation-harness/issues/347), if you want to use it as a library, you need more flexible dependency version pins.

To work around this I have copied the entire evaluation harness codebase into my codebase at https://github.com/allenai/catwalk/pull/106. This is not a great long-term solution. I would much rather just specify `lmeval>=1.0` in my `requirements.txt`.

Is there a release forthcoming, one that would allow me to depend on `lmeval` the same way other libraries work?

@dirkgr Thanks for reporting! The latest `0.3.0` update has been released. Let us know if you run into any more issues.
Repository: EleutherAI/lm-evaluation-harness
Issue: Results not matching with reported ones
Root cause: Validation Gap

I'm trying to replicate the results presented at https://huggingface.co/EleutherAI/gpt-neo-1.3B for Wikitext. However, when I run

```
python main.py \
    --model gpt2 \
    --model_args pretrained=EleutherAI/gpt-neo-1.3B \
    --tasks wikitext \
    --device 0
```

I don't get the same results. Am I doing something wrong? Tks!

I also could not get the same results as the one reported (https://huggingface.co/gpt2-large).
I got the below result for gpt2-large:
|      Task      |Version|Metric| Value |   |Stderr|
|----------------|------:|------|------:|---|-----:|
|lambada_standard|      0|ppl   |22.1788|±  |0.7741|
|                |       |acc   | 0.4040|±  |0.0068|

@Yujin-Yujin OpenAI does not report standard Lambada scores, which is why the numbers don’t match. I think we are ultimately not going to allow replication of their numbers, see [here](https://github.com/EleutherAI/lm-evaluation-harness/issues/350#issuecomment-1364543050) for more info and discussion.

@pedrogengo I shall have someone look into this
Repository: EleutherAI/lm-evaluation-harness
Issue: Switch to a "sliding window with stride of 1" strategy for perplexity evaluation?
Root cause: Documentation Deficiency

In the docstring for `base.LM.loglikelihood_rolling` the authors state that they aim to ["maximize the amount of context for each prediction"](https://github.com/EleutherAI/lm-evaluation-harness/blob/57f8ff24ca492b4cbd9c86fd9db7cfa519d79284/lm_eval/base.py#L49), however in both the within-docstring example and the [default implementation in `BaseLM`](https://github.com/EleutherAI/lm-evaluation-harness/blob/57f8ff24ca492b4cbd9c86fd9db7cfa519d79284/lm_eval/base.py#L187-L219) it appears that the implementation uses a sliding window with a stride of `self.max_length`.

Is this expected?  Should the implementation instead employ a stride of 1 to correctly factorize the sequence probability and in doing so both: 1. maximize the amount of context for each prediction, and 2. yield the most favorable perplexity/bits-per-byte result?  

@tgunter see lines 56 to 68,
the intended meaning of "maximizing amount of context" is different from the one you are mentioning
btw, your understanding of the meaning is exactly what I thought initially as well

Closed as we have solved this problem.