zhiminy/EvalEng huggingface dataset:
Repository: EleutherAI/lm-evaluation-harness
Issue: What is the correct way to evaluate thinking model via mmlu_generative?
Root cause: Configuration Error

As stated in the title, I was trying to evaluate the generative mode mmlu for thinking model such as:
```
# export HF_ENDPOINT=https://hf-mirror.com;
# export HF_DATASETS_CACHE='/root/.cache'
export LOGLEVEL=DEBUG;
export VLLM_WORKER_MULTIPROC_METHOD=spawn;
export CUDA_VISIBLE_DEVICES=0,1,2,3;
TASK_NAME=mmlu_generative;
MODEL_NAME=Qwen3-4B;
MODEL_PATH=/models/Qwen3-4B;
# MODEL_NAME=Qwen3-4B-Base-am-nothinking-ep2;
# MODEL_PATH=/ckpts/Qwen3-4B-Base/sft/AM-DeepSeek-R1-0528-Distilled-Nothinking/ep2;
lm-eval --model vllm \
    --model_args pretrained=$MODEL_PATH,served_model_name=$MODEL_NAME,max_model_len=32000,tensor_parallel_size=4,gpu_memory_utilization=0.9,enable_thinking=True \
    --tasks $TASK_NAME \
    --num_fewshot 1 \
    --batch_size auto \
    --max_batch_size 2048 \
    --output_path results/$TASK_NAME-results \
    --log_samples \
    --use_cache ./.cache/${TASK_NAME}_${MODEL_NAME} \
    --cache_requests refresh \
    --include_path custom_tasks/ \
    --apply_chat_template \
    --fewshot_as_multiturn \
    # --gen_kwargs '{"temperature":0.7, "top_p":0.8}' \
```

However, as a result, I received zero across all subjects. Could someone give me a hint on how to proceed?

You might consider setting `max_gen_toks` to `2048` or `4096` within `--model_args`.

The default `max_gen_toks` value, which is `256`, can often be quite restrictive for reasoning models. This is because `max_gen_toks` accounts for **both reasoning tokens and output tokens**. Increasing it could potentially help! 

Here's the relevant section for reference:
https://github.com/EleutherAI/lm-evaluation-harness/blob/0c9072191e790204c4dfeb3285925b586a548c2e/lm_eval/models/vllm_causallms.py#L125

---
I've tested with `mmlu_abstract_algebra_generative` using the following command:

```shell
lm_eval --model local-completions \
    --model_args model=Qwen/Qwen3-4B,base_url=http://localhost:8000/v1/completions,tokenizer=Qwen/Qwen3-4B,max_length=8192,max_gen_toks=2048,num_concurrent=16 \
    --tasks mmlu_abstract_algebra_generative \
    --output_path results \
    --batch_size 1 \
    --log_samples \
    --seed 42 \
    --gen_kwargs temperature=0.0 \
    --apply_chat_template
```

Here are the results I obtained from that test:

```
|     Tasks      |Version|Filter |n-shot|  Metric   |   |Value|   |Stderr|
|----------------|------:|-------|-----:|-----------|---|----:|---|-----:|
|abstract_algebra|      3|default|     0|exact_match|↑  | 0.81|±  |0.0394|
```

Hope this helps:)
Repository: EleutherAI/lm-evaluation-harness
Issue: final answer extraction
Root cause: Documentation Deficiency

Hi, thank you for your great work.
how the final answer in extracted from thinking and the solution for comparison with the ground truth in datasets like aime24?
thank you

[This](https://github.com/EleutherAI/lm-evaluation-harness/blob/c0fc717240032aec738c3199ae344887b5f34c23/lm_eval/tasks/aime/utils.py#L5) is the function used to extract the answer. It checks for the answer in both the `$$` and `\boxed{}` formats. You can also use few_shot prompting to set the format you want.

If you want to look at the generations, use the log_samples flag while running the lm_eval command. Make sure to also specify the output_path flag with log_samples.

Hello, can Farmark support LLM-based evaluation for answers that are mathematically equivalent but not string-matched?
For example:
Golden response: 1 / sqrt(2)
Student answers: 1 + sqrt(2) or 1 - sqrt(2)
In such cases, the filter function fails to recognize correct or equivalent forms. Could we enable LLM-based evaluation as a fallback? If the answer is marked wrong by the filter, we could re-evaluate it using an LLM to check for mathematical equivalence or contextual correctness.
Repository: EleutherAI/lm-evaluation-harness
Issue: lm_eval is slow (ifeval task)
Root cause: Algorithmic Error

Greetings! I want to preface this post by saying I am not knowledgable in matters of AI, so I hope my questions won't be too stupid.

---

The version of LM Eval in use is the last (0.4.9.1). I am trying to test gemma3-270m-it model for IFEval.

Two different versions are tested:

lm_eval --model local-chat-completions --tasks ifeval --model_args model=unsloth/gemma-3-270m-it-GGUF/,base_url=http://127.0.0.1:xxxx/v1/chat/completions --apply_chat_template --batch_size 1
For F16 GGUF (running on CUDA llama.cpp v1.52.0)

and
lm_eval --model hf --tasks ifeval --model_args pretrained=unsloth/gemma-3-270m-it,dtype=float32 --apply_chat_template --batch_size 1
For FP16 variance (running on Torch 3.8.0 with CUDA 11.8)

I tried two different config files (ifeval.yaml) — one is default and unchanged, one is with recommended Gemma 3 settings.

First issue — officially presented [zero-shot] IFEval [for gemma-3-270m-it] is 0.512, while measurements with commands above only yield 0.399 for F16 GGUF and 0.345 for FP16 (inst_level_loose_acc, the highest measure for both models). It is strange in two ways — why are the local values lower (with both configs) and why is FP16 value [14%] lower than F16 GGUF value, if they have an almost equal size, and unquantized models should have a higher precision? 

I am running these models on one RTX 3060 GPU.
Second issue — FP16 variance with dtype=float32 fulfills one request 20 seconds. With dtype=float16 the duration raises to 70 seconds. Although F16 GGUF only takes 3 seconds on average. Why such a difference, if quantized and unquantized models [of the same precision] are expected to run on the same speed (if they fit fully in VRAM), or even slower for quantized (which is not the case here)?

Third issue — in both cases the evaluation was bottlenecked by utilizing only one thread of CPU. For F16 GGUF [evaluation] GPUs' max usage is 75% (with rare moments of going up to 83%), for FP16 it is only 40%. Is it an expected behavior? Or is this exact task quite imparallelable? Or is there a way to utilize more threads?


Hi, did you manage to find any work around? I am using lm_eval from a python script and facing a similar issue. 


I found a solution for the second issue — by increasing batch_size from 1 to 80 (that's the max 3060 can fit) for FP16 variance, time-per-request decreased to around 1.5 seconds.
Repository: EleutherAI/lm-evaluation-harness
Issue: Cache file name too long bug using vllm deployed model for mmlu tasks
Root cause: Architectural Constraint

```
OSError: [Errno 36] File name too long: '/lm-evaluation-harness/.process_cache/requests-mmlu_elementary_mathematics_generative-5shot-rank0-world_size1-chat_template-fewshot_as_multiturn-tokenizer__ckpts__Qwen3-4B-Base__sft__AM-DeepSeek-R1-0528-Distilled-Nothinking__ep2.57669194e25afae85ea14f096de3ea7c6247d45e62770904575c34cbce2478f0.pickle'
```

As aforementioned, I encountered a `file name too long` bug for the full mmlu-generative task while using vllm to deploy model. I found that the sanitised model name always use the `pretrained` argument and thus also for the cache name. Is there a method that I can specify the model name myself such the name can be shorter?
Repository: EleutherAI/lm-evaluation-harness
Issue: Add task: LLMSQL
Root cause: Unimplemented Feature Gap

Hello everyone!
This issue is for adding the benchmark LLMSQL. The benchmark evaluates the models on Text2SQL benchmark created from old WikiSQL benchmark.

Paper link with benchmark introduction and results on popular models: https://arxiv.org/abs/2510.02350

Dataset version for lm-eval: [llmsql-bench/llmsql-benchmark-lm-evaluation-harness](https://huggingface.co/datasets/llmsql-bench/llmsql-benchmark-lm-evaluation-harness)

Codebase: https://github.com/LLMSQL/llmsql-benchmark



I plan to implement this benchmark myself and open a PR shortly.
Repository: EleutherAI/lm-evaluation-harness
Issue: [Feature request] platinum-bench support
Root cause: Unimplemented Feature Gap

As per title.

It'd be nice to support out of the box https://huggingface.co/datasets/madrylab/platinum-bench

Reference
https://github.com/MadryLab/platinum-benchmarks/blob/main/src/run_benchmark.py
https://arxiv.org/pdf/2502.03461
Repository: EleutherAI/lm-evaluation-harness
Issue: math_verify[antlr4_11_0] requires python 3.10+
Root cause: Configuration Error

Currently, pyproject sets python version to >=3.9. For tests that require dependencies for math tasks (`pip install -e .[math]`), they fail because math_verify requires python 3.10+. Could also make the math dependencies conditional in pyproject.toml instead, or wrap tests in try/except. Will anything break if python version was bumped to 3.10?
Repository: EleutherAI/lm-evaluation-harness
Issue: unable to run evals on anthropic-completions
Root cause: External Dependency Breakage

I am trying to run evaluations on Claude models with the following command: 

```
lm_eval --model anthropic-completions --model_args model=claude-3-haiku-20240307 --tasks gsm8k -L20
```

First the command fails with the error ```ModuleNotFoundError: attempted to use 'anthropic' LM type, but package `anthropic` is not installed. please install anthropic via pip install 'lm-eval[anthropic]' or pip install -e '.[anthropic]'```

These errors come from `lm-evaluation-harness/lm_eval/models/anthropic_llms.py` but neither of these suggested commands work and the file seems outdated based on the `pyproject.toml`.

When I try installing the pypi project `anthropic` and run the eval again, I get a different error. Here is the stacktrace:
```
AttributeError: 'Anthropic' object has no attribute 'get_tokenizer'
```

There are multiple issues here. The `lm-evaluation-harness/lm_eval/models/anthropic_llms.py` seems to have the suggested commands like `pip install -e '.[anthropic]'` from `v0.4.3`. The other issue is that there isn't a compatibility with the anthropic library. How do you suggest I should go about this?

Quick update, the [anthropic-sdk-python](https://github.com/anthropics/anthropic-sdk-python/blob/v0.39.0/src/anthropic/_client.py) had the `get_tokenizer` command removed from `v0.38.0` to `v0.39.0`. However, pinning `anthropic` pypi project to `v0.38.0` also throws an error - `TypeError: __init__() got an unexpected keyword argument 'proxies'`
Repository: EleutherAI/lm-evaluation-harness
Issue: Repeats is not working in Class
Root cause: Documentation Deficiency

I created the python task class and want to apply repeats. So, I tried passing the repeats in **Instance** but still it was creating single response.
Please help me out.

```python
def construct_requests(
    self, doc, ctx, chat_template=None, apply_chat_template=False, **kwargs
):
    task=kwargs["metadata"][0]
    doc_id=kwargs["metadata"][1]
    repeats=5
    kwargs.pop("metadata", None)
    return  [
        Instance(
            request_type="generate_until",
            doc=doc,
            arguments=(ctx, {"until": ["\n"], "max_gen_toks": 1048}),
            metadata=(task, doc_id, repeats),
            idx=0,
            **kwargs,
        )
    ]
````
Repository: EleutherAI/lm-evaluation-harness
Issue: support openlm format models
Root cause: Unimplemented Feature Gap

do you support working with openlm format models like https://huggingface.co/TRI-ML/DCLM-1B? Since OpenLM is not the architecture of the Transformers library. 

Usually I launch with accelerate launch -m lm_eval --model hf     --model_args pretrained=TRI-ML/DCLM-1Btrust_remote_code=False --tasks /home/jovyan/aakazakova_1/metrics_ru/lm-evaluation-harness/lm_eval/tasks/lambada_our  --batch_size 8

Could launching with vllm, sglang or another method be helpful?

Thanks for answer


----------------------
[rank3]: ValueError: The checkpoint you are trying to load has model type `openlm` but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.

[rank3]: You can update Transformers with the command `pip install --upgrade transformers`. If this does not work, and the checkpoint is very new, then there may not be a release version that supports this model yet. In this case, you can get the most up-to-date code by installing Transformers from source with the command `pip install git+https://github.com/huggingface/transformers.git` 

Also with
https://huggingface.co/Qwen/Qwen3-VL-30B-A3B-Instruct

[rank6]:     raise ValueError(
[rank6]: ValueError: Unrecognized configuration class <class 'transformers.models.qwen3_vl_moe.configuration_qwen3_vl_moe.Qwen3VLMoeConfig'> for this kind of AutoModel: AutoModelForCausalLM.

iirc you have to write a wrapper python script to use lm-eval-harness with openlm
Repository: EleutherAI/lm-evaluation-harness
Issue: long wait between the completion of generation and the output of metrics
Root cause: Algorithmic Error

What causes the long wait (many times longer than the generation time) between the completion of generation and the output of metrics? How can this be reduced?

<img width="927" height="385" alt="Image" src="https://github.com/user-attachments/assets/b62e5e81-d040-429e-b8d4-e9f7a482f80a" />


accelerate launch -m lm_eval --model hf     --model_args pretrained=unsloth/Qwen3-30B-A3B-Base,trust_remote_code=False --tasks /home/jovyan/aakazakova_1/metrics_ru/lm-evaluation-harness/lm_eval/tasks/lambada_our  --batch_size 8 --log_samples --output_path /home/jovyan/aakazakova_1/metrics_ru/lm-evaluation-harness/lm_eval/tasks/lambada_our/out


It is true for all models, metrics etc
Repository: EleutherAI/lm-evaluation-harness
Issue: Offline mmlu_redux_generative tries to reach “cais/mmlu”; only works if I point it to mmlu template (likely wrong template mapping + dataset id)
Root cause: Configuration Error

Hi, I’m trying to run mmlu-redux in offline mode with a locally cached dataset, but the task keeps attempting to hit the Hub and errors out. It only works if I switch the template to the mmlu one, which suggests mmlu_redux_generative might be mapped to the wrong template and/or the dataset id is wrong in the template.

Steps to reproduce

Environment:
HF_DATASETS_OFFLINE=1
Model served locally at http://127.0.0.1:8000/v1/completions

Command:
lm_eval --model local-completions --model_args "model=DeepSeek-R1-Distill-Qwen-1.5B,base_url=http://127.0.0.1:8000/v1/completions,tokenized_requests=False,max_tokens=256,trust_remote_code=True" --tasks mmlu_redux_generative --output_path ./ --batch_size auto

Local dataset:
I have mmlu-redux-2.0 and mmlu downloaded and cached locally.

Observed behavior

Even with HF_DATASETS_OFFLINE=1, I get:
ConnectionError: Couldn't reach 'cais/mmlu' on the Hub (OfflineModeIsEnabled)
If I edit ./lm_eval/tasks/mmlu/generative/_default_template_yaml to use dataset_path: mmlu and run the same command, the run proceeds, but that would be using the mmlu dataset rather than mmlu-redux, which is not what I want.

Expected behavior
mmlu_redux_generative should load the locally cached mmlu-redux dataset (no network calls), and not fall back to mmlu or try to access any Hub repo.

Hi! `mmlu-redux` points to `edinburgh-dawg/mmlu-redux-2.0` (https://huggingface.co/datasets/edinburgh-dawg/mmlu-redux-2.0) (it's modified from the original). It shouldn't really be calling the requesting `cais` one

I 

> Hi! `mmlu-redux` points to `edinburgh-dawg/mmlu-redux-2.0` (https://huggingface.co/datasets/edinburgh-dawg/mmlu-redux-2.0) (it's modified from the original). It shouldn't really be calling the requesting `cais` one

I modified the dataset_path in ./lm_eval/tasks/mmlu-redux/generative/_default_template_yaml to point to my local dataset edinburgh-dawg/mmlu-redux-2.0, but it throws the error: “ConnectionError: Couldn't reach 'cais/mmlu' on the Hub (OfflineModeIsEnabled).”

In lm_eval/tasks/mmlu-redux/generative/_mmlu.yaml

group: mmlu_redux_generative
group_alias: mmlu_redux (generative)
task:
  - group: stem
    task:
      - mmlu_stem_generative
    aggregate_metric_list:
      - metric: exact_match
        weight_by_size: true

I changed mmlu_stem_generative to mmlu_stem_generative_1.

And in lm_eval/tasks/mmlu-redux/generative/mmlu_abstract_algebra.yaml

"dataset_name": "abstract_algebra"
"description": "The following are multiple choice questions (with answers) about abstract
  \ algebra.\n\n"
"tag": "mmlu_stem_generative"
"include": "_default_template_yaml"
"task": "mmlu_abstract_algebra_generative"
"task_alias": "abstract_algebra"

I changed:
"tag": "mmlu_stem_generative_1"
"tag": "mmlu_stem_generative_1"
"task": "mmlu_abstract_algebra_generative_1"

With these changes, I can use the offline dataset edinburgh-dawg/mmlu-redux-2.0 and run tests with --tasks mmlu_stem_generative_1.
Repository: EleutherAI/lm-evaluation-harness
Issue: Support GSM8K-Symbolic eval
Root cause: Unimplemented Feature Gap

As per title.

Reference: https://arxiv.org/html/2410.05229v1


e.g. used by IBM folks to evaluate Granite 4: https://www.ibm.com/new/announcements/ibm-granite-4-0-hyper-efficient-high-performance-hybrid-models
<img width="1284" height="770" alt="Image" src="https://github.com/user-attachments/assets/db13f440-f80d-476c-bf42-3ddde39b3ec6" />
Repository: EleutherAI/lm-evaluation-harness
Issue: Different eval batch size get different results? 
Root cause: Algorithmic Error

In my setting, I use zero shot (num-fewshot is 0) and single GPU card
I try different eval batch such as 1, 2, 4, 8, 16. the acc results on lambada are different
Is this a bug? 


lm-eval==0.2.0

 Hi,
I'm seeing this issue too - 
did you find a way to fix this, or is this expected?
Repository: EleutherAI/lm-evaluation-harness
Issue: Subject: [BUG] Zero Accuracy on aime Task, but Non-Zero Results on aime24 with Identical Configuration
Root cause: Configuration Error

Hi team,

First off, thank you for your incredible work on this project. It's an invaluable tool for the community.

I seem to have encountered an issue while evaluating a model on the aime dataset. When I run the evaluation, the exact_match score is consistently 0.0. However, what's interesting is that when I switch to the aime24 dataset, using the exact same configuration otherwise, I get a non-zero accuracy score as expected.

Here are the results for reference:

aime task results:

"results": {
    "aime": {
        "alias": "aime",
        "exact_match,none": 0.0,
        "exact_match_stderr,none": 0.0
    }
}

aime24 task results

"results": {
    "aime24": {
        "alias": "aime24",
        "exact_match,none": 0.3,
        "exact_match_stderr,none": 0.0850962943396763
    }
}

Below is the command I used to launch the evaluation. The only change between the two runs was switching --tasks aime to --tasks aime24.

export VLLM_WORKER_MULTIPROC_METHOD=spawn

lm-eval --model vllm \
        --model_args pretrained=***,dtype=auto,gpu_memory_utilization=0.8,enable_thinking=True \
        --tasks aime \
        --apply_chat_template \
        --fewshot_as_multiturn \
        --num_fewshot 0 \
        --log_samples \
        --show_config \
        --batch_size=auto \
        --output_path=*** \
        --device cuda \
        --seed 42 \
        --trust_remote_code \
        --gen_kwargs '{"max_gen_toks": 16384}'

(Note: I have redacted the model and output paths for privacy.)

I'm not sure if this is an issue with the dataset configuration for aime within the harness or something I might be missing. Any guidance or insight you could provide would be greatly appreciated.

Thanks again for your time and help!

good catch, `aime.yaml` is missing the process_docs function that extracts results. opened a PR in #3353, but you can also add the line `process_results: !function utils.process_results` in `aime.yaml`

Hi @jannalulu 

That's fantastic! I've just re-run the evaluation per your proposal, and I can confirm that the issue is now resolved. I am now getting the expected accuracy on the aime task. Here is the new result log:

|Tasks|Version|Filter|n-shot| Metric | |Value | |Stderr|
|:---|---:|:---|---:|:---|---:|---:|---:|---:|
|aime | 0|none | 0|exact_match|↑ |0.4469|± |0.0163|

Thank you so much for your quick response and effective fix. I really appreciate your help! This issue can be closed now.

Best regards.
Repository: EleutherAI/lm-evaluation-harness
Issue: Groups for Longbench v1
Root cause: Unimplemented Feature Gap

[Longbench v1](https://github.com/THUDM/LongBench/tree/main/LongBench) has single-doc qa, multi-doc qa, summarization, few-shot learning, code completion, and synthetic task groupings. [Jet-nemotron paper](https://www.arxiv.org/abs/2508.15884) reports grouped scores with longbench too
Repository: EleutherAI/lm-evaluation-harness
Issue: Key Error: 'afrimmlu_direct_amh' when running simple_evaluate on AfriMGSM and AfriMMLU tasks
Root cause: Documentation Deficiency

I am trying to reproduce the results from [IrokoBench](https://aclanthology.org/2025.naacl-long.139/), but the evaluation does not run on [AfriMMLU](https://github.com/masakhane-io/lm-evaluation-harness/tree/main/lm_eval/tasks/afrimmlu) and [AfriMGSM](https://github.com/masakhane-io/lm-evaluation-harness/tree/main/lm_eval/tasks/afrimgsm), and I get the following error:

```
...
File "/mnt/task_runtime/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 312, in _load_individual_task_or_group
    subtask_list = self._get_tasklist(name_or_config)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/task_runtime/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 232, in _get_tasklist
    return self.task_index[name]["task"]
           ~~~~~~~~~~~~~~~^^^^^^
KeyError: 'afrimmlu_direct_amh'
```

Here is the code I use to run evaluation:

```python
from lm_eval.tasks import TaskManager
from lm_eval import simple_evaluate

task_manager = TaskManager()

# Smoke test: 5 examples per task
results = simple_evaluate(
    model        = eval_model,
    tasks        = ['afrimmlu_direct_amh'],
    num_fewshot  = 0,
    limit        = 5,
    task_manager = task_manager,
    log_samples  = True
)
```

It works well with [AfriXNLI](https://github.com/masakhane-io/lm-evaluation-harness/tree/main/lm_eval/tasks/afrixnli). It seems that AfriMGSM and AfriMMLU are not included in the list of tasks returned by that `get_tasklist` method.

I just realised with the commands `lm-eval --tasks list | grep afrimgsm` and `lm-eval --tasks list | grep afrimmlu` that there is an additional `_prompt_{nb}` at the end of each subset name where `{nb}` is a number between 1 and 5. There is also no `_direct` suffix for **AfriXNLI** and **AfriMGSM** prompts as stated in the Readmes, there is an inconsistency regarding task naming.
> It is important to note that there are two different versions of Irokobench ([v1](https://arxiv.org/pdf/2406.03368v1) and [v2](https://arxiv.org/pdf/2406.03368v2)) using different prompts. This must be taken into account in the `simple_evaluate` function in order to replicate the expected results. The absence of the `_direct` suffix mentioned above thus applies to **v2 prompts**, not v1 prompts.

So, the line `tasks = ['afrimmlu_direct_amh']` in the previous comment should be replaced for example by `tasks = ['afrimmlu_direct_amh_prompt_1']`.

The Readmes in [AfriXNLI](https://github.com/masakhane-io/lm-evaluation-harness/tree/main/lm_eval/tasks/afrixnli), [AfriMMLU](https://github.com/masakhane-io/lm-evaluation-harness/tree/main/lm_eval/tasks/afrimmlu) and [AfriMGSM](https://github.com/masakhane-io/lm-evaluation-harness/tree/main/lm_eval/tasks/afrimgsm) need to be updated in consequence.
Repository: EleutherAI/lm-evaluation-harness
Issue: Question: Does support multi-node evaluation with vllm?
Root cause: Documentation Deficiency

Hi, I want to know if lm_eval supports multi-node evaluation with vllm.
If it is supported, how can I use it?
Thanks!

I use the local API method when evaluating with vllm

> I use the local API method when evaluating with vllm

Thanks for your reply!