zhiminy/EvalEng huggingface dataset:
Repository: EleutherAI/lm-evaluation-harness
Issue: Couldn't find file squad-v1.1/train-v1.1.json when evaluate Qwen3-A3B with vllm pipeline
Root cause: External Dependency Breakage

```bash
  File "/root/.cache/huggingface/modules/datasets_modules/datasets/Rakuten--JGLUE/982de5a6b7515b0fcf1918cd7070994fff6b4736b8455f794cc144fdcc3ed036/JGLUE.py", line 622, in _split_generators
    return self.__split_generators(dl_manager)
  File "/root/.cache/huggingface/modules/datasets_modules/datasets/Rakuten--JGLUE/982de5a6b7515b0fcf1918cd7070994fff6b4736b8455f794cc144fdcc3ed036/JGLUE.py", line 603, in __split_generators
    file_paths = dl_manager.download_and_extract(_URLS[self.config.name])
  File "/opt/miniconda3/lib/python3.10/site-packages/datasets/download/download_manager.py", line 434, in download_and_extract
    return self.extract(self.download(url_or_urls))
  File "/opt/miniconda3/lib/python3.10/site-packages/datasets/download/download_manager.py", line 257, in download
    downloaded_path_or_paths = map_nested(
  File "/opt/miniconda3/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 511, in map_nested
    mapped = [
  File "/opt/miniconda3/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 512, in <listcomp>
    _single_map_nested((function, obj, batched, batch_size, types, None, True, None))
  File "/opt/miniconda3/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 380, in _single_map_nested
    return [mapped_item for batch in iter_batched(data_struct, batch_size) for mapped_item in function(batch)]
  File "/opt/miniconda3/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 380, in <listcomp>
    return [mapped_item for batch in iter_batched(data_struct, batch_size) for mapped_item in function(batch)]
  File "/opt/miniconda3/lib/python3.10/site-packages/datasets/download/download_manager.py", line 313, in _download_batched
    return [
  File "/opt/miniconda3/lib/python3.10/site-packages/datasets/download/download_manager.py", line 314, in <listcomp>
    self._download_single(url_or_filename, download_config=download_config)
  File "/opt/miniconda3/lib/python3.10/site-packages/datasets/download/download_manager.py", line 323, in _download_single
    out = cached_path(url_or_filename, download_config=download_config)
  File "/opt/miniconda3/lib/python3.10/site-packages/datasets/utils/file_utils.py", line 201, in cached_path
    output_path = get_from_cache(
  File "/opt/miniconda3/lib/python3.10/site-packages/datasets/utils/file_utils.py", line 630, in get_from_cache
    raise FileNotFoundError(f"Couldn't find file at {url}")
FileNotFoundError: Couldn't find file at https://raw.githubusercontent.com/yahoojapan/JGLUE/main/datasets/jsquad-v1.1/train-v1.1.json
```

The evaluate script is 
```bash
lm_eval --model vllm --model_args pretrained="/nfs-95/Qwen3/Qwen3-30B-A3B",max_model_len=8192,tensor_parallel_size=1,dtype=auto,data_parallel_size=1,trust_remote_code=True --tasks japanese_leaderboard --batch_size 2
```

How to solve this?
Repository: EleutherAI/lm-evaluation-harness
Issue: Support for using a remote /tokenize API endpoint as the tokenizer
Root cause: Unimplemented Feature Gap

I am trying to evaluate a fully closed source model that neither model nor tokenizer publicly available. The model is only accessible via API endpoints, including an endpoint for tokenization.

**command:**
lm_eval --model local-completions --tasks mmlu \
--model_args model=custom_model, base_url=http://XX.XX.XX.XX:8000/v1/completions,num_concurrent=1,max_retries=3,tokenized_requests=False 

**result:**
OSError: custom_model is not a local folder and is not a valid model identifier listed on 'https://huggingface.co/models'
If this is a private repository, make sure to pass a token having permission to this repo either by logging in with `huggingface-cli login` or by passing `token=<your_token>`

lm-eval attempts to instantiate a tokenizer using custom_model as if it were a local path or Hugging Face Hub identifier.

Would it be possible to add support in lm-eval to use such a remote /tokenize API endpoint for its internal tokenization needs?



Hi! We would welcome a PR.  The main changes would be adding a new conditional block to set `self.tokenizer` with the new `tokenizer_backend`, similar to how the HF one is set. (e.g., storing the tokenizer API endpoint URL from model_args)
https://github.com/EleutherAI/lm-evaluation-harness/blob/7aaceeec2e7b686d95d0e55b43af641dc2484b4a/lm_eval/models/api_models.py#L193-L201

and appropriately call it in`tok_encode` to call the remote tokenizer API with the input string and return the received token IDs.

https://github.com/EleutherAI/lm-evaluation-harness/blob/7aaceeec2e7b686d95d0e55b43af641dc2484b4a/lm_eval/models/api_models.py#L368
Repository: EleutherAI/lm-evaluation-harness
Issue: Chat completions does not support batching. Defaulting to batch size 1.
Root cause: Documentation Deficiency

how to add dynamic batchsize in the "local-chat-completions"?

What API endpoint are you using? Chat completions APIs that follow a format similar to `/v1/chat/completions` typically do not support multiple sequences. You can make multiple concurrent calls though, using the `num_concurrent=N` arg.

thank you,sir. it works.
command:lm_eval --model local-chat-completions --tasks ifeval --model_args model=deepseek,base_url=http://localhost:8084/v1/chat/completions,num_concurrent=16 --apply_chat_template --limit 1000
Repository: EleutherAI/lm-evaluation-harness
Issue: Issue with quantization_config argument
Root cause: Interface Contract Mismatch

Commit 758c5ed for lm_eval/models/huggingface.py added an extra quantization_config argument. 
However, this quantization_config seems to be a dict type. 

In transformers/quantizers/auto.py, the merge_quantization_configs function will consume this new argument and merged with other quantization configs. An error occurs here as merge_quantization_configs  expects quantization_config_from_args to be Optional[QuantizationConfigMixin]. When the quantization is in [GPTQConfig, AwqConfig, AutoRoundConfig, FbgemmFp8Config, CompressedTensorsConfig], it calls `loading_attr_dict = quantization_config_from_args.get_loading_attributes()`, which gives `AttributeError: 'dict' object has no attribute 'get_loading_attributes'`, making the lm_eval fails. 


I encounter this issue when trying to evaluate a model generated with llm_compressor. 
`recipe = [
    SmoothQuantModifier(smoothing_strength=0.8),
    GPTQModifier(scheme="W8A8", targets="Linear", ignore=["lm_head"]),
]
oneshot(
    model="TinyLlama/TinyLlama-1.1B-Chat-v1.0",
    dataset="open_platypus",
    recipe=recipe,
    output_dir="TinyLlama-1.1B-Chat-v1.0-INT8",
    max_seq_length=2048,
    num_calibration_samples=512,
)`

and
`lm_eval --model hf \
    --model_args pretrained=./TinyLlama-1.1B-Chat-v1.0-INT8,add_bos_token=True \
    --tasks wikitext \
    --device cuda:0 \
    --batch_size auto \
    --num_fewshot 5 \
    --limit 250`

Hi! Thanks for reporting this and providing the details. Would you be interested in submitting a PR? Would appreciate it. Also we should add some tests. 

Hi, I just send a PR. I don't have a gentle solution for it right now. My current solution is brute, simply commenting out the quantization_config argument. It thus loses the functionality implemented by previous commit. 
I'm still a beginner for lm-eval and for using Git to commit a change. Let me know if you observe anything I did wrong.
Repository: EleutherAI/lm-evaluation-harness
Issue: Add Support Conditional Generation Models like Mistral3
Root cause: Unimplemented Feature Gap

I am trying to perform evaluations for [Mistral 3 Small 3.1 24B Instruct 2503l](https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503). This model is not loadable by the `AutoModelForCausalLM` factory, and as a result an exception is raised when trying to perform evaluation with the `HFLM` class.

Is it possible to add support for loading `*ConditionalGeneration` models like Llama 4 and Mistral 3 to the `HFLM` class in your evaluation harness?

It is possible to patch in support for individual models by patching internal attributes of the factory methods in transformers, such as 
```python
transformers.AutoModelForCausalLM._model_mapping[transformers.Mistral3Config] = (
    transformers.models.mistral3.modeling_mistral3.Mistral3ForConditionalGeneration  # pyright: ignore[reportAttributeAccessIssue]
)
transformers.AutoModelForCausalLM._model_mapping._model_mapping["mistral3"] = "Mistral3ForConditionalGeneration"  # pyright: ignore[reportAttributeAccessIssue]
transformers.AutoModelForCausalLM._model_mapping._reverse_config_mapping["Mistral3Config"] = "mistral3"
```
but this does not feel like the right approach, as it seems intentional that these models are not loaded by the `AutoModelForCausalLM` factory methods.

Hi! #2999 allows the `hf-multimodal` and `vllm-vlm` model backends to run text tasks as well (by calling the super methods). 

You probably will have to add-to/remove this assertion
https://github.com/EleutherAI/lm-evaluation-harness/blob/82a99365abdc801a11d41095182a4da506591aec/lm_eval/models/huggingface.py#L893-L896

and I might just remove it altogether.



> Hi! [#2999](https://github.com/EleutherAI/lm-evaluation-harness/pull/2999) allows the `hf-multimodal` and `vllm-vlm` model backends to run text tasks as well (by calling the super methods).
> 
> You probably will have to add-to/remove this assertion
> 
> [lm-evaluation-harness/lm_eval/models/huggingface.py](https://github.com/EleutherAI/lm-evaluation-harness/blob/82a99365abdc801a11d41095182a4da506591aec/lm_eval/models/huggingface.py#L893-L896)
> 
> Lines 893 to 896 in [82a9936](/EleutherAI/lm-evaluation-harness/commit/82a99365abdc801a11d41095182a4da506591aec)
> 
>  assert self.AUTO_MODEL_CLASS in ( 
>      transformers.AutoModelForCausalLM, 
>      transformers.AutoModelForVision2Seq, 
>  ) 
> and I might just remove it altogether.

Thanks for sharing that other PR that just landed! We will definitely look into using both of those backends, especially the VLLM.

Ya, I have my own modifications to get my work done and I also identified this particular assert as a blocker for ConditionalGeneration models. When I dug around it mostly seemed there just to enforce the factories as the interface for loading, which makes sense if it were not for the sporadic handling of how Auto factories load multi-modal models by HF.

I'll keep eye on this issue, and set some time/engineering effort aside to see if those other backends help solve our problem in general.

Thanks!
Repository: EleutherAI/lm-evaluation-harness
Issue: Boolq perplexity
Root cause: Documentation Deficiency

What is done when the perplexity of (True) is equivalent to the perplexity of (False)? Which answer is picked?

I believe it will choose `no` as it's the first one in the [list](https://github.com/EleutherAI/lm-evaluation-harness/blob/8bc4afff22e73995883de41018388428e39f8a92/lm_eval/tasks/super_glue/boolq/default.yaml#L11) and that's the default behaviour of `np.argmax`
Repository: EleutherAI/lm-evaluation-harness
Issue: self-defined architecture model eval
Root cause: Documentation Deficiency

how to eval self-defined architecture model ?
Any document to instruct?

'''
ValueError: The checkpoint you are trying to load has model type `ComplexFormer` but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.
'''

I'm also curious about this thing. So have you gotten a proper solution to solving this?
Repository: EleutherAI/lm-evaluation-harness
Issue: llama4 MMMU prompts
Root cause: Validation Gap

when evaluated Llama-4-Scout-17B-16E MMMU as follows:

python -m lm_eval --model vllm-vlm \
     --model_args "$modelargs" \
     --tasks mmmu_val \
     --batch_size auto 

/vllm/vllm/multimodal/processing.py", line 1560, in _validate_mm_placeholders                           [rank0]:     raise RuntimeError(
[rank0]: RuntimeError: Expected there to be 1 prompt updates corresponding to 1 image items, but instead found 0 prompt updates! Either the prompt text has missing/incorrect tokens for multi-modal inputs, or there is a problem with your implementation of merged multi-modal processor for this model (usually arising from an inconsistency between `_call_hf_processor` and `_get_prompt_updates`).

how to set llama4 format correct prompt?
Repository: EleutherAI/lm-evaluation-harness
Issue: Cache fails when repeats are greater than 1
Root cause: Algorithmic Error

In evaluator.py at line 560, the request (req) is expanded using req.repeats. However, this process doesn't differentiate between the individual repeated queries. Consequently, the cache only stores the prediction for the first query instance.

```
# create `K` copies of each request `req` based off `K = req.repeats`
cloned_reqs = []
for req in reqs:
    cloned_reqs.extend([req] * req.repeats)
```

```
for generated_text, context in zip(
                    self.parse_generations(
                        outputs=outputs,
                        contexts=contexts,
                    ),
                    contexts,
                ):
    if generated_text is not None:
        res.append(generated_text)
        # partial caching
        if context is not None:
            self.cache_hook.add_partial(
                "generate_until",
                (context, all_gen_kwargs[0]),
                generated_text,
            )
            pbar.update(1)
```
Repository: EleutherAI/lm-evaluation-harness
Issue: why my japanese_leaderboard metric so low?
Root cause: Documentation Deficiency

# Leaderboad
https://rinnakk.github.io/research/benchmarks/lm/index.html

# Command
```bash
lm_eval --model vllm --model_args pretrained="/nfs-69/Qwen3/Qwen3-30B-A3B",max_model_len=8192,tensor_parallel_size=8,dtype=auto,data_parallel_size=1 --tasks japanese_leaderboard,ceval-valid,mmlu --batch_size 2 --trust_remote_code --output_path /qwen3-a3b-base  --log_samples
```

# Eval Results
| Tasks                               | Version | Filter      | n-shot | Metric        |   | Value  |   | Stderr   |
|-------------------------------------|---------|-------------|--------|---------------|---|--------|---|----------|
| japanese_leaderboard                 | N/A     |             |        |               |   |        |   |          |
| - ja_leaderboard_jaqket_v2          | 2       | whitespaces | 1      | f1            | ↑ | 0.7378 | ± | N/A      |
| - ja_leaderboard_jcommonsenseqa     | 1       | none        | 3      | acc           | ↑ | 0.5013 | ± | 0.0150   |
| - ja_leaderboard_jnli               | 1       | none        | 3      | balanced-acc  | ↑ | 0.3530 | ± | N/A      |
| - ja_leaderboard_jsquad             | 1       | whitespaces | 2      | f1            | ↑ | 0.5212 | ± | N/A      |
| - ja_leaderboard_marc_ja            | 1       | none        | 3      | balanced-acc  | ↑ | 0.6455 | ± | N/A      |
| - ja_leaderboard_mgsm               | 1       | none        | 5      | acc           | ↑ | 0.0840 | ± | 0.0176   |
| - ja_leaderboard_xlsum              | 1       | whitespaces | 1      | rouge2        | ↑ | 0.8167 | ± | N/A      |
| - ja_leaderboard_xwinograd          | 1       | none        | 0      | acc           | ↑ | 0.7612 | ± | 0.0138   |


I havn't test Qwen2.5-32B, but the Qwen3-30B-A3B's results are unacceptable, far from the leaderboard of Qwen2.5-32B, which i think unusual.

Why this huge gap appear? Was it the lm-eval problem?
Repository: EleutherAI/lm-evaluation-harness
Issue: Support for LongBenchV2
Root cause: Documentation Deficiency

Hi, when using the LongBench dataset, the readme mentions the LongBenchV2 benchmark but the implementation is only for LongBench. These are two different papers and they correspond to different datasets. 

Here's the current README, which is confusing: 


```
### Paper

Title: `LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks`

Abstract: `This paper introduces LongBench v2, a benchmark designed to assess the ability of LLMs to handle long-context problems requiring deep understanding and reasoning across real-world multitasks. LongBench v2 consists of 503 challenging multiple-choice questions, with contexts ranging from 8k to 2M words, across six major task categories: single-document QA, multi-document QA, long in-context learning, long-dialogue history understanding, code repository understanding, and long structured data understanding.`
More actions
Homepage: `https://github.com/THUDM/LongBench`
```


The LongbenchV2 dataset is available here https://huggingface.co/datasets/THUDM/LongBench-v2 


Can the README for longbench task be revised and a new task be added for LongbenchV2? 

Hi! Sorry this was an oversight on my part! Will fix
Repository: EleutherAI/lm-evaluation-harness
Issue: Add a module for evaluation with adaptive question selection
Root cause: Unimplemented Feature Gap

# Motivation

Evaluating large language models (LLMs) can be prohibitively expensive. By leveraging adaptive testing, which adaptively selects the most informative question tailored to each LLM’s performance profile, we can significantly reduce the number of questions required while preserving the reliability of the evaluation. Our large-scale empirical study shows that our method can reduce the query complexity by 50% on average and up to 82% across 183 LLMs and 22 benchmarks. In this context, we propose to add a module for adaptive question selection to lm-evaluation-harness.

More details can be found in our [ICML paper](https://arxiv.org/abs/2503.13335). Our method has already been integrated into HELM, supporting 22 benchmarks ([tutorial](https://crfm-helm.readthedocs.io/en/latest/reeval/), [PR](https://github.com/stanford-crfm/helm/pull/3397), [Blog](https://crfm.stanford.edu/2025/06/04/reliable-and-efficient-evaluation.html)).

# Objective

Our adaptive-testing extension aims to carry out adaptive testing on the evaluated model with the pre-calibrated difficulty information loaded from HuggingFace.

# Implemented techniques

We achieve this through the Rasch model from Item Response Theory (IRT). The Rasch model captures how LLM ability and question difficulty affect the correctness at the resolution of each (LLM, question) pair. The first phase is the calibration phase, where the data is analyzed to estimate the ability parameter of each LLM and the difficulty parameter of each question. According to the Rasch model, the difference between ability and difficulty directly specifies the likelihood that the LLM correctly answers the question.

The difficulty parameters derived from the calibration phase play a crucial role in reducing the cost of evaluation through the second phase: adaptive testing. Adaptive testing dynamically selects the most informative questions, tailoring the evaluation process to each LLM’s estimated ability. Rather than exposing each LLM to an exhaustive and static set of questions, adaptive testing optimizes efficiency by continuously choosing the questions that best elicit information about the LLM’s ability, thus also preserving the reliability of the evaluation.
Repository: EleutherAI/lm-evaluation-harness
Issue: Why does the same dataset and the same model produce different accuracy results with different batch sizes?
Root cause: Algorithmic Error

I tested the DeepSeek-V2-Lite model using the BoolQ dataset. When the batch size is 16, the accuracy is 0.7797; when the batch size is 32, the accuracy is 0.8018. Moreover, I read the model’s output from the cache and calculated the accuracy directly based on the probabilities of "True" and "False", but this result is also different from the one reported by lm-eval. Which one should be considered the correct result?

Hi! There's usually some trivial difference due to floating point precision, but generally the results should be within the standard error. Did you use the `hf` or `vllm` backend? I'll take a look

> Hi! There's usually some trivial difference due to floating point precision, but generally the results should be within the standard error. Did you use the `hf` or `vllm` backend? I'll take a look

hf backend, H20 GPU. Will different padding lengths under different batch sizes cause accuracy differences?

![Image](https://github.com/user-attachments/assets/4b36c8cd-01d2-4725-8ef6-81d22998d48f)

![Image](https://github.com/user-attachments/assets/c857544b-0660-4b84-b277-ecc8455661db)

> hf backend, H20 GPU. Will different padding lengths under different batch sizes cause accuracy differences?

Generally yes, due to differences in error accumulation across different batch sizes. These results look mostly typical.

> but this result is also different from the one reported by lm-eval.

You can save per-sample outputs using `--log_samples` and the per choice logprobs are saved in the 'resps' field (the second element indicates if that continuation would be chosen by greedy decoding, but very few tasks use that flag)
```
"resps": [[["-5.862434387207031", "False"]], [["-7.120368957519531", "False"]]]
```

closing now, but feel free to open!
Repository: EleutherAI/lm-evaluation-harness
Issue: AI2 OLMo support
Root cause: Unimplemented Feature Gap

I ran into the problem where I wanted to run evals on an AI2 OLMo model (e.g., [`allenai/DataDecide-dolma1_7-1B`](https://huggingface.co/allenai/DataDecide-dclm-baseline-75p-dolma1.7-25p-1B)), loaded from Hugging Face.
This not possible however, considering that OLMo uses a custom model implementation (more info [here](https://huggingface.co/allenai/OLMo-7B#uses)).  The following code should be added in order to utilize these OLMo models:
```py
from hf_olmo import OLMoForCausalLM  # pip install ai2-olmo

olmo = OLMoForCausalLM.from_pretrained("allenai/DataDecide-dolma1_7-1B", revision="step69369-seed-default")
```

Is there currently any way that lm-eval supports running these models (`trust_remote_code` in `--model_args` unfortunately does not work)? And if not, would it be good to extend the library to facilitate these models (I am willing to help with this, but I first want to make sure that extension is required)?

I forked the lm-evaluation-harness repo and added OLMo compatibility. 

If you want to do evals on OLMo models, you can install from repo from my fork, and use `--model hf_olmo` in CLI to start the evals. For now, I put it on the `hf_olmo` branch, so you have to use `git clone --depth 1 --branch hf_olmo https://github.com/dharmsen/lm-evaluation-harness`.

Do note that my fork is not tested, I did the bare minimum to get the OLMo models up and running. I'm open to working this out further and submitting a proper PR to the main repo if this is wanted.

Link to fork/branch: https://github.com/dharmsen/lm-evaluation-harness/tree/hf_olmo

Hi! Is the main change loading the model/tokenizer from `OLMoForCausalLM` rather than `AutoModelForCausalLM`? If so we can look into letting the user set the AutoModel when initializing

Hi Baber, yes this is the main change. I extended the HF class and changed `AutoModelForCausalLM` with `OLMoForCausalLM`, and the same for the tokenizer. I registered the model separately, so that you can load it with `--model hf_olmo`, but if you could supply a specific AutoModel when initializing, you can just import it yourself in the script and pass it yes.
Repository: EleutherAI/lm-evaluation-harness
Issue: Evalutating the text only side of a multimodal model.
Root cause: Validation Gap

When I try to evaluate LLama 3.1 based model I get this :

> ValueError: Attempted to run tasks: ['hellaswag'] which are text-only, but used a model type which only currently supports multimodal tasks.

I don't really care about the multimodal (image) part. I just want to see how the text based part works. Is it possible?

I just git pulled the latest and it works now. Thanks.