zhiminy/EvalEng huggingface dataset:
Repository: EleutherAI/lm-evaluation-harness
Issue: How to evaluate gemma-3-4b-it on winogrande, hellaswag
Root cause: Documentation Deficiency

can i use the framework for gemma3-vision to text  on text or vision data.
hwo to evaluate gemma3-4b-it ? is there another framework like this.

To evaluate gemma-3-4b-it on benchmarks such as Winogrande, Hellaswag, or similar tasks, you can use the [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) framework. Here’s a slightly generalized example command to run the evaluation:
```
lm_eval --model hf \
        --model_args pretrained=google/gemma-3-4b-it \
        --tasks winogrande,hellaswag \
        --device mps \
        --batch_size 16 \
        --verbosity INFO \
        --write_out \
        --output_path results \
        --log_samples \
        --limit 20 \
        --num_fewshot 0
```
Explanation of arguments:
	•	--model hf: Uses Hugging Face models.
	•	--model_args: Specifies the model checkpoint.
	•	--tasks: Lists the evaluation datasets you want to use.
	•	--device mps: For running the model efficiently on Apple Silicon GPUs. For CUDA GPUs, replace it with --device cuda.
	•	--batch_size: Adjust depending on your GPU’s capacity.
	•	--limit: Restricts the number of samples to evaluate, remove or adjust to evaluate the entire dataset.
	•	--num_fewshot: Controls the number of few-shot examples (set to 0 for zero-shot evaluation).

You can run similar commands for other evaluation tasks supported by the harness.


I also wrote a blog detailing how I ran mmlu-pro https://slyracoon23.github.io/blog/posts/2025-03-18_exploring_gemma_3_model.html

As for vision tasks, the current version of lm-evaluation-harness does not yet support Gemma-3 tokenization for combined vision and text data. Support for multimodal evaluation (vision + text) with Gemma-3 would require custom integration or modifications to the tokenizer and input processing pipeline.

> As for vision tasks, the current version of lm-evaluation-harness does not yet support Gemma-3 tokenization for combined vision and text data. Support for multimodal evaluation (vision + text) with Gemma-3 would require custom integration or modifications to the tokenizer and input processing pipeline.


Hello, thank you for sharing. 

I also encountered this problem for multimodal evaluation. The error is as follows.
```
ValueError: Attempted to run tasks: ['mmmu_val_art'] which require multimodal input, but the selected model type does not currently implement this. Multimodal support is currently restricted to the ['hf-multimodal', 'vllm-vlm'] model type.
```
 I guess the type of Gemma-3 is not correctly identified. Is there a multimodal evaluation framework that complements lm-evaluation-harness? Thank you very much :)




thank you i am only interested to text to text module evaluation. last time i check it was not working this time i see it work,.
thank you.
Repository: EleutherAI/lm-evaluation-harness
Issue: Implement the English Grammar Correction evaluation
Root cause: Unimplemented Feature Gap

From the GPT-3 paper

> Another task well suited for few-shot learning is correcting English grammar. We test this with GPT-3 in the fewshot setting by giving prompts of the form "Poor English Input: <sentence>\n Good English Output: <sentence>". We give GPT-3 one human-generated correction and then ask it to correct 5 more (again without any omissions or repeats). Results are shown in Figure 3.17.

This is a task where we will have to create our own dataset.

- [ ] Data processing code implemented
- [ ] Evaluation implemented

The evaluation code should be modeled after the interface in `lm_eval/base.py` and the example of the `BoolQ` task in `lm_eval/tasks/suerglue.py`
Repository: EleutherAI/lm-evaluation-harness
Issue: generate_until vs loglikelihood
Root cause: Resource Mishandling

Dear authors,

Many thanks for this incredible repo! When running inference with, e.g. `lm_eval --model hf --model_args pretrained=gemma-2-2b-it --tasks squadv2 --device cuda:0 --batch_size 64`, I noticed that `generate_until` runs successfully, but `loglikelihood` reports `CUDA OOM`. Why would that be the case? I am curious if `loglikelihood` uses significantly more memory than `generate_until`?

same error, do u have fixed it?
Repository: EleutherAI/lm-evaluation-harness
Issue: best_f1 and best_exact?

Dear authors,

Many thanks for this brilliant repo! When evaluating on the squad dataset, I understand that `has_ans` and `no_ans` refers to the case where the question can(not) be answered, but what does `best_f1` and `best_exact` mean? I don't recall seeing that metrics in the squad paper?

Hi! It [uses](https://github.com/EleutherAI/lm-evaluation-harness/blob/6fbebb4b3867e7861a26d3a05e922c35723fac06/lm_eval/tasks/squadv2/task.py#L165) the squad metric from the [evaluate](https://huggingface.co/spaces/evaluate-metric/squad_v2) library. I believe it computes the best by tuning the answerable threshold (amongst all samples).
Repository: EleutherAI/lm-evaluation-harness
Issue: MATH different from open llm leaderboard
Root cause: Algorithmic Error

https://github.com/EleutherAI/lm-evaluation-harness/pull/2773#issuecomment-2716980020

https://huggingface.co/blog/math_verify_leaderboard

Hi! this should be fixed in #2817!
Repository: EleutherAI/lm-evaluation-harness
Issue: Unreasonable VRAM Usage
Root cause: Resource Mishandling

I have been trying to evaluate a fine-tuned version of LLaMA 3 (fine-tuned with PEFT) with 8B parameters on the leaderboard task using lm-eval 0.4.8. I have set it to evaluate in FP16.
Unfortunately, whenever I try doing it, whether on vLLM or HuggingFace backend, I get an error saying "`torch.OutOfMemoryError: CUDA out of memory`". It tried allocating 7.83 GiB of VRAM on top of the 79.25 GiB already allocated. I invariably get this error right when it begins running loglikelihood requests.
For context, I am using an Nvidia A100 GPU with 80 GB VRAM, a Miniconda virtual environment, and a Vast.ai system. I have faced similar issues whenever I tried using smaller VRAM amounts. Additionally, it seems unreasonable for the amount of GPU VRAM used to evaluate leaderboard on an 8B model to be anywhere close to 80 GiB. 

Is there some setting I'm missing to reduce VRAM usage? Or is there a memory leak somewhere?

Command used: `lm-eval --model vllm --model_args pretrained=/root/merged_model,dtype=auto,gpu_memory_utilization=0.9 --tasks leaderboard --batch_size auto --device gpu --output_path /root/results/finetuned
`

Terminal Screenshot:

<img width="1440" alt="Image" src="https://github.com/user-attachments/assets/603c42c0-b947-44d8-a2a2-c43d36d310d6" />

I would really appreciate your help.

Hi! This is a known issue with vllm for logliklihood based tasks. You could try setting `gpu_memory_utilization` quite low (it's used by both the model weights and the kv-cache, so is model/GPU dependent). Could also try setting the batch size manually.

Are you experiencing the issue with `hf` as well? Generally that should work, if you set the batch_size appropriately. Maybe a bug in the `auto` computation when it switches over from loglikehood or generation tasks?
Repository: EleutherAI/lm-evaluation-harness
Issue: How to specify the precision for model inference to Float32
Root cause: Documentation Deficiency

Currently, the default precision for model testing is fp16. I would like to set it to float32 precision for testing. How should I set this?

For running a local model, you can use the `--model_args` flag to provide arbitrary arguments to the model constructor. In this example, you can use `torch_dtype=torch.float32` for models in the `transformers` library that support it.

@StellaAthena 
Hello, I added `torch_dtype=torch.float32` to `--model_args` as you suggested, but I encountered an error: `TypeError: transformers.models.auto.auto_factory._BaseAutoModelClass.from_pretrained() got multiple values for keyword argument 'torch_dtype'`. Is there any way to avoid this issue?
Repository: EleutherAI/lm-evaluation-harness
Issue: ValueError: Tasks claim_binary were not found
Root cause: Validation Gap

Traceback (most recent call last):
  File "/home/czr/anaconda3/envs/py310lmeval/bin/lm_eval", line 8, in <module>
    sys.exit(cli_evaluate())
  File "/home/czr/pycharmproject/lm-evaluation-harness/lm_eval/__main__.py", line 181, in cli_evaluate
    raise ValueError(
ValueError: Tasks claim_binary were not found. Try `lm-eval --tasks list` for list of available tasks.

How can we deal with this problem? 


+1
Repository: EleutherAI/lm-evaluation-harness
Issue: Review new GitHub Actions
GitHub labels: help wanted

A GH Action we were using, tj-actions/changed-files, [is severely compromised](https://semgrep.dev/blog/2025/popular-github-action-tj-actionschanged-files-is-compromised/). I've merged #2828 by @Tautorn (huge thank you for bringing this to our attention) which replaces it with another similar library.

I have checked that the reported vulnerability seems to be legit and that there is no specific reason to be suspicious of the replacement, but have not checked whether this PR breaks anything. The compromise appears to be sufficiently severe that I would rather the functionality break than leave the old library in use.

This issue exists to flag that the new GitHub action requires further review.

Reverted back to using `tj-actions/changed-files@v46.0.5` in #2918 as the replacement library wasn't actively maintained. From my understanding the security issue has been resolved and using the latest tagged release as suggested by the update in the link above.
Repository: EleutherAI/lm-evaluation-harness
Issue: how to load local ceval-exam dataset？
Root cause: Documentation Deficiency

i modify the dataset_path to load the local ceval dataset but always filed, can you tell me how to modify the “_default_ceval_yaml”?  thank you very much!!

I aslo want to know！！！！！！

> I aslo want to know！！！！！！
It seems to be a problem with the data format saved in online datasets. The data loaded by datasets will be saved in “.arrow“ data format, and there is also a json file related to datset (used to save data information). However, the local data is not in this format. It seems that it needs to be saved in "arrow" format by calling datasets.save_to_disk() before it can be loaded.

chinese：
猜测就是需要调用datasets.save_to_disk()将本地的数据保存成arrow的格式，然后再加载。。。
我还没实操，你可以试一下



I aslo want to know
Repository: EleutherAI/lm-evaluation-harness
Issue: Why didn't my filter_list in the YAML file take effect?
Root cause: Configuration Error

My YAML file contains the following configurations:

filter_list:
  - name: "custom_processing"
    filter:
      - function: "custom_filter"
 
I added printing in api.task.cpy:

`eval_logger.debug(f"The class TaskConfig(dict) self is {self}")`
`eval_logger.debug(f"The ConfigurableTask(Task) config is: {config}")`

 The displayed result is: 

`2025-03-25:10:27:42,929 DEBUG    [lm_eval.api.task:713] The ConfigurableTask(Task) config is: {'task': 'linkthings_qa', 'dataset_path': 'json', 'dataset_name': None, 'dataset_kwargs': {'data_files': '/home/zhpan/lm-evaluation-harness/lm_eval/tasks/linkthings/hongdian4.json'}, 'test_split': 'train', 'training_split': None, 'validation_split': None, 'fewshot_split': None, 'process_docs': <function process_docs at 0x7fbe0795e790>, 'doc_to_text': "{{ instruction }}{% if input != '' %}\n{{ input }}{% endif %}\n回答：", 'doc_to_target': '{{ output }}', 'target_delimiter': ' ', 'output_type': 'generate_until', 'metric_list': [{'metric': 'exact_match', 'higher_is_better': True, 'aggregation': 'mean'}, {'metric': 'bleu', 'higher_is_better': True, 'aggregation': 'bleu', 'tokenize': 'zh', 'force': True, 'lowercase': True}], 'generation_kwargs': {'do_sample': True, 'temperature': 0.3, 'top_p': 0.9, 'until': ['。', '！', '？']}, 'metadata': {'version': 1.0}}`

`2025-03-25:10:27:42,929 DEBUG    [lm_eval.api.task:102] The class TaskConfig(dict) self is TaskConfig(task='linkthings_qa', task_alias=None, tag=None, dataset_path='json', dataset_name=None, dataset_kwargs={'data_files': '/home/zhpan/lm-evaluation-harness/lm_eval/tasks/linkthings/hongdian4.json'}, training_split=None, validation_split=None, test_split='train', fewshot_split=None, process_docs=<function process_docs at 0x7fbe0795e790>, doc_to_text="{{ instruction }}{% if input != '' %}\n{{ input }}{% endif %}\n回答：", doc_to_target='{{ output }}', doc_to_image=None, doc_to_audio=None, unsafe_code=False, doc_to_choice=None, process_results=None, use_prompt=None, description='', target_delimiter=' ', fewshot_delimiter='\n\n', fewshot_config=None, num_fewshot=None, metric_list=[{'metric': 'exact_match', 'higher_is_better': True, 'aggregation': 'mean'}, {'metric': 'bleu', 'higher_is_better': True, 'aggregation': 'bleu', 'tokenize': 'zh', 'force': True, 'lowercase': True}], output_type='generate_until', generation_kwargs={'do_sample': True, 'temperature': 0.3, 'top_p': 0.9, 'until': ['。', '！', '？']}, repeats=1, filter_list=None, should_decontaminate=False, doc_to_decontamination_query=None, gen_prefix=None, metadata={'version': 1.0})`

couldn't read the filter_list configuration from the YAML file.

I am very devastated

Bro，This is my fault.Remember never to back up YAML files in the tasks directory
Repository: EleutherAI/lm-evaluation-harness
Issue: Fix slow gguf tests
Root cause: Algorithmic Error

I fixed the performance problem for gguf tests. By modifying the llama-cpp-python and building a new API model, we only need to use about 1/10 time compared to previous methods.
For details, please refer to [https://github.com/abetlen/llama-cpp-python/issues/1983](https://github.com/abetlen/llama-cpp-python/issues/1983) and [https://github.com/For-rest2005/lm-evaluation-harness](https://github.com/For-rest2005/lm-evaluation-harness)
The API model is not well written and require the modification in llama-cpp-python. If needed, I can make a pull request.
This problem is solved with the help of @PolluxyShi

Hi! A PR will be appreciated!
Repository: EleutherAI/lm-evaluation-harness
Issue: Missing standard kmmlu (multiple-choice) task
Root cause: Unimplemented Feature Gap

### Description

The standard multiple-choice kmmlu task (evaluated with accuracy) is missing, despite being listed in the README. Only the generative kmmlu-direct task currently exists.

### Details

The standard kmmlu task appears to have been removed about a year ago, possibly unintentionally.

This contrasts with kmmlu-hard, which correctly includes both multiple-choice (kmmlu-hard) and generative (kmmlu-hard-direct) versions.

kmmlu ([Dataset](https://www.google.com/url?sa=E&q=https%3A%2F%2Fhuggingface.co%2Fdatasets%2FHAERAE-HUB%2FKMMLU)) is a widely-used Korean benchmark. The standard accuracy-based evaluation is still common practice, as seen in recent work like the EXAONE 3.5 paper ([arXiv:2412.04862](https://www.google.com/url?sa=E&q=https%3A%2F%2Farxiv.org%2Fabs%2F2412.04862)).

### Solution

I've submitted a PR to re-add the standard kmmlu multiple-choice task using the acc metric. It has been tested locally.

closed by #2849
Repository: EleutherAI/lm-evaluation-harness
Issue: Significantly different results with different backends
Root cause: Interface Contract Mismatch

For the model `kaitchup/Qwen2.5-72B-Instruct-AutoRoundGPTQ-8bit`, the `leaderboard_ifeval` results differ significantly between the HF backend and the vLLM backend. Could you provide insights into the possible reasons or help debug the issue? Thanks in advance!



### HF backend
~~~ bash
CUDA_VISIBLE_DEVICES=0,1 lm-eval --model hf --model_args pretrained=./,parallelize=True,dtype=float16 --tasks leaderboard_ifeval --batch_size 16 --limit 10
~~~



hf (pretrained=./,parallelize=True,dtype=float16), gen_kwargs: (None), limit: 10.0, num_fewshot: None, batch_size: 16

| Tasks              | Version | Filter | n-shot | Metric                  |      |  Value |      | Stderr |
| ------------------ | ------: | ------ | -----: | ----------------------- | ---- | -----: | ---- | ------ |
| leaderboard_ifeval |       3 | none   |      0 | inst_level_loose_acc    | ↑    | 0.8333 | ±    | N/A    |
|                    |         | none   |      0 | inst_level_strict_acc   | ↑    | 0.7222 | ±    | N/A    |
|                    |         | none   |      0 | prompt_level_loose_acc  | ↑    | 0.7000 | ±    | 0.1528 |
|                    |         | none   |      0 | prompt_level_strict_acc | ↑    | 0.5000 | ±    | 0.1667 |


### vllm backend
~~~bash
CUDA_VISIBLE_DEVICES=0,1 lm-eval  --model vllm --model_args pretrained=./,tensor_parallel_size=2,dtype=float16 --tasks leaderboard_ifeval --batch_size auto  --limit 10
~~~

vllm (pretrained=./,tensor_parallel_size=2,dtype=float16), gen_kwargs: (None), limit: 10.0, num_fewshot: None, batch_size: auto

limit 10

| Tasks              | Version | Filter | n-shot | Metric                  |      |  Value |      | Stderr |
| ------------------ | ------: | ------ | -----: | ----------------------- | ---- | -----: | ---- | ------ |
| leaderboard_ifeval |       3 | none   |      0 | inst_level_loose_acc    | ↑    | 0.2222 | ±    | N/A    |
|                    |         | none   |      0 | inst_level_strict_acc   | ↑    | 0.2222 | ±    | N/A    |
|                    |         | none   |      0 | prompt_level_loose_acc  | ↑    | 0.1000 | ±    | 0.1000 |
|                    |         | none   |      0 | prompt_level_strict_acc | ↑    | 0.1000 | ±    | 0.1000 |

For the mlc-llm and llama-cpp-python, I have the similar problem

@kunxiongzhu For llama-cpp-python, the problem may come from [https://github.com/abetlen/llama-cpp-python/issues/1983](https://github.com/abetlen/llama-cpp-python/issues/1983).
Repository: EleutherAI/lm-evaluation-harness
Issue: How can I evaluate mlc-llm framework?
Root cause: Documentation Deficiency

Can anybody tell me how to evaluate [mlc-llm](https://github.com/mlc-ai/mlc-llm)?

I followed the steps below to evaluate a model running on MLC:
1. mlc_llm serve --host localhost  HF://mlc-ai/Llama-3.2-1B-Instruct-q4f16_1-MLC
2. lm_eval --model local-completions --tasks commonsense_qa --model_args model=mlc-ai/Llama-3.2-1B-Instruct-q4f16_1-MLC,base_url=http://localhost:8000/v1/completions,num_concurrent=1,max_retries=3,tokenized_requests=False

There are two strange phenomena. 
1. The accuracy of the model measured this way is much lower than that on Hugging Face, dropping from around 0.6 to 0.2 (I'm using the same model, whether it's Llama or GPT, and the same dataset). This clearly seems to be an error. 
2. When I evaluate different models with MLC, the results are still the same (when the models are different, the results should theoretically be different). This clearly isn't an issue with MLC. 

Can anybody tell me why this is happening? Did I make a mistake somewhere?


![Image](https://github.com/user-attachments/assets/58c82567-342e-4a67-a577-191e66ea02c1)

![Image](https://github.com/user-attachments/assets/9e9d0092-d3de-4d87-8b2a-33de6eaf0c31)

It seems like your model is not generating anything at all. Could you print the incoming request on the server? Where did you find the value of 0.6?

Could you show me how to print the request?

When I directly used the following cmd: 
lm_eval --model hf   --model_args pretrained=meta-llama/Llama-3.2-1B-Instruct    --tasks commonsense_qa
the result of the same dataset is about 0.6
![Image](https://github.com/user-attachments/assets/ff67424a-0332-4dc8-adc7-9572308ce9ca)

Also, have you tried using MLC to run other models on the same dataset? I found that all the models generated the same results, which is pretty weird

@kunxiongzhu I run into the same problem running it locally on my machine. Do not have a solution yet. I agree that the result of 0.2 is not correct, however please be aware that the MLC model was quantised so the accuracy might be different from the transformers model. 

@falkbene Thanks, yeah, the worst part is that when I switch to an fp16 type model or other models, the result remains the same at 0.2, which doesn't make sense. 

Can you run other models on the same dataset using MLC and check if your evaluation results remain unchanged (always giving an incorrect result)?
Repository: EleutherAI/lm-evaluation-harness
Issue: How to load Phi-4 model?
Root cause: Documentation Deficiency

I tried to eval the phi-4 model using llm-harness and it gave me an error where it couldn't find these following files. Which is very weird. 

![Image](https://github.com/user-attachments/assets/1f7833d8-ea5c-4210-9887-59aee3f2f347)
Repository: EleutherAI/lm-evaluation-harness
Issue: Performance of Llama3-8B on WiC (Word-in-Context) and RACE
Root cause: Documentation Deficiency

Im trying to evaluate llama3-8b on WiC. The cmd is as follows
```
lm_eval --model hf \
    --model_args pretrained=meta-llama/Meta-Llama-3-8B \
    --tasks wic \
    --device cuda:0 \
    --batch_size 4
```
And the results is 
```
|Tasks|Version|Filter|n-shot|Metric|   |Value |   |Stderr|
|-----|------:|------|-----:|------|---|-----:|---|-----:|
|wic  |      1|none  |     0|acc   |↑  |0.5047|±  |0.0198|
```
The results is similar to random selection. Why it would be like this? How can I get a normal acc value?
The version of `lm_eval` is 0.4.5

Also, I evaluate llama3-8b on RACE as follows
```
lm_eval --model hf \
    --model_args pretrained=meta-llama/Meta-Llama-3-8B \
    --tasks race \
    --device cuda:0 \
    --batch_size 4
```
The results is 
```
|Tasks|Version|Filter|n-shot|Metric|   |Value |   |Stderr|
|-----|------:|------|-----:|------|---|-----:|---|-----:|
|race |      2|none  |     0|acc   |↑  |0.4057|±  |0.0152|
```
While the llama3 report has this 

![Image](https://github.com/user-attachments/assets/7623a5c8-5507-4cd6-a23c-a2377b974b2f)

Is there any setting i made wrong? 

Hi!  I can't find reference scores for 8B models on WiC. Do you know of any?

> Hi! I can't find reference scores for 8B models on WiC. Do you know of any?

Me neither. I just trying to evaluate the language comprehension of LLM. 
If WiC is not a common dataset of LLM evaluation, is there more appropriate datasets for LLM to evaluate its language comprehension ability?
Repository: EleutherAI/lm-evaluation-harness
Issue: How can we modify different generation settings in transformers/generation when evaluating ?
Root cause: Documentation Deficiency

There are many generation methods in Transformers

![Image](https://github.com/user-attachments/assets/3d1ea56f-1b78-4115-bc1c-c07a0720844f)


The official method to change generation settings in Transformers is using model.generate() function. For example, if we  
want Dola generation, just add dola_layers="high" to  generate() like this:

`outputs = model.generate(**inputs, max_new_tokens=50, dola_layers="high", do_sample=False)`

So where is the model.generate() part in this repository or anywhere can we change the generation method when evaluating? 


I also tried `--gen_kwargs dola_layers="high" \`, but there is no change

Hi! It should pass the kwargs to `generate` appropriately. You could confirm by setting the `generation_kwargs` here [manually](https://github.com/EleutherAI/lm-evaluation-harness/blob/1514ac1e87548b8ab901e64422281d86f3a9ad0f/lm_eval/models/huggingface.py#L909)