Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Understanding the gap in benchmark results for Qwen3VL models
Root cause: Documentation Deficiency

Qwen3-VL reported benchmark results were generated using **VLMEvalKit**, but we would like to reproduce them using **lmms-eval**.
Some benchmarks like MMStar show a significant score gap when evaluated with lmms-eval.
I will use MMStar as the example, but the same conclusion extends to many other benchmarks.

## Reported vs Reproduced Scores
####  [Qwen3-VL-2B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-2B-Instruct) reported results on MMStar: 58.3
<img width="165" height="119" alt="Image" src="https://github.com/user-attachments/assets/5dc7e99f-f016-4947-a60a-1a530851c271" />




#### Results using `lmms-eval` (task: `mmstar`): 44.7
```
accelerate launch --num_processes 8 --main_process_port 12380 -m lmms_eval \
    --model qwen3_vl \
    --model_args "pretrained=Qwen/Qwen3-VL-2B-Instruct,attn_implementation=flash_attention_2,max_pixels=6422528,min_pixels=524288" \
    --tasks "mmstar" \
    --batch_size 1 \
    --log_samples \
    --output_path ./logs/ 

qwen3_vl (pretrained=Qwen/Qwen3-VL-2B-Instruct,attn_implementation=flash_attention_2,max_pixels=6422528,min_pixels=524288), gen_kwargs: (), limit: None, num_fewshot: None, batch_size: 1
|Tasks |Version|Filter|n-shot|        Metric         |   |Value |   |Stderr|
|------|-------|------|-----:|-----------------------|---|-----:|---|------|
|mmstar|Yaml   |none  |     0|average                |↑  |0.4469|±  |   N/A|
|mmstar|Yaml   |none  |     0|coarse perception      |↑  |0.7465|±  |   N/A|
|mmstar|Yaml   |none  |     0|fine-grained perception|↑  |0.5421|±  |   N/A|
|mmstar|Yaml   |none  |     0|instance reasoning     |↑  |0.6254|±  |   N/A|
|mmstar|Yaml   |none  |     0|logical reasoning      |↑  |0.3545|±  |   N/A|
|mmstar|Yaml   |none  |     0|math                   |↑  |0.1801|±  |   N/A|
|mmstar|Yaml   |none  |     0|science & technology   |↑  |0.2330|±  |   N/A|
```

#### Results using `lmms-eval` (task: `mmstar_oc`): 56.8 (commit: https://github.com/EvolvingLMMs-Lab/lmms-eval/pull/907)

```
accelerate launch --num_processes 8 --main_process_port 12380 -m lmms_eval \
    --model qwen3_vl \
    --model_args "pretrained=Qwen/Qwen3-VL-2B-Instruct,attn_implementation=flash_attention_2,max_pixels=6422528,min_pixels=524288" \
    --tasks "mmstar_oc" \
    --batch_size 1 \
    --log_samples \
    --output_path ./logs/ 
qwen3_vl (pretrained=Qwen/Qwen3-VL-2B-Instruct,attn_implementation=flash_attention_2,max_pixels=6422528,min_pixels=524288), gen_kwargs: (), limit: None, num_fewshot: None, batch_size: 1
|  Tasks  |Version|Filter|n-shot|        Metric         |   |Value |   |Stderr|
|---------|-------|------|-----:|-----------------------|---|-----:|---|------|
|mmstar_oc|Yaml   |none  |     0|average                |↑  |0.5681|±  |   N/A|
|mmstar_oc|Yaml   |none  |     0|coarse perception      |↑  |0.7372|±  |   N/A|
|mmstar_oc|Yaml   |none  |     0|fine-grained perception|↑  |0.5772|±  |   N/A|
|mmstar_oc|Yaml   |none  |     0|instance reasoning     |↑  |0.6676|±  |   N/A|
|mmstar_oc|Yaml   |none  |     0|logical reasoning      |↑  |0.4784|±  |   N/A|
|mmstar_oc|Yaml   |none  |     0|math                   |↑  |0.5116|±  |   N/A|
|mmstar_oc|Yaml   |none  |     0|science & technology   |↑  |0.4368|±  |   N/A|
```

## Why the Gap Appears
### VLMEvalKit and lmms-eval process the data differently
- `lmms-eval` uses HF datasets: [Lin-Chen/MMStar](https://huggingface.co/datasets/Lin-Chen/MMStar)
- VLMEvalKit uses a `.tsv` (see code  [here](https://github.com/open-compass/VLMEvalKit/blob/main/vlmeval/dataset/image_mcq.py#L91)) **with a different column layout that enables Qwen3-VL-specific prompts** (code [here](https://github.com/open-compass/VLMEvalKit/blob/19c0e386c0967936b5ab4357abdabd670ba5d361/vlmeval/vlm/qwen3_vl/prompt.py#L93))
- I uploaded that version to HF (see here: https://huggingface.co/datasets/morpheushoc/MMStar_opencompass)

Using this dataset we can reproduce how VLMEvalkit create the prompt for Qwen3VL models [lmms_eval/tasks/mmstar/utils.py](https://github.com/ArdalanM/lmms-eval/blob/add/mmstar_opencompass/lmms_eval/tasks/mmstar/utils.py#L134)

#### Current lmms-eval prompt 
```
from datasets import load_dataset
doc = load_dataset("Lin-Chen/MMStar")["val"][0]
text = mmstar_doc_to_text(doc, {})
visual = mmstar_doc_to_visual(doc)
print(text)

# Which option describe the object relationship in the image correctly?
# Options: A: The suitcase is on the book., B: The suitcase is beneath the cat., C: The suitcase is beneath the bed., D: The suitcase is beneath the book.
```

`lmms-eval` using VLMEvalKit style prompt (commit: https://github.com/EvolvingLMMs-Lab/lmms-eval/pull/907)
```
from datasets import load_dataset
doc = load_dataset("morpheushoc/MMStar_opencompass")["val"][0]
text = mmstar_oc_doc_to_text(doc)
visual = mmstar_oc_doc_to_visual(doc)
print(text)

# Question: Which option describe the object relationship in the image correctly?
# Options:
# A. The suitcase is on the book.
# B. The suitcase is beneath the cat.
# C. The suitcase is beneath the bed.
# D. The suitcase is beneath the book.
# Answer with the option letter only.
```

These minor differences in prompt construction are responsible for the benchmark gap between lmms-eval and VLMEvalKit.
VLMEvalKit is using model specific prompts (see example for Qwen3VL [here](https://github.com/open-compass/VLMEvalKit/blob/19c0e386c0967936b5ab4357abdabd670ba5d361/vlmeval/vlm/qwen3_vl/prompt.py). What would be the best way to add model specific template/prompt into `lmms-eval`





Given the amount of deviation in benchmark performance caused by these little changes in prompting, doesn't that suggest that the model is overfitting on these exact benchmark questions (with that exact formatting)?