zhiminy/EvalEng huggingface dataset:
Repository: open-compass/VLMEvalKit
Issue: A bug in qwen3vl
Root cause: Validation Gap

Thanks for the author's great efforts on this work. I found that the new version of qwen3vl code seems to have a bug. In qwen3_vl/model.py l81, the generation config does not have the “do_sample” parameter when it is set. It seems that this will cause a conflict.

Is there any evaluation code about Qwen3-VL? I try to search on GitHub, but it's difficult to find running code that we can directly use. The official evaluation code from the `Qwen3-vl` repo is actually not a complete code to reproduce the results in the report. 

> Is there any evaluation code about Qwen3-VL? I try to search on GitHub, but it's difficult to find running code that we can directly use. The official evaluation code from the `Qwen3-vl` repo is actually not a complete code to reproduce the results in the report.

I just use the newest VLMEvalKit code to evaluate the results. Also, that's true. I can't reproduce the results either.

> > Is there any evaluation code about Qwen3-VL? I try to search on GitHub, but it's difficult to find running code that we can directly use. The official evaluation code from the `Qwen3-vl` repo is actually not a complete code to reproduce the results in the report.
> 
> I just use the newest VLMEvalKit code to evaluate the results. Also, that's true. I can't reproduce the results either.

The results I reproduced from testing the original weights of qwen3vl-8B-Instruct are lower than those in the paper.

Hello~ how many GPUs do you use so that can running qwen3vl-8B-Instruct? I use 8 A100 GPU but even qwen3vl-4B-Instruct will get the error of OOM

> Hello~ how many GPUs do you use so that can running qwen3vl-8B-Instruct? I use 8 A100 GPU but even qwen3vl-4B-Instruct will get the error of OOM

Hi, I use 8 A800 for deploying and sft qwen3vl-8B-Instruct, and it works well.

Hello! could you please specify which Benchmark scores are inconsistent? How large is the gap between the results measured with VLMEvalKit and those officially reported by Qwen3-VL? We are working hard to align the evaluation settings. Thank your for your time!

> Hello! could you please specify which Benchmark scores are inconsistent? How large is the gap between the results measured with VLMEvalKit and those officially reported by Qwen3-VL? We are working hard to align the evaluation settings. Thank your for your time!

Thanks for your fast reply. Four results we produced are quite low: MMMU:64.1(-11.9),MMStar:71.5(-6.2),AI2D:85.4(-4.1),MathVista:80.3(-3.5). And one result is super hight: Hallusion:74.3(+10.5)
Repository: open-compass/VLMEvalKit
Issue: llava_v1.5_7b wrong results on Seedbench_IMG
Root cause: Validation Gap

Hi,

I have checked the saved result of llava7b on the seedbench_image benchmark and found that: for some problems, the llava7b can give the right prediction but in the evaluation framework it shows that it will return wrong answers.

For example, for index = 1198,4307
![output1198](https://github.com/open-compass/VLMEvalKit/assets/41474500/508a380e-2b98-4d2f-8889-293637c157c5)
Which object is likely found in the boy's hand? A: A book B: A soccer ball C: A calculator D: A pencil

![output4307](https://github.com/open-compass/VLMEvalKit/assets/41474500/8dc1f2f7-e230-47ed-a036-4ee51be16476)
Where is the priest located in the image? A: In front of the stained glass window B: To the right of the bride and groom C: To the left of the bride and groom D: Behind the bride and groom

Can you help explain this? Thanks for your help!


Hi, @BonitoW , 
Honestly, I'm not sure about the reason. Do you mean you got the right prediction when directly infer llava_v1.5_7b with those two questions? If so, I'll give a try. 

Hi, @BonitoW ,
any updates?
Repository: open-compass/VLMEvalKit
Issue: Qwen3-VL-238B-A22B-Instruct model occur ERROR：CUDA out of memory during Evaluation
Root cause: Resource Mishandling

hello~ When I evaluate the Qwen3-VL-238B-A22B-Instruct model with MMMU etc. benchmarks, at beginning everything is ok, the model can load and infer data, but for a while there would occur an error : torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 476.00 MiB. GPU 1 has a total capacity of 79.33 GiB of which 473.81 MiB is free. every benchmark will break in middle.
I use 8 A100 GPU, tp=8, how can i fix it ?

<img width="1811" height="164" alt="Image" src="https://github.com/user-attachments/assets/49165031-9fc2-4d4b-8d4d-a3135bc50cd3" />

<img width="754" height="227" alt="Image" src="https://github.com/user-attachments/assets/05c7ead7-6e44-4f1f-a216-2f4ff1a2bdb8" />

<img width="598" height="297" alt="Image" src="https://github.com/user-attachments/assets/74fa7271-0345-4e63-a4bb-e5f33fe3b918" />

the error image:
<img width="2504" height="969" alt="Image" src="https://github.com/user-attachments/assets/2d6ecc24-d189-429f-8de1-48585139d7bf" />

@mjuicem can you help take a look at this problem?

@jmq2025 Hello! We've also encountered this issue. It seems to be a common problem — you can refer to the discussion here for more details: https://github.com/QwenLM/Qwen3-VL/issues/1522#issuecomment-3345196749
Hope this helps!

> [@jmq2025](https://github.com/jmq2025) Hello! We've also encountered this issue. It seems to be a common problem — you can refer to the discussion here for more details: [QwenLM/Qwen3-VL#1522 (comment)](https://github.com/QwenLM/Qwen3-VL/issues/1522#issuecomment-3345196749) Hope this helps!

thanks for your suggestions, it's useful.
Repository: open-compass/VLMEvalKit
Issue: 测试会一直卡住，没有csv生成
Root cause: Resource Mishandling

<img width="308" height="340" alt="Image" src="https://github.com/user-attachments/assets/2dd87b51-7a66-4c58-a6d7-e41a8fdf9644" />



torchrun --nproc-per-node=8 run.py --data MMStar --model Qwen2.5-VL-rl  --verbose --work-dir ./outputs/Qwen2.5-VL-rl


只生成了几个pkl


请问该问题解决了吗，我在测试的时候也卡住了，用的数据集是WorldSense 模型是 Qwen2.5-Omni-7B
Repository: open-compass/VLMEvalKit
Issue: How can I use VLMEvalKit to test the DeepSeek-OCR model and obtain its accuracy score on OmniDocBench?
Root cause: Documentation Deficiency

How can I use VLMEvalKit to test the DeepSeek-OCR model and obtain its accuracy score on OmniDocBench?

Hello, I would like to ask if you have been successful?
Repository: open-compass/VLMEvalKit
Issue: Inquiry on Reproducing Qwen3-VL Tool-Use Results
Root cause: Unimplemented Feature Gap

The results of Qwen3-VL using tools for fine-grained perception tasks, as shown in the official report, are very impressive. I was wondering if it is currently possible to reproduce these results using VLMEvalKit?

<img width="4099" height="5608" alt="Image" src="https://github.com/user-attachments/assets/9fc14b1d-ca9f-4c0b-b96f-ec8696e36c22" />
Repository: open-compass/VLMEvalKit
Issue: ScienceQA_TEST的评测怀疑与官方的测评集不对齐！
Root cause: Configuration Error

根据ScienceQA论文和官网 https://scienceqa.github.io/index.html#home，训练集、验证集、测试集是3：1：1，测试集应该有4241张图片。

modelscope里面的 
https://modelscope.cn/datasets/swift/ScienceQA 的文档写的数据量是正确的，

执行具体代码
```
from modelscope.msdatasets import MsDataset
ds =  MsDataset.load('AI-ModelScope/ScienceQA', subset_name='default', split='test')
len(ds)==4241
```

也是正确的，是True。

但是VLMEvalKit的评测结果是错的，实测发现 只有2017条数据。

这是我的运行命令
```bash
swift eval \
    --eval_backend VLMEvalKit \
    --infer_backend pt \
    --eval_dataset ScienceQA_TEST \
    --model QwenQwen3-VL-4B-Instruct
```

结果文件在
./eval_output/vlmeval/时间/Qwen3-VL-4B-Instruct/Qwen3-VL-4B-Instruct_ScienceQA_TEST_openai_result.xlsx
中，这个只有一个表格，只有2017条数据。

看起来VLMEvalKit的 ScienceQA_TEST 没有正确对齐官方数据，请问这里面是有什么原因吗？是不是有些数据被特意筛选呢？还是存在一些技术问题？

我做了一个修复，修正后的数据集上传到Huggingface了，准备提个PR

经过初步查看，VLMEvalKit现在的数据确实和官方不符，删除了2224数据，经过初步判断，1. 这些数据都是正常内容，没有违规内容，不存在审核问题 2. 这些数据和已有的问题并不重复（通过question、answer、choices一起判断是否是唯一的问题）。我目前也没有找到这些是某个论文或者文献定义的子集的证据。可见应该是VLMEvalKit在制作数据时文件存储和预处理存在问题，可能导致了数据丢失，影响了VLMEvalKit在这个评测集上的正确性和可靠性

@2catycm 

ScienceQA 应该是一个 text-only QA 和 vqa 都有的数据集，我印象中可能是去掉了 text-only 部分，你可以看下是不是这样。

如果你新提 pr，可以将新的测试集命名为 ScienceQA_TEST，原有的命名为 ScienceQA_TEST_VQAOnly
Repository: open-compass/VLMEvalKit
Issue: ModuleNotFoundError: No module named 'vlmeval.dataset.utils.sr3d'
Root cause: Unimplemented Feature Gap

Hello!When running 3DSRBench, I encountered the following issue. Could you provide the relevant code?

ERROR - RUN - run.py: main - 492: Model Qwen3-VL-2B-Instruct x Dataset 3DSRBench combination failed: No module named 'vlmeval.dataset.utils.sr3d', skipping this combination.
Traceback (most recent call last):
  File "/data/luozehang/VLMEvalKit/run.py", line 463, in main
    eval_results = dataset.evaluate(result_file, **judge_kwargs)
  File "/data/luozehang/VLMEvalKit/vlmeval/dataset/image_mcq.py", line 2397, in evaluate
    from .utils.sr3d import parse_3dsr_prediction, eval_3dsr
ModuleNotFoundError: No module named 'vlmeval.dataset.utils.sr3d'


Hi @wufeim, can you help to take a look at this problem? 

Hi, sorry I didn't push this part of the code. Could you refer to [this commit](https://github.com/open-compass/VLMEvalKit/commit/c9729aac202c4acacdba43150ca1fd7489493aa3) and maybe contact the author?

+1 seems that the s3rd.py file was never committed

hi authors, any updates on this? @kennymckormick @FangXinyu-0913 @wufeim 

+1

Any updates on the integration of this dataset? It is been officially supported in recent update but the key s3rd.py file is still missing @wufeim
Repository: open-compass/VLMEvalKit
Issue: [Bug] MiMo
Root cause: Validation Gap

For starting with 'MiMo', I've changed it in `run.py`. Please review it if it is correct to run right.

```python
            if 'MiMo' in model_path:
                chat = ChatTemplateConfig(model_name=model_path)
            else:
                chat = ChatTemplateConfig(model_name='qwen2d5-vl')
            self.model = pipeline(
                model_path,
                backend_config=TurbomindEngineConfig(session_len=32768, cache_max_entry_count=0.1, tp=num_gpus),
                chat_template_config=chat)
```
Repository: open-compass/VLMEvalKit
Issue: Does ImageCaptionDataset have issues with COCO_VAL metrics calculation?
Root cause: Validation Gap

In the COCO_Caption_Scorer calculation process(class ImageCaptionDataset), jieba was not used for word segmentation; Instead, labels and predictions were directly used for metric calculation. Is this reasonable?


I used VLMEvalKit to perform predictions on the qwen3vl-32b model on the COCO_VAL dataset, and the model output was stored as an xlsx file.
python3 run.py --data COCO_VAL --model Qwen3-VL-32B-Instruct --workdir /my_path/ --verbose

I successfully obtained the prediction results file /my_path/Qwen3-VL-32B-Instruct_COCO_VAL.xlsx

Then I used the ImageCaptionDataset in VLMEvalKit to calculate the model prediction result metrics.
But, these metrics is very strange.

{'Bleu': [12.798238059985842, 6.888599261016476, 3.364287445868528, 1.6706468881999559], 'ROUGE_L': 12.974969371645798, 'CIDEr': 2.026869510712234e-08}

I noticed that the answer in the stored file are styled as follows:
"['A child holding a flowered umbrella and petting a yak.', 'A young man holding an umbrella next to a herd of cattle.', 'a young boy barefoot holding an umbrella touching the horn of a cow', 'A young boy with an umbrella who is touching the horn of a cow.', 'A boy holding an umbrella while standing next to livestock.']"
and the prediction in the stored file are styled as follows:
"A young boy holding an umbrella and feeding a water buffalo in a rural setting."

However, the data processing in ImageCaptionDataset.evaluate does not perform any word segmentation; it directly calculates metrics from the data in the labels and the response.

<img width="630" height="370" alt="Image" src="https://github.com/user-attachments/assets/4a4be884-3939-4619-96fc-8289ac8f3be3" />
Repository: open-compass/VLMEvalKit
Issue: ai2d数据集结果不正常
Root cause: Validation Gap

我在测试Qwen3-VL-4B-Instruct，执行
`CUDA_VISIBLE_DEVICES="5,6" python VLMEvalKit/run.py --model "$MODEL_NAME" --work-dir "$OUTPUT_PATH"  --data "AI2D_TEST"`
测试结果不正常，太低了
```
-------------------------  -------------------
split                      none
Overall                    0.19203367875647667
atomStructure              0.125
eclipses                   0.22580645161290322
faultsEarthquakes          0.03571428571428571
foodChainsWebs             0.20395869191049915
lifeCycles                 0.1366906474820144
moonPhaseEquinox           0.18050541516245489
partsOfA                   0.26078028747433263
partsOfTheEarth            0.23076923076923078
photosynthesisRespiration  0.11392405063291139
rockCycle                  0.14925373134328357
rockStrata                 0.07317073170731707
solarSystem                0.4166666666666667
typesOf                    0.16326530612244897
volcano                    0.3125
waterCNPCycle              0.06818181818181818
-------------------------  -------------------
```
Repository: open-compass/VLMEvalKit
Issue: ChartQA 评测中存在误判问题
Root cause: Validation Gap

## 问题描述
在对 **ChartQA** 进行评测时，我发现评测框架在部分情况下会因为数值格式差异而出现误判。

## 示例
**问题：**  
> What's the percentage of U.S adults who refused?

**图表：**  
![chart](https://github.com/user-attachments/assets/a3058e16-ce10-4ef4-9c59-149719107783)

**模型回答：**  
`2%`

**标准答案：**  
`2`

**评测结果：**  
`False`

## 说明
该示例中，模型输出的 “2%” 与标准答案 “2” 实际上语义一致，均表示 2%。此类情况应视为正确答案。  

如果这是我个人评测配置的问题，也请官方团队指正，非常感谢！

I meet the same question. 
And it can not distinguish "Blue" and "Blue." 🤣
Repository: open-compass/VLMEvalKit
Issue: 为什么judge模型一定要是指定的几个GPT模型
Root cause: Architectural Constraint

很多文件里都有`assert model in ['chatgpt-0125', 'exact_matching', 'gpt-4-0125']`这一行，比如`image_mcq.py`，这一行的作用是什么？我想用其他Openai兼容格式的模型，我可以注释掉它吗

Hi! As far as I know, in VLMEvalKit the “judge model” (the model used to score other models’ outputs) is currently only supported for a small set of models that have been specifically adapted and validated. I think this is mainly because:

- Prompt dependency: The judging prompts (system + user) are written and tested for those specific models, so the output format is predictable.
- Hard-coded parsing: The post-processing/parsing (e.g., extracting A/B/C/D, end markers, confidence, etc.) assumes those models’ output patterns, and hasn’t been adapted for other models.
- Avoiding evaluation failures: If you switch to another OpenAI or OpenAI-compatible model without prompt tuning and parser support, it may produce unexpected formats, fail to parse, or give unreliable scores.

Technically, you could comment out the assert or add a new model name, but I think it would be much safer to do something like:
1. Add the model name to the allowed list.
2. Review/design the judging prompt to make sure the model returns the expected judge-format output.
3. Add or adjust the post-processing/parsing logic (some models may respond more freely, so the parser might need changes).
4. Run plenty of regression tests across tasks/benchmarks to confirm the judge outputs are reasonable and stable.

Hope this can help! I'm looking forward to your reply!
Repository: open-compass/VLMEvalKit
Issue: MathVision evaluation sub-thread error: signal only works in main thread of the main interpreter
Root cause: Resource Mishandling

During MathVision evaluation, the following error is reported:
WARNING - mathv.py: post_check - 127: <class 'ValueError'>: signal only works in main thread of the main interpreter

@SYuan03 I'm encountering the same issue hhh

> [@SYuan03](https://github.com/SYuan03) I'm encountering the same issue hhh

Hi @yph22 , this is just a warning due to signal being used in a sub-thread. It shouldn’t affect the evaluation results, so it’s safe to ignore.
Repository: open-compass/VLMEvalKit
Issue: How to calculate overall score for HallusionBench
Root cause: Documentation Deficiency

It seems it can produce aACC, qACC and fACC for the bench, but your [leaderboard](https://rank.opencompass.org.cn/leaderboard-multimodal) shows of its overall score. Do I have to turn to its original paper to access overall score? 
Thx!

Hi, @WizardMx ,
The original paper did not include an "Overall" metric, we define it because we want to use a single number to represent the overall performance on HallusionBench. 
We define Overall Score as the average of aACC, qACC, fACC.
Repository: open-compass/VLMEvalKit
Issue: There are some bugs within RefCOCO dataset
Root cause: Validation Gap

'RefCOCO': "https://huggingface.co/datasets/mjuicem/RefCOCO-VLMEvalKit/resolve/main/RefCOCO.tsv"

e.g.
119	/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAgGBgcGB8aAP......./2Q==	Please provide the bounding box coordinate of the region this sentence describes: <ref>rightest guyin black</ref>	[0.088250, 0.033709, 0.469672, 0.474155]	RefCOCO+ test-A
--- I suppose the prompt should be "guy in", not "guyin". 
Hi @mjuicem Could you help check this out, and see if there are other issues about the dataset. Thanks :)
Repository: open-compass/VLMEvalKit
Issue: Unable to reproduce Intern-S1-mini on XLRS-Bench-lite
Root cause: Unimplemented Feature Gap

Hi，
I used the following command to reproduce the results of the Intern-S1-mini model on XLRS-Bench-lite, and only achieved a score of 14.51, which is significantly different from the published score of 51.63 for Intern-S1-mini.
python run.py --data XLRS-Bench-lite --model Intern-S1-mini --verbose
Upon troubleshooting, I found that the class method XLRSBench.evaluate() of the XLRS-Bench-lite dataset only uses rule-based methods to extract answers. Since this class inherits from the base class ImageMCQDataset, could you modify the source code to add judge_model?



Best regards
Repository: open-compass/VLMEvalKit
Issue: Qwen3-VL on ScreenSpot datasets
Root cause: Configuration Error

Hi, I got very low score for Qwen3-VL on ScreenSpot datasets. According to Qwen3-VL's docs, it's grounding output is normed to a 0-1000 value. Should I change the normalization method in ScreenSpot datasets to `click_point = (click_point[0] / 1000, click_point[1] / 1000)` ?

@mjuicem, can you help take a look at it?
Repository: open-compass/VLMEvalKit
Issue: MMVet 数据集问题

请问官方代码中给出的 https://opencompass.openxlab.space/utils/VLMEval/MMVet.tsv 是MMVet v2版本的吗？

Hi! To the best of my understanding, the URL https://opencompass.openxlab.space/utils/VLMEval/MMVet.tsv used in the official code refers to the original MM-Vet v1 version, since the link does not include any “v2” designation and the repository or configuration does not reference a v2 resource.
Hope this can help! I'm looking forward to your reply!
Repository: open-compass/VLMEvalKit
Issue: Bug: Evaluation on the "CMMMU" benchmark may incorrectly load images from "MMMU".
Root cause: Validation Gap

The cause appears to be lines 15-16 in `vlmeval/dataset/image_base.py`,

```
if 'MMMU' in dataset:
        return 'MMMU'
```

which may incorrectly return the MMMU image root when the dataset is CMMMU.

Hi @Murray-Wang, we have already resolved this issue. Thank you for bringing this issue to our attention.
Repository: open-compass/VLMEvalKit
Issue: Benchmark必须把图片保存到本地吗？
Root cause: Unimplemented Feature Gap

想问两个问题：
1. Benchmark必须把图片保存到本地吗？可不可以不保存，直接用base64，比如在build prompt里面不是传入图片路径，而是传入base64，因为图片比较多转化时间非常长（几个小时）
2. 如果必须保存，每次转化是否有缓存机制，检测到图片已经存在就不再转换了

谢谢
Repository: open-compass/VLMEvalKit
Issue: Minor Inconvenience with MIA-Bench Evaluation Parsing
Root cause: Validation Gap

Sometimes MIA-Bench can't parse the evaluation output correctly because of a missing space.
```
score_ = first_sentence[i].split(':')[1][1:].split('/')
```
[https://github.com/open-compass/VLMEvalKit/blob/main/vlmeval/dataset/miabench.py#L73](url)
```
total_score_ = first_sentence[i + 1].split(':')[1][1:].split('/')
```
[https://github.com/open-compass/VLMEvalKit/blob/main/vlmeval/dataset/miabench.py#L76](url)
Both lines require each score item to look something like this: 'score of component 1: 2/4', and this 'score of component1:3/6' will fail. I understand the former is in accordance with the evaluation prompt, but some models (even strong commercial models) may not follow the format precisely. 

My current fix is replacing "[1:]" with ".strip()" as it works for both cases.
Repository: open-compass/VLMEvalKit
Issue: Pos_tokens.reshape error while testing mmmu on internlm-xcomposer2-7b 
Root cause: Validation Gap

We meet error while testing mmmu on internlm-xcomposer2-7b 
<img width="1318" alt="bug-2" src="https://github.com/open-compass/VLMEvalKit/assets/68582965/49863ef6-8caa-45fd-98dd-08a5431a58be">
Looking forward to your reply.
Repository: open-compass/VLMEvalKit
Issue: Add Support for ScreenSpot-Pro and CVBench
Root cause: Documentation Deficiency

Hello VLMEvalKit team,

Thank you for your excellent work on building a unified evaluation toolkit for vision-language models. I noticed that in the documentation, ScreenSpot-Pro and CVBench are already listed as benchmark entries, but they are not yet supported in the current release.

Since these benchmarks are important for fine-grained grounding and comprehensive computer vision evaluation, I would like to request their full integration into VLMEvalKit. This would allow users to run evaluations more seamlessly without relying on separate scripts, and it would greatly improve the coverage and usability of the toolkit.

Thanks for considering this enhancement!

You can use CV-Bench-2D and CV-Bench-3D.
It seems that nobody maintains the DOC 🤣

Thanks a lot!
Repository: open-compass/VLMEvalKit
Issue: Hope to include Qwen3
Root cause: Unimplemented Feature Gap

Qwen3 LLMs series have been a heat recently and achieve impressed progress in many aspects, hope to see Qwen3s in the supported model list soon!

@Gao-anqi I don't think language models will be supported here, as this is a VLM evaluation toolkit.
Maybe checkout [opencompass](https://github.com/open-compass/opencompass)?