zhiminy/EvalEng huggingface dataset:
Repository: open-compass/VLMEvalKit
Issue: Reproducibility of deepseek-vl inference results
Root cause: Algorithmic Error

Hello, I am using the deepseek-vl model "deepseek-ai/deepseek-vl-7b-chat" and have observed significant variability in its outputs: different runs produce different results. I have already tried setting `do_sample=False` and `temperature=0`, but the outputs remain non-deterministic. Could you advise what settings or steps are required to obtain reproducible responses?
Thank you very much.

Hi, based on my understanding and some experiments on my side:

When running inference with do_sample=False and temperature=0, the model is using greedy decoding. 
The output variability I observed with DeepSeek-VL on GPU seems more related to numerical non-determinism, such as CUDA parallel kernels, FP16/BF16 mixed-precision computation, and the sensitivity of the multimodal (vision + language) encoding pipeline to small floating-point differences.
Even without any random sampling, tiny differences in logits can sometimes flip the argmax token at certain positions.

In my experience, switching to FP32 inference makes the results noticeably more stable, although at a significant cost in speed.
Overall, under GPU-based mixed-precision multimodal inference, it seems difficult to achieve strict token-level reproducibility, and in practice we can usually only expect semantic-level consistency.
Hope this can help. I'm looking forward to your reply! 

Thank you very much for your reply. I still have a small question. When I run deepseek-vl on ScienceQA and MMMU datasets multiple times using the default settings in VLMEvalKit, the results differ between runs — that is, they are not reproducible. However, when I use deepseek-vl2-tiny, repeated runs produce largely consistent results. Have you encountered this behavior, and do you know what might cause it?


Yes, I’ve also encountered this behavior, and personally I think it’s quite reasonable. From my experience, it mainly comes down to differences in model scale and logit margins.

For deepseek-vl-7b-chat, the model is larger and the multimodal alignment pipeline is more complex. At some positions, the top-1 and top-2 logits can be extremely close. Under FP16/BF16 GPU inference, very small numerical fluctuations can sometimes flip the argmax, which leads to differences across runs.

In contrast, deepseek-vl2-tiny is much smaller and tends to produce “coarser” logit distributions, where the top-1 token is usually much higher than the others. Because it’s less often in these borderline cases, the outputs appear more stable across repeated runs.

Hope this helps, and please feel free to correct me if I’m missing something.

Thank you for your reply. However, yesterday I tried the deepseek-vl2 model (which is larger than deepseek-vl-7B) and found that the outputs from two runs were identical. Therefore, besides the model-size issue you mentioned, could there be other causes?
Repository: open-compass/VLMEvalKit
Issue: 如何使用提前下载好的本地数据集，在哪里能配置路径呢
Root cause: Documentation Deficiency


https://github.com/open-compass/VLMEvalKit/blob/240f1d7cc12b52f3a5f6bec9e1acab47d514b63c/vlmeval/smp/file.py#L69
这有个变量设置数据路径，默认是~/LMUData，把对应的文件放在那里
Repository: open-compass/VLMEvalKit
Issue: OSError: Incorrect path_or_model_id: 'xtuner/llava-internlm2-20b/projector'. Please provide either the path to a local folder or the repo_id of a model on the Hub.
Root cause: Validation Gap

torchrun --nproc-per-node=8 --nnodes=1 --node_rank=0 --master_addr 10.255.xxx.xxx --master_port 8109 run.py --data LLaVABench --model llava-internlm2-20b --verbose

But I met the following problem below:
Traceback (most recent call last):
  File "/train-xxx/code/xxx/src/test_scripts/test.py", line 6, in <module>
    projector = AutoModel.from_pretrained(projector_path,
  File "/usr/local/lib/python3.10/dist-packages/transformers/models/auto/auto_factory.py", line 484, in from_pretrained
    resolved_config_file = cached_file(
  File "/usr/local/lib/python3.10/dist-packages/transformers/utils/hub.py", line 462, in cached_file
    raise EnvironmentError(
OSError: Incorrect path_or_model_id: 'xtuner/llava-internlm2-20b/projector'. Please provide either the path to a local folder or the repo_id of a model on the Hub.

I also used the following script to have a test but got the same error by my side.
```
import os.path as osp
import torch
from transformers import AutoModel

projector_path="xtuner/llava-internlm2-20b/projector"
projector = AutoModel.from_pretrained(projector_path,
                                              trust_remote_code=True,
                                              torch_dtype=torch.float16,
                                              device_map='cpu')
```




Hi, @starlitsky2010 ，
Would you please paste the full error traceback here, seems I can not find codes related to VLMEvalKit here in your traceback.
 

Feel free to re-open the issue for any further updates.
Repository: open-compass/VLMEvalKit
Issue: Image Base64 Decoding Error in SimpleVQA
Root cause: External Dependency Breakage

The image base64 in the sample with index 83 cannot be normally decoded.
When calling the `decode_base64_to_image` function, it throws the following error:
`PIL.UnidentifiedImageError: cannot identify image file`

Hi @Bujiazi @FangXinyu-0913, could you please take a look at this issue. Thanks!

This issue can be solved by upgrading `pillow` to 12.0.0
Repository: open-compass/VLMEvalKit
Issue: 我的环境现在只支持我要使用的模型，但VLMEvalKit 在导入时尝试加载所有模型的依赖
Root cause: Architectural Constraint

我的环境现在只支持我要使用的模型，但VLMEvalKit 在导入时尝试加载所有模型的依赖
Repository: open-compass/VLMEvalKit
Issue: evalscope和这个vlmeval版本对应关系是什么
Root cause: External Dependency Breakage

我发现evalscope导入了一些现在版本没有的东西
from vlmeval.utils.arguments import Arguments as VLMEvalArguments

并不确定 evalscope 所使用的 vlmevalkit 版本，请完全按照 evalscope 的教程进行安装
Repository: open-compass/VLMEvalKit
Issue: 模型评估结果和官方公布结果不一致
Root cause: Configuration Error

从官方下载到本地的Qwen3-VL-4B-Instruct模型，在MMBench_DEV_CN_V11测评集中的测评结果，和官方的相差很大，官方公布的是83.5，而我测出来才0.498，模型推理参数设置如下：
"Qwen3-VL-4B-Instruct":{
            "class": "Qwen3VLChat",
            "model_path": "/Qwen3VL/Qwen3-VL-4B-Instruct",
            "use_custom_prompt":false,
            "use_vllm":true,
            "temperature":0.7, 
            "max_new_tokens":16384,
            "repetition_penalty":1.0,
            "presence_penalty":1.5,
            "top_p":0.8,
            "top_k":20,
            "gpu_utils":0.7
        }
请问可能是什么原因导致的？
Repository: open-compass/VLMEvalKit
Issue: Qwen3-VL-4B-Instruct模型评估ocrbench结果和官方公布结果不一致
Root cause: Configuration Error

问题1：评测结果不一致：
本地评测出来结果为87，详细信息如下：
官方公布的结果为88.1,相差一个点，
[2025-12-30 11:26:43] INFO - RUN - run.py: main - 469: The evaluation of model Qwen3-VL-4B-Instruct x dataset OCRBench has finished!
[2025-12-30 11:26:43] INFO - RUN - run.py: main - 470: Evaluation Results:
[2025-12-30 11:26:43] INFO - RUN - run.py: main - 472:
{
    "Text Recognition": 265,
    "Scene Text-centric VQA": 180,
    "Doc-oriented VQA": 184,
    "Key Information Extraction": 181,
    "Handwritten Mathematical Expression Recognition": 60,
    "Final Score": 870,
    "Final Score Norm": 87.0
}

评测配置如下：
    "Qwen3-VL-4B-Instruct": partial(
        Qwen3VLChat,
        # model_path="Qwen/Qwen3-VL-4B-Instruct",
        model_path="./model/Qwen3-VL-4B-Instruct",
        use_custom_prompt=False,
        use_vllm=False,
        temperature=0.7, 
        max_new_tokens=200,
        repetition_penalty=1.0,
        presence_penalty=1.5,
        top_p=0.8,
        top_k=20
    ),

问题2: qwen3vl的在mmstar之类的选择题上选项解释非常多，导致评测结果很长。默认的配置（ use_custom_prompt=False）也没有使用qwen3vl自己的prompt实现（实测use_custom_prompt=True之后ocrbench下降到85）。是否有合适的配置？
Repository: open-compass/VLMEvalKit
Issue: ModuleNotFoundError: No module named 'xtuner.parallel'
Root cause: External Dependency Breakage

I met this problem when test by cmd:
torchrun --nproc-per-node=8 --nnodes=1 --node_rank=0 --master_addr 10.255.244.33 --master_port 8109 run.py --data LLaVABench --model llava-internlm2-20b --verbose

Traceback (most recent call last):
  File "/code/src/VLMEvalKit/run.py", line 153, in <module>
    main()
  File "/code/src/VLMEvalKit/run.py", line 83, in main
    model = infer_data_job(
  File "/code/src/VLMEvalKit/vlmeval/inference.py", line 210, in infer_data_job
    model = infer_data(
  File "/code/src/VLMEvalKit/vlmeval/inference.py", line 142, in infer_data
    response = model.generate(prompt=struct['text'], image_path=struct['image'], dataset=dataset_name)
  File "/code/src/VLMEvalKit/vlmeval/vlm/llava_xtuner.py", line 177, in generate
    from xtuner.model.utils import prepare_inputs_labels_for_multimodal
  File "/usr/local/lib/python3.10/dist-packages/xtuner/model/__init__.py", line 3, in <module>
    from .sft import SupervisedFinetune
  File "/usr/local/lib/python3.10/dist-packages/xtuner/model/sft.py", line 16, in <module>
    from xtuner.parallel.sequence import (get_sequence_parallel_world_size,
ModuleNotFoundError: No module named 'xtuner.parallel'
[2024-04-03 03:58:31,981] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 1) local_rank: 0 (pid: 2207729) of binary: /bin/python

Did you install the latest version of xtuner? We recommend you to install xtuner from the source code (latest main branch at https://github.com/InternLM/xtuner). 
BTW, do you have any idea on the potential reason of this error @LZHgrla 

@starlitsky2010 Hi!

There are some bugs on pypi package of xtuner==0.1.16.

Install xtuner>=0.1.17 can solve this issue.

```
pip install 'xtuner>=0.1.17'
```

> Did you install the latest version of xtuner? We recommend you to install xtuner from the source code (latest main branch at https://github.com/InternLM/xtuner). BTW, do you have any idea on the potential reason of this error @LZHgrla

I used xtuner==0.1.15 and that fixes it. Thanks
Repository: open-compass/VLMEvalKit
Issue: Video Holmes llm judge is not used in evaluation?
Root cause: Validation Gap

```
    def evaluate(self, eval_file, **judge_kwargs):

        from .utils.videoholmes import get_dimension_rating, extract_option

        assert get_file_extension(eval_file) in ['xlsx', 'json', 'tsv'], 'data file should be an supported format (xlsx/json/tsv) file'  # noqa: E501

        tmp_file = get_intermediate_file_path(eval_file, '_tmp', 'pkl')
        tgt_file = get_intermediate_file_path(eval_file, '_rating', 'json')
        score_file = get_intermediate_file_path(eval_file, '_score')

        if not osp.exists(score_file):
            model = judge_kwargs.get('model', 'exact_matching')

            if model == 'exact_matching':
                model = None
            else:
                model = build_judge(**judge_kwargs)
                if not model.working():
                    warnings.warn('OPENAI API is not working properly, will use exact matching for evaluation')
                    warnings.warn(DEBUG_MESSAGE)
                    model = None
            res = {} if not osp.exists(tmp_file) else load(tmp_file)
            res = {k: v for k, v in res.items() if FAIL_MSG not in v}

            data = load(eval_file)
            data_un = data[~pd.isna(data['prediction'])]

            for idx in data['index']:
                ans = data.loc[data['index'] == idx, 'answer'].values[0]
                pred = str(data.loc[data['index'] == idx, 'prediction'].values[0])

                predicted_answer = extract_option(pred)

                data.loc[idx, 'score'] = int(predicted_answer == ans)

            rejected = [x for x in data['score'] if x == -1]

            print(
                f'Among {len(data)} questions, failed to obtain prediction for {len(data) - len(data_un)} questions, '
                f'failed to obtain the score for another {len(rejected)} questions. '
                f'Those questions will be counted as -1 score in ALL rating, and will not be counted in VALID rating.'
            )

            dump(data, score_file)
```


so llm judge is not used in VideoHolmes?
Repository: open-compass/VLMEvalKit
Issue: screenspot 分数对齐
Root cause: Algorithmic Error

<img width="507" height="364" alt="Image" src="https://github.com/user-attachments/assets/c5e41d65-3282-4035-9661-6160889db48e" />
而 https://huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct 中展示了 结果为 84.7，是什么情况呢

如果..我这么计算 总体分数 = (80.48 × 502 + 41.02 × 334 + 78.21 × 436) / (502 + 334 + 436) ≈ 69.34% 可以吗，这样子的话，就有不小的差距。
Repository: open-compass/VLMEvalKit
Issue: A major problem with the multiple-choice evaluation
Root cause: Documentation Deficiency

There is a major problem with the multiple-choice evaluation.
I am testing MMBench-dev-en here, I use the result document generated by the llava framework--llava_MMBench_DEV_EN.xlsx, the result of your test here is 0.68.

Because his prediction is all a single word--just the options, so I tried to match it myself, just `if item['prediction']==item['answer']`, and I found that the final result is 0.77, so your test standard is seriously wrong, or I missed something, please let me know.

If you want the result file to test, I can send you the result file, or you can just have a check.

![image](https://github.com/open-compass/VLMEvalKit/assets/33275244/6ecf0c9f-d93d-4bb7-8b2f-eb9ba03791c9)

Evaluation code is too complex, but sometimes the results generated by the model are simple

Would you please upload the excel file and share with me so that I can have a check?
Besides, please note that we use **CircularEval** for MMBench, thus the final accuracy is not simply the average of all atomic tests. In fact, in CircularEval, you fail on a question if you fail on any of the 4 circular passes. 
To learn more about CircularEval, you can refer to the original paper of MMBench: https://arxiv.org/abs/2307.06281

I understand, but if you save the response directly in the xlsx file, and then you evaluate the the results based on the xlsx file, I will be right in my way.
[llava_v1.5_13b_MMBench_DEV_EN_Base_DB2.xlsx](https://github.com/open-compass/VLMEvalKit/files/14902176/llava_v1.5_13b_MMBench_DEV_EN_Base_DB2.xlsx)

Has this evaluation method been used on other multi-choise datasets?

> Has this evaluation method been used on other multi-choise datasets?

No, it's only used for MMBench Series (MMBench, MMBench-CN, CCBench)

> No, it's only used for MMBench Series (MMBench, MMBench-CN, CCBench)

I see. But after looking at the source code, I still don’t quite understand where you performed this operation. The only difference is following `evaluate/multi-choise.py`:
```python
    if listinstr(['mmbench', 'ccbench'], dataset.lower()):
        data = load(eval_file)
        data['index'] = [int(x) for x in data['index']]
        dump(data, eval_file)
```
Where do you distinguish the evaluation scheme of mmbench from other datasets?
Has the index already been processed in the downloaded dataset?



A simple `if item['prediction']==item['answer']` does not consider the rotate evaluation design in the MMBench series, maybe you should read the paper carefully before challenging the correctness of the code.
Repository: open-compass/VLMEvalKit
Issue: Error Encountered in Multi-Node Evaluation Using Distributed Arguments
Root cause: Unimplemented Feature Gap

I encountered an issue while attempting to perform a multi-node evaluation using PyTorch's torchrun with specific distributed arguments. Below is the command I used, including the distributed arguments setup and the execution command:

=
```

DISTRIBUTED_ARGS=" \
    --nproc_per_node 3 \
    --nnodes 4 \
    --node_rank $NODE_RANK \
    --master_addr $MASTER_ADDR \
    --master_port $MASTER_PORT"

torchrun $DISTRIBUTED_ARGS run.py \
    --data MME MMBench_DEV_EN MMBench_DEV_CN CCBench SEEDBench_IMG MMMU_DEV_VAL MathVista_MINI HallusionBench LLaVABench \
    MMBench_TEST_EN MMBench_TEST_CN \
    --model llava
```
Upon execution, I received the following error message:

`RUN - ERROR - No such file or directory: './llava/312_MME.pkl'`
It seems like only the .pkl file on node0 was saved correctly. Only 012,112,212 was saved.
Thank you in advance for your assistance!

Hi, @jdy18 ,
Currently, we have not supported multi-node test yet. I would further check if it's easy to add this new feature. 

> Hi, @jdy18 , Currently, we have not supported multi-node test yet. I would further check if it's easy to add this new feature.

I have successfully resolved the issue. The function get_rank_and_world_size() should return rank instead of local_rank, and the call to torch.cuda.set_device(rank) should be modified to torch.cuda.set_device(local_rank).

Great, are you willing to create a PR to support this new feature? I think it's compatible with the original single node multi-GPU parallel inference. 

The modification has been updated into the main branch.
Repository: open-compass/VLMEvalKit
Issue: Can not reach the info_vqa dataset
Root cause: External Dependency Breakage

The link of [info_vqa](https://opencompass.openxlab.space/utils/VLMEval/InfoVQA_TEST.tsv ) is unreachable.

the same as `DocVQA_TEST` `InfoVQA_VAL` `InfoVQA_TEST`


It should be OK now, you can have a retry. 

Thanks!
Repository: open-compass/VLMEvalKit
Issue: MBench_TEST_CN和MMBench_TEST_EN的tsv里没有answer列
Root cause: Documentation Deficiency

您好，这里下载的MMBench_TEST_CN和MMBench_TEST_EN数据集里没有answer，无法进行评估

Hi, @Jiadwu2 ,
We do not provide answers for MMBench_TEST_CN/EN, since they are confidential.
To obtain evaluation results for these two datasets, submit your prediction files here: https://mmbench.opencompass.org.cn/mmbench-submission
Repository: open-compass/VLMEvalKit
Issue: [Feature Request] RealWorldQA Benchmark
Root cause: Unimplemented Feature Gap

Would you like to add this benckmark ([introduction](https://x.ai/blog/grok-1.5v), [dataset](https://data.x.ai/realworldqa.zip)) to VLMEval? It's a multi-choice dataset and can be easily added to this repo.

Some examples:
<img width="433" alt="image" src="https://github.com/open-compass/VLMEvalKit/assets/33491471/0ef012c2-1102-4da4-8f33-aefe47658cbe">
<img width="446" alt="image" src="https://github.com/open-compass/VLMEvalKit/assets/33491471/4875d368-a908-46bd-a6da-76fd82101a27">
<img width="428" alt="image" src="https://github.com/open-compass/VLMEvalKit/assets/33491471/e6a8166e-49c5-4491-857e-7f8a51799b73">
<img width="425" alt="image" src="https://github.com/open-compass/VLMEvalKit/assets/33491471/eace298b-ae2a-4fab-b7ed-fa1772924ba7">


@junming-yang 

Thanks! I find it's just supported!
Repository: open-compass/VLMEvalKit
Issue: IndexError: index 1 is out of bounds for dimension 0 with size 1
Root cause: Interface Contract Mismatch

  cur_image_features = image_features[cur_image_idx]
                         ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
IndexError: index 1 is out of bounds for dimension 0 with size 1


Is there any reason why previous version can evaluate normally, but updated from git got such error? 

And this error happens only during evaluation at about 6%

Is there zero images samples in eval MMBench?

Hi, @lucasjinreal , 
Which VLM are you evaluating? I think MMBench does not have zero-image samples. As an ablation, you can try to evaluate some other VLMs like qwen_chat on MMBench to localize the bug.
BTW, VLMEvalKit has recently updated its generate interface (check https://github.com/open-compass/VLMEvalKit/pull/140). I'm not sure if this error is due to the refactoring. 

Am not sure as well, but just got this, I used llava model with qwen as LLM. The task is MMBenchCN

I used a try ... catch to generate_inner, not it can run, but looks like would effect the metric

@lucasjinreal 
Please provide a command so that I can try to reproduce this problem. 

Sorry, the exception got is on MMMU, not mmbench.



I am also experiencing the same problem when trying to run on MMMU. (I tried it with llava_v1.5_7b.)

@kennymckormick Can u take a look, actually didn't get this in previous version, some models actually didn't support more than 2 images in a single chat like llava.


> @kennymckormick Can u take a look, actually didn't get this in previous version, some models actually didn't support more than 2 images in a single chat like llava.

OK, I will give a try for the llava_v1.5_7b x  MMMU configuration. 

Hi, @icrto @lucasjinreal 
After investigation I find this is a bug due to the refactor of interface. We have fixed it now. Please have a retry with the latest main branch.
Repository: open-compass/VLMEvalKit
Issue: opencompass多模态榜单上的分数是exact_matching还是GPT辅助计算的分数
Root cause: Documentation Deficiency

如题，请问opencompass多模态榜单上的分数是exact_matching还是GPT辅助计算的分数？如果没有openai Key的话，是不是得不到opencompass上榜单的分数？谢谢

Hi, @jiezhangGt ,
1. 为 GPT 辅助计算的分数
2. 视模型而定，若模型指令遵循能力够强，则不需要 GPT 辅助

> Hi, @jiezhangGt ,
> 
> 1. 为 GPT 辅助计算的分数
> 2. 视模型而定，若模型指令遵循能力够强，则不需要 GPT 辅助

收到您的回复，谢谢~~
Repository: open-compass/VLMEvalKit
Issue: llava 34B 评测时CUDA out of memory
Root cause: Unimplemented Feature Gap

你好，我在评测llava 34B 时，有8张卡可用，发现只占了一张卡进行推理，导致CUDA out of memory。请问llava 34B的评测支持多卡评测吗？谢谢

已解决

请问您是怎么解决的呢？我遇到了相同的问题
Repository: open-compass/VLMEvalKit
Issue: A problem with version conflicts
Root cause: Configuration Error

![image](https://github.com/open-compass/VLMEvalKit/assets/128620656/0f0920b8-2dc1-48a2-88e3-d300e07781ea)
Hi! Thank you for your great work. But I have this issue while installing. So do you have other versions of `requirements.txt`? Or some other methods to fix this issue? Thanks!

Hi, @JesseZZZZZ , 
You can simply remove the version requirement and retry to install. I use torch==2.0.1 and didn't encounter this error. 

> Hi, @JesseZZZZZ , You can simply remove the version requirement and retry to install. I use torch==2.0.1 and didn't encounter this error.

THX! It worked!
Repository: open-compass/VLMEvalKit
Issue: There are some known issues with VQA tasks like OCRVQA, TextVQA, ChartQA, etc. We will fix them asap.
Root cause: Documentation Deficiency

请问这些问题修复了吗

Hi, @BlueBlueFF , 
They are fixed now. But notice that we use zero-shot evaluation for all those benchmarks, which may show some difference compared to few-shot evaluation. 

> Hi, @BlueBlueFF , They are fixed now. But notice that we use zero-shot evaluation for all those benchmarks, which may show some difference compared to few-shot evaluation.

How to get DocVQA Score? should i use "anls"? but i got very low prefermence(5.3)
Repository: open-compass/VLMEvalKit
Issue: Support Blink Dataset
Root cause: Unimplemented Feature Gap

一个更加关注 感知能力的数据集 Blink，非常纯粹的视觉感知benchmark。

https://zeyofu.github.io/blink/

<img width="457" alt="image" src="https://github.com/open-compass/VLMEvalKit/assets/18586273/2f9f21ec-808d-4d6c-aae5-ab018f2b27b5">

可以看出，GPT4V与人类水平有比较大的差距


Hi, @Ezra-Yu , thanks for your suggestion and we will have a review on this dataset.

VLMEvalkit has supported BLINK.
Repository: open-compass/VLMEvalKit
Issue: POPE evaluation
Root cause: Unimplemented Feature Gap

Will VLMEvalKit support the POPE benchmark in the near future?

Yes, we will support it in the next week. By default, we will adopt the adversarial track as our default one. 

Hi, @phquang , 
The multi-modal benchmark POPE has been supported and merged into VLMEvalKit.