zhiminy/EvalEng huggingface dataset:
Repository: open-compass/VLMEvalKit
Issue: DocVQA_TEST和InfoVQA_TEST无法评测
Root cause: Validation Gap

你好！我在评测DocVQA_TEST和InfoVQA_TEST时模型生成了prediction，但是会报这个错误，我打开tsv文件也没有answer，请问你是怎么进行评测的？
File "/share/project/open-compass/VLMEvalKit/vlmeval/dataset/image_vqa.py", line 46, in evaluate
    assert 'answer' in data and 'prediction' in data
AssertionError


Hi, @helloworld01001 ,

对于你提到的这两个数据集，由于数据文件中不包含 GT answer，因此代码库目前仅支持推理，不支持输出评测精度。推理文件应输出在 {model_name}_{dataset_name}.xlsx 中
Repository: open-compass/VLMEvalKit
Issue: 无法评测llava1.5-7b
Root cause: Environment Incompatibility

transformers 版本是 4.37， 下载llava1.5-7b权重至本地后vlmeval/config.py 中llava-v1.5-7b的模型地址也做了修改，第一次run会正常下载数据集，但没有正常运行，也没有报错，直接结束了，请问有什么解决方法吗？
![image](https://github.com/user-attachments/assets/48369f79-de4c-4b09-b4eb-34e4d2378267)
![1844546e868c9c73ce1b5caa25167ee](https://github.com/user-attachments/assets/66b57c5d-4fe9-479f-b6c3-67ca8345fbc3)


您好！可能是您没有正确安装llava所引起下列报错导致退出，您可以安装后再次进行尝试。
![image](https://github.com/user-attachments/assets/9895521a-e523-4a3b-85d7-27520e3d60d4)



![微信图片_20241014101911](https://github.com/user-attachments/assets/4ca5f168-e154-4540-9450-be292cebf423)
我加载模型的时候很慢很慢 并且运行报错是为什么呀


这个提示是正常的，您加载完模型推理过程中，有出现其他报错吗
![image](https://github.com/user-attachments/assets/5f8536f1-8861-4776-ba89-8f77aad36fea)


Feel free to re-open this issue for any further need.
Repository: open-compass/VLMEvalKit
Issue: XComposer cannot evaluate on Bench1_TEST
Root cause: Documentation Deficiency

Hello my command is as follows:
python run.py --data Q-Bench1_TEST --model XComposer --verbose

But the result is comfusing:
<img width="794" alt="截屏2024-10-09 14 23 30" src="https://github.com/user-attachments/assets/a488acd6-494b-4622-8c5e-8e1964d1fa95">


Hi, @yansuoyuli , 
Sorry for the confusion, Q-Bench1_TEST is a TEST split that does not include the ground-truth answer. I have updated the codes to add warning messages and information in the README.

BTW, here is the evaluation results of XComposer on Q-Bench1_VAL: 

![image](https://github.com/user-attachments/assets/b856004d-5a4d-4c34-95d0-368f3601e3a8)


How about Q-bench1_VAL？



---Original---
From: "Haodong ***@***.***&gt;
Date: Wed, Oct 9, 2024 16:27 PM
To: ***@***.***&gt;;
Cc: ***@***.******@***.***&gt;;
Subject: Re: [open-compass/VLMEvalKit] XComposer cannot evaluate onBench1_TEST (Issue #507)




 
Hi, @yansuoyuli ,
 Sorry for the confusion, Q-Bench1_TEST is a TEST split that does not include the ground-truth answer. I have updated the codes to add warning messages and information in the README.
 
BTW, here is the evaluation results of XComposer on Q-Bench1_VAL:
 
image.png (view on web)
 
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.Message ID: ***@***.***&gt;

> How about Q-bench1_VAL？
> […](#)
> ---Original--- From: "Haodong ***@***.***&gt; Date: Wed, Oct 9, 2024 16:27 PM To: ***@***.***&gt;; Cc: ***@***.******@***.***&gt;; Subject: Re: [open-compass/VLMEvalKit] XComposer cannot evaluate onBench1_TEST (Issue #507) Hi, @yansuoyuli , Sorry for the confusion, Q-Bench1_TEST is a TEST split that does not include the ground-truth answer. I have updated the codes to add warning messages and information in the README. BTW, here is the evaluation results of XComposer on Q-Bench1_VAL: image.png (view on web) — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***&gt;

Q-Bench1_VAL is OK and I have pasted the results.

Hello, thanks for your answers. I want to further ask about the meaning of the metrics:
type_0_concern_0  type_0_concern_1  type_0_concern_2  type_0_concern_3  type_1_concern_0  type_1_concern_1  type_1_concern_2  type_1_concern_3  type_2_concern_0  type_2_concern_1  type_2_concern_2  type_2_concern_3
Repository: open-compass/VLMEvalKit
Issue: 评测LLaVa系列模型时有bug
Root cause: Environment Incompatibility

您好！我在评测LLava系列模型时，会出现运行后，自动退出的情况，有时候不退出，有时候退出，不知道是什么原因

您好！可能是您没有正确安装llava触发了该错误所导致的退出，您可以在确认安装后再次进行尝试
![image](https://github.com/user-attachments/assets/c2951e13-8f56-4d40-b00b-8570a83eff26)


> 您好！可能是您没有正确安装llava触发了该错误所导致的退出，您可以在确认安装后再次进行尝试 ![image](https://private-user-images.githubusercontent.com/81462011/375667588-c2951e13-8f56-4d40-b00b-8570a83eff26.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3Mjg3MDA5NDQsIm5iZiI6MTcyODcwMDY0NCwicGF0aCI6Ii84MTQ2MjAxMS8zNzU2Njc1ODgtYzI5NTFlMTMtOGY1Ni00ZDQwLWIwMGItODU3MGE4M2VmZjI2LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDEwMTIlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQxMDEyVDAyMzcyNFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTRkYzAyZmMyZGE4Njc2MDViZTAxOTU0NWFhOWVmNzA4ZTBhYjcwMGYwYjczYWY0MGJiNTg3ZGY1OWJhMjZmMTAmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0._ny_CSPoMpVv2nM7JqH0uzQ7TyX0Xq6eIMGUjDHAckw)

您好！我check了代码，发现是load_pretrained_model函数会跳出
![image](https://github.com/user-attachments/assets/91c28921-665a-4aab-a40f-4b8435ff1fbc)

这个函数是llava内部自带的，您可以尝试在debug时把justMyCode设为false，进入函数内部查看具体报错

找到错误了，因为网络问题，llava文件下的函数报错了，挂VPN就解决了。
Repository: open-compass/VLMEvalKit
Issue: About evaluating existing result file (xlsx for MMBench)
Root cause: Documentation Deficiency

Hello, I have locally generated the result file for **_LLaVA-1.5-13B_** on _MMBench-Dev-EN_ and obtained `llava-v1.5-13b.xlsx`. I would like to evaluate this result file directly. However, it seems that currently, only evaluation from scratch after generating the result file is supported as introduced in _Quick Start_. Could you please tell me if the current repository supports this feature?

Hi,  @ligeng0197 , 
Would you please paste the file here so I can check what's wrong? 
Besides, the name of the result file should be llava-v1.5-13b_MMBench_DEV_EN.xlsx, if you didn't explicitly specify it. 

Thank you very much for your prompt response! I apologize if my explanation was unclear. The `llava-v1.5-13b.xlsx` file was generated using the original _LLaVA 1.5 code_. However, I would like to use VLMEvalKit locally to directly calculate the MMBench-dev metrics for the generated xlsx file, but I am not sure if this functionality feature is currently supported.

> Hi, @ligeng0197 , Would you please paste the file here so I can check what's wrong? Besides, the name of the result file should be llava-v1.5-13b_MMBench_DEV_EN.xlsx, if you didn't explicitly specify it.

[llava-v1.5-13b.xlsx](https://github.com/user-attachments/files/17338203/llava-v1.5-13b.xlsx)
The generated file for reference.

Hi, @ligeng0197 , 
You have to log in to use our evaluation service. 
The approach to evaluate MMBench DEV set w. VLMEvalKit has been sent to you via email.
Repository: open-compass/VLMEvalKit
Issue: Integration of MuirBench
Root cause: Documentation Deficiency

Hi everyone, first of all the framework is great 😊

I had seen that you want to add [MuirBench](https://muirbench.github.io/) to the framework. The test data set was published on [Hugging Face](https://huggingface.co/datasets/MUIRBENCH/MUIRBENCH) with the corresponding answers. So I wanted to ask if an integration of the benchmark is planned?

Hi, @nicokossmann , 
MUIRBench is already supported in VLMEvalKit (in PR #383 )
Repository: open-compass/VLMEvalKit
Issue: Support for Falcon2 VLM
Root cause: Unimplemented Feature Gap

I would like to get the evaluation reproduced for Falcon2-VLM-11B model. Can you add the support for that model as part of the vlm class and config.py?

Thanks

Hi, @Venkatesh-Balavadani , 
We have added it to our todo list and are working on it. 

Added in #556
Repository: open-compass/VLMEvalKit
Issue: Are images compressed in tsv files?
Root cause: Documentation Deficiency

Thanks for your contribution to providing a collection of VLM datasets and models. I'm wondering why the tsv versions of datasets in this repository are smaller than the official versions. For example, the RealworldQA dataset downloaded from the official website [RealWorldQA](https://data.x.ai/realworldqa.zip) has 677MB, while the tsv version in this repo [RealWorldQA_tsv](https://opencompass.openxlab.space/utils/VLMEval/RealWorldQA.tsv) only has 175MB. You are using base64 to encode images into texts and store them directly in tsv columns, which should be lossless. So why has the data size been reduced significantly? It seems that other datasets are having the similar situation.

Hello, @ywwynm 
The images in the dataset on the official RealWorldQA website are in webp format, whereas when we converted the original dataset to tsv format, we uniformly converted it to JPEG format during encode, you can refer to the [code](https://github.com/open-compass/VLMEvalKit/blob/9871435fd08e6c0e1e5a3bccb4150e641bc1e4ce/vlmeval/smp/vlm.py#L82-L93) here in our repo.

@SYuan03 Thanks for your explanation. For other datasets like SeedBench or MMTBench, is such processing also performed? If the original images have already been in JPEG format, will you compress it again using the same code?

Hello, @ywwynm 
In fact, if the original image is in jpeg format, there will not be such a significant change in data size even after our processing. We just convert it to the tsv format we need for the convenience of unified processing.
Repository: open-compass/VLMEvalKit
Issue: Is A-OKVQA support discontinued?
Root cause: Documentation Deficiency

A while back A-OKVQA was supported, and I did try that out. Now, I am not seeing that in the ImageMCQDataset class. 
https://github.com/open-compass/VLMEvalKit/blob/058f6733a82954fd4c8c72dfc22c6b3881c93638/vlmeval/dataset/image_mcq.py#L43

Hi @MOSHIIUR. Sorry for accidentally deleting the link to the file due to our mistake, we've added it here (https://github.com/open-compass/VLMEvalKit/blob/main/vlmeval/dataset/image_mcq.py#L94), thank you very much for reminding us!
Repository: open-compass/VLMEvalKit
Issue: none


Hi, @Wu-Zongyu , 
可以贴出不一致结果的示例吗？
我们在推理过程中，已有设定 temperature 为 0，但由于底层的随机性，仍可能在不同次的推理中有不同结果。但据观察，这点不太影响统计上的指标 (如数据集精度等)
Repository: open-compass/VLMEvalKit
Issue: Cant use POPE
Root cause: Environment Incompatibility

The following error comes out when I try to eval POPE dataset with official LLaVA-v1.5-7B model.
![image](https://github.com/user-attachments/assets/25029994-b7e9-4d23-b36c-9cb497e997f5)


The problem solved after redownloading the dataset.
Repository: open-compass/VLMEvalKit
Issue: Adding new dataset: worldmedQA-V to VLMEvalKit 
Root cause: Unimplemented Feature Gap

Hello everyone,

Thank you for open-sourcing such a great evaluation repo!

I’m Shan Chen, primarily working in the field of health AI at Harvard.

We’ve recently developed a multimodal evaluation dataset, cleaning up medical exam data from four countries (Japan, Israel, Spain, and Brazil) over several years.

This work is being submitted to NAACL 2025, and the format of our data is the VLMEvalKit MCQ format (identical to MMbench, etc.). We hope it can be integrated into the official repo. We can provide you guys the huggingface dataset link for smooth transformation. 

Thanks a lot!
Shan Chen  

Has replied via email. 

All the tsv datasets are here: https://huggingface.co/datasets/WorldMedQA/V/tree/main

Thanks for integrating WorldMedQA-V into VLMEvalkKit!

@shan23chen  PR #572 has been merged now.
Repository: open-compass/VLMEvalKit
Issue: How to eval with multigpu when using device_map='auto'
Root cause: Unimplemented Feature Gap

when use 34B LLM model, I can not put all the model to single gpu. so I use device_map='auto' to put several part to multi gpus.
but I found inference time cost too much, so how to solve this problem.

when I use 32 frames per video, It cost 100s for visual encoder.

I believe that implementing TP (tensor parallel) would be helpful to improve inference speed (currently we are using model parallel, which is slow). However, currently supporting TP is out of our scope. 

Another possible solution is that you can try to find if the corresponding model is supported by some deployment frameworks (vLLM, LMDeploy, etc.), if so you can first deploy the model and call it via API.
Repository: open-compass/VLMEvalKit
Issue: 评测InternVL2-8B出现OOM
Root cause: Configuration Error

python run.py --data MMMU_DEV_VAL --model InternVL2-8B --work-dir ./outputs  
使用8张显卡计算，每张显存是24GB


InternVL2-8B模型按照给定的split_model函数切分到8张卡上，跑MMMU_DEV_VAL数据集时会不定在某个case上报OOM错误，怀疑是显存被其他程序占用了导致网络中间的feature数据申请不到足够显存，如果是这样，VLMEvalKit有没有工具能够提前给feature预分配足够的显存避免OOM？

另：官方给出的多模态大模型评测结果，是在单卡显存80GB上跑出的吗？

![捕获](https://github.com/user-attachments/assets/034e548d-ded4-4f8c-a93f-90063b353894)



Hi, @AACengineer , 
PR #566 has supported this feature, now you can set the environment variable `AUTO_SPLIT=1`, and use command
`python run.py --model InternVL2-8B --data MMMU_DEV_VAL` to use multiple GPUs for one VLM. We will soon update the related doc. 

AUTO_SPLIT: When set to 1 and launched N processes on a node w. M GPUs, each VLM will be distributed on M / N GPUs.

Here is a screen shot when I evaluate InternVL2-8B x MMMU_DEV_VAL on two 4090 GPUs, it works. 

![image](https://github.com/user-attachments/assets/25ecb190-ae71-4024-98ba-158504be66fc)


@AACengineer Hi, has your question been solved?
Repository: open-compass/VLMEvalKit
Issue: AI2D数据集无法加载
Root cause: Documentation Deficiency

我在使用SOLO-7B模型进行evolution时无法加载AI2D数据集
![微信图片_20241023000148](https://github.com/user-attachments/assets/0c0080f3-8c44-45ad-9dc7-4c0f7376e1fa)


Hi, @zyy98123 , 
You should use AI2D_TEST or AI2D_TEST_NO_MASK (content in corresponding boxes not masked when choices are capital letters).
Repository: open-compass/VLMEvalKit
Issue: 测试Video-MME_CN

测试Video-MME_CN是否可行？
有建议怎么修改代码吗？比如是不是要把prompt改了

感谢！


Hi @jun0yayay, 想问一下您指的是哪个数据集呢，Video-MME好像暂未有cn版本

> Hi @jun0yayay, 想问一下您指的是哪个数据集呢，Video-MME好像暂未有cn版本

我方了没有是吗，那应该是我们自己弄的.............。

那是不是您觉得改改prompt应该就可以了？把原来英文的变成中文prompt

感谢！

> Hi @jun0yayay, 想问一下您指的是哪个数据集呢，Video-MME好像暂未有cn版本

您好不知道您有建议的修改路线吗？

Seems the benchmark is not an officially supported one. You can DM @FangXinyu-0913 for additional advice.
Repository: open-compass/VLMEvalKit
Issue: dataset_config.py在哪里
Root cause: Documentation Deficiency

https://github.com/open-compass/VLMEvalKit/blob/main/docs/zh-CN/get_started/Quickstart.md#:~:text=%2D%2Ddata%20(list%5Bstr%5D)%3A%20%E8%AE%BE%E7%BD%AE%E5%9C%A8%20VLMEvalKit%20%E4%B8%AD%E6%94%AF%E6%8C%81%E7%9A%84%E6%95%B0%E6%8D%AE%E9%9B%86%E5%90%8D%E7%A7%B0%EF%BC%88%E5%9C%A8%20vlmeval/utils/dataset_config.py%20%E4%B8%AD%E5%AE%9A%E4%B9%89%EF%BC%89
vlmeval/utils文件夹下面没有dataset_config.py

https://github.com/open-compass/VLMEvalKit/issues/402


The README has also been updated accordingly.
Repository: open-compass/VLMEvalKit
Issue: qwen2-vl-7b设置device_map为auto时报错
Root cause: Interface Contract Mismatch

和之前的一些issue一样，使用V100 32G单卡推理时某些case会OOM，所以开了device_map希望把模型切分到多张卡上，但是用python和torchrun启动都会报错Expected all tensors to be on the same device, but found at least two devices, cuda:1 and cuda:0!，debug看了一下，具体问题出现在qwen2vl forward时，input_ids本来是在cuda:0上，但是embedding层被放在了cuda：1上，所以经过embeding后inputs_embeds的device就变成了cuda：1，而后续的一些其他变量仍然在cuda：0上。（transformers/models/qwen2_vl/modeling_qwen2_vl.py的1687-1695行）

想知道有没有什么简单的办法可以解决多卡推理一个模型的问题，实在是没有80g的卡

Hi, @wwwbq , 
#562 这个 PR 应该可以解决你的问题，逻辑是：
当 world_size 为 1 (使用 python 启动) 及显存大小小于 40 GB 时，使用 “auto" 方式加载模型 (平均显存加载到所有卡上)。以下截图是利用 2 张 4090 测试 Qwen2-VL-7B 时的显存占用。

Command: `python run.py ---model Qwen2-VL-7B-Instruct --data $YOUR_DATASET`

![image](https://github.com/user-attachments/assets/bcbbc2cf-2767-41ed-aa52-3372d235cfa7)


对于其他模型，你可以照类似方式支持，或是提出需求

Now there are some changes: 

Please set the environment variable `AUTO_SPLIT=1` if you want to distributed a model on multiple GPUs. Currently, the feature supported Qwen2-VL, Llava-OneVision, Llama-3.2, and InternVL2. 

If you need further VLMs to support this feature, you can comment here or create a PR follow #566 

@wwwbq @kennymckormick "qwen2vl forward时，input_ids本来是在cuda:0上，但是embedding层被放在了cuda：1上，所以经过embeding后inputs_embeds的device就变成了cuda：1，而后续的一些其他变量仍然在cuda：0上" 这个问题现在多卡还是存在的，请问是怎么解决的？
Repository: open-compass/VLMEvalKit
Issue: gpt api_base不支持非develop模式
Root cause: Validation Gap

![image](https://github.com/user-attachments/assets/0b98f912-9c35-4659-a2bf-b9a42e082883)
gpt api如果没给deployment_name值会直接assert报错，但实际上有些api是非develop的，可参考azure内部实现
![image](https://github.com/user-attachments/assets/7f342b29-f029-488f-b2f9-799724c3b2ce)


Hi，@huakyouin ，
这个是否方便提个 PR 修改一下？ Azure API 的 feature 本来是社区用户支持的，我们由于没有 Azure API  所以不好 DEBUG


Feel free to re-open the issue for any further updates. 

any update on how to use azure api, any steps to do that? Thanks very much
Repository: open-compass/VLMEvalKit
Issue: What is this code doing?

What is this code doing?????

https://github.com/open-compass/VLMEvalKit/blame/490bb4e0fea0381eeeb43b75d289e7c6c2531e02/vlmeval/vlm/internvl_chat.py#L105

Some times you need to run N procs on M GPUs (2 procs on 8 GPUs, etc.), 
the script will get the number of GPUs used by each proc under this circumstance.

啊啊啊啊啊啊啊啊啊啊啊啊啊啊啊！
Repository: open-compass/VLMEvalKit
Issue: MMBench这种选择题，是否可以考虑对prediction进行清洗。
Root cause: Documentation Deficiency

<img width="561" alt="image" src="https://github.com/user-attachments/assets/0dc2e466-61a6-4591-8bca-b9309ac0d1ad">

模型已经预测了“选项B”，却算作错误。如果对prediction进行清洗，或者更改下评测代码，“选项B”也应该算作正确答案。

Hi, @xushilin1 您好
事实上我们已经进行了“清洗”，您看到的这条数据被认为hit=0应该是因为MMBench采用了CircularEval Strategy。
即每次都对选项和答案进行循环移位，以生成新的提示供VLM使用。只有当VLM在所有循环传递中正确预测答案时，它才被认为成功解决了该问题。具体您可查看[MMBench](https://arxiv.org/pdf/2307.06281)的Section 4.3内容。
Repository: open-compass/VLMEvalKit
Issue: The result of gpt4o on MathVista is 61.6, which is lower than the 62.7 on the list.
Root cause: External Dependency Breakage

After reproducing the gpt-4o on mathvista，we found the result of gpt4o on MathVista is 61.6, which is lower than the 62.7 on the list. Is there something i missed to get that scores. We have also test the gpt-4o with the prompt ending with 'solution:‘ and got a score at 65.3%。 

Hi, @kkk123698745 , 
I will rerun the evaluation and post the results here (fact check: are you using gpt-4o-20240806?). By the way, we are happy to update the evaluation result if you can create a PR to integrate ur better evaluation setting. 

@kennymckormick  Thanks for your reply，
Expecting your result of rerun. Yeah, we used gpt-4o-20240806. In deed，we just run your code and got a score at 61.6% without any modification on prompt and got a score at 65.3% with the prompt adding '\nsolution:' at the tail.The full prompt like this:'Hint: Please answer the question and provide the correct option letter, e.g., A, B, C, D, at the end.
Question: Is the number of purple metallic things that are behind the small green motorbike less than the number of blue metal articulated buss?
Choices:
(A) Yes
(B) No
solution:' 

Hi, @kkk123698745 , 
I have performed two checks:
1. Reperform the evaluation (get accuracy based on old prediction): we find the accuracy is almost the same. We got 62.6% accuracy while our reported one is 62.7%. You can reproduce the results w. our released prediction files here: https://huggingface.co/datasets/VLMEval/OpenVLMRecords. 

Screenshot:

![image](https://github.com/user-attachments/assets/c39ff03e-1434-4e8e-8f9b-22c7f6a8ac98)

2. Re-conduct the entire evaluation process: At the time there is larger difference. I only got 60.9% accuracy, which is about 2% lower than our reported results.

Screenshot: 

![image](https://github.com/user-attachments/assets/e530ba33-4a2e-4d23-bf73-099df0d8f322)

I think the most possible reason for such difference is that, since GPT-4o is a proprietary model (a black box for us), maybe OpenAI  has made some modifications to the model behind so the results now are different. For API models, our evaluation results only stands for the performance at the corresponding timestamp.
Repository: open-compass/VLMEvalKit
Issue: LLaVA v1.5 7b very low performance on TextVQA
Root cause: Configuration Error

Dear authors,

Thanks for sharing this great codebase.

I tried to test LLaVA-v1.5-7b (`llava_v1.5_7b`) model on `TextVQA_VAL` and only get about 21.88 accuracy, which is much lower than the 58.2 performance reported in the paper. I understand that the codebase is not aimed to reproduce the exact results but it seems that the gap is too big.

Is there any reason why this happens and any quick fix?

Thanks very much!

Same to you, did you solve it?

> Same to you, did you solve it?

No I haven't figured it out. 🥲

I've also got the same problem.
Repository: open-compass/VLMEvalKit
Issue: [Bug] RuntimeError: world_size (8) is not equal to tensor_model_parallel_size (1) x pipeline_model_parallel_size (1) 
Root cause: Configuration Error

I used the interface from the vllm repository (https://github.com/vllm-project/vllm) to load the model and ran 
```bash
torchrun --nproc-per-node=8 run.py --data Video-MME --model Qwen2_VL-M-RoPE-80k
```
 for evaluation, but I got the error 
```bash
RuntimeError: world_size (8) is not equal to tensor_model_parallel_size (1) x pipeline_model_parallel_size (1). 
```
Could you please advise on how to resolve this?


Here is the interface
```
from vllm import LLM
llm = LLM("/mnt/hwfile/mllm/weixilin/cache/Qwen2-VL-7B-Instruct", 
            max_model_len=100000,
            limit_mm_per_prompt={"video": 10},
            )
```

https://github.com/vllm-project/vllm/blob/3ea2dc2ec49d1ddd7875045e2397ae76a8f50b38/vllm/distributed/parallel_state.py#L1025
Seems that the error occur at this assertion, so how can I modify my vlmeval program to fit the assertion, thanks.

Looks like a problem related to vLLM. 
I think you can first launch VLM models as API services and then perform the evaluation via API calling. It's a better practice to avoid problems been coupled.
Repository: open-compass/VLMEvalKit
Issue: 关于结果保存
Root cause: Documentation Deficiency

存储的时候能不能重新跑继续用之前跑过的存储结果呀？这个代码会重新来吗，因为有时候跑到半路会死，每次都要重新跑前面的实例吗？

The intermediate results will be automatically saved and reused.
Repository: open-compass/VLMEvalKit
Issue: Error when evaluating MMBench_TEST_EN
Root cause: Validation Gap

Looks like the annotation in MMBench_TEST_EN does not contains the key "answer"? Yet the prefetch_acc has to access this key, as shown in https://github.com/open-compass/VLMEvalKit/blob/7f099c1f4b3bc99d9607651ce4f590bcf889e709/vlmeval/inference.py#L156

Because The test_file have no key "`answer`", I think they do not add the answer of the choise in the file `MMBENCH_TEST_EN.tsv`.

Hi, @KosumosuL , 
The label of testing samples is confidential, to obtain the evaluation accuracy of MMBench test split, please upload it to the evaluation service of send it to opencompass@pjlab.org.cn 

> Hi, @KosumosuL , The label of testing samples is confidential, to obtain the evaluation accuracy of MMBench test split, please upload it to the evaluation service of send it to [opencompass@pjlab.org.cn](mailto:opencompass@pjlab.org.cn)

I can understand that. My point is, however, the code should consider this situation for avoiding this error, or the readme should point this out.

> > Hi, @KosumosuL , The label of testing samples is confidential, to obtain the evaluation accuracy of MMBench test split, please upload it to the evaluation service of send it to [opencompass@pjlab.org.cn](mailto:opencompass@pjlab.org.cn)
> 
> I can understand that. My point is, however, the code should consider this situation for avoiding this error, or the readme should point this out.

Hi, @KosumosuL , that's a good point, I have fixed the problem in this commit: https://github.com/open-compass/VLMEvalKit/commit/e99204665a2cea7e6a006750c6526d820dacda7d, the evaluation skip for MMBench_TEST will be skipped on any non-official servers. You can try again on the latest main branch.