zhiminy/EvalEng huggingface dataset:
Repository: open-compass/VLMEvalKit
Issue: MiniCPM-V似乎不兼容文档中的测试代码
Root cause: Validation Gap

根据文档的测试代码：
'''
如何测试一个 VLM 是否可以正常运行:

from vlmeval.config import supported_VLM
model = supported_VLM['idefics_9b_instruct']()
# 前向单张图片
ret = model.generate(['assets/apple.jpg', 'What is in this image?'])
print(ret)  # 这张图片上有一个带叶子的红苹果
# 前向多张图片
ret = model.generate(['assets/apple.jpg', 'assets/apple.jpg', 'How many apples are there in the provided images? '])
print(ret)  # 提供的图片中有两个苹果
'''
我首先尝试了InternVL2-8B，没有问题；
但在使用MiniCPM-V测试时报错：

Traceback (most recent call last):
  File "/mnt/hwfile/chenlj22/VLMEvalKit/test.py", line 4, in <module>
    ret = model.generate(['assets/apple.jpg', 'What is in this image?'])
  File "/mnt/hwfile/chenlj22/VLMEvalKit/vlmeval/vlm/base.py", line 116, in generate
    return self.generate_inner(message, dataset)
  File "/mnt/hwfile/chenlj22/VLMEvalKit/vlmeval/vlm/minicpm_v.py", line 71, in generate_inner
    if DATASET_TYPE(dataset) == 'MCQ':
  File "/mnt/hwfile/chenlj22/VLMEvalKit/vlmeval/dataset/__init__.py", line 178, in DATASET_TYPE
    if 'openended' in dataset.lower():
AttributeError: 'NoneType' object has no attribute 'lower'

你好，感谢对VLMEvalkit的关注！

您的问题在于，MiniCPM中的[generate_inner](https://github.com/open-compass/VLMEvalKit/blob/889822d94ec97ccad900a5bf3321be8baec262ac/vlmeval/vlm/minicpm_v.py#L71)函数针对不同类型的Benchmark改变了max_new_tokens。由于您在直接使用generate的过程中没有使用dataset，所以导致`dataset=None`。

为了解决这个问题，您可以直接将[minicpm_v.py](https://github.com/open-compass/VLMEvalKit/blob/main/vlmeval/vlm/minicpm_v.py#L71)中的以下代码：
```
if DATASET_TYPE(dataset) == 'MCQ':
    max_new_tokens = 20
elif DATASET_TYPE(dataset) == 'Y/N':
    max_new_tokens = 100
else:
    max_new_tokens = 1024
```
更改为：
`max_new_tokens = 1024`
这样只会更改模型的输出长度，并不会影响评测结果。

如果您有问题，欢迎继续提问。

了解了，感谢您的回复！







------------------ 原始邮件 ------------------
发件人: Xiangyu Zhao ***@***.***&gt;
发送时间: 2025年2月17日 18:16
收件人: open-compass/VLMEvalKit ***@***.***&gt;
抄送: chenlj22 ***@***.***&gt;, Author ***@***.***&gt;
主题: Re: [open-compass/VLMEvalKit] MiniCPM-V似乎不兼容文档中的测试代码 (Issue #784)
Repository: open-compass/VLMEvalKit
Issue: Could you support custom system prompt?
Root cause: Unimplemented Feature Gap

Is there support for adding custom system prompt like R1?

```python
SYSTEM_PROMPT = (
    "A conversation between User and Assistant. The user asks a question, and the Assistant solves it. The assistant "
    "first thinks about the reasoning process in the mind and then provides the user with the answer. The reasoning "
    "process and answer are enclosed within <think> </think> and <answer> </answer> tags, respectively, i.e., "
    "<think> reasoning process here </think><answer> answer here </answer> {Question}"
)
```
Repository: open-compass/VLMEvalKit
Issue: 使用qwen2.5_VL fine-tune后评测出现错误，是怎么回事
Root cause: Validation Gap

You are using a model of type qwen2_5_vl to instantiate a model of type qwen2_vl. This is not supported for all configurations of models and can yield errors.
ERROR - RUN - run.py: main - 432: Model Qwen2.5-VL-7B-Instruct-SFT x Dataset MME combination failed: No such file or directory:  xxxx

您好，请问有完整的报错日志吗？

```
FileNotFoundError: [Errno 2] No such file or directory: 'workspace/qwen25vl_3b_sft/T20250402_G1774d8cd/qwen25vl_3b_sft_DocVQA.xlsx'
```

combination  没有成功，为啥还是会读这个 `xxx_DocVQA.xlsx ` 文件呢？这里应该是读 `xxx_DocVQA_TEST.xlsx`。

麻烦帮忙看下。

你好 请问解决了吗

同样的问题

路径问题，自己的weights需要包含Qwen2.5-VL-7B-Instruct

即把你的weights放在 .../Qwen2.5-VL-7B-Instruct/内
Repository: open-compass/VLMEvalKit
Issue: Janus-pro-7B 精度

我测出来它MME-p是1520, 官方report的值是1567, 这个差距是合理的吗

Hello, 

Thank you for your patience and for raising this issue. A 40-point difference in the MME benchmark is relatively reasonable and could be influenced by the versions of your environment dependencies, such as `transformers`, `torch`, and `cuda`. To ensure consistency, we recommend updating your environment to match the official setup used for Janus-pro-7B. Once updated, please re-run the tests.

Best regards,
Repository: open-compass/VLMEvalKit
Issue: Lmdeploy cannot be used to inference and eval
Root cause: Documentation Deficiency

I carefully read the instructions on the LMDeploy for inference and evaluation page (https://github.com/open-compass/VLMEvalKit/blob/main/docs/zh-CN/EvalByLMDeploy.md). However, when I actually execute the evaluation command, it still loads the local model. It seems that there is no proper communication established between the server and the evaluation in run.py. Are there any additional steps I should take? Thank you.

Hi,  

Thank you for your interest in VLMEvalKit!  

Could you please share the execution code you used? This will help us better understand the context and provide more accurate guidance.  

If you'd like to use LMDeploy for evaluation, please refer to the model configuration in [config.py](https://github.com/open-compass/VLMEvalKit/blob/2d8e44e465bb89ff55ffaf54f989449d96c32af5/vlmeval/config.py#L115) and update the IP address to match your LMDeploy API endpoint. 

Additionally, you can review the [LMDeploy class](https://github.com/open-compass/VLMEvalKit/blob/2d8e44e465bb89ff55ffaf54f989449d96c32af5/vlmeval/api/lmdeploy.py#L308) code to explore any further customization options or requirements.  

If you encounter any issues or have additional questions, feel free to reach out—we’re happy to assist!  

Best regards,

> I carefully read the instructions on the LMDeploy for inference and evaluation page (https://github.com/open-compass/VLMEvalKit/blob/main/docs/zh-CN/EvalByLMDeploy.md). However, when I actually execute the evaluation command, it still loads the local model. It seems that there is no proper communication established between the server and the evaluation in run.py. Are there any additional steps I should take? Thank you.

maybe you should use this command "python run.py --data xxx --model lmdeploy --verbose --api-nproc 64"
Repository: open-compass/VLMEvalKit
Issue: 评估qwen2.5-vl模型时报错：ValueError: Trying to set a tensor of shape torch.Size([2048]) in "bias" (which has shape torch.Size([1280])), this looks incorrect.
Root cause: Validation Gap

我git了最新的[VLMEvalKit](https://github.com/open-compass/VLMEvalKit)代码
torch                    2.5.1+cu124
torchaudio               2.5.1+cu124
torchvision              0.20.1+cu124
tqdm                     4.67.1
transformers             4.49.0
triton                   3.1.0
typer                    0.15.1
typing_extensions        4.12.2
tzdata                   2025.1
urllib3                  2.3.0
uvicorn                  0.34.0
validators               0.34.0
vlmeval                  0.1.0        /mnt/general/ganchun/code/VLMEvalKit

仍然报错，qwen2.5-vl我使用官方的原生模型不会报错，但是我自己微调了之后，传的是checkpoint路径，就会报这个错误：ValueError: Trying to set a tensor of shape torch.Size([2048]) in "bias" (which has shape torch.Size([1280])), this looks incorrect.

> 我git了最新的[VLMEvalKit](https://github.com/open-compass/VLMEvalKit)代码 torch 2.5.1+cu124 torchaudio 2.5.1+cu124 torchvision 0.20.1+cu124 tqdm 4.67.1 transformers 4.49.0 triton 3.1.0 typer 0.15.1 typing_extensions 4.12.2 tzdata 2025.1 urllib3 2.3.0 uvicorn 0.34.0 validators 0.34.0 vlmeval 0.1.0 /mnt/general/ganchun/code/VLMEvalKit
> 
> 仍然报错，qwen2.5-vl我使用官方的原生模型不会报错，但是我自己微调了之后，传的是checkpoint路径，就会报这个错误：ValueError: Trying to set a tensor of shape torch.Size([2048]) in "bias" (which has shape torch.Size([1280])), this looks incorrect.

已解决！

您好，麻烦请问一下您是怎么解决的呢？



> 您好，麻烦请问一下您是怎么解决的呢？

在这个VLMEvalKit/vlmeval/vlm/qwen2_vl/model.py中100行左右：
if '2.5' in model_path or **'2_5' in model_path**: # 添加上加粗的部分，这里是通过路径来判断是2.5还是2.0的vl模型的
    from transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor
    MODEL_CLS = Qwen2_5_VLForConditionalGeneration
    self.processor = AutoProcessor.from_pretrained(model_path)
else:
    from transformers import Qwen2VLForConditionalGeneration, Qwen2VLProcessor
    MODEL_CLS = Qwen2VLForConditionalGeneration
    self.processor = Qwen2VLProcessor.from_pretrained(model_path)

you are a legend
Repository: open-compass/VLMEvalKit
Issue: 测评结果文件信息矛盾
Root cause: Documentation Deficiency

![Image](https://github.com/user-attachments/assets/45fb22ac-6ef1-4d6f-8d3b-cfd9cc9f7787)
- answer、prediction、log三栏的数据矛盾
- 下面这几行（红色框框）exact_match失败
谢谢！

您好，方便提供这个结果文件吗？这可能有助于我们更快速为您解决问题。

> 您好，方便提供这个结果文件吗？这可能有助于我们更快速为您解决问题。

好的，谢谢！

[Qwen2-VL-7B-Instruct_MMBench_DEV_EN_openai_result.xlsx](https://github.com/user-attachments/files/18963016/Qwen2-VL-7B-Instruct_MMBench_DEV_EN_openai_result.xlsx)

您好，您遇到的这个问题是因为MMBench在评估的时候采用了`circular eval`的方式，即将一个多选题的选项打乱顺序重复问，只有多次全部答对才视为答对。而在评估日志文件中，模型仅保存了第一次的`prediction`，在之后几次match的过程中，如果match失败，将输出当前的`prediction`到`log`中，因此会导致看到的`log`结果和保存的`prediction`中的不匹配。

感谢感谢！
Repository: open-compass/VLMEvalKit
Issue: 两次相同的测试结果，同一个评估模型，结果差异很大
Root cause: Validation Gap

![Image](https://github.com/user-attachments/assets/217c094e-cfd5-42d0-81ac-4907f58a2c3c)
上面两个结果是一致的，对过了，但是评估模型给的分数差异比较大，用的是MiniCPM-V-2_6，是评估模型哦能力不行吗？因为是内网用不了openai，推荐用哪个本地LLM

您好，   

感谢您的提问！   

从您提供的信息来看，这似乎是MMVet Benchmark。由于这项工作属于较早期的研究，当时使用的评估模型是 GPT-4-turbo。需要注意的是，不同的评估模型可能会给出差异较大的评分结果，因此我们也很难判断哪种模型的评分更加准确或优越。   

如果您希望与其他模型的结果保持一致，我们建议您使用默认的评估模型进行对比。如果您无法调用openai，我们建议您部署Qwen2.5系列等较强的语言模型。这样可以更好地确保结果的一致性和可比性。   

同一个评估模型，对同一个结果两次给出的评分差异也很大？这个问题怎么解决

您好，保持所有设置相同的情况下，评测之间的差异应该很小。您这两次的MMVet分数评测分别是多少呢？

分别是50.0和42.0，具体数值没有记录，看了下是因为评价模型针对同一个结果给出的评分差异比较大，参考上图。

您好，这种情况在我们的日常使用中很少出现，请问方便share一下gpt打分的记录文件(MMVet_gpt4o_turbo.xlsx)吗？

.xlsx文件导不出来，直接看图
![Image](https://github.com/user-attachments/assets/482cd465-6b56-4349-992c-3a2c2ce4b788)
两个答案一摸一样。

![Image](https://github.com/user-attachments/assets/53b88d9b-b465-4cc4-b01a-531c5b59000a)
评分模型一个给出了1，一个给出了0.5，导致两次最终评分为51.9、48，差异有点大，这个评分模型怎么消除随机性。


您使用的是VLMEvalkit中的GPT-4-turbo作为评估模型吗？

用的是本地lmdeploy部署的MiniCPM-v-2_6模型

不建议使用VLM作为评估模型，MiniCPM2.6本身能力不足，建议部署Qwen2.5-72B进行评测并设置`temperature=0`。
Repository: open-compass/VLMEvalKit
Issue: 78B模型推理加速（78B Modeling Reasoning Acceleration）
Root cause: Configuration Error

您好，我现在有八张H100想评测internvl2_5-78B. 目前是通过AUTO_SPLIT拆分模型到三张卡上python推理，请问可以在此基础上加快速度变成ddp嘛

Hello, I now have eight H100 want to review internvl2_5-78B. Currently through the AUTO_SPLIT split model to three cards on the python reasoning, please ask can be based on this to speed up to ddp?

您好，  


DDP（Distributed Data Parallel）策略通常用于模型训练时的并行化处理。对于 Internvl2.5-78B 模型，如果您希望进行并行推理，可以尝试使用`torchrun --nproc_per_node=2`来部署。
通过该命令，您可以同时部署两个模型。按照您之前提到的切分策略，每个模型实例会被分配到三张 GPU 卡上运行。因此，在这种情况下，总共会占用六张卡的资源。当然，具体的部署策略可以根据您机器的实际硬件资源上限进行灵活调整。  



那请问AOTU_SPLIT这个参数怎么设置呢，我在将原来的python换成torchrun --nproc_per_node=2会oom

我的sh脚本是这样的：
export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
export AUTO_SPLIT=1
 torchrun --nproc_per_node=4 run.py --data "$data" --model Qwen2.5-VL-72B-Instruct --verbose --mode infer --reuse，看起来只用了前四张卡，没有像python那样进行模型拆分

对于Qwen2.5-72B模型而言，具体的[split_model](https://github.com/open-compass/VLMEvalKit/blob/eb58e9e05d4f64316608128988c60a3cb200307d/vlmeval/vlm/qwen2_vl/model.py#L34)定义了模型的切分方法。对于72B模型，只用两张卡切分，很容易导致OOM问题。一般建议至少四张卡进行推理。

很抱歉，这样的代码也会报错
export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
export AUTO_SPLIT=1
torchrun --nproc_per_node=2 run.py --data "$data" --model Qwen2.5-VL-72B-Instruct --verbose --mode infer --reuse
AUTO_SPLIT的设置只能为1嘛

您的报错仍旧是OOM吗？那建议您使用`--nproc_per_node=1`试试

> 很抱歉，这样的代码也会报错 export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 export AUTO_SPLIT=1 torchrun --nproc_per_node=2 run.py --data "$data" --model Qwen2.5-VL-72B-Instruct --verbose --mode infer --reuse AUTO_SPLIT的设置只能为1嘛

我也试了  看起来只用了前两张卡，没有像python那样进行模型拆分
Repository: open-compass/VLMEvalKit
Issue: MMMU's evaluation does not display results
Root cause: External Dependency Breakage

I have successfully conducted several evaluations, but when MMMU evaluation has finished (100%), it has been stuck for hours and the results are not displayed
(By the way, I changed the vlmeval/config.py file and used a local InternVL2_5-8B model)

![Image](https://github.com/user-attachments/assets/df2c79b4-3395-4d0b-9798-769bfbee52f2)

We recommend rerunning the command, as the results should then become visible. 

But I have already run this two or three times, and it always works like this 



> We recommend rerunning the command, as the results should then become visible.



Changing the torch and transformers to a lower version will solve this problem.
The version which can't run correctly is transformers==4.46.0 torch==2.6.0
Repository: open-compass/VLMEvalKit
Issue: 关于llava-cot的ChartQA的评估问题
Root cause: Documentation Deficiency

请问llava-cot在评估ChartQA的时候，因为输出的是COT序列，但generate_inner返回的好像是模型的整个推理序列而不单是最终答案

那么请问在评估llava-cot的ChartQA时候，框架会在哪里抽取得到最终答案呢？
Repository: open-compass/VLMEvalKit
Issue: 关于使用 --config 的问题
Root cause: Documentation Deficiency

run.py中有一句：

> When launching with `--config`, args for API VLMs, such as `--retry`, `--verbose`, will be ignored.

请问，为什么在使用 --config 配置时，不能使用 api-nproc这些参数呢？基于什么样的考虑呢？
Repository: open-compass/VLMEvalKit
Issue: Unable to reproduce Qwen2.5-VL-7B-Instruct
Root cause: Environment Incompatibility

How nice a work [VLMEvalKit](https://github.com/open-compass/VLMEvalKit) it is!!!

I tested Qwen2.5-VL-7B-Instruct on mmstar and it was only 61.8, but it was 63.9 in the text. The environment version is as follows

`transformers==4.49.0
torch==2.6.0
flash_attn==2.7.4.post1`

![Image](https://github.com/user-attachments/assets/141dfd47-a07b-452a-8fe2-2529ae6083cb)

![Image](https://github.com/user-attachments/assets/23e1995f-2fff-4490-a2c5-9e98633bcf4d)

Please use `pip install git+https://github.com/huggingface/transformers accelerate`  to build transformers from source. We use latest transformers and `torch=2.5.1`.
Repository: open-compass/VLMEvalKit
Issue: Unable to reproduce Llama-3.2-11B-Vision-Instruct on MMstar
Root cause: Environment Incompatibility

Nice work!!!

Unable to reproduce Llama-3.2-11B-Vision-Instruct on MMstar.

Results when the hyperparameters are as follows 

{'do_sample': True, 'max_new_tokens': 512, 'temperature': 0.6, 'top_p': 0.9}
![Image](https://github.com/user-attachments/assets/3323d62e-6386-4f57-875a-a68bcb1974d4)


{'do_sample': False, 'max_new_tokens': 2048, 'temperature': 0.0, 'top_p': None, 'num_beams': 1}
![Image](https://github.com/user-attachments/assets/c7c20c68-4836-4eee-abe5-cfcc431a0e49)


kwargs_default = dict(do_sample=True, max_new_tokens=2048, temperature=0.6, top_p=0.9)
![Image](https://github.com/user-attachments/assets/30445895-b458-4fba-bd72-eef57d463f06)


kwargs_default = dict(do_sample=True, max_new_tokens=2048, temperature=0.1, top_p=0.7)
![Image](https://github.com/user-attachments/assets/4e113851-92a9-4fa8-8908-21e62df556b0)

We conduct evaluation again based on below environment:
```
torch=2.1.2
transformers=4.45.1
cuda=11.8
flash_attn=2.5.5
```
and the final results is `47.1`. 
The environment may cause significant effect on model's performance. Please change the environment and try again.

However, it is still far from the MMstar results, which is 49.8  on the [leaderboard](https://huggingface.co/spaces/opencompass/open_vlm_leaderboard)

![Image](https://github.com/user-attachments/assets/318bccd8-c95a-4c0c-9168-06a1b497e5f3)

It's worth noting that the model's performance can be significantly influenced by the environment. Therefore, if you're aiming for higher performance, we recommend experimenting with different environments to identify the most optimal setup.
Repository: open-compass/VLMEvalKit
Issue: 如何将阿里百炼 api 中的模型作为评判者或选择提取器？
Root cause: Unimplemented Feature Gap

好像只支持open-ai的gpt家族。
求各位大佬教教！
`CUDA_VISIBLE_DEVICES=2 py run.py --data MME --model llava_v1.5_7b --verbose --judge qwen-turbo`
类似这个命令不可用

您好，VLMEvalkit中的`build_judege`函数定义于[judge_util.py](https://github.com/open-compass/VLMEvalKit/blob/c350e6e29a7fa6dcd5dbb7f7404ee64797480cb0/vlmeval/dataset/utils/judge_util.py#L7)，目前支持了[硅基流动(SiliconFlow)](https://siliconflow.cn/)的api以及openai的api。

您可以仿照[SiliconFlowAPI](https://github.com/open-compass/VLMEvalKit/blob/c350e6e29a7fa6dcd5dbb7f7404ee64797480cb0/vlmeval/api/siliconflow.py#L40)构建一个基于阿里百炼的类，用于支持阿里百炼作为judge model。

感谢回复，确实需要自己实现~
Repository: open-compass/VLMEvalKit
Issue: 请问如何评测多模态模型（如Qwen-2.5-vl）在语言模型评测数据集（如MMLU）上的性能？
Root cause: Documentation Deficiency

能通过使用 Opencompass 自定义模型API的方式 或者 使用 VLMEvalKit 自定义数据集的方式实现吗

> 能通过使用 Opencompass 自定义模型API的方式 或者 使用 VLMEvalKit 自定义数据集的方式实现吗

用另一个文本评测库Opencompass
Repository: open-compass/VLMEvalKit
Issue: [BUG?] 使用同样的模型，在此 repo下跑推理与模型官方（例如 LLaVA）的推理 demo 结果不同
Root cause: Algorithmic Error

例如：
```
from vlmeval.config import supported_VLM
model = supported_VLM['llava_v1.5_7b']()
ret = model.generate([{'type': 'image', 'value': '${your_home}/LMUData/images/MME/945.jpg'}, {'type': 'text', 'value': 'This is a picture from a real scene. Is there only two real apples in this picture? Please answer yes or no.'}])
print(ret)
```
得出来输出为`1`！似乎看起来 instruction-following 能力很弱 : (

然而，使用 llava 官方提供的 CLI 测试结果输出为 Yes/No！！！

---
需要注意的是，如果把 `Please answer yes or no.` 前面仅有一个空格改为两个空格，在本 repo 代码下也能输出 Yes/No？？？
@kennymckormick 麻烦您帮忙看下是不是有 bug，测试的几个结果不好的似乎都是由于没有按照要求输出，也就是“仅在本 repo”出现的 instruction-following 弱的情况。
Repository: open-compass/VLMEvalKit
Issue: 是否支持采样评估部分数据集？
Root cause: Unimplemented Feature Gap

请问支持采样评估部分数据集么 全量跑评估时间太久了。

@zhangyimi 目前暂时不支持，您是想评测视频类数据集还是图像类数据集呢？

You can seek the solution via https://github.com/open-compass/VLMEvalKit/issues/840

The feature is currently not supported. If you wish to do so, you can create custom small-scale datasets (for example, create a smaller version of tsv file that only includes a subset of records) and add them to your own fork.
Repository: open-compass/VLMEvalKit
Issue: Ola模型自测与OpenCompass榜单差距很大
Root cause: Environment Incompatibility

基于VLMEvalKit评测，裁判模型gpt-4-turbo

![Image](https://github.com/user-attachments/assets/b8bda505-a9d5-4ae9-8c14-7a10db1fe067)

https://github.com/Ola-Omni/Ola/issues/14

@dongyh20 Could you please assist in resolving this question?

I will check this asap

We have reported the results on our own machine. The results have a slight difference with the official benchmark, but it is acceptable and it may caused by different machines or different envs. 
#https://github.com/Ola-Omni/Ola/issues/14
Repository: open-compass/VLMEvalKit
Issue: Failed to download MMBench_V11.tsv  error
Root cause: Validation Gap

Hi, 
I'm trying to run the evaluation on the published records (https://huggingface.co/datasets/VLMEval/OpenVLMRecords), and for the MMBench file (for example Claude3-5V_Sonnet_MMBench_V11.xlsx) I receive the following error?

Exception: Failed to download https://opencompass.openxlab.space/utils/benchmarks/MMBench/MMBench_V11.tsv 

I'm running  vlmutil eval "Claude3-5V_Sonnet_MMBench_V11.xlsx" for example.

Thanks!

Hi,  

Thank you for your interest in VLMEvalKit!  

Please note that **MMBench_V11.tsv** is not publicly available. If you'd like to evaluate **MMBenchV11_TEST_EN**, you can use the following command:  

```bash
python run.py --model xx --data MMBench_TEST_EN_V11
```

After generating the results, please submit them to the [MMBenchOnline platform](https://mmbench.opencompass.org.cn/mmbench-submission) to obtain the evaluation score.  

If you have any further questions or need additional assistance, feel free to reach out!  

Best regards,  


Thanks for the reply. I already have the predictions from your evaluation records data. I'm only looking to evaluate the response. 

Per your instruction. I"m running 

 vlmutil eval "Claude3-5V_Sonnet_MMBench_V11.xlsx" for example.
There are also MMBench.xlsx files in the records. But they don't work as well
Repository: open-compass/VLMEvalKit
Issue: run.py不支持传入本地模型
Root cause: Configuration Error

模型在本地路径，并且export MMEVAL_ROOT=/opt/test, 运行： python run.py --data MMBench_V11_MINI --model /opt/Qwen/Qwen2-VL-7B-Instruct
会提示PermissionError: [Errno 13] Permission denied: '/opt/Qwen/Qwen2-VL-7B-Instruct/T20250305_G'

期望路径为/opt/test下
另外，当没有指定MMEVAL_ROOT时，期望写入路径不在模型路径下

如果你想更改模型路径，请在[config.py](https://github.com/open-compass/VLMEvalKit/blob/eda4a6296c008cce586bfa0f934017b5269bc35b/vlmeval/config.py#L406)中将`model_path`更改为你的路径，如`/opt/Qwen/Qwen2-VL-7B-Instruct`。
Repository: open-compass/VLMEvalKit
Issue: 无法支持云端模型api？（比如硅基流动）
Root cause: Validation Gap

看到好几个地方out_file = f'{work_dir}/{model_name}_{dataset_name}_supp.pkl'
问题是调用硅基流动的模型比如deepseek-ai/deepseek-vl2，model_name里面就有/。
out_file就是outdir/deepseek-ai/deepseek-vl2/deepseek-ai/deepseek-vl2_supp.pkl，路径不存在，然后就报错

Please show the complete error log.

> Please show the complete error log


定位出来就是work_dir={output}/{model_name}，会自动创建work_dir，然后在work_dir创建{model_name}_{dataset_name}_supp.pkl，但是如果model_name中带斜杠，就是目录不存在。
eval_config-model-type，使用任意带/的模型，就会这么报错
```
[2025-03-07 10:20:21,862] ERROR - RUN - run.py: run_task - 420: Model qvq/qvq-72b-preview x Dataset vqa combination failed: , skipping this combination.
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/vlmeval/run.py", line 306, in run_task
    model = infer_data_job(
  File "/usr/local/lib/python3.10/site-packages/vlmeval/inference.py", line 168, in infer_data_job
    model = infer_data(
  File "/usr/local/lib/python3.10/site-packages/vlmeval/inference.py", line 104, in infer_data
    supp = infer_data_api(
  File "/usr/local/lib/python3.10/site-packages/vlmeval/inference.py", line 62, in infer_data_api
    track_progress_rich(gen_func, structs, nproc=api_nproc, chunksize=api_nproc, save=out_file, keys=indices)
  File "/usr/local/lib/python3.10/site-packages/vlmeval/utils/mp_util.py", line 25, in track_progress_rich
    assert osp.exists(osp.dirname(save)) or osp.dirname(save) == ''
AssertionError
```

I see. So you're using a model named `qvq/qvq-72b-preview`? Is that a model path?

> I see. So you're using a model named `qvq/qvq-72b-preview`? Is that a model path?

![Image](https://github.com/user-attachments/assets/3f130f7b-caad-4896-872f-8f14123d60f1)

Yes, all the models above are in the format of manufacturer/model name. I think it's possible to read the model name and replace "/" to other characters instead of confusing it with the path



Please refer to [SiliconFlowAPI](https://github.com/open-compass/VLMEvalKit/blob/eda4a6296c008cce586bfa0f934017b5269bc35b/vlmeval/api/siliconflow.py#L40C7-L40C21) class for detail implementation. If you want to use `qvq/qvq-72b-preview`, we would rather recommend you to customize the config, for example, [TeleMMAPI](https://github.com/open-compass/VLMEvalKit/blob/eda4a6296c008cce586bfa0f934017b5269bc35b/vlmeval/api/siliconflow.py#L133):
```
'TeleMM': partial(TeleMMAPI, model='TeleAI/TeleMM', temperature=0, retry=10),
```
Repository: open-compass/VLMEvalKit
Issue: where is the vlmeval/utils/data_util.py
Root cause: Documentation Deficiency

![image](https://github.com/open-compass/VLMEvalKit/assets/55678087/14a5e92c-0d20-4c01-8ea7-9ca947fb40b3)
Excellent work:) 
But I can`t fine the data_util file in the lasted version..
And  I would like to ask when the Internlm2 7b model will be supported.Thanks!

``` shell
# When running with `python`, only one VLM instance is instantiated, and it might use multiple GPUs (depending on its default behavior). 
# That is recommended for evaluating very large VLMs (like IDEFICS-80B-Instruct).

# IDEFICS-80B-Instruct on MMBench_DEV_EN, MME, and SEEDBench_IMG, Inference and Evalution
python run.py --data MMBench_DEV_EN MME SEEDBench_IMG --model idefics_80b_instruct --verbose 
# IDEFICS-80B-Instruct on MMBench_DEV_EN, MME, and SEEDBench_IMG, Inference only
python run.py --data MMBench_DEV_EN MME SEEDBench_IMG --model idefics_80b_instruct --verbose --mode infer

# When running with `torchrun`, one VLM instance is instantiated on each GPU. It can speed up the inference. 
# However, that is only suitable for VLMs that consume small amounts of GPU memory. 

# IDEFICS-9B-Instruct, Qwen-VL-Chat, mPLUG-Owl2 on MMBench_DEV_EN, MME, and SEEDBench_IMG. On a node with 8 GPU. Inference and Evaluation.
torchrun --nproc-per-node=8 run.py --data MMBench_DEV_EN MME SEEDBench_IMG --model idefics_80b_instruct qwen_chat mPLUG-Owl2 --verbose
# Qwen-VL-Chat on MME. On a node with 2 GPU. Inference and Evaluation. 
torchrun --nproc-per-node=2 run.py --data MME --model qwen_chat --verbose
```

Hi, @TousenKaname ,
For now, please refer to the `dataset_config.py`. The document has been updated accordingly. 

Thanks!And I would like to ask when the Internlm2 7b model will be supported.