zhiminy/EvalEng huggingface dataset:
Repository: open-compass/VLMEvalKit
Issue: [bug] 反馈两个bug，模型和数据集用config.json配置会出现一些问题
Root cause: Configuration Error

![Image](https://github.com/user-attachments/assets/d1ec5cf5-0e8a-4d17-a8a0-a244e05ba026)
需要return一下model


![Image](https://github.com/user-attachments/assets/cf2e7b8a-9c7a-493a-adac-325975ae05d3)
增加一下初始化参数
Repository: open-compass/VLMEvalKit
Issue: version of transfomers
Root cause: External Dependency Breakage

which version of transformers should i apply when employing qwen2vl_series  and mplug_owl3 respectively.
i encounter the following problems:
for qwen2vl：

`ImportError: cannot import name 'Qwen2VLForConditionalGeneration' from 'transformers'`
for mplug_owl3:
`File "/data1/VLMEvalKit/vlmeval/vlm/mplug_owl3.py", line 146, in __init__  
    self.tokenizer = AutoTokenizer.from_pretrained(  
  File "/usr/local/lib/python3.10/dist-packages/transformers/models/auto/tokenization_auto.py", line 733, in from_pretrained  
    raise ValueError( 
ValueError: Tokenizer class Qwen2Tokenizer does not exist or is not currently imported.`

Hi @The-Git-Rain.
For Qwen2VL, we recommend that you use transformers>=4.48.0
For mPLUG-Owl3, it seems to be the same problem as Qwen2vl. You can also try to upgrade the transformers

For Qwen2VL, I use the latest transformers version and solve the problem 
For mPLUG-Owls, i use the latest version but I still have the problem below:
`  File "/data1/VLMEvalKit/vlmeval/vlm/mplug_owl3.py", line 320, in generate_inner  
    if listinstr(['MVBench'], dataset):  
  File "/data1/VLMEvalKit/vlmeval/smp/misc.py", line 121, in listinstr  
    if item in s:  
TypeError: argument of type 'NoneType' is not iterable`
it seems the configuration is not compatible to mPLUG-Owl3
Repository: open-compass/VLMEvalKit
Issue: There is a long gap between the validation accuracy of the dataset of vlmevalkit and the model paper
Root cause: Documentation Deficiency

On the TextVQA dataset, the paper in `Instructblip 13b` indicates that its precision is `50.7`, and the paper in `Qwen VL Chat` shows an accuracy of `63.75`. 
In terms of the accuracy measured by the vlmevalkit official, the accuracy of `Instructblip 13b` is about `30`, and the accuracy of `QWEN VL Chat` is `10.5`, what do you think is the problem? 
Also, I tested the accuracy of Instructblip 13b on textVQA and found that I ran with an accuracy of 16.7, what went wrong? These are all the results of prefech, and GPT is not used.

Hi, @YongLD , 
Actually, the support of VQA datasets is still in progress (we only share some preliminary results for now). We still cannot obtain the corresponding accuracies reported by the VLM papers. Potential reasons might be different prompt used or different inference hyper parameters. 

Hello, I find that for TextVQA dataset, LLaVA evaluation with with reference token like:
What kind of beer is this?\nReference OCR token: NINK, NK, BOWING, CC, STON, SUE, ED, Sublimely, SELF, ELF-RICHEE, swAaVd, KGy, ALE\nAnswer the question using a single word or phrase.
in VLMEvalKit does not apply that token, would you like to add an option for users to choose to add the reference tokens or not?

@kennymckormick Can we use the azure openai key in VlmEvalKit? How can I change the base_url of azure?

> @kennymckormick Can we use the azure openai key in VlmEvalKit? How can I change the base_url of azure?

Currently, VLMEvalKit does not support openai api key (cuz I do not have azure api access can cannot debug). You can follow the azure doc to support an azure openai wrapper in VLMEvalKit. 

I've noticed that most of the results are dissimilar compared to those in the research paper. I believe that the framework that's in use should be rectified, making it more corresponding with the original results.

> I've noticed that most of the results are dissimilar compared to those in the research paper. I believe that the framework that's in use should be rectified, making it more corresponding with the original results.

Hi, @geknow ,
We understand the known issues with the preliminary results on VQA tasks might be misleading. We have know removed these results and will re-upload them once ready.
Repository: open-compass/VLMEvalKit
Issue: 很多模型使用vlmevalkit评测不同的数据集时效果都有较大的差距
Root cause: Validation Gap

mPLUG code with paper榜单上coco数据集的BLEU-4为46.5，CIDER为155.1
实际我跑的BLEU-4为33.4，CIDER为95.61，结果如下：

![Image](https://github.com/user-attachments/assets/6887aa67-d5bd-4cdb-a362-30f2dc00d123)
（另外，评测指标能不能尽量完整一些，补全METEOR和ROUGE-L）
还有minicpmv2.6在mme-video上面的结果也较差，这个我在其他的issue当中也看到了类似的问题，这里再次提出，希望能够得到解决。
以下是我的验证脚本，使用的swift。
CUDA_VISIBLE_DEVICES=0,1,2,3 \
swift eval \
    --adapters /share/user/output/mPLUG-Owl3-7B-240728/v1-20250421-170456/checkpoint-4405 \
    --eval_backend VLMEvalKit \
    --infer_backend pt \
    --eval_dataset COCO_VAL \
    --temperature 0
感谢您和您团队的工作，希望能够尽快完善！

我建议你采用模型进行评测，规则判断的分数都很低

> 我建议你采用模型进行评测，规则判断的分数都很低

caption任务现在还是有很多都是采用规则判断评测的，要与之前的工作对比就无法避开这一点。而令我比较困惑的是评测的指标相差居然会如此之大，换算成百分比大约差了20%。而且我也测试过mme-video数据集，minicpmv2.6在该数据集上取得的分数为54.7，相较于官方的榜单的63.7相差仍然是十分大，而该benchmark个人认为不能算规则判断的吧。

可以对比你实测的结果也行！
也可以分析一下回答的结果



---原始邮件---
发件人: ***@***.***&gt;
发送时间: 2025年4月23日(周三) 晚上7:40
收件人: ***@***.***&gt;;
抄送: "Kyrie ***@***.******@***.***&gt;;
主题: Re: [open-compass/VLMEvalKit] 很多模型使用vlmevalkit评测不同的数据集时效果都有较大的差距 (Issue #941)


    
我建议你采用模型进行评测，规则判断的分数都很低
  
caption任务现在还是有很多都是采用规则判断评测的，要与之前的工作对比就无法避开这一点。而令我比较困惑的是评测的指标相差居然会如此之大，换算成百分比大约差了20%。而且我也测试过mme-video数据集，minicpmv2.6在该数据集上取得的分数为54.7，相较于官方的榜单的63.7相差仍然是十分大，而该benchmark个人认为不能算规则判断的吧。

—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you commented.Message ID: ***@***.***&gt;
  lanyuki left a comment (open-compass/VLMEvalKit#941)
  
我建议你采用模型进行评测，规则判断的分数都很低
  
caption任务现在还是有很多都是采用规则判断评测的，要与之前的工作对比就无法避开这一点。而令我比较困惑的是评测的指标相差居然会如此之大，换算成百分比大约差了20%。而且我也测试过mme-video数据集，minicpmv2.6在该数据集上取得的分数为54.7，相较于官方的榜单的63.7相差仍然是十分大，而该benchmark个人认为不能算规则判断的吧。
 
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you commented.Message ID: ***@***.***&gt;
Repository: open-compass/VLMEvalKit
Issue: How to add new custom dataset?
Root cause: Documentation Deficiency

Thanks for the project! I’m curious—what’s the best way to add a custom dataset? Is there any helpful documentation or reference you could point me to?

please refer to [Development Guide](https://github.com/open-compass/VLMEvalKit/blob/main/docs/en/Development.md) for details.
Repository: open-compass/VLMEvalKit
Issue: Did not detect the .env file  when using torchrun
Root cause: Environment Incompatibility

when using python run.py --data ChartQA_TEST--model 7B --verbose，it goes fine but too much slow
when using torchrun command like，torchrun --nproc-per-node=4 run.py --data ChartQA_TEST--model 7B --verbose， it comes with the following issue

W0423 17:11:07.226000 3089305 site-packages/torch/distributed/run.py:793] *****************************************
[2025-04-23 17:11:11] ERROR - misc.py: load_env - 212: Did not detect the .env file at /data3/xxf/VLMEvalKit/.env, failed to load. 
[2025-04-23 17:11:11] ERROR - misc.py: load_env - 212: Did not detect the .env file at /data3/xxf/VLMEvalKit/.env, failed to load. 
[2025-04-23 17:11:11] ERROR - misc.py: load_env - 212: Did not detect the .env file at /data3/xxf/VLMEvalKit/.env, failed to load. 
[2025-04-23 17:11:11] ERROR - misc.py: load_env - 212: Did not detect the .env file at /data3/xxf/VLMEvalKit/.env, failed to load. 
[2025-04-23 17:11:11,812] WARNING - RUN - run.py: main - 174: --reuse is not set, will not reuse previous (before one day) temporary files
[2025-04-23 17:11:11] WARNING - run.py: main - 174: --reuse is not set, will not reuse previous (before one day) temporary files
[2025-04-23 17:11:11] ERROR - misc.py: load_env - 212: Did not detect the .env file at /data3/xxf/VLMEvalKit/.env, failed to load. 
[2025-04-23 17:11:11] ERROR - misc.py: load_env - 212: Did not detect the .env file at /data3/xxf/VLMEvalKit/.env, failed to load. 
[2025-04-23 17:11:11] ERROR - misc.py: load_env - 212: Did not detect the .env file at /data3/xxf/VLMEvalKit/.env, failed to load. 

could u help me to solve it，many thanks

I test with `torchrun --nproc-per-node=1 run.py --data ChartQA_TEST --model Eagle-X5-7B --verbose` and it works fine. Please check if the env file `/data3/xxf/VLMEvalKit/.env` exists

> I test with `torchrun --nproc-per-node=1 run.py --data ChartQA_TEST --model Eagle-X5-7B --verbose` and it works fine. Please check if the env file `/data3/xxf/VLMEvalKit/.env` exists

many thanks, and I have another question: Is the [VLM-EvalKit] suitable for evaluating models trained with LoRA? Specifically, I fine-tuned qwenvl using LoRA and want to test its performance. Would the toolkit support this scenario, or are there any compatibility considerations I should be aware of?

If you fine-tune a supported model using VLMEvalKit, evaluating it should be straightforward. You'll need to define your model, inherit from the base model architecture, and specify the path to your fine-tuned weights in `vlmeval/config.py` as follows:
```python
qwen_series = {
    "qwen_lora": partial(QwenVL, model_path="path/to/model_path"),
    "qwen_base": partial(QwenVL, model_path="Qwen/Qwen-VL"),
    ...,
}
```

see [config.py/qwenvl_series](https://github.com/open-compass/VLMEvalKit/blob/b643eed3d6d6577bde967b1678ce2fc9273779aa/vlmeval/config.py#L532) for details
Repository: open-compass/VLMEvalKit
Issue: 无法找到MMStar在config.json文件中对应的class
Root cause: Documentation Deficiency

您好！
我正在使用--config的方法对MMStar数据集进行测试，但是无法在vlmeval/dataset/__init__.py中找到MMStar所对应的class。
config.json文件需要我填写MMStar的"class"才能工作，请问该在哪里找到呢？
非常期待您的回复！

MMStar 对应的class是 `ImageMCQDataset` , 部分数据集的评估格式是一样的，因此只提供数据集链接，参考 [MMStar](https://github.com/open-compass/VLMEvalKit/blob/b643eed3d6d6577bde967b1678ce2fc9273779aa/vlmeval/dataset/image_mcq.py#L86)
Repository: open-compass/VLMEvalKit
Issue: Janus-Pro Series on VQA
Root cause: Validation Gap

I experimented with the **Janus-Pro** series on the **ChartQA_TEST** dataset using the following command:

`torchrun --nproc-per-node=4 run.py --data ChartQA_TEST --model Janus-Pro-7B --verbose
`

###  Performance

| Split           | Accuracy |
|-----------------|----------|
| test_augmented  | 30.56    |
| test_human      | 20.80    |
| **Overall**     | **25.68** |

Upon inspecting the predictions, I noticed that the answers were **all concatenated with trailing dots**, such as:
"Yes.", "1995.", "No."

### After Removing Trailing Dots:

| Split           | Accuracy |
|-----------------|----------|
| test_augmented  | 75.04    |
| test_human      | 43.44    |
| **Overall**     | **59.24** |

This suggests the punctuation are not post-processed and might be affecting the evaluation script or ground truth comparison - for other tasks too.

By default, ChartQA uses `relaxed_accuracy` to judge its answer, which is a rule-based metric, see [ChartQA Section 5.1](https://arxiv.org/pdf/2203.10244) for more details. In VLMEvalKit, we follow the same way to judge the results, so there may be a gap between the real performance and reported performance.

Yes, I’m aware the evaluation uses `relaxed_accuracy`, but Janus-series models still need custom post-processing to remove trailing punctuation; without that, the results aren’t reliable. @MaoSong2022 , have you run the command and seen the behavior?

It seems other models exhibit similar behavior. As mentioned in the design philosophy ([VLMEvalKit is not designed to](https://github.com/open-compass/VLMEvalKit?tab=readme-ov-file#-the-goal-of-vlmevalkit)), we currently adhere to default settings for models and benchmarks.

To address this, there are two potential solutions: 

1. Modify the `generative_inner` function within the model to better adapt its output for various benchmarks, or 
2. Adjust the evaluation logic of the benchmark itself.
Repository: open-compass/VLMEvalKit
Issue: how to test pass@k?
Root cause: Unimplemented Feature Gap

I wonder if VLMEvalKit can test the pass@k of MLLMs on some benchmarks?

Hi, @digbangbang , 

Short Answer: No. 

Currently we adopt temperature=0 for most VLMs, so the pass@k metric may not be meaningful. 

If you would like to test pass@k w. VLMEvalKit, you can set the model temperature to non-zero and run the inference multiple times, then you can aggregate all evaluation results and calculate the pass@k. 

Thanks, @kennymckormick. I will try this method.
Repository: open-compass/VLMEvalKit
Issue: Error when using TextVQA_VAL
Root cause: Validation Gap

```python
[2025-04-26 23:28:58,209] ERROR - RUN - run.py: main - 482: Model Qwen2-VL-2B-Instruct x Dataset TextVQA_VAL combination failed: , skipping this combination.
Traceback (most recent call last):
  File "LLaMA-Factory/VLMEvalKit/run.py", line 325, in main
    model = infer_data_job(
  File "LLaMA-Factory/VLMEvalKit/vlmeval/inference.py", line 185, in infer_data_job
    model = infer_data(
  File "LLaMA-Factory/VLMEvalKit/vlmeval/inference.py", line 148, in infer_data
    response = model.generate(message=struct, dataset=dataset_name)
  File "LLaMA-Factory/VLMEvalKit/vlmeval/vlm/base.py", line 110, in generate
    message = self.preproc_content(message)
  File "LLaMA-Factory/VLMEvalKit/vlmeval/vlm/base.py", line 91, in preproc_content
    assert item["type"] == "text"
AssertionError
```

The benchmark below is the same:
MME-RealWorld-Lite
POPE

你好，能问下后面是怎么解决的吗，我遇到了类似的问题


> 你好，能问下后面是怎么解决的吗，我遇到了类似的问题

你可以去看看你的 LMUData （应该叫这个名字）里面的数据是什么情况。

测试的数据在第一次的时候都会下载到这个目录里面，有些情况下测试数据里提到的本地目录可能不是很对（比如图片指向了不正确的地方），比如修改过 LMUData 的位置可能发生这种情况。这时候可以选择重新下载一次测试数据，或者手动进去批量地把不正确的目录都改成正确的目录。
Repository: open-compass/VLMEvalKit
Issue: The inference on the AMBER dataset is very slow.
Root cause: Configuration Error

The AMBER dataset is a yes-or-no dataset, but the questions in this dataset do not explicitly state: "Please answer yes or no." As a result, during evaluation, the model generates very long responses, which slows down the entire inference process. Is it possible to address this issue?

Hi, thanks for pointing out the problem. I checked the the original AMBER dataset, they didn't provide such suffix like "Please answer yes or no.". For consistency, we did not use these suffix too.

If you want to generate shorter responses, you can modify the dataset file mannually or consider using custom prompt for the evaluated model.

That somehow makes sense, I can add this additional instruction to the test prompt of AMBER. 

> Hi, thanks for pointing out the problem. I checked the the original AMBER dataset, they didn't provide such suffix like "Please answer yes or no.". For consistency, we did not use these suffix too.
> 
> If you want to generate shorter responses, you can modify the dataset file mannually or consider using custom prompt for the evaluated model.

Maybe it can be handled another way? Such as let the max new token when evaluating model at amber benchmark be 1?

Hi, @pspdada . The problem has been resolved in https://github.com/open-compass/VLMEvalKit/pull/961. 

> > Hi, thanks for pointing out the problem. I checked the the original AMBER dataset, they didn't provide such suffix like "Please answer yes or no.". For consistency, we did not use these suffix too.
> > If you want to generate shorter responses, you can modify the dataset file mannually or consider using custom prompt for the evaluated model.
> 
> Maybe it can be handled another way? Such as let the max new token when evaluating model at amber benchmark be 1?

I do not recommend this alternative, since most API VLMs may not say Yes / No in the beginning of their responses.

> That somehow makes sense, I can add this additional instruction to the test prompt of AMBER.

@kennymckormick I can help to do this
Repository: open-compass/VLMEvalKit
Issue: dataset如何在离线的服务器使用呢？
Root cause: Documentation Deficiency

您好，以MMMU_DEV_VAL数据集为例，我在有网络的环境下会自动下载，但我现在想要将这一套测评框架部署到服务器上，而服务器上是没有外网的，因此我想要找到缓存的文件，然后配置到一个文件夹中，然后到时候可以离线使用，但找了好久，都没有找到相关的文件和配置方法。

如果没有显式设置的话，默认的文件路径为 `$HOME/LMUData`, 参见 [LMUDataRoot](https://github.com/open-compass/VLMEvalKit/blob/6ea82af1812a04debdcd1817a3ddbb9dce9a5068/vlmeval/smp/file.py#L68)

Hi, @RuipingWang1986 , 

所有 TSV 文件对应的下载链接均在数据集类中有提供，可先在有网络环境进行下载，再将整个 LMUData 目录拷贝至离线机器。

> 如果没有显式设置的话，默认的文件路径为 `$HOME/LMUData`, 参见 [LMUDataRoot](https://github.com/open-compass/VLMEvalKit/blob/6ea82af1812a04debdcd1817a3ddbb9dce9a5068/vlmeval/smp/file.py#L68)

非常感谢，问题已经解决



> Hi, [@RuipingWang1986](https://github.com/RuipingWang1986) ,
> 
> 所有 TSV 文件对应的下载链接均在数据集类中有提供，可先在有网络环境进行下载，再将整个 LMUData 目录拷贝至离线机器。

非常感谢，问题已经解决
Repository: open-compass/VLMEvalKit
Issue: API模型怎么并行处理
Root cause: Documentation Deficiency

比如GPT4o评测，怎么并行加速评测？

python run.py --model {API_MODEL_NAME} --data {DATASET} --api-nproc XXX

You can set `--api-nproc` to a large number (like 8 or 16) to call APIs in parallel, the default value of `--api-nproc` is 4. 

Great
Repository: open-compass/VLMEvalKit
Issue: 可以使用自己搭建的VLLM进行测试吗？
Root cause: Configuration Error


https://github.com/open-compass/VLMEvalKit/issues/789
提到了解决方法
python run.py --data MMBench_DEV_EN --model lmdeploy --verbose --api-nproc 16
不过需要手动调整一下vlmeval/config.py中的端口
```python
# line 386
    "lmdeploy": partial(
        LMDeployAPI,
        api_base="http://0.0.0.0:8000/v1/chat/completions",  # 默认是23333
        temperature=0,
        retry=10,
    ),
```

是不是要自己实现generate_inner方法就可以了？
Repository: open-compass/VLMEvalKit
Issue: Turn Off Saving Images in LMUData
Root cause: Unimplemented Feature Gap

When running experiments, encoded images from the TSV are automatically saved to the `LMUData` folder. Is there a way to skip saving them to optimize disk space usage? Would be nice to have such a `arg`. 

Thanks!

Hi, currently, all images will be converted into an image file before feed into LLM. The process happens in `build_prompt` function in each dataset, for example:
```python
if self.meta_only:
    tgt_path = toliststr(line['image_path'])
else:
    tgt_path = self.dump_image(line)
...
msgs = [dict(type='image', value=tgt_path)]
```

to pass image in the base64 format directly to LLM, you can comment the above lines and use the following code

```python
base64_image = line["image"]
msgs = [
    {"image" : f"data:image;base64,{base64_image}",
]
```

To control whether or not saving images, this requires further discussion. As far as I known, base64 image format has drawbacks in terms of size, efficiency, and debuggability. Maybe @kennymckormick can give more insightful opinions.
Repository: open-compass/VLMEvalKit
Issue: 多线程同时读取tsv文件时有线程读取不成功
Root cause: Resource Mishandling

报错位置L35[https://github.com/open-compass/VLMEvalKit/blob/8c3908f892c39ea16acd31814ac11921200625fc/vlmeval/dataset/video_base.py#L31-L36](https://github.com/open-compass/VLMEvalKit/blob/8c3908f892c39ea16acd31814ac11921200625fc/vlmeval/dataset/video_base.py#L31-L36)

我在运行评测时，有概率出现以下错误：

 ```

  File "/nfs196/zzy/Code/VLMEvalKit-main/vlmeval/dataset/video_base.py", line 36, in __init__

    assert 'question' in self.data and 'video' in self.data

```

在assert前一行加入

`print（self.data.keys()）`

发现在报错时会有线程读取的时错误的

```

Index(['task_type', 'prefix', 'video', 'question', 'answer', 'candidates',

       'index'],

      dtype='object')

Index(['task_type', 'prefix', 'video', 'question', 'answer', 'candidates',

       'index'],

      dtype='object')

Index(['Unnamed: 0', '1890', 'index'], dtype='object')

Index(['task_type', 'prefix', 'video', 'question', 'answer', 'candidates',

       'index'],

      dtype='object')

```
因为是有概率发生，所以感觉可能是多线程的bug

可以看一下您运行的指令吗，方便我们复现

torchrun --nproc-per-node=4  run.py --data MVBench --model Qwen2-VL-7B-Instruct --verbose
有概率发生，所以可能不太好复现 ：）

 似乎知道原因了，因为我的MVBench是保存到本地的，我稍微改了下代码让他每次都生成一遍tsv，可能是由于多线程有线程写tsv的时候另一个线程在读就会有读数据出问题，一般来说不是每次都生成tsv就不会有这个问题
Repository: open-compass/VLMEvalKit
Issue: Adding LiveXiv - a live benchmark
Root cause: Unimplemented Feature Gap

Hi,
Thanks for the super important work of standardize evaluation! 
I want to add my recent ICLR 2025 benchmark [LiveXiv ](https://huggingface.co/datasets/LiveXiv/LiveXiv) Which is a live and updated benchmark.
Is there a support for live benchmark or I need to add each version each time?

Thanks,
Nimrod

Hi, currently, we do not support live benchmarks, since all dataset will be saved into a `.tsv` file, you need to add each version each time. 

If your splits share a same format, you can just add new splits to `DATASET_URL` and `DATASET_MD5` without changing code when there is a new splits. For example, in [CVBench](https://github.com/open-compass/VLMEvalKit/blob/34dceec3deb14e64e8845fffd944b2363462ad1c/vlmeval/dataset/image_mcq.py#L889), there are two splits, `CV-Bench-2D` and `CV-Bench-3D`.

Hoping this helps, let me know if you have any problems or further support.
Repository: open-compass/VLMEvalKit
Issue: 测试api based model时preprocess_message_with_role()会造成system prompt无限叠加
Root cause: Algorithmic Error

https://github.com/open-compass/VLMEvalKit/blob/7edc329f3107e14fc1c25c40dbdbc08e86aa3894/vlmeval/api/base.py#L197-L215

如题，我在测试自建数据集的时候发现system无限叠加，原因是我的数据集返回的prompt里面有systemprompt属性，这样每次过preprocess_message_with_role函数时，system prompt都会叠加一遍L205和L213。理论上只要有system属性的，如MVbench也会出现这个问题？

```
    def preprocess_message_with_role(self, message):
        system_prompt = ''
        new_message = []

        for data in message:
            assert isinstance(data, dict)
            role = data.pop('role', 'user')
            if role == 'system':
                system_prompt += data['value'] + '\n'
            else:
                new_message.append(data)
                
        self.system_prompt = self.system_prompt_base
        
        if system_prompt != '':
            if self.system_prompt is None:
                self.system_prompt = system_prompt
            else:
                self.system_prompt += '\n' + system_prompt

        return new_message
```

把self.system_prompt_base和self.system_prompt同样初始化，然后加上一个self.system_prompt = self.system_prompt_base应该可以解决这个问题

@kennymckormick 
这个问题我觉得是不是应该在new_message里返回system prompt，类似这样

```python
if system_prompt != '':
    if self.system_prompt is not None:
        system_prompt += self.system_prompt + '\n' + system_prompt
    new_message.insert(0, dict(role='system', content=system_prompt))
```
这样可以防止`self.system_prompt`被修改。但是问题是所有的api模型都需要重新处理一下`self.system_prompt`.
Repository: open-compass/VLMEvalKit
Issue: 结果Excel能否保存每个样本正确或错误或者得分
Root cause: Unimplemented Feature Gap


目前的xlsx应该都是有对应的Prediction和score的，请问您评测是哪个benchmark，我们查看一下。

感觉好多都没有，测了OCRbench、mathvista等等都没有

有计划增加这个吗？@MaoSong2022
Repository: open-compass/VLMEvalKit
Issue: 关于qwen2.5omni的性能

你好，我使用[VLMEvalKit](https://github.com/open-compass/VLMEvalKit) 对qwen2.5-omni进行评估时发现在几个数据集上的性能和论文中的差距较大，请问这是为什么，下面是我评测的几个数据集性能

|数据集|论文|评估|命令|
|---|---|---|---|
|MMMU_val|59.2|51.6|python run.py --data MMMU_DEV_VAL --model Qwen2.5-Omni-7B|
|MMMU_pro| 36.6|31|python run.py --data MMMU_Pro_10c MMMU_Pro_V --model Qwen2.5-Omni-7B|
|MMStar| 64.0|60.6|python run.py --data MMStar --model Qwen2.5-Omni-7B|



你好，VLMEvalKit不保证能复现原始论文中的结果，影响结果的原因有很多，包括采样设置，prompt等原因等。你可以在[qwen2.5-omni Github](https://github.com/QwenLM/Qwen2.5-Omni) 提Issue反应相应问题。

@MaoSong2022 顺带请问下：MMMU-Pro评估有4个key，请问分别是什么含义呢？
MMMU_Pro_10c、MMMU_Pro_10c_COT、MMMU_Pro_V、MMMU_Pro_V_COT

> [@MaoSong2022](https://github.com/MaoSong2022) 顺带请问下：MMMU-Pro评估有4个key，请问分别是什么含义呢？ MMMU_Pro_10c、MMMU_Pro_10c_COT、MMMU_Pro_V、MMMU_Pro_V_COT

MMMU_Pro_10c和MMMU_Pro_10c_COT指的是10options数据集，MMMU_Pro_V和MMMU_Pro_V_COT指的是vision数据集，问题和选项都在图片中，需要模型自己去解析；带COT的是思维链推理方式，输入的prompt和对应的后处理评估方法会有所不同