zhiminy/EvalEng huggingface dataset:
Repository: open-compass/VLMEvalKit
Issue: cannot use Gemini
Root cause: Unimplemented Feature Gap

I cannot use Gemini

when use: GeminiProVision
2024-09-02 18:57:19,592 - ChatAPI - ERROR - 404 Gemini 1.0 Pro Vision has been deprecated on July 12, 2024. Consider switching to different model, for example gemini-1.5-flash.

when use: GeminiPro1-5 
    assert model == 'gemini-1.0-pro'
AssertionError

Hi, how can I incorporate gemini-1.5-flash model in this framework?  Or can you please update the Gemini model? Thanks!

Hi,

The latest version of vlmevalkit has has resolved this issue. Please refer to [gemini](https://github.com/open-compass/VLMEvalKit/blob/b779baa7681d73155489f3cb33c8c62a77be0842/vlmeval/api/gemini.py#L25) and use Gemini-1.5-pro. 


Thank you!

> Hi, how can I incorporate gemini-1.5-flash model in this framework? Or can you please update the Gemini model? Thanks!

We will consider adding support for the gemini-1.5-flash model soon.
Repository: open-compass/VLMEvalKit
Issue: cannot use gpt-4v.
Root cause: External Dependency Breakage

Log: {
    "error": {
        "message": "The model `gpt-4-1106-vision-preview` has been deprecated, learn more here: https://platform.openai.com/docs/deprecations",
        "type": "invalid_request_error",
        "param": null,
        "code": "model_not_found"
    }
}

Hi,

As indicated by the error message, **the model gpt-4-1106-vision-preview has been deprecated**. Please switch to using other recent GPT models. All model names can be found in [utils/gpt](https://github.com/open-compass/VLMEvalKit/blob/b779baa7681d73155489f3cb33c8c62a77be0842/vlmeval/dataset/utils/judge_util.py#L8).

Thank you!
Repository: open-compass/VLMEvalKit
Issue: Deploy local LLM as judge  llm on VLLM tools,but it didn't use it to evaluate
Root cause: Configuration Error

vllm cli: ```CUDA_VISIBLE_DEVICES=6,7 python -m vllm.entrypoints.openai.api_server --served-model-name Qwen2-7B-Instruct --port 8001 --model ../models/Qwen2-7B-Instruct --tensor-parallel-size 2 --api-key sk-123456```

for evluating mmstar,mathvista_mini datasets

.env settings as following: 
```
LMUData=./data
OPENAI_API_KEY=sk-123456
OPENAI_API_BASE=http://0.0.0.0:8001/v1/chat/completions     
LOCAL_LLM=Qwen2-7B-Instruct
```

+1! Doploy internlm2-1.8B as judge using lmdeploy, but it didnot use when evaluate!
`OPENAI_API_KEY=sk-123456
OPENAI_API_BASE=http://0.0.0.0:23333/v1/chat/completions
LOCAL_LLM=./VLM/ckpt/internlm2-1.8/`
Repository: open-compass/VLMEvalKit
Issue: The performance of Cambrian-1-8B.
Root cause: Validation Gap

Hello, I am using this code to evaluate the official checkpoint of Cambrian-1-8B. However, I fail to reproduce the results. Specifically, I got only 64.78 on MMBench-DEV-EN, while the reported score is 75.9 (Table 8 of the paper).

Could you help me to reproduce the results?

Hi,

Thank you for your findings! We have identified that the issue is due to Cambrian-8b’s instruction-following capabilities. We recommend modifying the prompt in [MMBench](https://github.com/open-compass/VLMEvalKit/blob/1e2c6bb20a9803552d8bd56d5cb0ce876ae2a648/vlmeval/dataset/image_mcq.py#L171) as follows:

```
if len(options):
    prompt += options_prompt
    # prompt += 'Please select the correct answer from the options above. \n'
    prompt +=  '\n' + "Answer with the option's letter from the given choices directly."
```
With this change, the evaluation results of MMBench-Dev-EN for Cambrian-8B should improve to **76.29**.

Best regards.

Thanks for the reply! I have successfully reproduced the results!
Repository: open-compass/VLMEvalKit
Issue: Custom prompt for API?
Root cause: Unimplemented Feature Gap

I find that testing models via API is not consistent with testing models directly, especially the prompt. API do not support custom prompt now. Will VLMEvalKit support custom prompt for API?

testing API：https://github.com/open-compass/VLMEvalKit/blob/main/vlmeval/inference.py#L33
testing model：https://github.com/open-compass/VLMEvalKit/blob/main/vlmeval/inference.py#L111

Hi, @geweihgg , 
We have added custom prompt for APIs now. Please have another try.
Repository: open-compass/VLMEvalKit
Issue: MME-RealWorld Not Supported.
Root cause: Unimplemented Feature Gap

Hello, I found that MME-RealWorld has not been supported:
```bash
ERROR - Dataset MME-RealWorld is not valid, will be skipped.
```

Hi,
this problem has been fixed just now.
Thank you!

Thanks for the prompt reply! The fix works for me.
Repository: open-compass/VLMEvalKit
Issue: 测试程序经常中断
Root cause: Environment Incompatibility

请问在跑测试程序时，为什么会经常中断？不改任何东西又可以继续跑，又会中断...

您好，感谢你的反馈！能否提供更多相关信息，比如具体的运行命令、报错信息或中断时的日志？这样我们可以更好地帮助你解决问题。谢谢！

换了个环境，暂时没出现中断的问题，谢谢

有任何问题可以随时重新打开或新建 issue。测试程序如果再次中断，欢迎提供更多信息，我们会尽量帮忙协助解决。
Repository: open-compass/VLMEvalKit
Issue: 评测时PermissionError: [Errno 13] Permission denied
Root cause: Environment Incompatibility

Traceback (most recent call last):
  File "D:\opencompass1\VLMEvalKit\run.py", line 223, in <module>
    main()
  File "D:\opencompass1\VLMEvalKit\run.py", line 137, in main
    model = infer_data_job(
  File "D:\opencompass1\VLMEvalKit\vlmeval\inference.py", line 162, in infer_data_job
    model = infer_data(
  File "D:\opencompass1\VLMEvalKit\vlmeval\inference.py", line 102, in infer_data
    supp = infer_data_api(
  File "D:\opencompass1\VLMEvalKit\vlmeval\inference.py", line 60, in infer_data_api
    track_progress_rich(gen_func, structs, nproc=api_nproc, chunksize=api_nproc, save=out_file, keys=indices)
  File "D:\opencompass1\VLMEvalKit\vlmeval\utils\mp_util.py", line 188, in track_progress_rich
    raise e
  File "D:\opencompass1\VLMEvalKit\vlmeval\utils\mp_util.py", line 174, in track_progress_rich
    ans = load(save)
  File "D:\opencompass1\VLMEvalKit\vlmeval\smp\file.py", line 177, in load
    return handlers[suffix](f)
  File "D:\opencompass1\VLMEvalKit\vlmeval\smp\file.py", line 150, in load_pkl
    return pickle.load(open(pth, 'rb'))
PermissionError: [Errno 13] Permission denied


Hi, @40459447 , 
The toolkit is developed for Linux and we are not sure if it runs properly on Windows platform
Repository: open-compass/VLMEvalKit
Issue: Input length of input_ids is 0, but max_length is set to -2009.
Root cause: Validation Gap

When I run the latest  code with the following command
"python run.py --data MMBench_TEST_EN_V11 MMBench_TEST_CN_V11  --model InternVL2-76B --verbose"

Error：
ValueError: Input length of input_ids is 0, but max_length is set to -2009. This can lead to unexpected behavior. You should consider increasing max_length or, better yet, setting max_new_tokens.

python package:
transformers                   4.40.0
typing_extensions              4.9.0
torch                          2.2.1+cu121
torchaudio                     2.2.1+cu121
torchvision                    0.17.1+cu121

In this environment, I can run the original  InternVL2-76B without any errors.

Hi,  @wangli68, 
I have fixed the `split_model` method for later torch versions (2.2+) in this commit: https://github.com/open-compass/VLMEvalKit/commit/7957e996434e2022cfa55772de31890310953934 (though I'm not sure if it is related to your question). 

Under a similar environment, the code run well on 4 A800 GPUs (when `load_in_8bit` is False, the evaluation of InternVL2-76B requires at least three 80G GPUs). The environment is:

```
transformers 4.40.0
typing_extensions 4.9.0
torch 2.2.2
torchaudio 2.2.2
torchvision 0.17.2
```

Evaluation screenshot (The `device logs are added for debugging): 

![image](https://github.com/user-attachments/assets/5e8f0f1a-f406-4b10-89c3-09af5ad1fa86)

GPU MEM Usage: 

![image](https://github.com/user-attachments/assets/76202dec-bf80-4421-925f-2cd5f9ac22c5)
Repository: open-compass/VLMEvalKit
Issue: word_count_mean of COCO Caption
Root cause: Documentation Deficiency

How is the "word_count_mean" in the COCO Caption results computed? I can't find it in the scripts.

Hi, @moiSusu , `word_count_mean` is counted based on the prediction files of COCO_VAL, and the script is not included in this repo. Below is the function we used for word counting, in case you need:

```python
def word_count(inText):
    inText = str(inText)
    outText = inText
    punct = [';', r'/', '[', ']', '"', '{', '}', '(', ')', '=', '+', '\\', '_', '-', '>', '<', '@', '`', ',', '?', '!']
    commaStrip = re.compile('(\d)(,)(\d)')  # noqa: W605
    periodStrip = re.compile('(?!<=\d)(\.)(?!\d)')  # noqa: W605
    for p in punct:
        if (p + ' ' in inText or ' ' + p in inText) or (re.search(
                commaStrip, inText) is not None):
            outText = outText.replace(p, '')
        else:
            outText = outText.replace(p, ' ')
    outText = periodStrip.sub('', outText, re.UNICODE)
    return len(outText.split())
```
Repository: open-compass/VLMEvalKit
Issue: qwen2vl_series需要什么版本的transformers
Root cause: Environment Incompatibility

 你好！请问qwen2vl_series需要什么版本的transformers，我评测Qwen2-VL-2B-Instruct时会出现这种错误，或者我需要额外配置什么吗，麻烦告知，感谢！
 from transformers import Qwen2VLForConditionalGeneration, Qwen2VLProcessor
ImportError: cannot import name 'Qwen2VLForConditionalGeneration' from 'transformers' (/root/miniconda3/envs/qwen/lib/python3.10/site-packages/transformers/__init__.py)


目前还只能从git源码安装

```bash
pip install git+https://github.com/huggingface/transformers
```

![image](https://github.com/user-attachments/assets/ae005515-2cef-4f75-aba1-4aca9dae3704)
But why does this problem persist even when I have installed transformers from source?

> ![image](https://private-user-images.githubusercontent.com/128971236/382229295-ae005515-2cef-4f75-aba1-4aca9dae3704.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MzA2ODQ0NzUsIm5iZiI6MTczMDY4NDE3NSwicGF0aCI6Ii8xMjg5NzEyMzYvMzgyMjI5Mjk1LWFlMDA1NTE1LTJjZWYtNGY3NS1hYmExLTRhY2E5ZGFlMzcwNC5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQxMTA0JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MTEwNFQwMTM2MTVaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT0yODA2YzM3YTI5ODJlZTZmOWQ0YWYzNzFkZDlkZWQwZjA3NGZjZjM2ZTZmMTRjMTI0ZDhlOTkxYTY2MWNmNDEzJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCJ9.NDKU9KqPzGSuZVTtodKwWppUqjDXg_rnQdbTxX8ZG7g) But why does this problem persist even when I have installed transformers from source?

我当时是可以安装的，你可以试试先把transformers仓库下载下来，在本地安装，也可以看一下Qwen的要求： https://github.com/QwenLM/Qwen2-VL
Repository: open-compass/VLMEvalKit
Issue: InternVL2 Truncated Output
Root cause: Configuration Error

![image](https://github.com/user-attachments/assets/e38cad21-490a-434e-9c45-1b3a7764f34f)
When I use InternVL2-40B to do single image inference using personal data, the output is truncated.

Hi, @TJ-Ouyang , 
That is strange, apparently the token number of the output is much smaller than the max_tokens set. Would you please share the content of your test.py so we may help with debugging? 

> Hi, @TJ-Ouyang , That is strange, apparently the token number of the output is much smaller than the max_tokens set. Would you please share the content of your test.py so we may help with debugging?

We have solved the problem by setting "kwargs_default = dict(do_sample=False, max_new_tokens=512, top_p=None, num_beams=1)" before self.model.chat at line 359 in internvl_chat.py. We used ipdb and found the process did not jump into chat_inner function, where max_new_tokens was set before. 
![image](https://github.com/user-attachments/assets/2ef8e550-9b23-4311-988e-7adc5946edac)


@TJ-Ouyang , 
I have figured out the cause of this problem: the default `max_new_tokens` for InternVL2-40B is just 20. I have changed the default kwargs value so that other users will not meet this problem again.

https://github.com/open-compass/VLMEvalKit/blob/e542caea3cf5d362131bccdb97a70ce50839c332/vlmeval/vlm/internvl_chat.py#L177
Repository: open-compass/VLMEvalKit
Issue: 'gpt-3.5-turbo-0613'已被废弃，建议将'gpt-3.5-turbo-0613' -> gpt-3.5-turbo
Root cause: External Dependency Breakage

![image](https://github.com/user-attachments/assets/899f6351-96e7-449c-a1cf-10e73fbf6402)
![image](https://github.com/user-attachments/assets/a7cf1b86-c906-4b4a-bc2b-ad9dbb52892d)
![image](https://github.com/user-attachments/assets/3ccf7f77-28e9-4a8a-bdeb-08143ca188a6)


Hi, @Cooperx521 , 
我们最新的代码中应该已经移除了 gpt-3.5-0613 的使用， 请基于我们最新的代码进行评测。
Repository: open-compass/VLMEvalKit
Issue: infovqa_test测试时会报错
Root cause: Validation Gap

![image](https://github.com/user-attachments/assets/80e2f259-207e-43d9-895e-e4ab1728c1cf)


Hi, @Cooperx521 , 
InfoVQA_TEST does not include gt answer, so you can not obtain the metrics of InfoVQA_TEST w. VLMEvalKit.
You can evaluate the InfoVQA_val instead.
Repository: open-compass/VLMEvalKit
Issue: HallusionBench skips samples without images
Root cause: Validation Gap

Hello, 
     I follow your [Evaluation](https://github.com/open-compass/VLMEvalKit/blob/main/docs/en/get_started/Quickstart.md#step-2-evaluation) and evaluate HallusionBench using the script as follows:
`python run.py --data HallusionBench --model model --verbose --nproc=1`
The model generated 951 results, which skipped the samples without images. The HallusionBench benchmark should evaluate 1,129 samples, including some pure text samples. 

I found the possible cause lies in the [skip setting](https://github.com/open-compass/VLMEvalKit/blob/d1aafac1a7986881872ff8a80a8cc03cc6677691/vlmeval/dataset/image_base.py#L34). Running when setting skip_noimg=False will raise error. 

Do I miss something? Thanks for your time!

Hi, @ChuanyangZheng , 
In VLMEvalKit we did not involve the pure-text samples, cuz they are not related to multi-modal evaluation. 

Thanks for your reply, I also want to ask is the MMBench V1.1 in the [opencompass leaderboard](https://rank.opencompass.org.cn/leaderboard-multimodal/?m=REALTIME) evaluated in a way as follows?
`python run.py --data MMBench_V11 --model model --verbose --nproc=1`
But MMBench_V11 is a [Internal Only file](https://github.com/open-compass/VLMEvalKit/blob/1f96d21fbd81ce0d49c257e9cd343be89a5f23fb/vlmeval/dataset/image_mcq.py#L56).
We can only access MMBench_TEST_EN_V11 and MMBench_TEST_CN_V11. So can we use the following commands
```
python run.py --data MMBench_TEST_EN_V11 --model model --verbose --nproc=1
python run.py --data MMBench_TEST_CN_V11 --model model --verbose --nproc=1
```
and average these two results to get the result in the leaderboard?


@ChuanyangZheng 

Yeah, and you need to submit the prediction file to https://mmbench.opencompass.org.cn/mmbench-submission to obtain the metrics.
Repository: open-compass/VLMEvalKit
Issue: 【提问】关于语音模型的评测
GitHub labels: Feature Request

hi，
想咨询一下，后续会有关于语音相关模型的评测吗？

@xx0412 会考虑接入，您有推荐的评测集吗

@kennymckormick 在试用 opencompass中， opencompass 能不能自定义支持语音的评测，还是基于 VLMEvalKit 框架来做语音模型的支持更合适？ 在探索阶段，不知如何选😄
Repository: open-compass/VLMEvalKit
Issue: 如何测评大模型的效率
Root cause: Unimplemented Feature Gap

测评使用起来非常方便，请问是否支持测评模型的效率？
如吞吐量这样的参数，后续是否会支持该功能。

Hi, @wenyu1009 , 
一个模型的效率，会由多方面的因素决定：硬件、推理引擎、量化策略等。 对模型效率的评测超出了本 repo 的范畴。你可能可以在 vllm 或是 lmdeploy 中找到相应资源。

好嘞，感谢回复
Repository: open-compass/VLMEvalKit
Issue: DocVQA测评无法正常使用
Root cause: Validation Gap

当我运行
```
CUDA_VISIBLE_DEVICES=1 python run.py --data DocVQA_TEST --model Qwen2-VL-2B-Instruct
```
得到以下结果
```
100%|█████████████████████████████████████████████▉| 5164/5168 [2:13:48<00:06,  1.53s/it][{'role': 'user', 'content': [{'type': 'image', 'image': '/vlmeval/images/DocVQA_TEST/16380.jpg', 'min_pixels': 1003520, 'max_pixels': 12845056}, {'type': 'text', 'text': 'What is the extension number of Jo Spach ?\nPlease try to answer the question with short words or phrases if possible.'}]}]
7240.
100%|█████████████████████████████████████████████▉| 5165/5168 [2:13:50<00:04,  1.48s/it][{'role': 'user', 'content': [{'type': 'image', 'image': '/vlmeval/images/DocVQA_TEST/57341.jpg', 'min_pixels': 1003520, 'max_pixels': 12845056}, {'type': 'text', 'text': 'What is the net worth in 2012 (Rs. Cr.)?\nPlease try to answer the question with short words or phrases if possible.'}]}]
2849
100%|█████████████████████████████████████████████▉| 5166/5168 [2:13:51<00:02,  1.48s/it][{'role': 'user', 'content': [{'type': 'image', 'image': '/vlmeval/images/DocVQA_TEST/61872.jpg', 'min_pixels': 1003520, 'max_pixels': 12845056}, {'type': 'text', 'text': 'What is the cost of supplies for the 3rd year?\nPlease try to answer the question with short words or phrases if possible.'}]}]
400
100%|█████████████████████████████████████████████▉| 5167/5168 [2:13:53<00:01,  1.48s/it][{'role': 'user', 'content': [{'type': 'image', 'image': '/vlmeval/images/DocVQA_TEST/57343.jpg', 'min_pixels': 1003520, 'max_pixels': 12845056}, {'type': 'text', 'text': 'What is the dividend payout in 1996?\nPlease try to answer the question with short words or phrases if possible.'}]}]
61
100%|██████████████████████████████████████████████| 5168/5168 [2:13:54<00:00,  1.55s/it]
Traceback (most recent call last):
  File "/VLMEvalKit/run.py", line 226, in <module>
    main()
  File "/VLMEvalKit/run.py", line 208, in main
    eval_results = dataset.evaluate(result_file, **judge_kwargs)
  File "/VLMEvalKit/vlmeval/dataset/image_vqa.py", line 48, in evaluate
    assert 'answer' in data and 'prediction' in data
AssertionError
```
我是最近拉取的仓库代码

Hi, @M3Dade , 

我们只支持 DocVQA_TEST 的推理，不支持评测 (因为没有 GroundTruth)。 你可以评测 DocVQA_VAL
Repository: open-compass/VLMEvalKit
Issue: liuhaotian/llava-v1.6-vicuna-7b 评测时报错
Root cause: Environment Incompatibility

![image](https://github.com/user-attachments/assets/3f59a672-b776-4ac5-8b09-d1662d63d7c7)
已经将model_path更换为自己下载模型的位置，transformer也已经更新到最新版，这里的报错有点奇怪，因为官方仓库里面也没有这个文件
![image](https://github.com/user-attachments/assets/ebb341b7-0272-4adc-9d31-c6ddba3a2024)


Please use this repo: https://huggingface.co/llava-hf/llava-v1.6-vicuna-7b-hf
Repository: open-compass/VLMEvalKit
Issue: how to run on multi-gpu with device_map='auto'
Root cause: Documentation Deficiency

when I use 34B llm model, single-gpu will report OOM.
so I set device_map='auto', but It seems cant use torchrun, It takes too much time to inference.
how to solve this problem?

Hi, which VLM are you using? 

Close the issue due to no response in weeks, plz reopen if needed

I use yi-34B LLM.
It seems visual encoder cost too much time.（32 frames per video）
Repository: open-compass/VLMEvalKit
Issue: Reproducing QWen2VL Results on Video Benchmarks with VLMEvalKit
Root cause: Unimplemented Feature Gap

Thanks for the evaluation toolkit. Could you provide scripts to reproduce QWen2VL results on Video Benchmarks?

It actually can not reproduced...

@aniki-ly @luohao123 

Qwen-VL Team is working on fixing the issues. 

> @aniki-ly @luohao123
> 
> Qwen-VL Team is working on fixing the issues.

hi, now this code can do that? 

@FangXinyu-0913 Has this issue already been solved?
Repository: open-compass/VLMEvalKit
Issue: MMBench_TEST 评估结果是否可以自动提交
Root cause: Unimplemented Feature Gap

评测 MMBench_TEST 后得到xlsx，需要手动提交到 MMB 的评估网站，是否可以在VLMEvalKit 中实现自动提交到评估服务器的功能？谢谢！

您好 @Sync-yxh, 目前我们尚不支持自动提交到评估服务器，需要您手动提交。该功能的开发目前暂时还未计划中。
Repository: open-compass/VLMEvalKit
Issue: MME-RealWorld-CN的评测集适配问题
Root cause: Environment Incompatibility

使用工具进行MME-RealWorld-CN评测后，发现结果xlsx中部分题目的选项内容存在错误：
如index=4820-5616，每个选项都少了第一个字符，排查发现选项内容是这样提取的：'A': item['multi-choice options'][0][4:]，
对于大部分选项是成立的（如"（A） 直走。"）但是对于这部分内容的选项会缺少一个空格（如“(A)图像的右侧”），导致解析从第二个字符开始。
请检查该问题是由于我下载的原始数据集版本特殊还是一个共性问题。

是json文件的问题，json文件已经进行了更新，请下载最新的

https://huggingface.co/datasets/yifanzhang114/MME-RealWorld-Base64/blob/main/MME-RealWorld-CN/MME_RealWorld_CN_base64_0.json
Repository: open-compass/VLMEvalKit
Issue: Qwen2-VL-2B-Instruct与榜单结果对不齐
Root cause: Environment Incompatibility

你好！请问为什么我测的Qwen2-VL-2B-Instruct与榜单上的结果差了那么多，Qwen2-VL需要哪些额外的配置吗，transformers我用的也是最新的：git+https://github.com/huggingface/transformers.git。
![image](https://github.com/user-attachments/assets/c0420075-d0d1-40f0-a065-be2ac51e9a75)

我在测试Qwen2-VL-2B-Instruct时显示了这个，请问会有影响吗，代码可以正常运行。
Unrecognized keys in `rope_scaling` for 'rope_type'='default': {'mrope_section'}
Unrecognized keys in `rope_scaling` for 'rope_type'='default': {'mrope_section'}
Unrecognized keys in `rope_scaling` for 'rope_type'='default': {'mrope_section'}

![image](https://github.com/user-attachments/assets/be1e9249-c4cb-4888-91b4-1b0b55fd7c38)


hi @helloworld01001. transformers版本请参照Qwen-VL-2官方设定
`pip install git+https://github.com/huggingface/transformers@21fac7abba2a37fae86106f87fcf9974fd1e3830 accelerate`
你测试时显示的这一问题可能是transformers版本不对应导致的，我测试时暂未遇见该报错。

Please follow the comment of @FangXinyu-0913 and use the corresponding transformer commit for evaluation.
Repository: open-compass/VLMEvalKit
Issue: 在MCQ任务中，一道题目具有多张图片，我应该如何构建框架需要的tsv数据集？
Root cause: Documentation Deficiency

在MCQ任务中，一道题目具有多张图片，我应该如何构建框架需要的tsv数据集？

Hi, @Nefefilibata ,
你可以参照 BLINK 的数据文件，这个数据集中选择题包含多张图片。
Repository: open-compass/VLMEvalKit
Issue: Adding Pixtral from Mistral Team
Root cause: Unimplemented Feature Gap

Hey @kennymckormick @jinyu121 @FangXinyu-0913 :

I was planning to add : https://huggingface.co/mistralai/Pixtral-12B-2409 model to the eval framework.

The Mixtral team recommends using vllm for inference. Is it okay to introduce that dependency for the repo ?

If its okay I can go ahead and create a PR for the same

Hi, @amitbcp , 
Since the repository supports hundreds of different MLLMs, we are not going to add the dependencies of a specific MLLM into the codebase dependency.

Sure @kennymckormick !

Will wait for Mixtral to integrate with transformers and then create a PR for the same
Repository: open-compass/VLMEvalKit
Issue: The evaluation result were all wrong after upgrade
Root cause: Environment Incompatibility

the evaluation got far worse metric after uprade, what could going wrong?

My code didn't change anything. 

That's wired, MMBench_DEV_CN, and MMStar, same model, previous evaluted as 95.8 and 56 repectively, but using latest vlmeval, the result got 79 and 48, the result far more lower,

Any reason for this?

> That's wired, MMBench_DEV_CN, and MMStar, same model, previous evaluted as 95.8 and 56 repectively, but using latest vlmeval, the result got 79 and 48, the result far more lower,
> 
> Any reason for this?

The previous results look weird: AFAIK, the latest state-of-the-art VLM can not achieve 90+% Acc on MMBench-DEV. Would you please provide the specific commit id you previously used so that  I can try to reproduce the error. 

We intended to do this, to remove possibility of model issue, we let MMBench_DEV_CN trained in dataset. So it's extremly weired so low evaluation metric on DEV datasets. (MMStar not trained, but our score previous is normal, not normal after changed evaluation pipline as well)

I have re-evaluated MMBench_DEV_CN and MMStar on multiple VLMs (which are evaluated more than half a year ago) with the latest code, including:

GPT4V_20240409, qwen_chat, llava_v1.5_7b, llava_v1.5_13b, deepseek_vl_7b, deepseek_vl_1.3b, MMStar. 

Basically, all evaluation results align with the accuracies we reported on Open VLM Leaderboard, with the maximum accuracy difference less than 2% (since there is some small changes about the choice extractor: gpt-3.5-0613 is no longer supported by OpenAI so we switch to gpt-3.5-0125). 
Moreover, we have provided all prediction files at: https://huggingface.co/datasets/VLMEval/OpenVLMRecords for user comparisons. 

If you want to go further with this issue, please provide an officially supported DATASET x MODEL pair, which you find the evaluation result is significantly different from our reported one. Also, a friendly reminder is, slight changes in the environment (for example, the version of some python packages) may also significantly affect the performance of your custom model. One can try to detect it by comparing old & new predictions. 



Oh, sorry for my mistake, it's normal now!

> Oh, sorry for my mistake, it's normal now!哦，抱歉，我的错误，现在已经正常了！

Could you teach me how to solve it?