zhiminy/EvalEng huggingface dataset:
Repository: open-compass/VLMEvalKit
Issue: 请问关于TextVQA的评测现在是正常的吗？
Root cause: Unimplemented Feature Gap

您好，之前在issue中有提到说关于TextVQA等VQA的数据集的评测仍然在完善中，请问现在关于TextVQA数据集的评测是否完善了呢？
Repository: open-compass/VLMEvalKit
Issue: MMbench1.1??
Root cause: Documentation Deficiency

I came across the benchmark table for Qwen2.5-VL and decided to reach out. I would like to evaluate my model using MMBench v1.1. How does MMBench v1.1 differ from v1, and how can I run the evaluation?



If you want to use MMBenchV1.1-EN-TEST/MMBenchV1.1-CN-TEST, you can use the command:
```
python run.py --model YOUR_MODEL --data MMBench_TEST_EN_V11/MMBenchV1.1-CN-TEST
```
and then commit the evaluation results to [MMBenchOnlineEvaluation](https://mmbench.opencompass.org.cn/mmbench-submission) to get the final score.

Thank you for your response. I have one more question. What are the differences between MMBench and MMBench1.1?



> Thank you for your response. I have one more question. What are the differences between MMBench and MMBench1.1?

Basically, MMBench 1.1 is an updated version of MMBench 1.0, with the following differences:
1. We remove some low-quality questions in MMBench 1.0 (the answer is wrong / the question can be solved without the image)
2. We add some new questions, which are relatively more difficult. 

MMBench 1.1 is the version that aligns with our ECCV paper: https://arxiv.org/abs/2307.06281

> > Thank you for your response. I have one more question. What are the differences between MMBench and MMBench1.1?
> 
> Basically, MMBench 1.1 is an updated version of MMBench 1.0, with the following differences:
> 
> 1. We remove some low-quality questions in MMBench 1.0 (the answer is wrong / the question can be solved without the image)
> 2. We add some new questions, which are relatively more difficult.
> 
> MMBench 1.1 is the version that aligns with our ECCV paper: https://arxiv.org/abs/2307.06281

Thank you for your kind and detailed answer. 👍
Repository: open-compass/VLMEvalKit
Issue: Qwen2.5-vl-instruct-3b and MathVerse_MINI
Root cause: Validation Gap

i run Qwen2.5-vl-instruct-3b with MathVerse_MINI_Vision_only， the score (9.0) is much lower than opencompass web -> 31.2 ， how can i debug this? 
Qwen2.5-vl-instruct-3b dont use custom prompt, MathVerse_MINI_Vision_only's prompt is simple vqa promot, "question" is only "According to the question shown in the image, please directly answer the question and provide the correct option letter, e.g., A, B, C, D."
is there any wrong with my run ?

Could you please share your running command and config?

> Could you please share your running command and config?

I have figured out the issue.  Thank you for your relay!

For the scoring prompt in MathVerse, Qwen2.5-72B's output is not just 0 or 1, but also includes some analysis, which causes the score determination to fail.

By changing "score = int(res) == 1" to "score = int(res[0]) == 1",  we can obtain an approximate score.

I hope this can help those who are unable to use OpenAI and are using local vLLM instead.

> > Could you please share your running command and config?
> 
> I have figured out the issue. Thank you for your relay!
> 
> For the scoring prompt in MathVerse, Qwen2.5-72B's output is not just 0 or 1, but also includes some analysis, which causes the score determination to fail.
> 
> By changing "score = int(res) == 1" to "score = int(res[0]) == 1", we can obtain an approximate score.
> 
> I hope this can help those who are unable to use OpenAI and are using local vLLM instead.

hello!Do you have any successful reproduction results of Qwen2.5VL-3B on the Mathverse dataset? Looking forward to your reply

  针对指令跟谁能力不强的评价模型，提取第一个出现的0或1字符
        
if FAIL_MSG in res or not re.search(r'[01]', res):
            log += f'Try {i}: output is {prediction}, res is {res}, failed to parse.\n'
        else:
        # 提取第一个出现的0或1字符
            extracted_value = re.search(r'[01]', res).group()
            log += f'Succeed (extracted {extracted_value})'
            return dict(log_score=log, score=int(extracted_value) == 1)
Repository: open-compass/VLMEvalKit
Issue: Does Video-MME apply to model Qwen2.5-VL-7B-Instruct with VLMEvalKit?
Root cause: Environment Incompatibility

The error is as follow, what does it mean？Is there anything I can do? 
>ERROR - RUN - run.py: main - 432: Model Qwen2.5-VL-7B-Instruct x Dataset Video-MME_0.5fps_subs combination failed: ('Connection broken: IncompleteRead(571046212 bytes read, 1053658347 more expected)', IncompleteRead(571046212 bytes read, 1053658347 more expected)), skipping this combination.

Hi @huanghuang358. I'm sorry for not getting back to you sooner. It seems to be an error related to file downloading or reading. Do you have more detailed error information?

> Hi [@huanghuang358](https://github.com/huanghuang358). I'm sorry for not getting back to you sooner. It seems to be an error related to file downloading or reading. Do you have more detailed error information?

You mentioned it! It turns out to be a network problem, and after preparing a local dataset it works.
Repository: open-compass/VLMEvalKit
Issue: InternVL2.5模型在USE_COT测试时精度均下降了？

InternVL2.5模型在USE_COT测试时精度均下降了？
请问除了Use_COT=“1”之外，还有哪些需要修改吗？

您好，在哪个bench上精度下降了呢


我测试MMMU时，精度下降，请问是测评的时候需要调整代码的哪些地方吗？
CUDA_VISIBLE_DEVICES=0,1,2 python run.py --model InternVL2_5-38B --data MMMU_DEV_VAL --verbose
CUDA_VISIBLE_DEVICES=0,1,2 USE_COT="1" python run.py --model InternVL2_5-38B --data MMMU_DEV_VAL --verbose
Repository: open-compass/VLMEvalKit
Issue: MCQ多图像推理 传入的message的图片的type都是text，但是推理能出现答案
Root cause: Validation Gap

你好，我在使用多图像推理的时候，我使用了Qwen2-VL-2B-Instruct的模型，我非常好奇如何做到多图像推理的，所以我去看了源码，但是我惊讶的发现，我在messages中传入的全是type为text，然后是路径的图片，并且这个路径还不一定是正确的，比如如下
{'role': 'user', 'content': [{'type': 'text', 'text': 'images/celeba_multiple_mcq/question_1_1.png'}, {'type': 'text', 'text': 'images/celeba_multiple_mcq/question_1_2.png'}, {'type': 'text', 'text': 'images/celeba_multiple_mcq/question_1_3.png'}, {'type': 'text', 'text': 'images/celeba_multiple_mcq/question_1_4.png'}, {'type': 'text', 'text': 'images/celeba_multiple_mcq/question_1_5.png'}, {'type': 'text', 'text': 'Which attribute is most frequent among the images?\nPlease answer the question and provide only the correct option letter, e.g., A, B, C, D.\n(A) Bags_Under_Eyes\n(B) Eyeglasses\n(C) Wearing_Hat\n(D) Male\n'}]}]

经过apply template之后会变成这样
['
system\nYou are a helpful assistant.
\n
user\nimages/celeba_multiple_mcq/question_1_1.pngimages/celeba_multiple_mcq/question_1_2.pngimages/celeba_multiple_mcq/question_1_3.pngimages/celeba_multiple_mcq/question_1_4.pngimages/celeba_multiple_mcq/question_1_5.pngWhich attribute is most frequent among the images?\nPlease answer the question and provide only the correct option letter, e.g., A, B, C, D.\n(A) Bags_Under_Eyes\n(B) Eyeglasses\n(C) Wearing_Hat\n(D) Male\n
\n
assistant\n']
但是即便是这样，好像也能输出一个结果，我觉得非常奇怪，在Qwen的示例里面都是给定了image格式，我不知道你有没有这样的问题，或者这种做法是正确的么？

感觉推理都是这样，我觉得很奇怪，我看到有个verbose参数可以打印出message，全部都是type为text，图片根本没有读入进去成为token，打印input id也是，如果有什么没注意的，请跟我说一声，verbose参数设置也可以看到

发现代码没错，可能是另一个仓库传入的时候，image不见了，所以变成了text，感谢
Repository: open-compass/VLMEvalKit
Issue: Inputs without JPEG conversion
Root cause: Unimplemented Feature Gap

Hello, I've notice that the input images used in VLMEvalKit are converted to JPEG format. I wonder if there is version that use the original images instead of JPEG format as input?

Hi @kydxh, we do not provide evaluation sets using original images, but the image quality before and after conversion should not change much, and the impact on the evaluation results is quite small. Which dataset do you want to evaluate?

Thank you. I want to evaluate some VQA datasets, such as ScienceQA, MMMU, Mathvista and TextVQA.
Besides, I want to ask that, if the bugs in evaluating TextVQA dataset mentioned before has been modified?
Repository: open-compass/VLMEvalKit
Issue: LLava GPT Eval无法对齐
Root cause: Documentation Deficiency

在LLava中，Prompt是输出字母https://github.com/open-compass/VLMEvalKit/blob/main/vlmeval/vlm/llava.py#L96，然而在GPT Eval
中，https://github.com/open-compass/VLMEvalKit/blob/main/vlmeval/evaluate/multiple_choice.py#L75 却要求模型输出选项的内容，这里的Diff是为啥呢？

@BlueBlueFF 
需要澄清的是，[multiple_choice.py#L75](https://github.com/open-compass/VLMEvalKit/blob/main/vlmeval/evaluate/multiple_choice.py#L75) 与推理多模态模型（如 LLaVA）无关，只是我们评测选择题时处理 **无法直接匹配选项字母** 情况的一种方式（即利用语言模型进行语义匹配）。
Repository: open-compass/VLMEvalKit
Issue: 如何在本地测评时只测评部分数据集
Root cause: Unimplemented Feature Gap

请问vlmeval在做video evaluation的时候，如何只测评benchmark中的部分数据集欸，比如取10个。
我把VideoBaseDataset基类中self.videos = videos改成self.videos = videos[:10]后，用config文件运行run.py文件后，看终端还是会运行很多个iteration。谢谢

您好，我们目前暂时不支持只评测一部分数据集，您是想对哪一个数据集的部分进行评测呢？

您可以考虑使用这种方法，固定采样数据集的前n项进行评测

![Image](https://github.com/user-attachments/assets/b6a6f599-b3ef-4e40-b978-27bb9ac21278)

谢谢您，已经解决了，在您的帮助下，我把video base类里的self.data = self.data.iloc[:n]了，感谢
Repository: open-compass/VLMEvalKit
Issue: OCRVQA、GQA数据集的prediction里跟有.句号，导致准确率为0
Root cause: Validation Gap

不知道为什么，模型在OCRVQA、GQA数据集评测时，提取的prediction里跟有.句号，导致准确率为0。请问该如何解决？

![Image](https://github.com/user-attachments/assets/3aa7d87d-1b0d-4d3a-8322-bbf7508e901b)

@WalkerWorldPeace 您好这是哪一个模型评测的结果呢，我们这边尝试复现一下

> [@WalkerWorldPeace](https://github.com/WalkerWorldPeace) 您好这是哪一个模型评测的结果呢，我们这边尝试复现一下

我在测试 https://huggingface.co/HaochenWang/ross-qwen2-7b  时碰到了类似的问题
Repository: open-compass/VLMEvalKit
Issue: 评测Video-MME数据集时报错
Root cause: Documentation Deficiency

评测Video-MME时报错：fps and nframe should be set at least one valid value, skipping this combination.
执行代码：
model=InternVL2_5-2B
datasets="Video-MME"
python run.py --data ${datasets} --model ${model} --verbose
报错如下：
Traceback (most recent call last):
  File "/new_code/VLMEvalKit/run.py", line 237, in main
    dataset = build_dataset(dataset_name, **dataset_kwargs)
  File "/new_code/VLMEvalKit/vlmeval/dataset/__init__.py", line 219, in build_dataset
    return cls(dataset=dataset_name, **kwargs)
  File "/new_code/VLMEvalKit/vlmeval/dataset/videomme.py", line 52, in __init__
    super().__init__(dataset=dataset, nframe=nframe, fps=fps)
  File "/new_code/VLMEvalKit/vlmeval/dataset/video_base.py", line 43, in __init__
    raise ValueError('fps and nframe should be set at least one valid value')
ValueError: fps and nframe should be set at least one valid value
[2025-03-12 14:10:48] ERROR - run.py: main - 428: Model InternVL2_5-2B x Dataset Video-MME combination failed: fps and nframe should be set at least one valid value, skipping this combination.
用的是最新版本的代码

您好，您没有设定fps或nframe，您可以用--fps或--nframe设定视频的评测抽帧要求，或使用 https://github.com/open-compass/VLMEvalKit/blob/main/vlmeval/dataset/video_dataset_config.py 内的快捷配置进行评测，示例如下：

`python run.py --data Video-MME_8frame --model InternVL2_5-2B --verbose`

(使用8帧进行评测，不带字幕信息）


好的，谢谢。我还遇到一个问题，评测MVBench数据集时遇到的：
[2025-03-12 15:07:25] ERROR - run.py: main - 428: Model MagicVL x Dataset MVBench combination failed: video/nturgbd/S010C002P017R002A005_rgb.avi does not exist, skipping this combination.
Traceback (most recent call last):
  File "/new_code/VLMEvalKit/run.py", line 237, in main
    dataset = build_dataset(dataset_name, **dataset_kwargs)
  File "/new_code/VLMEvalKit/vlmeval/dataset/__init__.py", line 219, in build_dataset
    return cls(dataset=dataset_name, **kwargs)
  File "/new_code/VLMEvalKit/vlmeval/dataset/mvbench.py", line 74, in __init__
    super().__init__(dataset=dataset, nframe=nframe, fps=fps)
  File "/new_code/VLMEvalKit/vlmeval/dataset/video_base.py", line 21, in __init__
    ret = self.prepare_dataset(dataset)
  File "/new_code/VLMEvalKit/vlmeval/dataset/mvbench.py", line 180, in prepare_dataset
    generate_tsv(dataset_path)
  File "/new_code/VLMEvalKit/vlmeval/dataset/mvbench.py", line 142, in generate_tsv
    raise Exception(
Exception: video/nturgbd/S010C002P017R002A005_rgb.avi does not exist
我看了下huggingface下载的数据里并没有这个视频路径，请问下知道是什么原因吗

您好，是因为这个原因：

![Image](https://github.com/user-attachments/assets/f151888c-7b78-4b1d-be5a-908868a3c899)

他们的huggingface库内将这些视频删除了，需要您自行下载放到对应路径

知道了，谢谢

请问这是哪里的信息，有对应的链接吗

https://huggingface.co/datasets/OpenGVLab/MVBench
Repository: open-compass/VLMEvalKit
Issue: Merge of Q-Bench-Video
Root cause: Unimplemented Feature Gap

Please consider adding Q-Bench-Video

https://huggingface.co/datasets/zhangzicheng/Q-Bench-Video/tree/main

We are working on it, and you can review it at https://github.com/open-compass/VLMEvalKit/pull/847
Repository: open-compass/VLMEvalKit
Issue: About prompts in different models and datasets
Root cause: Documentation Deficiency

I've notice that the prompts used in different VLMs and datasets may be different (eg., direct answer or COT).  May I ask how you set up the prompts? Are the prompts set up according to the dataset or the models' official configuration? 
Additionally, for the evaluation on the same dataset, do different models use the same prompt?

The prompt is set up based on these principles:
1. For some models that have `custom_prompt` requirements for specific types of datasets (or specific datasets), use the model's `custom_prompt`
2. If there are no above requirements, use the model's default prompt

Basically, prompts are set up according to the dataset or the models' official configuration. 

Thank you very much. How to determine whether a model has "custom prompt"? Is it based on the official configuration of the model? And what is the default prompt?
I want to evaluate some models, including LLaVA, Qwen2-VL, DeepSeek-VL, GPT-4o, etc. Do these models have custom prompts?

You can check it via the `custom_prompt` function defined in each model. Usually, the model's custom prompt is based on its official configuration (or from their developers modification)

Thank you very much. Additional, is it fair to compare the results using different prompts between different models?


For some models, the `custom prompt` has a significant impact on their performance (for example, responding only with options or simple vocabulary, or requiring detailed answers). If you want to reproduce the performance demonstrated by the model in the paper/report, you may need to follow the custom prompt provided by the official source.
You can also use our codebase to evaluate the model uniformly to ensure the reproducibility of the results.
All in all, the codebase of our aim you can see in ReadME (https://github.com/open-compass/VLMEvalKit?tab=readme-ov-file#-the-goal-of-vlmevalkit).



I got it. Thank you very much.
Repository: open-compass/VLMEvalKit
Issue: TextVQA results on LLaVA 1.5 and 1.6
Root cause: Configuration Error

1. I'm confused as to why for llava1.5, the model used is "liuhaotian/llava-v1.5-7b", but for llava 1.6, it's "llava-hf/llava-v1.6-vicuna-7b-hf" instead of "liuhaotian/llava-v1.6-vicuna-7b"?
2. I run the code on "llava_v1.5_7b" but got result only 21.9, which is much lower than the official results of LLaVA. Why did such results occur?
3. For "llava_next_vicuna_7b", when I use the original configuration code ("llava-hf/llava-v1.6-vicuna-7b-hf"), the acc is bout 63.9. However, when I change the model to "liuhaotian/llava-v1.6-vicuna-7b" ("llava_next_vicuna_7b": partial(LLaVA, model_path="liuhaotian/llava-v1.6-vicuna-7b"),), the accuracy is suddenly drops to 25.47.
4. I notice that for llava, the authors use OCR tokens in the inference process. But it seems that in VLMEvalKit, the OCR tokens are not used?

I'm really confused about these questions. Looking forward to your reply.
Repository: open-compass/VLMEvalKit
Issue: VLMEvalKit如何加载本地数据集？ How to load local dataset?
Root cause: Documentation Deficiency

如果我想使用本地的数据集进行评估，请问该修改哪些代码？ 
谢谢 


您可以选择将自己的数据集打包成tsv，然后放在~/LMUData目录下即可

> 您可以选择将自己的数据集打包成tsv，然后放在~/LMUData目录下即可

感谢 

想请问下如果是本地数据，除打包成tsv并放到对应目录下之外，是不是需要修改对应数据集中的DATASET_URL呢？如果是本地数据集，DATASET_URL这里直接写本地路径吗？

<img width="652" alt="Image" src="https://github.com/user-attachments/assets/68dbc3b1-bbba-45fa-8777-1ae57586c6d7" />

> 您可以选择将自己的数据集打包成tsv，然后放在~/LMUData目录下即可

请问是否可以指定目录？

> > 您可以选择将自己的数据集打包成tsv，然后放在~/LMUData目录下即可
> 
> 请问是否可以指定目录？

目录默认为 ~/LMUData，也可以通过环境变量 `LMUData` 进行指定

> 想请问下如果是本地数据，除打包成tsv并放到对应目录下之外，是不是需要修改对应数据集中的DATASET_URL呢？如果是本地数据集，DATASET_URL这里直接写本地路径吗？
> 
> <img alt="Image" width="652" src="https://private-user-images.githubusercontent.com/83212399/424856274-68dbc3b1-bbba-45fa-8777-1ae57586c6d7.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NDU3NDQ4NDEsIm5iZiI6MTc0NTc0NDU0MSwicGF0aCI6Ii84MzIxMjM5OS80MjQ4NTYyNzQtNjhkYmMzYjEtYmJiYS00NWZhLTg3NzctMWFlNTc1ODZjNmQ3LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTA0MjclMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUwNDI3VDA5MDIyMVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPThkYjFlNzlmNGE1MjNjZmIwYmRiODkzMDk5NmJjODg3Mjk0NzU4NTM3NjMzM2VlYjIxNDAyNGViNWE3ZDI5YzkmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.emosxQRPnvlPMzOQtVcgg3wiFamGdSEEn1Bb9YdB6nU">

若利用本地数据：
1. 如果本地数据为官方支持的数据集，不需要对原本标注的 DATASET_URL 进行修改
2. 如果本地数据为用户自己构建的数据集，需要在 DATASET_URL 中增加条目：
    2.1. Key 为 dataset 名称 （自行指定，指定后评测时使用对应名称）
    2.2. URL 为一 fake URL 即可，仅最后部分需要与本地文件名称相同，形如：`http://your.custom.url/XXX/YYY/file_name.tsv`
Repository: open-compass/VLMEvalKit
Issue: IndexError: piece id is out of range.
Root cause: External Dependency Breakage

I want to inference llava v1.5, but encountered error "IndexError: piece id is out of range". The environment I used is "llava 1.1.3, transformers 4.31.0, sentencepiece 0.1.99, torch 2.0.1"
The detailed error information is listed as follows:
  output = self.tokenizer.batch_decode(output_ids, skip_special_tokens=True)[
  File "/root/anaconda3/envs/llava16/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 3485, in batch_decode
    return [
  File "/root/anaconda3/envs/llava16/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 3486, in <listcomp>
    self.decode(
  File "/root/anaconda3/envs/llava16/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 3525, in decode
    return self._decode(
  File "/root/anaconda3/envs/llava16/lib/python3.10/site-packages/transformers/tokenization_utils.py", line 931, in _decode
    filtered_tokens = self.convert_ids_to_tokens(token_ids, skip_special_tokens=skip_special_tokens)
  File "/root/anaconda3/envs/llava16/lib/python3.10/site-packages/transformers/tokenization_utils.py", line 912, in convert_ids_to_tokens
    tokens.append(self._convert_id_to_token(index))
  File "/root/anaconda3/envs/llava16/lib/python3.10/site-packages/transformers/models/llama/tokenization_llama.py", line 204, in _convert_id_to_token
    token = self.sp_model.IdToPiece(index)
  File "/root/anaconda3/envs/llava16/lib/python3.10/site-packages/sentencepiece/__init__.py", line 1045, in _batched_func
    return _func(self, arg)
  File "/root/anaconda3/envs/llava16/lib/python3.10/site-packages/sentencepiece/__init__.py", line 1038, in _func
    raise IndexError('piece id is out of range.')
How can I solve the problem?

It seems a problem on sentencepiece and transformers, you can refer to https://github.com/haotian-liu/LLaVA for solutions.

I have already verified that the versions transformers and sentencepiece meet the requirements specified in the corresponding version of llava, but this error still persists. 
Besides, I wonder when I run llava v1.5 in VLMEvalKit, is the version of llava I need v1.1.3 or the latest v1.2.2.post1?

> I have already verified that the versions transformers and sentencepiece meet the requirements specified in the corresponding version of llava, but this error still persists. Besides, I wonder when I run llava v1.5 in VLMEvalKit, is the version of llava I need v1.1.3 or the latest v1.2.2.post1?

For llava v1.5, it is recommended to use the earlier versions of LLaVA for evaluation.
Repository: open-compass/VLMEvalKit
Issue: 怎样对llava codebase微调得到的lora模型进行评测？
Root cause: Documentation Deficiency

请问使用https://github.com/haotian-liu/LLaVA/ 框架微调得到的lora模型，应该怎么进行评测呢？


我看到https://github.com/haotian-liu/LLaVA/ 这个仓库里面有https://github.com/haotian-liu/LLaVA/blob/main/scripts/merge_lora_weights.py 支持合并的权重，但是这个权重和https://huggingface.co/liuhaotian/llava-v1.5-7b 的形式不太一样，huggingface中没有打包vision encoder的权重，而使用上述代码得到的权重会包括vision tower的权重，这个问题怎么能优雅地解决呢？vlmevalkit中llava的示例似乎是希望一个huggingface格式的权重。谢谢

![Image](https://github.com/user-attachments/assets/0fa2b008-7f2f-4135-b534-249c7f9a4178)

![Image](https://github.com/user-attachments/assets/e2777180-d2c9-4049-b221-0d176df16237)
huggingface中没有打包vision部分权重

There can be various approaches: 
1. Create a checkpoint that complies with HuggingFace format, and then use our sample code for inference.
2. Serve an API (OpenAI format) based on your checkpoint, and then conduct the evaluation based on API.
Repository: open-compass/VLMEvalKit
Issue: 测试minigpt时根据模型输出显示并未读到图片
Root cause: Validation Gap

我在测试minigpt-med时，模型输出如图

![Image](https://github.com/user-attachments/assets/2d54bbd8-c297-4023-9dff-ab1ed82edb0f)
可以发现它并未读取到图片。
但我检查了vis_processor和embs可以确定我给出的mixed_embs中是包含图像信息的，这是为什么？

mixed_embs的信息如下图

![Image](https://github.com/user-attachments/assets/64cba7dc-b0c2-4f9e-a9d9-0e66ca65d224)

您好，可以首先确认从模型传入的messages这一变量中是否含有图像，再向下debug查看

![Image](https://github.com/user-attachments/assets/78fd7604-8545-4e76-a0ea-b005fa3983de)您好，我在generate inner这个函数中可以确定传来的message成功解析出了prompt和image path，在upload image和encode image我有尝试直接传入图片路径和传入pil打开后的图片两种方式，但得到的输出是一样的，根据后面的内容显示mixed embs应该也合并了text和image的token

问题已解决，更换prompt就可以，谢谢！


好的，如果您有其他的问题，欢迎在此issue下留言。
Repository: open-compass/VLMEvalKit
Issue: Why resize images in MMMU for DeepSeek-VL2?
Root cause: Documentation Deficiency

I notice that in evaluation of DeepSeek-VL2 on MMMU, the code resize the first image. But I don't know why. Could you please tell me the reasons?

<img width="467" alt="Image" src="https://github.com/user-attachments/assets/73bc87ca-29da-4874-bd2d-bd209209c29b" />

(codes from "generate_inner" in VLMEvalKit/vlmeval/vlm/deepseek_vl2.py)

The corresponding Pull Request was created by a member from the official DeepSeekVL2 team. We may need to seek help from the PR author @gnobitab 

Thank you very much.
Repository: open-compass/VLMEvalKit
Issue: How to set detailed configuration when using lmdeploy?
Root cause: Documentation Deficiency

For example, I want to test the Qwen2-VL-2B-Instruct model with some configurations like `max_pixels= 6272000`. How to set this when using lmdeploy?

What's more, how to use custom prompt when using lmdeploy?

Can you provide the example command?

@PhoenixZ810 

Hello @davidluciolu , this setting can be configured for the incoming text/images when passing messages, please refer to to https://github.com/InternLM/lmdeploy/blob/main/docs/en/multi_modal/qwen2_5_vl.md#offline-inference for detailed solution.
Repository: open-compass/VLMEvalKit
Issue: Could you please release the requirements.txt for each LVLM?
Root cause: Documentation Deficiency

Thank you for your excellent contribution!

Could you kindly provide the `requirements.txt` for each of the supported VLMs? I noticed some discrepancies in the `Readme`, such as `InternVL` requiring version `4.37.2` instead of `4.37.0`. We are conducting a large-scale LVLM experiment, and it would be incredibly helpful if you could share the requirements that work for you. Thanks so much!

`4.33.0` seems to not work for the `QWen series`. I found it should be `4.45.0` for `QWen2VL` and `4.49.0` for `QWen2.5VL`.
Repository: open-compass/VLMEvalKit
Issue: Can VLMEvalKit support API models using OpenRouter API
Root cause: Documentation Deficiency

I have a OpenRouter API key which can access gpt and Claude etc. Can I use OpenRouter apikey for api model

Hi, @UUUserna ,

That depends on whether `OpenRouter` complies with OpenAI format API. If so, you can directly adopt the pre-defined class GPT4V (based on OpenAIWrapper). 

If not, you need to implement a new API wrapper for OpenRouter on your own. 

It seems that OpenRouter is compatible with OpenAI SDK, so you can directly use the GPT4V class and only modify the key and api_base