zhiminy/EvalEng huggingface dataset:
Repository: open-compass/VLMEvalKit
Issue: 执行MVBench评测时，发现两个问题，修改已放在评论中的附件里
Root cause: Validation Gap


1.由于MVBench部分数据视频被删除，导致评测时这部分的视频推理结果无法生成，评测结果无法正常读取，出现报错；
2.在使用evaluate方法时，对推理结果没有进行合理地处理，没有去掉空格，导致评测分数不对
以上两个问题，我修改了部分代码，放在附件中的patch中
[eval.patch](https://github.com/user-attachments/files/17956417/eval.patch)


Hi @Xiaoqi-Zhou916. 感谢您提出的问题及给出的解决方案，我们会在今日对这个问题进行修复～

修复了嘛 最新的代码还是有问题，报部分视频不存在的错误

![image](https://github.com/user-attachments/assets/c088b8dd-a55e-4c38-9dad-fca39f2ca6d6)
mvbench的部分视频可能需要您自己手动下载，或者您是在推理完成评估之后报的错吗，是否有详细的报错信息呢

如果要跳过这些没下载的数据，具体代码需要改动哪里

![image](https://github.com/user-attachments/assets/15817f1f-e7ef-4637-9ff9-d92361c8ffb7)
需要在这里把else后面的报错去掉，这样构建tsv的时候就会跳过那些数据，但这样会使得mvbench这个数据集缺少一部分数据，可能会影响对模型性能的评估
Repository: open-compass/VLMEvalKit
Issue: Invalid url of English and Chinese split of OlympiadBench
Root cause: Configuration Error

The links below are invalid: 
https://opencompass.openxlab.space/utils/VLMEval/OlympiadBench_EN.tsv https://opencompass.openxlab.space/utils/VLMEval/OlympiadBench_CN.tsv 

Hi there,

Thank you for reaching out and for your patience.

We've noticed that the link provided for OlympiadBench is indeed not working as expected. We appreciate your understanding, and our team is actively working on resolving this issue to restore access as soon as possible.

We sincerely appreciate your attention to detail and bringing this to our notice. Once the link is fixed, we will ensure to update this thread with the correct information.

Hello,
We are sorry to inform that we will no longer provide separate support for OlympiadBench_EN and OlympiadBench_CN. Instead, VLMEvalKit now supports the unified [OlympiadBench](https://github.com/open-compass/VLMEvalKit/blob/2a66d654108837f5af4159e5543f27f407ac8de1/vlmeval/dataset/image_vqa.py#L435C9-L435C23) dataset.
We appreciate your understanding regarding this change. If you have any questions or need assistance with the transition, please do not hesitate to contact us.

Thank you for your understanding.

Thank you for your answer.
Repository: open-compass/VLMEvalKit
Issue: the infer speed is too slow
Root cause: Configuration Error

![image](https://github.com/user-attachments/assets/e82c9eff-1691-45ca-8441-17307a13e199)
I tested on textvqa based on the supported model, but why is the inference speed so slow as shown in the figure

I also tested several other data sets, and likewise infer was slow

@kennymckormick I'm sorry to bother you. Can you give me some advice

Hi, @TuuSiwei ,
That's not normal, which model are you evaluating?
BTW, the reason might be that your output is too long (hundreds of tokens, etc.)

I appreciate your insightful advice. I found that "max_new_tokens" were initialized to 1024 in the model file. As follows
https://github.com/open-compass/VLMEvalKit/blob/bb0fc6b9a6ab7f9fb4f508aad71da8ab1bf2620c/vlmeval/vlm/mgm.py#L61
Problem solved after I narrowed it down！
Repository: open-compass/VLMEvalKit
Issue: Can not download MM-Math and DynaMath dataset from the default links
Root cause: External Dependency Breakage

DynaMath: https://opencompass.openxlab.space/utils/VLMEval/DynaMath.tsv
MM-Math: https://opencompass.openxlab.space/utils/VLMEval/MM-Math.tsv

Hello, the links are not working for downloading. Could you please update them?

The links have been fixed now.
Repository: open-compass/VLMEvalKit
Issue: conflict between torch2.5.1 and typing_extension4.7.1 
Root cause: External Dependency Breakage

ERROR: Cannot install vlmeval and vlmeval==0.1.0 because these package versions have conflicting dependencies.

The conflict is caused by:
    vlmeval 0.1.0 depends on typing_extensions==4.7.1
    openai 1.3.5 depends on typing-extensions<5 and >=4.5
    torch 2.5.1 depends on typing-extensions>=4.8.0
    vlmeval 0.1.0 depends on typing_extensions==4.7.1
    openai 1.3.5 depends on typing-extensions<5 and >=4.5
    torch 2.5.0 depends on typing-extensions>=4.8.0
Repository: open-compass/VLMEvalKit
Issue: Does VLMEvalKit support Paged Attention evaluation?
Root cause: Unimplemented Feature Gap

Thanks for your great work.
I would love to know if VLMEvalKit supports the Paged Attention inference? If it does, how can I use the Paged Attention for inference? Thanks.

Hi,
Thank you for reaching out regarding the use of Paged Attention with VLMEvalkit.

Currently, VLMEvalkit does not provide specific support for Paged Attention inference. If you intend to utilize Paged Attention, please verify whether your model supports this feature before proceeding.

We appreciate your understanding and are happy to assist with any further questions or clarifications you may need.
Repository: open-compass/VLMEvalKit
Issue: 用Chartqa测评LLama3.2-11b-vision时，结果全部为0
Root cause: Validation Gap

在测评Llama3.2-11b-vision时，用ChartQa，中间有过程，显示了2500条数据的过程，但结果全部为0，结果如下

To disable this warning, you can either:
        - Avoid using `tokenizers` before the fork if possible
        - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
        - Avoid using `tokenizers` before the fork if possible
        - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
[2024-12-03 16:21:54,323] INFO - RUN - run.py: main - 393: The evaluation of model Llama-3.2-11B-Vision-Instruct x dataset ChartQA_TEST has finished! 
[2024-12-03 16:21:54] INFO - run.py: main - 393: The evaluation of model Llama-3.2-11B-Vision-Instruct x dataset ChartQA_TEST has finished! 
[2024-12-03 16:21:54,323] INFO - RUN - run.py: main - 394: Evaluation Results:
[2024-12-03 16:21:54] INFO - run.py: main - 394: Evaluation Results:
[2024-12-03 16:21:54,325] INFO - RUN - run.py: main - 400: 
--------------  -
test_augmented  0
test_human      0
Overall         0
--------------  -
[2024-12-03 16:21:54] INFO - run.py: main - 400: 
--------------  -
test_augmented  0
test_human      0
Overall         0
--------------  -

Hi,
Thank you for your inquiry.
Once the inference process has successfully completed, an .xlsx file will be generated in the designated output directory.
To provide you with more specific details regarding the contents of the .xlsx file, it would be helpful to know the particular setup or context in which the inference process is being run. 
Thanks.
Repository: open-compass/VLMEvalKit
Issue: LLAVA-one-vision MMMU 性能评测差别很大
Root cause: Configuration Error

这是评测的脚本：torchrun --nproc-per-node=1 run.py --data MMMU_DEV_VAL --model llava_onevision_qwen2_0.5b_ov --verbose
评测下来的指标是：
split                                validation           dev
Overall                              0.3522222222222222   0.31333333333333335
Accounting                           0.43333333333333335  0.0
Agriculture                          0.36666666666666664  0.2
Architecture_and_Engineering         0.23333333333333334  0.2
Art                                  0.3333333333333333   0.0
Art_Theory                           0.43333333333333335  0.6
使用的 gpt3.5 进行评测，和llava-one-vision 论文里面给的结果0.31，有很大的出入。这是怎么回事呢。

Hi, @linxid ,
我们推荐使用官方的 OPENAI API 进行评测，可以确认下你这边是否使用的是 OPENAI 官方 API。
依据我们的测试结果，与 0.31 没有显著差异 （见下图）：
![image](https://github.com/user-attachments/assets/2a5512ef-9f19-412e-9cd7-f8cc8ae81d77)


![image](https://github.com/user-attachments/assets/b8361ae4-232f-4b39-b382-7df87a9132af)
看样子还是有些差别,请问差别是来自哪里呀
Repository: open-compass/VLMEvalKit
Issue: llama3.2_11b_vision用MathVista_MINI测评时，由于不设置.env文件，会报错
Root cause: Documentation Deficiency

llama3.2_11b_vision用MathVista_MINI测评时，由于不设置.env文件，会报错。然后我又建立了.env文件，内部是空的，再执行，也一样报错

MathVision_MINI也是一样，.env文件内的Openai 密钥为空，会报错

MathVista和MathVision，必须要openai的密钥吗？还有哪几个模型必须要密钥，否则无法run通？

跟模型没关系，跟数据集有关系


Hello,

Thank you for reaching out.

I hope this message finds you well. I would like to inform you that certain complex benchmarks, such as MMVet and MathVista, require an OpenAI API key to retrieve accurate results. Should you wish to run these specific benchmarks, please ensure you have provided the necessary OpenAI API key. You can verify which benchmarks necessitate an OpenAI API key by reviewing the [run.py](https://github.com/open-compass/VLMEvalKit/blob/2a66d654108837f5af4159e5543f27f407ac8de1/run.py#L328) script in our repository.

好的，感谢
Repository: open-compass/VLMEvalKit
Issue: MMMU test set
Root cause: Unimplemented Feature Gap

It will be nice if VLMEvalKit can generate result for MMMU test set,

Hi, @mactavish91 ,
The feature has been updated in https://github.com/open-compass/VLMEvalKit/pull/75. 

@kennymckormick Thank you for you great effort! I wonder if there is a version that can be submitted to the MMMU leaderboard.
Repository: open-compass/VLMEvalKit
Issue: 您好，能否增加一些中文的评估benchmark啊？比如CMMMU，支持的中文benchmark太少了
Root cause: Unimplemented Feature Gap


Hi,

Thank you for your interest and engagement with our project.

We are pleased to inform you that we now support the evaluation of Splitted Chinese benchmarks such as MMBench-CN. Should you wish to contribute by adding new Chinese benchmarks, we would be delighted to review your pull request.

Hi, @GoogleAlphaZero , 
CMMMU has been incorporated in VLMEvalKit. 
If you have more suggestions, don't hesitate to comment here and we'll consider.
Repository: open-compass/VLMEvalKit
Issue: AutoScoringJudge文件
Root cause: Algorithmic Error


判断
str1 = "$(a+b)^2=a^2+b^2+2ab$"
str2 = "$(a+b)^2=a*a+b*b+2ab$"
是否等价时，返回错误。

![1](https://github.com/user-attachments/assets/26ecd861-b5f8-46a2-8366-156fff3e9b51)
这个是否可以用后面 equation_equal函数中的方式判断
equation = Eq(lhs_expr, rhs_expr)
simplified_eq = simplify(equation.lhs - equation.rhs)

Hi,
Thank you for bringing this to our attention and for your thoughtful input.

After careful consideration, we believe that `str1 = "$(a+b)^2=a^2+b^2+2ab$"` and `str2 = "$(a+b)^2=aa+bb+2ab$"` exhibit differences in notation that make them not entirely equivalent. Therefore, we would prefer to keep the judgment as it is to maintain mathematical accuracy.

We appreciate your understanding and welcome any further discussion on this topic.
Repository: open-compass/VLMEvalKit
Issue: MathVerse_MINI 指向的 DATASET_URL 失效
Root cause: External Dependency Breakage

[https://github.com/open-compass/VLMEvalKit/blob/main/vlmeval/dataset/image_vqa.py#L269](url)
你好！这里 MathVerse_MINI 对应的 url 似乎失效了
![image](https://github.com/user-attachments/assets/4e8fdd95-67bc-46cc-9f4d-0b3e0460f326)


另外，OlympiadBench_EN 和 OlympiadBench_CN 对应的 url 似乎也失效了？
[https://github.com/open-compass/VLMEvalKit/blob/main/vlmeval/dataset/image_vqa.py#L439](url)

Hi there,

Thank you for reaching out and for your patience.

We have reviewed the [MathVerse_MINI.tsv](http://opencompass.openxlab.space/utils/benchmarks/MathVerse/MathVerse_MINIV.tsv) file, and we can confirm that it remains valid in our network. However, we've noticed that the link provided for OlympiadBench is indeed not working as expected. We appreciate your understanding, and our team is actively working on resolving this issue to restore access as soon as possible.

We sincerely appreciate your attention to detail and bringing this to our notice. Once the link is fixed, we will ensure to update this thread with the correct information.

Hello,
We are sorry to inform that we will no longer provide separate support for OlympiadBench_EN and OlympiadBench_CN. Instead, VLMEvalKit now supports the unified [OlympiadBench](https://github.com/open-compass/VLMEvalKit/blob/2a66d654108837f5af4159e5543f27f407ac8de1/vlmeval/dataset/image_vqa.py#L435C9-L435C23) dataset.
We appreciate your understanding regarding this change.  If you have any questions or need assistance with the transition, please do not hesitate to contact us.

Thank you for your understanding.

Hi, MathVerse_MINI 目前已可用，此外，VLMEvalKit 目前仅支持 OlympiadBench，不再支持 EN / CN 的 split
Repository: open-compass/VLMEvalKit
Issue: The locally saved glm-4v-9b model encounters errors during evaluation, which are related to the way the tokenizer is loaded
Root cause: Configuration Error

local model: https://huggingface.co/OPEA/glm-4v-9b-int4-sym-inc
![image](https://github.com/user-attachments/assets/f873c481-cb1f-4bb7-9695-6475a90ccb4b)

See issue in glm-v-9b for a similar error：
https://huggingface.co/THUDM/glm-4v-9b/discussions/35



Hi, @WeiweiZhang1 , 
For custom models, you first need to add a wrapper and make sure the inference can be successfully performed. Then you can use VLMEvalKit for multi-modal evaluation. 

You can reopen the issue if there is any further problems
Repository: open-compass/VLMEvalKit
Issue: 测评MMVET，用llama3.2-11b-vision时，openai的密钥为空，会报错
Root cause: Validation Gap

测评MMVET，.env文件中的密钥为空，会报错，使用模型为llama3.2-11b-vision

Hi, @GoogleAlphaZero , 
MMVet 必须使用 API 进行测试

> Hi, @GoogleAlphaZero , MMVet 必须使用 API 进行测试

感谢！可否给出，哪个模型必须要API，哪个可以不要API

Hello,

Thank you for reaching out.

I hope this message finds you well. I would like to inform you that certain complex benchmarks, such as MMVet and MathVista, require an OpenAI API key to retrieve accurate results. Should you wish to run these specific benchmarks, please ensure you have provided the necessary OpenAI API key. You can verify which benchmarks necessitate an OpenAI API key by reviewing the [run.py](https://github.com/open-compass/VLMEvalKit/blob/2a66d654108837f5af4159e5543f27f407ac8de1/run.py#L328) script in our repository.

好的，感谢各位
Repository: open-compass/VLMEvalKit
Issue: 与榜单指标差距过大，MTVQA和MME
Root cause: Environment Incompatibility

用llama3.2-11b-vision
测试MTVQA的指标，您发布的为Overall:15.3，我实测为Overall:22.3812，
测试MME，您发布的指标为Perception:1380.9、Cognition:439.6，我实测为perception： 1319.5、Cognition:283
请问是什么原因？感谢，辛苦您解答或核实下榜单






@FangXinyu-0913 

您好，想问一下您是用我们的codebase进行评测吗，以及评测时的参数设定是怎么设定的呢，我们这边按您的设定再评测试一下

没改参数，只是config内设置了llama3.2_11b_vision的路径，.env中设置了openai的密钥。运行命令为
python run.py --data MME MTVQA_TEST \
    --model Llama-3.2-11B-Vision-Instruct --nproc 4 \
    --work-dir /workspace/mydir \
    --verbose
    
  感谢！

我们这边重新测试了一遍，和榜单上的指标没有过大差别
![image](https://github.com/user-attachments/assets/0ef84811-0153-4335-9fe5-fa3b9c55b26c)
![image](https://github.com/user-attachments/assets/842a32be-d80e-4e65-8d43-bbd5c6b2c94b)
这是我们的环境
![image](https://github.com/user-attachments/assets/d033b8a9-7161-42ad-ab03-cac67bd84275)
想问一下您的环境配置是怎样的呢

Hi @GoogleAlphaZero, 由于您过久未回复这个问题，因此暂时关闭这个issue了。如果有问题的话欢迎您随时reopen或在此处回复联系我，感谢您的理解
Repository: open-compass/VLMEvalKit
Issue: Local judge LLM?
Root cause: Documentation Deficiency

Is it possible to use locally deployed LLM like [LLaVa-Critic](https://llava-vl.github.io/blog/2024-10-03-llava-critic/) as judge LLM instead of calling GPT4 API?

Hi, @lyzhongcrd , 
Yeah. However, we recommend you use the same LLM as the judger for all LMMs to make it comparable. 
For MCQ or Y/N benchmarks, when LLMs are only used as choice extractor for more accurate evaluation, using different LLMs will not lead to significantly different results. 

@kennymckormick Could you tell me how to use locally deployed LLMs as judge LLM in the VLM eval kit? Thanks.

> @kennymckormick您能告诉我如何使用本地部署的LLM作为VLM评估套件中的判断LLM吗？谢谢。

请问一下贴主，如果我要评估的模型是VLM，本地部署的模型应该是LLM还是VLM呢

@kennymckormick Same question +1, how to use locally deployed LLMs as judge LLM in the VLM eval kit? Thanks.

> Hi, [@lyzhongcrd](https://github.com/lyzhongcrd) , Yeah. However, we recommend you use the same LLM as the judger for all LMMs to make it comparable. For MCQ or Y/N benchmarks, when LLMs are only used as choice extractor for more accurate evaluation, using different LLMs will not lead to significantly different results.

Hi, How much would it cost to evaluate on all major benchmarks with GPT4 API ?  and how to use locally LLMs as judge LLM ?
Repository: open-compass/VLMEvalKit
Issue: The image quality in VLMEvalKit is compressed and lower than that provided by Benchmark itself.
Root cause: Algorithmic Error

The image quality in VLMEvalKit is compressed and lower than that provided by Benchmark itself.

Hi, @Sync-yxh
During our pre-processing, we have converted images to JPEG format. We will try to build another version that do not perform the JPEG conversion and do the evaluation again to see the gap.
We have noticed this in [issues/578](https://github.com/open-compass/VLMEvalKit/issues/578) and are working on building a new version.
Repository: open-compass/VLMEvalKit
Issue: OlympiadBench数量跟论文不一致
Root cause: Documentation Deficiency

代码里的https://opencompass.openxlab.space/utils/VLMEval/OlympiadBench.tsv这个文件里question只有6765道题，但我看原论文里数学有6142，物理有2334，加起来有8476道题才对，请问这个数据是做了筛减吗？

Hi, @wdp-007 , 
We don't keep those Theorem Proving questions in the original dataset since those questions need to be verified by human.
Repository: open-compass/VLMEvalKit
Issue: local judge model not work, meet error when model.working()
Root cause: Validation Gap

i have followed quick start to deploy local llm as judge. But I meet this error 

**_assert model.working(), ('MathVista evaluation requires a working OPENAI API\n' + DEBUG_MESSAGE)
           ^^^^^^^^^^^^^^^
AssertionError: MathVista evaluation requires a working OPENAI API_**

i have get the model form judge build
![image](https://github.com/user-attachments/assets/aa17b4dc-eded-4a05-8746-911730d3d76c)

here is my .env
_**OPENAI_API_KEY=sk-123456
OPENAI_API_BASE=http://0.0.0.0:23333/v1
LOCAL_LLM=internlm/internlm2-chat-1_8b**_
and my deployment command is
**_CUDA_VISIBLE_DEVICES=3 lmdeploy serve api_server internlm/internlm2-chat-1_8b --server-port 23333_**
start eval command is 
**_python run.py --data MathVista_MINI --model InternVL2_5-8B --work-dir /data/eval_result --judge internlm/internlm2-chat-1_8b --nproc 1_**



Hi, the reason is that you local deployed judge model is not working. Would you please check if the api is actually working? 

Close stale issue. Feel free to reopen for further updates.
Repository: open-compass/VLMEvalKit
Issue: awq model eval support request
Root cause: Documentation Deficiency

when i use awq internvl model for evaluation, it will show that the model structure is wrong ( i use internvl config register)

Hi, @sanbuphy ,
You can deploy the AWQ model with lmdeploy and then evaluate w. the lmdeploy wrapper. 

Check the document here: https://github.com/open-compass/VLMEvalKit/blob/main/docs/en/EvalByLMDeploy.md
Repository: open-compass/VLMEvalKit
Issue: 有InternVL 2.5和deepseek vl2的支持计划么


internvl2.5目前已经支持了～另一个您指的是deepseek VL2吗，目前还在规划当中

> internvl2.5目前已经支持了～另一个您指的是deepseek VL2吗，目前还在规划当中

 似乎仓库中没有 internvl2.5?


在config里是有的，可以调用，我们今天也补充一下readme～

> 在config里是有的，可以调用，我们今天也补充一下readme～

我测了一下8B的，ChartQA指标是82.9，好像和paper里有差距，差了2个点

Hi, @JohnTang93 , 
目前两类模型均已支持
Repository: open-compass/VLMEvalKit
Issue: AUTO_SPLIT=1  如何搭配CUDA_VISIBLE_DEVICES=使用
Root cause: Architectural Constraint

和别人共享机器，只有4,5,6,7卡可以用时，有auto_split和visible_device一起使用的需求。但是目前的代码一起用会报cuda error错误。

您好@ZihaoZheng98! 想问一下您auto_split是针对哪个某些模型呢～部分模型因为一些原因，只支持在8卡上进行auto_split

> 和别人共享机器，只有4,5,6,7卡可以用时，有auto_split和visible_device一起使用的需求。但是目前的代码一起用会报cuda error错误。



Hi, @ZihaoZheng98 , 
目前的 model split 策略暂未考虑起始 GPU index 非 0 的情况，可以在对应模型的 building 过程中设置 device 以绕过
Repository: open-compass/VLMEvalKit
Issue: MME-realworld benchmark如何设置local path？
Root cause: Documentation Deficiency

代码中自动下载数据总是因网络停止，单独下载后如何修改代码来使用嗯？

解决了 image_mcq.py的665行改成本地路径就行。不知道VLMEvalKit是否可以在run.py里增加一个参数来统一指定本地路径呢？

Hi, @ZihaoZheng98 ,
可以通过设置 LMUData 环境变量来控制：

1. VLMEvalKit 默认将所有数据保存在 `$LMUData` （可设置的环境变量） 目录下，代码中由 `LMUDataRoot()` 获取这一路径；当 LMUData 变量未设置时，将默认为 `$HOME/LMUData/`
2. 如需使用本地的数据文件，下载完成后复制入 `$LMUData` 即可


参考代码：https://github.com/open-compass/VLMEvalKit/blob/e42058a1f5556b462c1840bf0ea43b47e19e471a/vlmeval/smp/file.py#L68
Repository: open-compass/VLMEvalKit
Issue: 关于 RealWorldQA 以及 MMMU_DEV_VAL 里面部分条目的问题
Root cause: Validation Gap

你好！我在研究 RealWorldQA 的时候发现，某些题目在处理的时候似乎出现了选项的遗漏和错乱，并且和原始的正确答案对不上了。例如 index=105 的题目，在 huggingface 上找到原版题目：[https://huggingface.co/datasets/lmms-lab/RealWorldQA/viewer/default/test?p=1&row=105](url)
![image](https://github.com/user-attachments/assets/abc19543-c077-4c25-8e15-873ed83cd3c7)
与此同时，从 [https://github.com/open-compass/VLMEvalKit/blob/main/vlmeval/dataset/image_mcq.py#L87](url) 下载的 tsv 文件，关于这道题的选项则变成了两个，分别是原题的 A 和 C 选项：
![image](https://github.com/user-attachments/assets/decf1269-37ad-49c1-90ae-d2bd03a123e4)
类似这样的情况我猜是原始处理的时候由于原题 B 选项后面缺少 . 分隔符导致处理的时候忽略了？但是这题原始的正确答案是 C，如果按照 tsv 里的版本就不对了。

另外，关于 MMMU_DEV_VAL 也有部分题目可能存在问题，例如 validation_Geography_15
[https://huggingface.co/datasets/MMMU/MMMU/viewer/Geography/validation?row=14](url)
![image](https://github.com/user-attachments/assets/f58efe00-3c4a-4946-96a2-c2837175d7d5)
由于这题 D 选项是 None，因此在 tsv 中被记录成了和不存在选项相同的 NaN 值，这里由于时间关系暂时存疑的是，这是否会导致推理时只展示 ABC 三个选项？由于正确答案是 D，因此若只展示三个选项则本题无论如何回答不对了。
![image](https://github.com/user-attachments/assets/2ca4b040-f8cc-4526-9f05-157db442eeed)


RealWorldQA 中 index=633 的行同样有类似的问题，C 选项被遗漏了：
![image](https://github.com/user-attachments/assets/1e8c4edf-1837-427b-b68a-d9033b472925)
![image](https://github.com/user-attachments/assets/45ab34d8-e088-4c20-af57-c9e354d55feb)


另外，RealWorldQA 中 index=731 的条目，原始数据集忘记将 answer 处理成选项形式了，在 tsv 中也没有修正过来：
![image](https://github.com/user-attachments/assets/d09f7570-747b-44d5-95f8-3035a3fd0661)


Hi, @lihytotoro 

好的，我们这边看一下

Hi, @lihytotoro , 
All mentioned issues have been fixed in the new MMMU_DEV_VAL.tsv and RealWorldQA.tsv. We do not find anymore problems corresponding to the two benchmarks.
Repository: open-compass/VLMEvalKit
Issue: ChartQA/DocVQA 分数异常
Root cause: Configuration Error

ChartQA/DocVQA 两个benchmark跑正常的模型出来分数很低并且接近零

Hi,
Could you please show more details of the inference results?

我测试Aquila-VL-2B-llava-qwen跑出来只有 74，torchrun --nproc-per-node=8 --master_port=29503 run.py --data DocVQA_VAL --model Aquila-VL-2B --verbose， 直接这样运行的，是哪里有问题吗

@Orlando-CS check out this example I did : https://github.com/moured/qwen-vl2.5-chartqa
Repository: open-compass/VLMEvalKit
Issue: WildVision raises exception
Root cause: Validation Gap

During the evaluation of WildVision, the following exception was raised.

<img width="779" alt="截屏2024-12-23 14 49 46" src="https://github.com/user-attachments/assets/5838b295-c66c-47b8-af6b-218cea2de542" />


Hi, 
Could you please show more details in the inference results of Wildvision?

Hi, the problem has been fixed in #687
Repository: open-compass/VLMEvalKit
Issue: Llama-3.2-11B-Vision-Instruct来测评MMStar和HallusionBench
Root cause: Configuration Error

用Llama-3.2-11B-Vision-Instruct来测评MMStar和HallusionBench时，模型输出句子不完整（生成的prediction），大多数分词之后集中在130token。但是测MMVet时，就不会出现这种情况。

您好，针对mcq和y/n数据集，我们将max_new_token设定成了128，您可以在https://github.com/open-compass/VLMEvalKit/blob/main/vlmeval/vlm/llama_vision.py#L200进行修改

知道啦，谢谢
Repository: open-compass/VLMEvalKit
Issue: ChartQA augmented & CMMMU
Root cause: Unimplemented Feature Gap

It would be nice if VLMEvalKit can support evaluation on ChartQA augmented set and CMMMU, since it has supported ChartQA human set and MMMU

Hi, @KosumosuL ,
ChartQA augmented is now incorporated. However, there is no news from the developers of CMMMU. 

VLMEvalkit has supported ChartQA.
Repository: open-compass/VLMEvalKit
Issue: 待测模型请求
Root cause: Documentation Deficiency

您好，请问必须选择supported给出的模型么？如果我对模型进行量化、剪枝后是否还能测试呢？谢谢！

Hi，只需模型能够正常推理即可用 VLMEvalKit 测试，但如原 model wrapper 无法正常处理，需自行支持新的 model wrapper。

收到，谢谢您的回复，我现在马上开始实验！