zhiminy/EvalEng huggingface dataset:
Repository: modelscope/evalscope
Issue: 测试数据集支持
Root cause: Unimplemented Feature Gap

我发现有以下数据集好像不支持：EvalPlus、MultiPL-E、CRUX-O、MGSM、MMMLU、INCLUDE
这些数据集好像在paper里面用的挺多。但是在evalscope里面好像没有。这些数据能不能也支持以下。
Repository: modelscope/evalscope
Issue: 【模型评测】执行评测脚本时judge模型报 'parameter.enable_thinking must be set to false for non-streaming calls'
Root cause: Configuration Error

在执行对模型评测时，报Error occurred during qwen3-32b@https://dashscope.aliyuncs.com/compatible-mode/v1 LLM judge evaluation: Error code: 400 - {'error': {'message': '**parameter.enable_thinking must be set to false** f**or non-streaming calls**', 'type': 'invalid_request_error', 'param': None, 'code': 'invalid_parameter_error'}错误，找不到哪个参数可以屏蔽掉enable_thinking.

下面是代码：

`import os
from evalscope import TaskConfig, run_task
from evalscope.constants import EvalType, JudgeStrategy

task_cfg = TaskConfig(
    model='/models/Qwen3-32B',
    api_url='http://127.0.0.1:8090/v1',
    api_key='123',
    eval_type=EvalType.SERVICE,
    datasets=['chinese_simpleqa'],
    stream=False,
    limit=1,
    judge_strategy=JudgeStrategy.AUTO,
    analysis_report=True,
    generation_config={"chat_template_kwargs": {"enable_thinking": False}},
    judge_model_args={
        'model_id': 'qwen3-32b',
        'api_url': 'https://dashscope.aliyuncs.com/compatible-mode/v1',
        'api_key': 'xxxx',
        'generation_config': {"chat_template_kwargs": {"enable_thinking": False}},
        'enable_thinking': False
    }
)

run_task(task_cfg=task_cfg)`


<img width="3341" height="666" alt="Image" src="https://github.com/user-attachments/assets/a00ed506-dd29-4b65-b875-c042cb77a493" />

这样设置再试一下 `'generation_config': {'extra_body':{'chat_template_kwargs': {'enable_thinking': False}}}`

@Yunnglin <img width="995" height="537" alt="Image" src="https://github.com/user-attachments/assets/2c4344d4-2dfb-4638-afa6-0142c3da764b" />
加上了，还是报上面的parameter.enable_thinking must be set to false for non-streaming calls，请查一下加的位置是对的吧

<img width="1822" height="297" alt="Image" src="https://github.com/user-attachments/assets/4c7a278b-152f-4e1c-8085-101e74744271" />

@zuowg 你可以设置 `'generation_config': {"stream": True}` 来解决这个问题

> 感谢你的反馈！我们将关闭此问题。如果您有任何疑问，请随时重新打开它。如果EvalScope对您有所帮助，欢迎给我们点个STAR以示支持，谢谢！
Repository: open-compass/VLMEvalKit
Issue: If i want to use model of  LLaVa in VLMEval, which version should i installed for these?
Root cause: Configuration Error

The error below appears when I install  LLava v1.1.3 after having installed latest VLMEval:

```
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.                                                                            
jupyterlab 4.1.2 requires httpx>=0.25.0, but you have httpx 0.24.0 which is incompatible.                                    
xtuner 0.1.13 requires transformers!=4.34.1,!=4.35.0,!=4.35.1,!=4.35.2,>=4.32.1, but you have transformers 4.31.0 which is incompatible.                                                                                                                  
vlmeval 0.1.0 requires gradio==4.15.0, but you have gradio 3.35.2 which is incompatible.                                     
vlmeval 0.1.0 requires transformers==4.33.0, but you have transformers 4.31.0 which is incompatible. 
```

Hi, @EmberYS ,
I think it's OK to have these error messages, you can continue to run the official LLaVA with no problem.
However, if you want to evaluate LLaVA models trained with XTuner, please install later transformers version like 4.33.0 or 4.36.0
Repository: open-compass/VLMEvalKit
Issue: 单机多卡测试,shape对不上
Root cause: Interface Contract Mismatch


你好，可以提供一下测试代码以及报错信息吗

@MaoSong2022 I meet similar problem:

`python run.py --data MMBench_TEST_EN --model Qwen2.5-VL-3B-Instruct --verbose` is ok, but 
`torchrun --nproc-per-node=8 run.py --data MMBench_TEST_EN --model Qwen2.5-VL-3B-Instruct --verbose` results in error:
```shell
  File "/opt/conda/envs/py3.11+pytorch2.6+cu124/lib/python3.11/site-packages/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py", line 1171, in forward
    layer_outputs = decoder_layer(
                    ^^^^^^^^^^^^^^
  File "/opt/conda/envs/py3.11+pytorch2.6+cu124/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/py3.11+pytorch2.6+cu124/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/py3.11+pytorch2.6+cu124/lib/python3.11/site-packages/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py", line 1033, in forward
    hidden_states, self_attn_weights, present_key_value = self.self_attn(
                                                          ^^^^^^^^^^^^^^^
  File "/opt/conda/envs/py3.11+pytorch2.6+cu124/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/py3.11+pytorch2.6+cu124/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/py3.11+pytorch2.6+cu124/lib/python3.11/site-packages/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py", line 804, in forward
    key_states = key_states.view(bsz, q_len, -1, self.head_dim).transpose(1, 2)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: shape '[1, 1490, -1, 128]' is invalid for input of size 47680
Infer Qwen2.5-VL-3B-Instruct/MMBench_TEST_EN, Rank 4/8:   0%|               | 0/833 [00:01<?, ?it/s]
```

你好，[Qwen2.5-VL](https://github.com/QwenLM/Qwen2.5-VL)的transformer版本为`transformers==4.51.3` 请确保在对应的版本下进行测试，参考相关issue #975
Repository: open-compass/VLMEvalKit
Issue: Detailed results of ScienceQA-IMG
Root cause: Documentation Deficiency

Thanks for the great effort of this repo! I see you provide the zero-shot results of several MLLMs on ScienceQA-IMG dataset. Could you please add the detailed results (i.e., NAT, SOC, LAN) of the TEST and VAL partitions?

Hi, @thecharm, 
I will add these results later. 

Looking forward to the result as well 🥳

Hi, @zhangwengyu999 , 
The detailed results are now available at https://openxlab.org.cn/apps/detail/kennyutc/open_mllm_leaderboard.
Repository: open-compass/VLMEvalKit
Issue: when using TextVQA_VAL，meet torch.OutOfMemoryError
Root cause: Resource Mishandling

Here is the problem：
main/VLMEvalKit/vlmeval/vlm/qwen2_vl/modeling_qwen2_5_vl.py", line 273, in forward
    attn_weights = torch.matmul(q, k.transpose(1, 2)) / math.sqrt(self.head_dim)
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 124.68 GiB. GPU 0 has a total capacity of 44.53 GiB of which 19.05 GiB is free. Including non-PyTorch memory, this process has 25.46 GiB memory in use. Of the allocated memory 24.89 GiB is allocated by PyTorch, and 73.93 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

The model is qwen2_5_vl_7B.

when set attn_implementation='eager', test TextVQA_VAL is ok.

 When  attn_implementation='eager', it has torch.OutOfMemoryError after several epochs
But it has no torch.OutOfMemoryError when using other dataset MMbench，chartQA.

Does anyone know the reason，please help me，I try a lot but failed

Hi! You don't have enough memory in your current CUDA GPU.

> Hi! You don't have enough memory in your current CUDA GPU.

The batch_size is 1. The CUDA  memory will increase suddenly  after several epochs

I test qwen2.5-vl 7B with 80GB and it works fine. use 40GB memory  to test ~7B LVLM may causes OOM error, especially for MMMU and MathVista, so it is better to use a GPU with higher memory or use a quantized version of the model :)

> I test qwen2.5-vl 7B with 80GB and it works fine. use 40GB memory to test ~7B LVLM may causes OOM error, especially for MMMU and MathVista, so it is better to use a GPU with higher memory or use a quantized version of the model :)

Thanks. It really helps
Repository: open-compass/VLMEvalKit
Issue: InternVL2_5-26B not working
Root cause: External Dependency Breakage

I could run InternVL2_5-26B last week, however when I now try to evaluate it again, even on datasets I have tried before, I now get this:

Command: CUDA_VISIBLE_DEVICES=2 python run.py --data TextVQA_VAL --model InternVL2_5-26B --verbose

I now get this warning, related to the errors I show below:

 "InternLM2ForCausalLM has generative capabilities, as `prepare_inputs_for_generation` is explicitly defined. However, it doesn't directly inherit from `GenerationMixin`. From 👉v4.50👈 onwards, `PreTrainedModel` will NOT inherit from `GenerationMixin`, and this model will lose the ability to call `generate` and other related functions.
  - If you're using `trust_remote_code=True`, you can get rid of this warning by loading the model with an auto class. See https://huggingface.co/docs/transformers/en/model_doc/auto#auto-classes
  - If you are the owner of the model architecture code, please modify your model class such that it inherits from `GenerationMixin` (after `PreTrainedModel`, otherwise you'll get an exception).
  - If you are not the owner of the model architecture class, please contact the model code owner to update it.
Warning: Flash attention is not available, using eager attention instead."

<img width="839" alt="Image" src="https://github.com/user-attachments/assets/f56c6bf7-c183-410e-9e81-529abb0cca5e" />

<img width="845" alt="Image" src="https://github.com/user-attachments/assets/a0913f18-2695-4b2b-a2ea-ee0a208788fd" />

<img width="845" alt="Image" src="https://github.com/user-attachments/assets/517ec267-0332-4f5a-91b4-46eedcbf30ff" />

How can we bypass this? Thank you.

 

This problem can be solved by downgrading the transformers version to  <4.50
Repository: open-compass/VLMEvalKit
Issue: 有什么方法可以从本地读取数据文件吗
Root cause: Documentation Deficiency

请问有什么方法可以从本地读取那些数据文件吗
需要修改什么 python 文件呀
因为国内网下载国外数据集经常卡死

Is there any method that I can read the data from local without downloading.

1. 使用HF代理下载，`export HF_ENDPOINT=https://hf-mirror.com`
2. 在网速较好的机器上下载好之后，传输到目标服务器上，下载好的数据集通常位于`$HOME/LMUData/`目录下

> 1. 使用HF代理下载，`export HF_ENDPOINT=https://hf-mirror.com`
> 2. 在网速较好的机器上下载好之后，传输到目标服务器上，下载好的数据集通常位于`$HOME/LMUData/`目录下
请问传输到目标服务器的`$HOME/LMUData/`目录下之后，怎样修改配置文件才能运行本地数据？
谢谢
Repository: open-compass/VLMEvalKit
Issue: The md5 hashtest of the dataset DocVQA_VAL hasn't passed.
Root cause: Validation Gap

I checked the Completeness of DocVQA_VAL after downloading it from [here](https://opencompass.openxlab.space/utils/VLMEval/DocVQA_VAL.tsv).  Not sure which one is correct, please take a correct.

![image](https://github.com/open-compass/VLMEvalKit/assets/35883846/2d662c9c-17c6-4a53-a594-aed3d9d2c846)

Hi, @EmberYS  ,
That's due to you are not using the latest code, please fetch the latest main branch and retry. 

Hi, @kennymckormick 
Although [PR#104](https://github.com/open-compass/VLMEvalKit/pull/104) fixed md5 of DocVQA_VAL,  the column of “answers” in DocVQA_VAL doesn't fit code here. 

![image](https://github.com/open-compass/VLMEvalKit/assets/35883846/285d8d2f-dd77-4028-b61a-ff0eec97fb32)

This problem of mismatch between them leads to:

![image](https://github.com/open-compass/VLMEvalKit/assets/35883846/a8d8856b-d41a-4c8a-b256-52dba2930ece)

This seems to require modifying the code for DocVQA_VAL ( I tested replacing "answer" with "answers".  Passed !) or dataset of DocVQA_VAL. Also pleased to open a pull request if helpful.

HI, @EmberYS 
The bug in DocVQA has been resolved, please refer to #111   to modify the dataset_md5_dict and then reload DocVQA dataset.

Hi, @EmberYS ,
As verified by myself, the problem of DocVQA has been fixed now. I tried to evaluate InternLM-XComposer2 on DocVQA_VAL, and obtained a 58.6 score.
Repository: open-compass/VLMEvalKit
Issue: azure or vertex
Root cause: Documentation Deficiency

如果我使用azure的gpt4o api，或者vertex的gemini 2.5 pro
我应该在.env文件里怎么写

If I use Azure's GPT - 4O API or Vertex's Gemini 2.5 Pro, what should I write in the .env file?

我这样写但是报错了

my .env文件
OPENAI_API_KEY="my key"
OPENAI_API_BASE="my link"

got error
/mnt/largeml-train-gui-agent/hankaiyang/packages/VLMEvalKit/vlmeval/dataset/image_mcq.py:254: UserWarning: OPENAI_API_KEY is not set properly, will use exact matching for evaluation
warnings.warn('OPENAI_API_KEY is not set properly, will use exact matching for evaluation')

ENVs for AZURE GPT-4o (the feature is supported by the community):
1. AZURE_OPENAI_ENDPOINT
2. AZURE_OPENAI_DEPLOYMENT_NAME
3. OPENAI_API_VERSION
4. Moreover, you need to set use_azure=True in class GPT4V instantiation

ENVs for Gemini Vertex:
1. GOOGLE_API_BACKEND=vertex

To use gemini with vertex backend, you need to first follow the official guide of vertex backend and complete the setup. Once all set, you don't need to provide an API key. 



> ENVs for AZURE GPT-4o (the feature is supported by the community):
> 
> 1. AZURE_OPENAI_ENDPOINT
> 2. AZURE_OPENAI_DEPLOYMENT_NAME
> 3. OPENAI_API_VERSION
> 4. Moreover, you need to set use_azure=True in class GPT4V instantiation
> 
> ENVs for Gemini Vertex:
> 
> 1. GOOGLE_API_BACKEND=vertex
> 
> To use gemini with vertex backend, you need to first follow the official guide of vertex backend and complete the setup. Once all set, you don't need to provide an API key.

sad, not work for me.😭

Hi, any update on this, how to use Azure api? Thanks very much
Repository: open-compass/VLMEvalKit
Issue: difference sample number with HallusionBench's paper
Root cause: Configuration Error

I try to evaluate models on HallusionBench.
The number of hallusionbench in lmevalkit is 951, which is 1129 in HallusionBench's huggingface.
I wonder why they have different number.
What's more, I get only about 30 score of seed1.5vl on hallusionbench, which is far from 60.3 in the seed1.5vl paper.
Could someone get the similar result with me?
Thanks.
Repository: open-compass/VLMEvalKit
Issue: 这个框架能评估VLM的纯文本能力吗？

这个框架能评估多模态大模型的纯文本能力吗？想看下模型的纯文本能力

感谢您的关注与支持。我们建议使用Opencompass评估模型的纯语言能力。当前VLMEvalkit不适用于纯文本评测。

Best Regards
Repository: open-compass/VLMEvalKit
Issue: internvl3-1b model test
Root cause: Validation Gap

/cloudml-dev-largeml-4/root/largelm-dev4/dl_work/songchen1/code/VLMEvalKit/vlmeval/vlm/internvl/internvl_chat.py:225: UserWarning: Following kwargs received: {'do_sample': False, 'max_new_tokens': 4096, 'top_p': None}, will use as generation config. 
  warnings.warn(f'Following kwargs received: {self.kwargs}, will use as generation config. ')
[2025-06-06 08:47:20] ERROR - RUN - run.py: main - 510: Model InternVL3-1B x Dataset MMMB_ar combination failed: str expected, not NoneType, skipping this combination.
Traceback (most recent call last):
  File "/cloudml-dev-largeml-4/root/largelm-dev4/dl_work/songchen1/code/VLMEvalKit/run.py", line 382, in main
    model = infer_data_job(
            ^^^^^^^^^^^^^^^
  File "/cloudml-dev-largeml-4/root/largelm-dev4/dl_work/songchen1/code/VLMEvalKit/vlmeval/inference.py", line 192, in infer_data_job
    model = infer_data(
            ^^^^^^^^^^^
  File "/cloudml-dev-largeml-4/root/largelm-dev4/dl_work/songchen1/code/VLMEvalKit/vlmeval/inference.py", line 124, in infer_data
    os.environ['WORLD_SIZE'] = ws_bak
    ~~~~~~~~~~^^^^^^^^^^^^^^
  File "<frozen os>", line 719, in __setitem__
  File "<frozen os>", line 793, in encode
TypeError: str expected, not NoneType

Is the problem occured again? it seems there are some bugs on GPU configuration.
Repository: open-compass/VLMEvalKit
Issue: Add support for 2 extra models
Root cause: Unimplemented Feature Gap

Would it be possible to add support to both [GOT-OCR2.0](https://github.com/Ucas-HaoranWei/GOT-OCR2.0)  and [Donut](https://github.com/clovaai/donut)?

Thank you

Hi, to support new models, you can refer to [Development Guide](https://github.com/open-compass/VLMEvalKit/blob/57ca395fd8f4e56fae90c8fad9dbef6e38fe408f/docs/en/Development.md)
Repository: open-compass/VLMEvalKit
Issue: GUI tsv file
Root cause: Configuration Error

where can i get the tsv files in gui folder?

ScreenSpot_Mobile.tsv
ScreenSpot_Desktop.tsv
ScreenSpot_Web.tsv
ScreenSpot_v2_Mobile.tsv
ScreenSpot_v2_Desktop.tsv
ScreenSpot_v2_Web.tsv


Hi, 

You can check our latest main branch, the URLs have been updated.

> Hi,
> 
> You can check our latest main branch, the URLs have been updated.

but i still cant download these files.
http://opencompass.openxlab.space/utils/benchmarks/GUI/ScreenSpot/ScreenSpot_v2_Mobile.tsv
http://opencompass.openxlab.space/utils/benchmarks/GUI/ScreenSpot/ScreenSpot_v2_Desktop.tsv
http://opencompass.openxlab.space/utils/benchmarks/GUI/ScreenSpot/ScreenSpot_v2_Web.tsv

![Image](https://github.com/user-attachments/assets/6414f783-6390-47c2-b619-906275efe8ba)

> > Hi,
> > You can check our latest main branch, the URLs have been updated.
> 
> but i still cant download these files. http://opencompass.openxlab.space/utils/benchmarks/GUI/ScreenSpot/ScreenSpot_v2_Mobile.tsv http://opencompass.openxlab.space/utils/benchmarks/GUI/ScreenSpot/ScreenSpot_v2_Desktop.tsv http://opencompass.openxlab.space/utils/benchmarks/GUI/ScreenSpot/ScreenSpot_v2_Web.tsv
> 
> ![Image](https://github.com/user-attachments/assets/6414f783-6390-47c2-b619-906275efe8ba)

Sorry, there are some mistakes in the original URLs, they are fixed now in the main branch.

The correct URL is XXXX/ScreenSpot_v2/ScreenSpot_v2_Mobile.tsv, etc.
Repository: open-compass/VLMEvalKit
Issue: Ambiguous questions in CountBenchQA
Root cause: Validation Gap

# Some questions are ambiguous in CountBenchQA

Origin dataset: https://huggingface.co/datasets/nielsr/countbench

Look like that https://huggingface.co/datasets/vikhyatk/CountBenchQA create dataset from origin dataset by using GPTs. Some questions are ambiguous. A human in six poses or expressions or six men?

<img width="1049" alt="Image" src="https://github.com/user-attachments/assets/f53594c0-1690-4b24-9123-ad51c6f101e1" />

<img width="1050" alt="Image" src="https://github.com/user-attachments/assets/6d652e25-71fb-48dd-a2fb-3208e3952fc7" />

<img width="1175" alt="Image" src="https://github.com/user-attachments/assets/70fd4670-c324-423f-85ad-5cab07dc3d6e" />



```
countbench_data/ad5a521d-b33a-4a03-a1f2-b0f2904fcb01_1.jpg
countbench_data/b354a1d4-dfea-4ca9-9e54-e620f0074193_1.jpg
countbench_data/76a104b1-c738-45c1-b1b8-ea131ec1b7d2_1.jpg
countbench_data/8b731c7c-0d03-415d-9523-4f11d188ae9a_1.jpg
countbench_data/0496cd59-be2d-4e6a-bc08-76a55c01fbd9_1.jpg
countbench_data/43379f7b-bfb7-4d33-af11-e6e9ba81cdfe_1.jpg
countbench_data/575ec778-2b78-4588-a9b0-28ee8b9ddb31_1.jpg
countbench_data/87b2bd92-bba6-4a9d-b1a1-cb5ea7a889d9_1.jpg
countbench_data/987ed382-84be-44ec-bdc7-b6c31b80be78_1.jpg
countbench_data/45255f95-4eed-4172-8088-c48a336d946f_1.jpg
countbench_data/1adf4020-9c05-45c3-976d-a362484f87cd_1.jpg
countbench_data/199562be-d94b-40b7-bb6f-f366ae8efb3a_1.jpg
countbench_data/d4ff9c73-f1fb-4ff4-a15d-a8ec024b946b_1.jpg
countbench_data/ac00b968-7d97-494d-aea1-7f1a8dcd0e2c_1.jpg
```


countbench_data/8b731c7c-0d03-415d-9523-4f11d188ae9a_1.jpg broken image
countbench_data/834681c8-8bd9-4e23-94fe-bba24acba8dd_1.jpg wrong answer
ountbench_data/987ed382-84be-44ec-bdc7-b6c31b80be78_1.jpg wrong answer


<img width="1146" alt="Image" src="https://github.com/user-attachments/assets/d4396b75-2022-49f3-8440-98e642589338" />

Thank you for your correction! We will refine these data.

We have already updated the dataset.
Repository: open-compass/VLMEvalKit
Issue: Support judge model for MAC&ShortVQA Question
Root cause: Unimplemented Feature Gap

# Support judge model for MCQ&ShortVQA Question

 https://huggingface.co/IAAR-Shanghai/xVerify-9B-C 。


支持一些 MAC&ShortVQA  问题的 judge model；

在实际评测中，为了测试一个 MCQ / VQA 数据集，会考虑非常多的 **PROMPT工程**以及**后处理**。但越面向用户的模型，输出的多样性越高。后处理的复杂度越高，写起来也越累，错误率还特别高，这让后处理越来越复杂。

在查看 ChartQA 测试的结果时，发现 规则提取的答案错误率非常高，尝试了 gpt4omini/deepseekv3 评价，发现错误率还是比较高，还需要设计 system_prompt 以及 few-shot。 当我使用 xVerify-9B-C 事情变得简单，正确率也高很多。

人工check所有的评测结果后，错误率大约是：

规则的错误率：       5% ～6 %
gpt4omini 错误率:  2% ~ 3%
xverify 的错误率： 0.5% ~ 0.6%

vllm 单卡的服务大约 40QPS （eval 40条数据每秒）

可能和这些模型训过 ChartQA 相关数据有影响，但后续使用在countbench 等数据集上时，表现非常好，很难发现误判。


最大的好处时 **较低成本下，可以忽视掉 prompt工程和后处理的代码！！！**






赞，我们计划在各个 VQA 测试集上验证下

The evaluation method of ChartVQA `relaxed_correctness` is indeed not very robust. However, if the goal is to compare it with other VLM models, it seems best to use the same evaluation method.

> # Support judge model for MCQ&ShortVQA Question
> https://huggingface.co/IAAR-Shanghai/xVerify-9B-C 。
> 
> 支持一些 MAC&ShortVQA 问题的 judge model；
> 
> 在实际评测中，为了测试一个 MCQ / VQA 数据集，会考虑非常多的 **PROMPT工程**以及**后处理**。但越面向用户的模型，输出的多样性越高。后处理的复杂度越高，写起来也越累，错误率还特别高，这让后处理越来越复杂。
> 
> 在查看 ChartQA 测试的结果时，发现 规则提取的答案错误率非常高，尝试了 gpt4omini/deepseekv3 评价，发现错误率还是比较高，还需要设计 system_prompt 以及 few-shot。 当我使用 xVerify-9B-C 事情变得简单，正确率也高很多。
> 
> 人工check所有的评测结果后，错误率大约是：
> 
> 规则的错误率： 5% ～6 % gpt4omini 错误率: 2% ~ 3% xverify 的错误率： 0.5% ~ 0.6%
> 
> vllm 单卡的服务大约 40QPS （eval 40条数据每秒）
> 
> 可能和这些模型训过 ChartQA 相关数据有影响，但后续使用在countbench 等数据集上时，表现非常好，很难发现误判。
> 
> 最大的好处时 **较低成本下，可以忽视掉 prompt工程和后处理的代码！！！**


你好，请问你用的lmdeploy部署的这个模型吗