zhiminy/EvalEng huggingface dataset:
Repository: open-compass/VLMEvalKit
Issue: Linting check is off

Logs for the latest run: https://github.com/open-compass/VLMEvalKit/actions/runs/14008567865/job/39225514111


<img width="717" alt="Image" src="https://github.com/user-attachments/assets/cb9a6626-f33c-4afb-98ae-c9618b218066" />


All files are skipped due to additional '|' in the end of line 6:

https://github.com/open-compass/VLMEvalKit/blob/e4e026375fffd6d7228f7d7625057dfddbfedb29/.pre-commit-config.yaml#L1-L7

oh, we will get it fixed. 

The problem is now fixed.
Repository: open-compass/VLMEvalKit
Issue: Mathverse vision only score different from leaderboard
Root cause: Documentation Deficiency

I tested qwen2.5-vl-7b but the result is much lower than leaderboard, is there any special command?
Command:
CUDA_VISIBLE_DEVICES=1 python run.py --data MathVerse_MINI_Vision_Only --model Qwen2.5-VL-7B 


Result:
--------------  ------------------
split           Vision Only
Overall         30.96446700507614
Solid Geometry  11.76470588235294
Functions       37.10691823899371
Plane Geometry  33.52941176470588
Area            21.978021978021978
Angle           35.233160621761655
Analytic        32.55813953488372
Length          36.81318681318682
Volume          7.8431372549019605
Applied         28.440366972477065
Coordinate      56.25
Expression      12.5
Property        38.028169014084504
--------------  ------------------

Hi @DietDietDiet, please refer to the official environment configuration (https://github.com/QwenLM/Qwen2.5-VL) and try to configure the environment using a higher version of torch

torch: 2.5.1 
transformers: 4.50.1
这个需要升级版本吗，想确认下榜单上登记的指标是MathVerse_MINI_Vision_Only 还是 MathVerse_MINI_Vision_Only_COT，看官方榜单COT与否正好也差十分

> torch: 2.5.1 transformers: 4.50.1 这个需要升级版本吗，想确认下榜单上登记的指标是MathVerse_MINI_Vision_Only 还是 MathVerse_MINI_Vision_Only_COT，看官方榜单COT与否正好也差十分

Same question.

@Lemok00 你测下来官方版本结果也差不多嘛？


> [@Lemok00](https://github.com/Lemok00) 你测下来官方版本结果也差不多嘛？

是的，MathVerse_MINI_Vision_Only 31.1，MathVerse_MINI_Vision_Only_cot 31.2
Repository: open-compass/VLMEvalKit
Issue: 怎么使用第三方平台转发的api key和api base
Root cause: Documentation Deficiency

在国内api平台购买的api key，现在已经有api的key和api base的网站，使用gpt评测的时候是只需要在.env文件中修改吗？

![Image](https://github.com/user-attachments/assets/5f2fa4b6-9e07-4a04-ba22-4355610ff2f1)
这里还需要修改吗？
Repository: open-compass/VLMEvalKit
Issue: Adding a New Benchmark - VELOCITI to VLMEvalKit
Root cause: Documentation Deficiency

Hi,

I am one of the creators of a VLM benchmark - VELOCITI, which is primarily designed to test the compositionality of VLMs in short 10s videos.
We test how well the models understand the presence and nuances of agents and actions in dense, action-rich, fast-paced movie videos!

Also, VELOCITI is accepted at CVPR 2025!

I'll gladly give you more details, as you need them.

Reaching out to request the procedure for our benchmark to incorporated into the VLEvalKit, please let me know how to go about this? and what I can do to have it approved and added to this toolkit.


Thanks for the amazing work and for maintaining this repository,

Best
V

Hi @varungupta31. Thank you for your support on our codebase.
If you want to contribute a dataset to us, you need to join in the form of a PR.
If your dataset is of a single type, you can refer to this PR for implementation:
https://github.com/open-compass/VLMEvalKit/pull/270
If your dataset is of a composite type, you can refer to this PR for implementation:
https://github.com/open-compass/VLMEvalKit/pull/847


Please let us know if you guys have any trouble during creating the pull request. 

Hey @kennymckormick,
Sorry for my absence here.

I wanted to reply once I made some progress but running a bit short on time to do this.

I will close the issue for now, hopefully try in some time and re-open if I face issues.
Repository: open-compass/VLMEvalKit
Issue: Running Video-MME with a different number of frames
Root cause: Documentation Deficiency

I tried to evaluate Qwen2.5-VL-3B-Instruct on the Video-MME data set but with a different number of frames. I added the following line to video_dataset_config.py:

`videomme_dataset = {
    'Video-MME_8frame': partial(VideoMME, dataset='Video-MME', nframe=8),
    'Video-MME_8frame_subs': partial(VideoMME, dataset='Video-MME', nframe=8, use_subtitle=True),
    'Video-MME_1fps': partial(VideoMME, dataset='Video-MME', fps=1.0),
    'Video-MME_0.5fps': partial(VideoMME, dataset='Video-MME', fps=0.5),
    'Video-MME_0.5fps_subs': partial(VideoMME, dataset='Video-MME', fps=0.5, use_subtitle=True),
    'Video-MME_768frame': partial(VideoMME, dataset='Video-MME', nframe=768), # <- added line
}```

I then ran the evaluation using:

python run.py --data Video-MME_768frame --model Qwen2.5-VL-3B-Instruct

I had two issues doing it this way. The first is that it is taking a lot longer than it should be. The other is that the amount of time for each video got a lot longer for longer videos, which should not be the case since the number of frames should be constant.
Is this the wrong way to do it?

Another thing I wanted to do is to evaluate the model using a constant frame rate of 2fps, but with a maximum number of frames capped at 768 frames for each video. Is this option supported? If so, how to do it?

Hi @Yazan-Almajnouni, Sorry for replying so late.
Regarding the first question, because the length of each video read is different, the reading time varies accordingly. At the same time, since you require reading 768 frames, the time taken to obtain the corresponding frames from the video will also be longer, resulting in a greater time cost.
Regarding the second question, you can use the 2fps parameter for evaluation. Since the maximum sampling frame for qwen2.5 is 768 frames by default, it can meet your requirements. For specific code, please refer to the details at https://github.com/QwenLM/Qwen2.5-VL/blob/main/qwen-vl-utils/src/qwen_vl_utils/vision_process.py#L226.
Repository: open-compass/VLMEvalKit
Issue: 使用MVBench数据集评测出现报错
Root cause: Validation Gap

![Image](https://github.com/user-attachments/assets/2d41acc5-697f-4a9d-9497-19472c136ba3)
报错如上截图，看了下报错的输入信息，原因是获取不到video/nturgbd/中以avi结尾的视频。
video/nturgbd/中的视频在huggingface中已经没有视频数据了，但是VLMEvalKit还是会下载数据集中的video/MVBench_videos_ntu.txt
导致读取数据时，出现报错！

@Xiaoqi-Zhou916 您好这个需要您自行下载mvbench中的nturgbd视频，放到对应路径下即可

> [@Xiaoqi-Zhou916](https://github.com/Xiaoqi-Zhou916) 您好这个需要您自行下载mvbench中的nturgbd视频，放到对应路径下即可

可是MVBench数据集已经没有nturgbd视频了，我要去哪里下载呢？我觉得这个是不是可以在你们的代码里面进行下校验，不然商用的话也是很麻烦的

![Image](https://github.com/user-attachments/assets/cd8c02f4-0e08-4a7b-9b30-c7b087365db7)-

您可以到nturgbd数据集官网结合给出的index自行下载。我们会考虑在此处加入校验，感谢您的建议。

> > [@Xiaoqi-Zhou916](https://github.com/Xiaoqi-Zhou916) 您好这个需要您自行下载mvbench中的nturgbd视频，放到对应路径下即可
> 
> 可是MVBench数据集已经没有nturgbd视频了，我要去哪里下载呢？我觉得这个是不是可以在你们的代码里面进行下校验，不然商用的话也是很麻烦的
> 
> ![Image](https://github.com/user-attachments/assets/cd8c02f4-0e08-4a7b-9b30-c7b087365db7)-

Hi, @Xiaoqi-Zhou916 , 

NTURGB+D 的视频需要联系原数据开发者下载：https://rose1.ntu.edu.sg/dataset/actionRecognition/，我们作为第三方无权提供下载方式
Repository: open-compass/VLMEvalKit
Issue: request to support skywork-r1v
Root cause: Unimplemented Feature Gap

https://github.com/SkyworkAI/Skywork-R1V

Hi @Orlando-CS. Thank you for your support of our community. You can add models to VLMEvalKit by submitting a pull request. You can refer to this pull request: https://github.com/open-compass/VLMEvalKit/pull/697. If you have any questions during the process, feel free to contact me.
Repository: open-compass/VLMEvalKit
Issue: 很多评测数据集的post_check函数报错
Root cause: Configuration Error

我在用Qwen2.5-VL-Instruct-3B模型评测时，在很多数据集的post_check函数中遇到下面的警告，例如MathVision
 mathv.py: post_check - 128: <class 'ValueError'>: signal only works in main thread of the main interpreter

<img width="584" alt="Image" src="https://github.com/user-attachments/assets/42940c8d-f5b6-4dbc-8efd-85a535e562b8" />

不清楚为什么会出现这个警告，以及这个警告是否会影响评测的精度？

+1

Changing the line to `@timeout_decorator.timeout(30, use_signals=False)` at https://github.com/open-compass/VLMEvalKit/blob/7c47fcaedcd9e3eff1546cb41da84c090a64679c/vlmeval/dataset/utils/mathv.py#L14
fixes this error on MathVision.
See https://github.com/pnpnpn/timeout-decorator#multithreading
Repository: open-compass/VLMEvalKit
Issue: qwen2.5-vl-7b 用mmmu报错combination failed
Root cause: Validation Gap

root@localhost:/VLMEvalKit# HIP_VISIBLE_DEVICES=7 torchrun --nproc-per-node=1 run.py --data MMMU_DEV_VAL --model Qwen2.5-VL-7B-Instruct --verbose
[2025-04-09 11:17:04] ERROR - misc.py: load_env - 212: Did not detect the .env file at /panhb/docker_in/VLMEvalKit/.env, failed to load. 
[2025-04-09 11:17:04] ERROR - misc.py: load_env - 212: Did not detect the .env file at /panhb/docker_in/VLMEvalKit/.env, failed to load. 
[2025-04-09 11:17:04,243] WARNING - RUN - run.py: main - 171: --reuse is not set, will not reuse previous (before one day) temporary files
[2025-04-09 11:17:04] WARNING - run.py: main - 171: --reuse is not set, will not reuse previous (before one day) temporary files
Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.48, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
<class 'FileNotFoundError'>: [Errno 2] No such file or directory: 'nvidia-smi'
[2025-04-09 11:17:06,241] ERROR - RUN - run.py: main - 438: Model Qwen2.5-VL-7B-Instruct x Dataset MMMU_DEV_VAL combination failed: , skipping this combination.
Traceback (most recent call last):
  File "/panhb/docker_in/VLMEvalKit/run.py", line 316, in main
    model = infer_data_job(
  File "/panhb/docker_in/VLMEvalKit/vlmeval/inference.py", line 176, in infer_data_job
    model = infer_data(
  File "/panhb/docker_in/VLMEvalKit/vlmeval/inference.py", line 110, in infer_data
    model = supported_VLM[model_name]() if isinstance(model, str) else model
  File "/panhb/docker_in/VLMEvalKit/vlmeval/vlm/qwen2_vl/model.py", line 114, in __init__
    assert max_gpu_mem > 0
AssertionError
[2025-04-09 11:17:06] ERROR - run.py: main - 438: Model Qwen2.5-VL-7B-Instruct x Dataset MMMU_DEV_VAL combination failed: , skipping this combination.
Traceback (most recent call last):
  File "/panhb/docker_in/VLMEvalKit/run.py", line 316, in main
    model = infer_data_job(
  File "/panhb/docker_in/VLMEvalKit/vlmeval/inference.py", line 176, in infer_data_job
    model = infer_data(
  File "/panhb/docker_in/VLMEvalKit/vlmeval/inference.py", line 110, in infer_data
    model = supported_VLM[model_name]() if isinstance(model, str) else model
  File "/panhb/docker_in/VLMEvalKit/vlmeval/vlm/qwen2_vl/model.py", line 114, in __init__
    assert max_gpu_mem > 0
AssertionError

Hi, I am running into the same error, may i ask how you solve the problem? Thanks!
Repository: open-compass/VLMEvalKit
Issue: How are models that use in-context examples handled?
GitHub labels: Feature Request

Hi, thanks for this great work! I was wondering how models which can use in-context examples are evaluated. Is everything in this benchmark kit zero-shot? 

Hi , @sachit-menon 
Everything is zero-shot now. We do not officially support few-shot since only few models support that. 

Hi, we developed an ICL benchmark for VLLMs here: https://github.com/ys-zong/VL-ICL. Welcome to try it out.

@kennymckormick I wonder if you have a plan to integrate VL-ICL into this very useful repo. I'm also happy to contribute:)


Thanks!
________________________________
From: Yongshuo Zong ***@***.***>
Sent: Friday, April 12, 2024 6:49:15 AM
To: open-compass/VLMEvalKit ***@***.***>
Cc: Sachit Menon ***@***.***>; Mention ***@***.***>
Subject: Re: [open-compass/VLMEvalKit] How are models that use in-context examples handled? (Issue #90)


Hi, we developed an ICL benchmark for VLLMs here: https://github.com/ys-zong/VL-ICL. Welcome to try it out.

@kennymckormick<https://github.com/kennymckormick> I wonder if you have a plan to integrate VL-ICL into this very useful repo. I'm also happy to contribute:)

—
Reply to this email directly, view it on GitHub<https://github.com/open-compass/VLMEvalKit/issues/90#issuecomment-2051524455>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AFD4ZUJGPM3SXJHDLZRRUMLY4632XAVCNFSM6AAAAABC6RF2OWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANJRGUZDINBVGU>.
You are receiving this because you were mentioned.Message ID: ***@***.***>


Definitely Yes! Welcome to implement your new dataset VL-ICL in VLMEvalKit. After implemented, we can help with the evaluation of API models. 
BTW, you can check our latest refactor PR #140 , and make sure you are developing based on the new interface. 


> Hi, we developed an ICL benchmark for VLLMs here: https://github.com/ys-zong/VL-ICL. Welcome to try it out.
> 
> @kennymckormick I wonder if you have a plan to integrate VL-ICL into this very useful repo. I'm also happy to contribute:)
Repository: open-compass/VLMEvalKit
Issue: 通过LMDeployAPI方式部署Qwen2-VL-7B-Instruct，测精度
Root cause: Unimplemented Feature Gap

- 当前Qwen2VLChat方式只支持本地加载方式
- LMDeployAPI方式不支持Qwen2-VL-7B-Instruct
Repository: open-compass/VLMEvalKit
Issue: textvqa和pope评测报错
Root cause: Validation Gap

使用qwen2.5vl，其他benchmark评测顺利完成，但是textvqa和pope评测时遇到以下问题
<img width="1316" alt="Image" src="https://github.com/user-attachments/assets/8755e02d-4ecb-4ad4-a93a-9fa7880a1824" />

我也遇到了同样的问题，请问你是如何解决的？

我也遇到了类似的问题：
```python
[2025-04-26 23:53:33,639] ERROR - RUN - run.py: main - 482: Model Qwen2-VL-2B-Instruct x Dataset POPE combination failed: , skipping this combination.
Traceback (most recent call last):
  File "LLaMA-Factory/./VLMEvalKit/run.py", line 325, in main
    model = infer_data_job(
  File "LLaMA-Factory/VLMEvalKit/vlmeval/inference.py", line 185, in infer_data_job
    model = infer_data(
  File "LLaMA-Factory/VLMEvalKit/vlmeval/inference.py", line 148, in infer_data
    response = model.generate(message=struct, dataset=dataset_name)
  File "LLaMA-Factory/VLMEvalKit/vlmeval/vlm/base.py", line 110, in generate
    message = self.preproc_content(message)
  File "LLaMA-Factory/VLMEvalKit/vlmeval/vlm/base.py", line 91, in preproc_content
    assert item["type"] == "text"
```

数据集手机加载时，LOCALIZE 函数会将 
1.base64编码的图像解码并保存为文件
2.添加 image_path 字段指向这些图像文件
3.生成 *_local.tsv 文件存储处理后的数据

删掉*_local.tsv 文件就可以了
Repository: open-compass/VLMEvalKit
Issue: Qwen2.5-VL-3B-Instruct在很多评测数据集的指标与官方榜单差距较大
Root cause: Configuration Error

torch版本2.5.1
transformers版本4.49.0.dev0
judge模型gpt-4o-turble
generation_config:
        top_p=0.001,
        top_k=1,
        temperature=0.01,
        repetition_penalty=1.0,

**自测结果：**
AI2D: 0.7836
**DynaMath (worst case overall): 0.067**
HallusionBench: 42.49
MMU(val): 0.48
MMStar: 0.544

**官方结果：**
AI2D: 0.814
**DynaMath (worst case overall): 0.132**
HallusionBench: 46.6
MMU(val): 0.512
MMStar: 0.563

在其他一些数据集上我测的结果比官方榜单高，总体avg差距不是特别大。
但是上述的DynaMath、HallusionBench、MMU(val)和榜单差距有点太大了。不知道是什么原因造成的呢？





您好，对于您的Generation Config：
```
top_p=0.001,
top_k=1,
temperature=0.01,
repetition_penalty=1.0,
```
我们在评测的时候一般仅设置temperature=0，用于确保结果的稳定性。此外，torch版本和transformers的版本也会造成影响。

> 您好，对于您的Generation Config：
> 
> ```
> top_p=0.001,
> top_k=1,
> temperature=0.01,
> repetition_penalty=1.0,
> ```
> 
> 我们在评测的时候一般仅设置temperature=0，用于确保结果的稳定性。此外，torch版本和transformers的版本也会造成影响。

您好，感谢您的回答。请问可以提供Qwen2.5-VL的官方测试环境吗，比如torch transformers版本？

Same problem, could the official team release the complete information about the testing environment?



> 您好，对于您的Generation Config：
> 
> ```
> top_p=0.001,
> top_k=1,
> temperature=0.01,
> repetition_penalty=1.0,
> ```
> 
> 我们在评测的时候一般仅设置temperature=0，用于确保结果的稳定性。此外，torch版本和transformers的版本也会造成影响。

您好，请问可以问一下qwen2.5-vl的详细参数设置吗？ 我测出来的docvqa与官方基本一致，所以想要详细参数

请问一下，judge model的选择对测试集指标的影响大吗？
Repository: open-compass/VLMEvalKit
Issue: Do you support customered models and local models on evaluation of image and video datasets?for example mistral-small-24b. If so, How to configure it?
Root cause: Documentation Deficiency


Yes. To evalute local models, it depends on cases:
- if it is derived from a existing architecture, for example llava, qwen or internvl, you just need to define the model name and the corresponding model path, for example:
```
"llava_v1.5_7b": partial(LLaVA, model_path="liuhaotian/llava-v1.5-7b")
```
see `vlmeval/config.py` for supported VLMs. 

- if the model is trained from strach with a new architecture, then you may need to implement it, see [Development Guide](https://github.com/open-compass/VLMEvalKit/blob/main/docs/en/Development.md) for more details.
Repository: open-compass/VLMEvalKit
Issue: The certificate of openxlab.space has expired
Root cause: External Dependency Breakage

Unable to download .tsv files from openxlab.space

Same issue

what is the file name

The same issue is as follows:
Exception: Failed to download https://opencompass.openxlab.space/utils/VLMEval/MME.tsv

+1

@kennymckormick 
Exception: Failed to download https://opencompass.openxlab.space/utils/VLMEval/LLaVABench.tsv

Add the following code at the beginning of the configuration to ignore SSL certificate verification:


```python
import ssl
# 全局禁用 SSL 验证（影响所有基于 `ssl` 的请求）
ssl._create_default_https_context = ssl._create_unverified_context
```

> ⚠️ WARN: This is a temporary measure, and it's best to wait for the official certificate update.

> Add the following code at the beginning of the configuration to ignore SSL certificate verification:
> 
> import ssl
> # 全局禁用 SSL 验证（影响所有基于 `ssl` 的请求）
> ssl._create_default_https_context = ssl._create_unverified_context
> > ⚠️ WARN: This is a temporary measure, and it's best to wait for the official certificate update.

We will temporarily bypass the SSL verification to fix this problem. After all, we have another md5 check. 

#923 as a temp solution
Repository: open-compass/VLMEvalKit
Issue: benchmark的build_prompt函数调用问题
Root cause: Documentation Deficiency

相同的benchmark，在build_prompt打一个断点，使用不同的模型，有的回进断点，有的不会进，这是bug吗？

并不是。
评估使用prompt取决于模型是否定义了  `use_custom_prompt`, 如果定义了这个函数，评估的时候会使用模型自定义的prompt，否则会使用dataset的`build_prompt`函数。文档可以参考 [Development Guide](https://github.com/open-compass/VLMEvalKit/blob/main/docs/zh-CN/Development.md#%E5%AE%9E%E7%8E%B0%E4%B8%80%E4%B8%AA%E6%96%B0%E7%9A%84%E6%A8%A1%E5%9E%8B) 支持自定义提示词构建部分，代码可以参考 [building prompt](https://github.com/open-compass/VLMEvalKit/blob/2f51add663ae53f0a5160d46ef60c84155280f53/vlmeval/inference.py#L139)
Repository: open-compass/VLMEvalKit
Issue: Unknown error when loading LLaVA model
Root cause: Environment Incompatibility

when I run the command
python run.py --data MMBench_DEV_EN MME SEEDBench_IMG --model llava_v1.5_13b --verbose
it shows
 warnings.warn('Unknown error when loading LLaVA model.')
<img width="659" alt="image" src="https://github.com/open-compass/VLMEvalKit/assets/31176427/ef253296-ab2f-477a-988a-f870e007ec0b">

how to deal with this?

Hi, @cyj95 ,
Looks like this error is encountered when you failed to execute the following command:
![image](https://github.com/open-compass/VLMEvalKit/assets/34324155/dcef8522-67f4-4bf4-8ad8-5af0d5103e08)

Please clone the LLaVA repo: https://github.com/haotian-liu/LLaVA, install LLaVA from source (`pip install -e .`) and try again. 

Hi, @kennymckormick 
I debug the run.py with args: "--data MME --model llava_v1.5_7b --verbose". The code always comes to here:
![image](https://github.com/open-compass/VLMEvalKit/assets/35883846/84a5c9ad-19a2-43a2-b86d-f5a8fb61c0b2)

The specific error shows:
![image](https://github.com/open-compass/VLMEvalKit/assets/35883846/cafce057-29a2-426c-902b-05ff814e649f)
Does it mean that environments without internet connectivity cannot works?

Installed verson of llava and vlmeval:
![image](https://github.com/open-compass/VLMEvalKit/assets/35883846/9bce2abf-7e50-489c-b23d-7d5c9190b73e)

Path of the dataset and model：
![image](https://github.com/open-compass/VLMEvalKit/assets/35883846/3a4c87ca-0a49-4b33-8ef5-390fb9542dd9)

any suggestions to resolve this issue, thanks!!!

Hi, @EmberYS, 
Looks like a question about internet connectivity: you are trying to fetch openai clip from huggingface but failed.
Repository: open-compass/VLMEvalKit
Issue: ccocr 报错 UnboundLocalError: local variable 'ccocr_evaluator_map' referenced before assignment
Root cause: Validation Gap

![Image](https://github.com/user-attachments/assets/f4a279c8-2b7e-482e-a11b-e7a6dd7d62e4)
如图在CCOCR 跑 Qwen-VL-Max-0809 发生错误
ERROR - RUN - run.Dv: main - 443: Model Owen-VL-Max-0809 x Dataset GOCR MultiSceneer Funsd combination failed
local variable 'ccocr_evaluator_map' referenced before
assignment,
skipping this combination.
Traceback (most recent call last):
File "/mnt/workspace/VLMEvalKit/run.py", Line 414, in main
eval results = dataset.evaluate(result file, **judge kwargs)
File "/mnt/workspace/VLMEvalKit/vlmeval/dataset/image_ccocr-py", Line 178, in evaluate
eval func = ccocr_evaluator map.get (group name, None)
UnboundLocalError: local variable "ccocr evaluator map' referenced before assignment

命令 python run.py --data CCOCR_MultiSceneOcr_Funsd --model Qwen-VL-Max-0809 --work-dir outputs/0415/multi_scene_ocr --verbose --api-nproc 1
环境 
modelscope:1.23.0
pytorch2.3.1
tensorflow2.16.1
python 3.10
ubuntu 22.04

Evluating results on CC-OCR requies installing extra packages, `zss` and `apted`. See [CC-OCR requirements.txt](https://github.com/AlibabaResearch/AdvancedLiterateMachinery/blob/main/Benchmarks/CC-OCR/requirements.txt) for details. 

Try `pip install zss apted` and check if the same problem occurs.

非常感谢 问题已经解决了
Repository: open-compass/VLMEvalKit
Issue: [Benchmark] Add support for CharXiv benchmark
Root cause: Unimplemented Feature Gap

[OpenAI GPT-4.1](https://openai.com/index/gpt-4-1/) uses [CharXiv](https://charxiv.github.io/) benchmark to show its vision capability.  I belive CharXiv benchmark should be supported by VLMEvalKit. 

I am working on supporting this benchmark, any suggestions are welcomed :)


- [x] descriptive_val.tsv file 
- [x] reasoning_val.tsv file
- [ ] test
Repository: open-compass/VLMEvalKit
Issue: Evaluation on a VQA dataset from huggingface
Root cause: Documentation Deficiency

Could any body suggest how to evaluate a model on a huggingface  dataset. For example HuggingFaceM4/ChartQA. 

Hi, supported benchmarks/datasets are listed in [Features](https://aicarrier.feishu.cn/wiki/Qp7wwSzQ9iK1Y6kNUJVcr6zTnPe?table=tblsdEpLieDoCxtb). I wonder if there is a relationship between  `HuggingFaceM4/ChartQA` and the listed ChartQA. 
- If they are same, then you  can use `python run.py --model your_model --data ChartQA_TEST` to get the results
- If they are different, then you need to implement it based on [Development Guide](https://github.com/open-compass/VLMEvalKit/blob/main/docs/en/Development.md), or you can raise a feature request issue, we will support it ASAP.

Thank you for your response. I understood 

@chandrabhuma maybe the following repo would help : https://github.com/moured/qwen-vl2.5-chartqa 
I tried to do it for Qwen model

Thank you for the inputs. It is working fine.
Repository: open-compass/VLMEvalKit
Issue: Error in analyzing video datasets with Qwen2.5-VL-3B-Instruct
Root cause: External Dependency Breakage

I am running the below command `python run.py --data Video-MME_8frame --model Qwen2.5-VL-3B-Instruct` but I am getting the error:
  File "VLMEvalKit_original/VLMEvalKit/vlmeval/vlm/qwen2_vl/model.py", line 192, in generate_inner
    images, videos = process_vision_info([messages])
  File "/usr/local/lib/python3.10/dist-packages/qwen_vl_utils/vision_process.py", line 332, in process_vision_info
    video_inputs.append(fetch_video(vision_info))
  File "/usr/local/lib/python3.10/dist-packages/qwen_vl_utils/vision_process.py", line 260, in fetch_video
    video = VIDEO_READER_BACKENDS[video_reader_backend](ele)
  File "/usr/local/lib/python3.10/dist-packages/qwen_vl_utils/vision_process.py", line 230, in _read_video_decord
    nframes = smart_nframes(ele, total_frames=total_frames, video_fps=video_fps)
  File "/usr/local/lib/python3.10/dist-packages/qwen_vl_utils/vision_process.py", line 161, in smart_nframes
    raise ValueError(f"nframes should in interval [{FRAME_FACTOR}, {total_frames}], but got {nframes}.")
ValueError: nframes should in interval [2, 2227], but got 0.

I am installing transformers==4.49.0 and qwen-vl-utils[decord]==0.0.8. Are there any issues with my packages?