Repository: open-compass/VLMEvalKit
Issue: 评测MVBench时报错
Root cause: Validation Gap

你好，我在评测MVBench时会多一个路径报错 FileNotFoundError: [Errno 2] No such file or directory: '/root/.cache/huggingface/hub/datasets--OpenGVLab--MVBench/snapshots/230a2d4fac8900333c61754641c7a13e069ac9c6/video/star/Charades_v1_480/K47J5.mp4/K47J5.mp4'

![image](https://github.com/user-attachments/assets/1ca452d3-b2ed-49d1-a7b9-78e1298bdfc2)


上面的问题我把这两行代码注释掉解决了
![image](https://github.com/user-attachments/assets/0d595806-4e73-4c03-a609-baaa02dab160)
但是运行一段时间后报了下面的错误
Moviepy - video ready /root/.cache/huggingface/hub/datasets--OpenGVLab--MVBench/snapshots/230a2d4fac8900333c61754641c7a13e069ac9c6/video/ssv2_video/28142.mp4                                                                            
Moviepy - Building video /root/.cache/huggingface/hub/datasets--OpenGVLab--MVBench/snapshots/230a2d4fac8900333c61754641c7a13e069ac9c6/video/ssv2_video/114567.mp4.
Moviepy - Writing video /root/.cache/huggingface/hub/datasets--OpenGVLab--MVBench/snapshots/230a2d4fac8900333c61754641c7a13e069ac9c6/video/ssv2_video/114567.mp4

479it [08:56,  1.07it/s]
[2024-10-26 00:39:33] WARNING - warnings.py: _showwarnmsg - 109: /share/project/zhaohuxing/anaconda3/envs/llavaonevision/lib/python3.10/site-packages/moviepy/video/io/ffmpeg_reader.py:123: UserWarning: Warning: in file /root/.cache/huggingface/hub/datasets--OpenGVLab--MVBench/snapshots/230a2d4fac8900333c61754641c7a13e069ac9c6/video/ssv2_video/114567.webm, 307440 bytes wanted but 0 bytes read,at frame 71/72, at time 5.92/5.92 sec. Using the last valid frame instead.
  warnings.warn("Warning: in file %s, "%(self.filename)+

Moviepy - Done !        
Moviepy - video ready /root/.cache/huggingface/hub/datasets--OpenGVLab--MVBench/snapshots/230a2d4fac8900333c61754641c7a13e069ac9c6/video/ssv2_video/114567.mp4                                                                           
944it [15:19,  1.03it/s]
[2024-10-26 00:45:56,445] ERROR - RUN - run.py: main - 229: Model llavaonevision-qwen1.5b-mlp2x_gelu-qwen2.5-stage2-1012-32n40g-node8 x Dataset MVBench combination failed: , skipping this combination.
Traceback (most recent call last):
  File "/share/project/zhangjialing/open-compass_date/open-compass_1021/VLMEvalKit/run.py", line 121, in main
    model = infer_data_job_video(
  File "/share/project/zhangjialing/open-compass_date/open-compass_1021/VLMEvalKit/vlmeval/inference_video.py", line 143, in infer_data_job_video
    model = infer_data(
  File "/share/project/zhangjialing/open-compass_date/open-compass_1021/VLMEvalKit/vlmeval/inference_video.py", line 98, in infer_data
    response = model.generate(message=struct, dataset=dataset_name)
  File "/share/project/zhangjialing/open-compass_date/open-compass_1021/VLMEvalKit/vlmeval/vlm/base.py", line 111, in generate
    message = self.preproc_content(message)
  File "/share/project/zhangjialing/open-compass_date/open-compass_1021/VLMEvalKit/vlmeval/vlm/base.py", line 92, in preproc_content
    assert item['type'] == 'text'
AssertionError


Hi @helloworld01001 您更新到最新的版本还存在这个问题吗，如果存在的话，想知道您传入的message内容是怎么样的呢


Close Stale Issue, Feel free to reopen for any further updates.