Repository: open-compass/VLMEvalKit
Issue: MMBench-Video Dataset Download Bug
Root cause: Validation Gap

### Hello, I have an issue with executing the evaluation with MMBench-Video dataset:
I try to execute `run.py` with _MMBench-Video_ dataset, but there are two scenarios that happen everytime I try to run the program. It either says that repo path is not a directory or downloads the dataset again (even though I have downloaded it already).

Here is my huggingface cache directory:
`(evalkit) root@autodl-container-3fac46b1d9-b311c924:~/autodl-tmp/hf_cache/huggingface/hub# pwd`
`/root/autodl-tmp/hf_cache/huggingface/hub`

`(evalkit) root@autodl-container-3fac46b1d9-b311c924:~/autodl-tmp/hf_cache/huggingface/hub# ls`
`MVLU                  datasets--opencompass--MMBench-Video       models--Qwen--Qwen2-VL-2B-Instruct
datasets--MLVU--MVLU  models--Efficient-Large-Model--VILA1.5-3b  version.txt`

When I export the path to the huggingface cache directory (without the hub included): 
`(evalkit) root@autodl-container-3fac46b1d9-b311c924:~/autodl-tmp/sanya/VLMEvalKit# export HF_HOME=/root/autodl-tmp/hf_cache/huggingface` 
and execute:
`python run.py --data MMBench-Video --model GPT4o`

Then I receive this error: 
`raise CorruptedCacheException(f"Repo path is not a directory: {repo_path}")
huggingface_hub.errors.CorruptedCacheException: Repo path is not a directory: /root/autodl-tmp/hf_cache/huggingface/datasets--opencompass--MMBench-Video`


When I export the path to the huggingface cache directory (the hub included):
`export HF_HOME=/root/autodl-tmp/hf_cache/huggingface/hub` 
and execute:
Then it starts to download a dataset under the following path:
`/root/autodl-tmp/hf_cache/huggingface/hub/hub`


Hi @Noctis-SC. Thanks for your advice. I have fixed the problem, you can pull the latest code and try again, if you still have problems with re-downloading：
```Then it starts to download a dataset under the following path:```
Before the redownload action begins, it will check the integrity of this function(https://github.com/open-compass/VLMEvalKit/blob/main/vlmeval/dataset/mmbench_video.py#L70), and it will turn to false if it can not find any of the videos or the md5 code isn't the same as setting. So could you please debug the code and tell me it turn to false in which part of this function.

> Hi @Noctis-SC. Thanks for your advice. I have fixed the problem, you can pull the latest code and try again, if you still have problems with re-downloading： `Then it starts to download a dataset under the following path:` Before the redownload action begins, it will check the integrity of this function(https://github.com/open-compass/VLMEvalKit/blob/main/vlmeval/dataset/mmbench_video.py#L70), and it will turn to false if it can not find any of the videos or the md5 code isn't the same as setting. So could you please debug the code and tell me it turn to false in which part of this function.

Thank you for your quick response, @FangXinyu-0913 I have pulled the updated code, it seems to work fine with `VILA1.5-3B`model.  However, I'm not sure is it related or not, but I have another issue, this time with `Qwen2-VL-7B-Instruct`

**When I run this command:**
`(qwen2) root@autodl-container-3fac46b1d9-b311c924:~/autodl-tmp/sanya/VLMEvalKit# AUTO_SPLIT=1 python run.py --model Qwen2-VL-7B-Instruct --data MMBench-Video  --verbose` 

**The output I get:**
`/root/autodl-tmp/miniconda3/envs/qwen2/lib/python3.10/site-packages/transformers/utils/hub.py:128: FutureWarning: Using `TRANSFORMERS_CACHE` is deprecated and will be removed in v5 of Transformers. Use `HF_HOME` instead.
  warnings.warn(
[2024-11-11 14:26:33,312] WARNING - RUN - run.py: main - 53: --reuse is not set, will start the evaluation from scratch
[2024-11-11 14:26:33] WARNING - run.py: main - 53: --reuse is not set, will start the evaluation from scratch
Downloading shards: 100%|████████████████████████████████████████████████████████████████████| 5/5 [00:00<00:00, 18542.46it/s]
`Qwen2VLRotaryEmbedding` can now be fully parameterized by passing the model config through the `config` argument. All other arguments will be removed in v4.46
Loading checkpoint shards: 100%|████████████████████████████████████████████████████████████████| 5/5 [00:05<00:00,  1.11s/it]
0it [00:00, ?it/s]Qwen2-VL-7B-Instruct is a video-llm model, nframe is set to 8, not using default
[{'role': 'user', 'content': [{'type': 'text', 'text': 'What is the name of the player who scored the first goal in the video?'}, {'type': 'video', 'video': 'file:///root/autodl-tmp/hf_cache/huggingface/datasets/hub/datasets--opencompass--MMBench-Video/snapshots/ef35e21df54488715a906c7e47146f5d9f4abbed/video/wZxzBvAgqxc.mp4', 'nframes': 8}]}]
qwen-vl-utils using decord to read video.
The player who scored the first goal in the video is wearing a red jersey with the name "PALMER" on the back.
The player who scored the first goal in the video is wearing a red jersey with the name "PALMER" on the back.
1it [00:04,  4.07s/it]
[2024-11-11 14:26:46,632] ERROR - RUN - run.py: main - 284: Model Qwen2-VL-7B-Instruct x Dataset MMBench-Video combination failed: '>' not supported between instances of 'NoneType' and 'int', skipping this combination.
Traceback (most recent call last):
  File "/root/autodl-tmp/sanya/VLMEvalKit/run.py", line 163, in main
    model = infer_data_job_video(
  File "/root/autodl-tmp/sanya/VLMEvalKit/vlmeval/inference_video.py", line 173, in infer_data_job_video
    model = infer_data(
  File "/root/autodl-tmp/sanya/VLMEvalKit/vlmeval/inference_video.py", line 98, in infer_data
    if getattr(model, 'fps', 0) > 0:
TypeError: '>' not supported between instances of 'NoneType' and 'int'
[2024-11-11 14:26:46] ERROR - run.py: main - 284: Model Qwen2-VL-7B-Instruct x Dataset MMBench-Video combination failed: '>' not supported between instances of 'NoneType' and 'int', skipping this combination.
Traceback (most recent call last):
  File "/root/autodl-tmp/sanya/VLMEvalKit/run.py", line 163, in main
    model = infer_data_job_video(
  File "/root/autodl-tmp/sanya/VLMEvalKit/vlmeval/inference_video.py", line 173, in infer_data_job_video
    model = infer_data(
  File "/root/autodl-tmp/sanya/VLMEvalKit/vlmeval/inference_video.py", line 98, in infer_data
    if getattr(model, 'fps', 0) > 0:
TypeError: '>' not supported between instances of 'NoneType' and 'int'`





Hi @Noctis-SC. I have already fixed this bug in the newest code. This is due to problems with `fps` and `nframe` Settings. You can pull the latest code and execute the same command, and there are no more problems in my attempts.

Hello, @FangXinyu-0913, thank you for your quick fix, everything works for me now.  Thank you again