zhiminy/EvalEng huggingface dataset:
Repository: Vchitect/VBench
Issue: Evaluation data issues 
Root cause: Documentation Deficiency

Hello, Is there an unified configs for the evaluation data (mp4 files), such as the fps, duration, resolution, and etc.

And does the different settings (such as duration and resolution) of mp4 files have an influence on the final evaluation results using VBench? I'm not sure about this.


Hi, thanks for your question! Currently there are several supported formats (like MP4 and GIF). See this function for video loading: https://github.com/Vchitect/VBench/blob/master/vbench/utils.py#L101

VBench has already considered the influence from frame rate and resolution during evaluation.

For example, some dimensions can be sensitive to video frame rate, like "Dynamic Degree". For higher frame rate videos, the dynamics (movement) between two consecutive frames are generally smaller, resulting in a false result of "low dynamic degree". Therefore, we eliminate frame rate's influence at the preprocessing step: https://github.com/Vchitect/VBench/blob/master/vbench/dynamic_degree.py#L86C9-L86C19

So during evaluation, you can directly supply the raw video without any preprocessing to VBench, and VBench will take care of any preprocessing needed for individual evaluation dimensions.
Repository: Vchitect/VBench
Issue: anyway to pre-download files into .cahce/vbench ?
Root cause: Documentation Deficiency

some annoying things happened when i ran vbench evaluation:
#########
--2025-03-25 23:08:28--  (try: 4)  https://huggingface.co/OpenGVLab/VBench_Used_Models/resolve/main/grit_b_densecap_objectdet.pth
Connecting to huggingface.co (huggingface.co)|199.96.61.1|:443... failed: Connection timed out.
Connecting to huggingface.co (huggingface.co)|2a03:2880:f10c:283:face:b00c:0:25de|:443... failed: Cannot assign requested address.
Retrying.
#########
and it still happen even i excute export HF_ENDPOINT=https://hf-mirror.com
so i wonder if i can download all the needed files into .cache/vbench before evaluate

Hey, you can run the following commands.
1. all vbench models are stored in ``./cache/vbench`` just cd into it.
2. Then ``mkdir grit_model`` to create the dir for this specific model.
3. Finally ``wget https://huggingface.co/OpenGVLab/VBench_Used_Models/resolve/main/grit_b_densecap_objectdet.pth`` to save the model in that path.
Repository: Vchitect/VBench
Issue: Release VBench-2.0 on Hugging Face

Hi @ziqihuangg 🤗

I'm Niels and work as part of the open-source team at Hugging Face. I discovered your work through Hugging Face's daily papers as yours got featured: https://huggingface.co/papers/2503.21755.
The paper page lets people discuss about your paper and lets them find artifacts about it (your dataset for instance),
you can also claim the paper as yours which will show up on your public profile at HF, add Github and project page URLs.

Would you like to host the datasets you've released on https://huggingface.co/datasets?
I see you're using Google Drive for it. Hosting on Hugging Face will give you more visibility/enable better discoverability, and will also allow people to do:

```python
from datasets import load_dataset

dataset = load_dataset("your-hf-org-or-username/your-dataset")
```

If you're down, leaving a guide here: https://huggingface.co/docs/datasets/loading.
We also support Webdataset, useful for image/video datasets: https://huggingface.co/docs/datasets/en/loading#webdataset.

Besides that, there's the [dataset viewer](https://huggingface.co/docs/hub/en/datasets-viewer) which allows people to quickly explore the first few rows of the data in the browser.

After uploaded, we can also link the datasets to the paper page (read [here](https://huggingface.co/docs/hub/en/model-cards#linking-a-paper)) so people can discover your work.

Let me know if you're interested/need any guidance.

Kind regards,

Niels

Hi Niels, thanks for the message! Indeed we will host the dataset via huggingface as well. Let me text you for any questions regarding the huggingface dataset. Thanks!

Hi @ziqihuangg I saw you closed this issue, however I don't see a dataset linked to https://huggingface.co/papers/2503.21755.

Are you planning to release that?

Hi Niels,

We’ve released a few datasets related to this work:

https://huggingface.co/datasets/Vchitect/VBench-2.0_human_anomaly

https://huggingface.co/datasets/Vchitect/VBench-2.0_sampled_videos

https://huggingface.co/datasets/Vchitect/VBench-2.0_human_annotation

Let us know the next steps!

Ok thanks, you can link them to the paper by mentioning https://huggingface.co/papers/2503.21755 in the dataset cards.

Also if you're interested, we could look into making the datasets compatible with the Datasets library and the viewer, allowing people to explore the data right from the browser.

Hi Niels,
We add the link in the three datasets above. For 'making the datasets compatible with the Datasets library and the viewer', we’d appreciate it if you could provide detailed instructions.
Repository: Vchitect/VBench
Issue: sampled videos for vbench2 unavailable
Root cause: Documentation Deficiency

hi all, will actual videos generated for getting vbench2 numbers be released for download just like vbench1.0?

Yes, they are being organized and will be released soon.



The sample videos of six models we evaluated are all released. See https://github.com/Vchitect/VBench/tree/master/VBench-2.0/sampled_videos#sampled-videos for details.

much appreciated!
Repository: Vchitect/VBench
Issue: Exactly same leaderboard results for Wan2.1-T2V-1.3B and Open-Sora-2.0(2025-03-18)
Root cause: Validation Gap

<img width="1263" alt="Image" src="https://github.com/user-attachments/assets/6138c90a-f524-4eae-8444-f82d27cba533" />

<img width="1271" alt="Image" src="https://github.com/user-attachments/assets/3e52a92f-6b29-4698-b29a-200eb2092fae" />

Apologies for the confusion — the leaderboard results have been updated accordingly.

That's wonderful! Great thanks!
Repository: Vchitect/VBench
Issue: Vbench2.0 docker
Root cause: Environment Incompatibility

May I ask if you could provide a Docker image for vbench 2.0?

I also tried to build docker for vbench 2.0 and found the dependencies are too complicated, especially the mmcv.  I launched several dockers,  where all of them failed due to the operator in mmcv.It would be nice if VBench team can provide a docker image.

+1 on the docker image request, mmcv is known for it's env compatibility with other packages
Repository: Vchitect/VBench
Issue: Technical Questions Regarding VBench T2V Video Submission Process
Root cause: Documentation Deficiency

Our team has developed a text-to-video generation model and wishes to submit samples for official evaluation. We would greatly appreciate clarification on the following technical points:  

1. File Organization
   Should all video files be placed in a single directory, or must they be organized into subfolders by dimensions (e.g./subject consistency/,/
motion smoothness/)?  

2. Duplicate Prompt Handling
   We encountered two identical prompts in our dataset. How should we name the corresponding video files to avoid conflicts?  

3. Submission Method
   As we noticed limited corporate submissions via GitHub Issues, is email submission acceptable for sharing:  
   • Google Drive download links (video samples)  
   • Model infomation 

4. Evaluation Timeline
   After successful submission, how long does it typically take for results to appear on the Vbench Leaderboard?  

We aim to ensure full compliance with VBench guidelines and appreciate your expertise in helping us navigate the submission process.  

Thank you for supporting open research initiatives!
Repository: Vchitect/VBench
Issue: Reproduction issues for Wan2.1-T2V-1.3B
Root cause: Configuration Error

1. Could you share the hyperparameters or shell for sampling videos?
2. what is the precise setting of guidance_scale, flow_shift, num_inference_steps, sampler, checkpoint(diffusers or not), negative prompt?

I tried to reproduce using guidance_scale=6.0, flow_shift=8.0, sampler=UniPC (diffusers implementation), checkpoint=Wan2.1-T2V-1.3B, num_inference_steps=50, resolution=81x480x832, negative_prompt="色调艳丽，过曝，静态，细节模糊不清，字幕，风格，作品，画作，画面，静止，整体发灰，最差质量，低质量，JPEG压缩残留，丑陋的，残缺的，多余的手指，画得不好的手部，画得不好的脸部，畸形的，毁容的，形态畸形的肢体，手指融合，静止不动的画面，杂乱的背景，三条腿，背景人很多，倒着走"

i used VBench/prompts/augmented_prompts/Wan2.1-T2V-1.3B/all_dimension_aug_wanx.txt

3. environment i used is: torch==2.6.0, cuda12.4, torchvision==0.21.0, transformers==4.46.3

However, there are many differences in results, such as subject_consistency (reproduced 93.04 vs. vbench leaderboard 96.34.



The Wan2.1-T2V-1.3B entry (dated 2025-03-13) on the VBench leaderboard was provided by the Wan Team. It was mistakenly labeled as “sampled by VBench Team,” and we apologize for the confusion this may have caused. The label has since been corrected on the VBench Leaderboard.

We are currently in communication with the Wan Team regarding the reimplementation of this entry and will share any updates as they become available.

In parallel, the VBench Team will allocate resources to sample the Wan2.1-T2V-1.3B model independently and will publicly release our reimplementation results.

We have updated the VBench-sampled results on the leaderboard and [prompt list](https://github.com/Vchitect/VBench/tree/master/prompts/augmented_prompts/Wan2.1-T2V-1.3B)

Thank you. What is the precise setting of guidance_scale, flow_shift, num_inference_steps, sampler, checkpoint(diffusers or not), negative prompt?

We followed the recommended parameters provided in the official repository [https://huggingface.co/Wan-AI/Wan2.1-T2V-1.3B], including guidance_scale = 6.0, num_inference_steps = 50, sampler = unipc, and the following negative prompt:
"色调艳丽，过曝，静态，细节模糊不清，字幕，风格，作品，画作，画面，静止，整体发灰，最差质量，低质量，JPEG压缩残留，丑陋的，残缺的，多余的手指，画得不好的手部，画得不好的脸部，畸形的，毁容的，形态畸形的肢体，手指融合，静止不动的画面，杂乱的背景，三条腿，背景人很多，倒着走".
The only deviation from the default configuration is that we set flow_shift = 3.0, as specifically recommended by the Wan Team. Additionally, we used the GitHub version of the checkpoint.

We have open-sourced  **Wan2.1-T2V-1.3B (2025-05-03)** sampled by VBench Team on Hugging Face (https://huggingface.co/spaces/Vchitect/VBench_Leaderboard). Hope this can be helpful to you.
Repository: Vchitect/VBench
Issue: Cannot reproduce VBench-I2V results on CogVideoX-5b-I2V
Root cause: Environment Incompatibility

We highly appreciate your continuous contribution in t2v, i2v benchmark

Following the guidance, we install the specified environment (cuda 12.1, transformers 4.33.2, torch 2.5.1). We download cogvideox-5b-i2v results from google drive. However, the reproduced results are different from vbench leaderboard. e.g. i2v subject 0.9738 v.s. 0.9719, and we cannot find the explanation. (setting: resolution 3-2, imaging_quality_preprocessing_mode longer)

here is the result json: 

[results_2025-04-27-19_36_33_eval_results.json](https://github.com/user-attachments/files/19929513/results_2025-04-27-19_36_33_eval_results.json)

Could you provide a list of your conda environments？ @Cuogeihong 

We downgraded pytorch to 2.0.1 and solve most inconsistency, but we cannot reproduce camera motion's score: 67.86% v.s. 67.68%

pip environment: [pip_list.txt](https://github.com/user-attachments/files/19953746/pip_list.txt)

our camera motion evaluate result: [results_2025-04-28-19_45_08_eval_results.json](https://github.com/user-attachments/files/19953749/results_2025-04-28-19_45_08_eval_results.json)

We used the pip list you provided and still got the same results as shown on the leaderboard. Could you please share the exact command you used, as well as your conda list (which includes more detailed dependencies)

We change pytorch to 2.1.1 and camera motion score: 67.65 v.s. 67.68%, still having a small gap

exec command: `python evaluate_i2v.py --videos_path $VIDEO_PATH --dimension camera_motion --ratio 3-2 --output_path ./evaluations/$OUTPUT_NAME --full_json_dir ./vbench2_beta_i2v/vbench2_i2v_full_info.json`

conda list:  [conda_list.txt](https://github.com/user-attachments/files/20061713/conda_list.txt)

new camera motion result:  [results_2025-05-06-18_26_32_eval_results.json](https://github.com/user-attachments/files/20061589/results_2025-05-06-18_26_32_eval_results.json)
Repository: Vchitect/VBench
Issue: What is minimum requirement for do "Evaluate Your Videos"?
Root cause: Documentation Deficiency

I evaluate my own dataset but face Cuda out of memory issue when set dimension like dynamic degree, motion_smoothness, and subject consistency. I have 4090 24 gb vram gpu, but ram allocation always reach 64 GB. What is recommended GPU for evaluate my videos?

It depends on the video’s length and resolution. You can try using VBench-Long, which splits longer videos into shorter clips before evaluation.

Are there any guidelines? It can be tough reserving cloud GPUs, so I'd hate to finally nab a server, only to find out it's too small.

Thank you for the suggestion @ziqihuangg.  So, I need to replace the script using the vbench2_beta_long folder
Repository: Vchitect/VBench
Issue: cannot download sa_0_4_vit_l_14_linear.pth
Root cause: Environment Incompatibility

when i download the "sa_0_4_vit_l_14_linear.pth"  through https://github.com/LAION-AI/aesthetic-predictor/blob/main/sa_0_4_vit_l_14_linear.pth?raw=true. The webpage shows "Access has been restricted".

![Image](https://github.com/user-attachments/assets/5b8673e9-99bb-4f6a-a3ea-1c1a51de587c)

I’m able to access the .pth file without issues. Please check your network or local configuration. Alternatively, you can manually download the file from GitHub and place it in your cache directory.
Repository: Vchitect/VBench
Issue: Reproduce CogVideoX-5B Evaluation Results
Root cause: Documentation Deficiency

Hi, thanks for your great work!

I'm trying to reproduce the results for CogVideoX-5B (diffusers) using the evaluation code and the provided videos. However, I'm consistently getting different scores across several dimensions. For example, I obtain a subject consistency score of 91.8%, whereas the leaderboard reports 96.45%.

Could you please share more detailed information about the exact environment setup and dependencies used to obtain the reported results? I’ve tried several different environments, but I keep getting the same (lower) result.

Thanks in advance!

Please provide the command you used for evaluation.

Hi, and thanks for the quick response!
I tried both of the following commands:
vbench evaluate --videos_path ../CogVideoX-5B/subject_consistency/ --dimension subject_consistency
python evaluate.py --videos_path ../CogVideoX-5B/subject_consistency/ --dimension subject_consistency

In both, I get an evaluation of 360 videos with a mean score of 91.8%

You should use VBench-Long for videos of 5.0 or longer.

Thank you!
Repository: Vchitect/VBench
Issue: Sampled videos for i2v
Root cause: Documentation Deficiency

Thanks for providing the sampled videos. Are the sampled videos for VBench-i2v provided as well? I couldn't find a link.

VBench-I2V sampled videos are provided [here](https://drive.google.com/drive/folders/1EsNeCDg3TvJ-BfKA7QuwWWXVlDqprhCM?usp=sharing).

Thanks!
Repository: Vchitect/VBench
Issue: which paper is for vbench 1.0 ?

I feel quite confused about following 2 papers:
1. https://arxiv.org/pdf/2311.17982
2. https://arxiv.org/pdf/2411.13503

Both are. The second link is an extended version (as commented in the arXiv abstract page) of the CVPR 2024 paper, which further includes VBench-Long, VBench-I2V, VBench-Trustworthiness, and more.

> Both are. The second link is an extended version (as commented in the arXiv abstract page) of the CVPR 2024 paper, which further includes VBench-Long, VBench-I2V, VBench-Trustworthiness, and more.

Got it! thanks.
Repository: Vchitect/VBench
Issue: is the huggingface dataset same with the one on google drive?
Root cause: Documentation Deficiency

is the huggingface dataset{  [dataset1](https://huggingface.co/datasets/Vchitect/VBench_sampled_video) ,  [dataset2](https://huggingface.co/datasets/Vchitect/VBench-2.0_sampled_videos)   } same with the one on [google drive](https://drive.google.com/drive/folders/13pH95aUN-hVgybUZJBx1e_08R6xhZs5X) ?

Hi, could you clarify which Hugging Face dataset you're referring to and share the link with us?

> Hi, could you clarify which Hugging Face dataset you're referring to and share the link with us?

sorry, I provide the wrong link before. could u check it again?
Repository: Vchitect/VBench
Issue: how to execute the project ? 
Root cause: Documentation Deficiency

its great effort from the authors. can anyone tell step by step explanation to run the project. your effort will be highly appreciated. 

Sorry for the late reply. 
The simplest way is to follow these steps: 
1. Install VBench by running `pip install vbench`. 
2. Generate the corresponding videos with your text-to-video model based on the dimensions listed in the [prompts list](https://github.com/Vchitect/VBench/tree/master/prompts) or using the [all dimension](https://github.com/Vchitect/VBench/blob/master/prompts/all_dimension.txt). 
3. Save the [VBench_full_info.json](https://github.com/Vchitect/VBench/blob/master/VBench_full_info.json) file in the current directory. 
4. Use the command `vbench evaluate --videos_path $VIDEO_PATH --dimension $DIMENSION` in the command line. 
5. Find the results for the corresponding dimension in the `evaluation_results` directory.

Since there is no more follow-up questions for a while, I'm closing the issue. Feel free to re-open it if you encounter more issues in executing the project.
Repository: Vchitect/VBench
Issue: Re: VBench I2V vs T2V

Hello,

Have been a long-time follower/fan of VBench as the best objective way to evaluate video gen models. I was a little curious regarding how comparable/inter-changeable the I2V benchmarks vs regular VBench is. If model 1 has higher total score on I2V bench vs model 2 with a lower total score on regular VBench, can model 1 be considered overall "better" in their video outputs?

Thank you!
Repository: Vchitect/VBench
Issue: what gpu can I use for vbench2
Root cause: Documentation Deficiency

I use a L20, but it outs "NotImplementedError: Cannot copy out of meta tensor; no data!"
I checked the issue from llava, they explained that this is a typical error for OOM. 
So I want to know how much memory should I use?

Hi, we show the memory cost below:
For 'Complex_Plot', 'Complex_Landscape', 'Human_Interaction' and 'Motion_Order_Understanding', it requires 50G
For 'Composition', 'Dynamic_Attribute', 'Dynamic_Spatial_Relationship', 'Human_Clothes', 'Material', 'Mechanics', 'Thermotics', 'Motion_Rationality', it requires 38G
'Human_Anatomy' : 5G
'Human_Identity': 2G
'Diversity': 28G
'Camera_Motion', 'Multi-View_Consistency': 18G
'Instance_Preservation': 3G




> Hi, we show the memory cost below: For 'Complex_Plot', 'Complex_Landscape', 'Human_Interaction' and 'Motion_Order_Understanding', it requires 50G For 'Composition', 'Dynamic_Attribute', 'Dynamic_Spatial_Relationship', 'Human_Clothes', 'Material', 'Mechanics', 'Thermotics', 'Motion_Rationality', it requires 38G 'Human_Anatomy' : 5G 'Human_Identity': 2G 'Diversity': 28G 'Camera_Motion', 'Multi-View_Consistency': 18G 'Instance_Preservation': 3G

Thank you for your reply!
Repository: Vchitect/VBench
Issue: Conflict of environment between hpsv2 and mediapipe for vbench2.0
Root cause: Environment Incompatibility

When I prepare the environment for vbench2.0, a conflict occurs between hpsv2==1.2.0 and mediapipe. The hpsv2==1.2.0 requires protohuf<4, while mediapipe requires protohuf>=4.25.0. I wonder how to fix this problem? Which version of mediapipe should I install?
Thanks!

Hi, you can use the version below:
hpsv2==1.2.0
mediapipe==0.10.11
protohuf==3.20.3
Repository: Vchitect/VBench
Issue: code of Line 47 makes no sense at all.
Root cause: Algorithmic Error

https://github.com/Vchitect/VBench/blob/3ef98d3e2c8fa082fc0b676d23b4629d41df39bd/vbench/third_party/grit_src/grit/predictor.py#L47
https://github.com/Vchitect/VBench/blob/3ef98d3e2c8fa082fc0b676d23b4629d41df39bd/vbench/third_party/grit_src/grit/predictor.py#L48
Repository: Vchitect/VBench
Issue: __init__() got an unexpected keyword argument 'cache_dir'
Root cause: Interface Contract Mismatch


when I evaluate videos on "human_action" dimension , I met the above error in both version in requirements.txt and latetst version.
then I have to add "kwargs.pop('cache_dir', None) "before the following line.

https://github.com/Vchitect/VBench/blob/3ef98d3e2c8fa082fc0b676d23b4629d41df39bd/vbench/third_party/umt/models/modeling_finetune.py#L351
Repository: Vchitect/VBench
Issue: data structures of sample videos in Google Drive  are not clear
Root cause: Documentation Deficiency

the data structure of some model are not clear. For example,"perdimension/AccVideo"is not clear

1. some  links or folder names looks a bit messy in [here](https://github.com/Vchitect/VBench/tree/master/sampled_videos)

2. these two links( [link1](https://drive.google.com/file/d/1iNBcAGakVLZge9BtIFgwccR0RD9w52it/view?usp=drive_link) [link2](https://drive.google.com/drive/folders/1TnABYBauOJicjpYkxW2ixr_t_klD0S67?usp=drive_link) )are too confusing.

3. google drive link of sample videos of `Hunyuan Video (2025-05-22)` is not provided

4. number of sample videos of `Sora` in the google drive link is not enough compared to other models

Thanks for your feedback.

In [sampled_videos](https://github.com/Vchitect/VBench/tree/master/sampled_videos), we maintain a table that lists all the video models we have sampled and the Google Drive links where the results are stored. We ensure every sampled model is recorded here with its corresponding video link. You can simply find the model you are interested in and its Google Drive link from it, without needing to worry about the actual directory structure within Google Drive.

For example:

The [link2](https://drive.google.com/drive/folders/1TnABYBauOJicjpYkxW2ixr_t_klD0S67) you mentioned corresponds to the [Open-Sora 2.0 (2025-03-18)](https://github.com/Vchitect/VBench/tree/master/sampled_videos#what-are-the-details-of-the-video-generation-models.) entry in this table;

Hunyuan Video (2025-05-22) is also listed in the same table.
Repository: Vchitect/VBench
Issue: The version of mmcv
Root cause: Configuration Error

I followed the instruction in Readme and installed mmcv==2.2.0. However, it results in "UnImplemented dimension Human_Anatomy! MMCV==2.2.0 is used but incompatible. Please install mmcv>=2.0.0rc4 < 2.1.0

The issue has already been mentioned in our installation guide.

’‘There will be an error about the mmcv version exceeds 2.1.0, users could directly change the mmcv_maximum_version to 2.3.0 in Your_PATH_to_Anaconda/env/vbench2/lib/python3.10/site-packages/mmdet/__init__.py and Your_PATH_to_Anaconda/env/vbench2/lib/python3.10/site-packages/mmyolo/__init__.py’‘