zhiminy/EvalEng huggingface dataset:
Repository: Vchitect/VBench
Issue: Cogvideo2B score
Root cause: Configuration Error

Hi, I see that the total score of cogvideo2B on Leaderboard is 80.94%, but after I use all_dimension_long. txt to inference, the total score measured is only 78.68%.
The video I produced with cogvideo2B was 8 frame rate, 6s length, and 480x720 resolution.
May I ask why my test result is so much lower than that on the leaderboard?
Looking forward to your reply，thanks a lot

Could you provide the details of the model checkpoint and sampling setting?

> Could you provide the details of the model checkpoint and sampling setting?

Model weights are download from https://huggingface.co/THUDM/CogVideoX-2b/tree/main here
Inference code is inference/cli_demo.py from Cogvideo2B repo
And sampling setting is the default setting, I only change some dir path

> > Could you provide the details of the model checkpoint and sampling setting?
> 
> Model weights are download from https://huggingface.co/THUDM/CogVideoX-2b/tree/main here Inference code is inference/cli_demo.py from Cogvideo2B repo And sampling setting is the default setting, I only change some dir path

Hello, here are our settings for sampling CogVideoX-2B (last line): https://github.com/Vchitect/VBench/tree/master/sampled_videos#what-are-the-details-of-the-video-generation-models. We use the [SAT](https://github.com/THUDM/CogVideo/tree/1c2e487820e35ac7f53d2634b69d48c1811f236c/sat) weights to sample videos for evaluation.

> > Could you provide the details of the model checkpoint and sampling setting?
> 
> Model weights are download from https://huggingface.co/THUDM/CogVideoX-2b/tree/main here Inference code is inference/cli_demo.py from Cogvideo2B repo And sampling setting is the default setting, I only change some dir path

And for evaluation, we use the [VBench-long code](https://github.com/Vchitect/VBench/tree/master/vbench2_beta_long) to evaluate sampled videos.

Hi, I also tried the sat weights to sample videos, and got a new result 79.75% which is still much lower than the report result.
For evaluation, I still use the old evaluation code, could these cause problem? Since the only difference between using the longer txt and the original is the length of the prompt

What prompt list did you use?

https://github.com/Vchitect/VBench/blob/master/prompts/gpt_enhanced_prompts/all_dimension_longer.txt
this one

Hi, we've released the following details for evaluating CogVideoX-2B: 
- videos used for evaluation: https://drive.google.com/file/d/1zuQ47Uvze4157o4YMta0Zqz9G8TdHcXZ/view?usp=share_link
- code used for evaluation: https://github.com/Vchitect/VBench/tree/master/vbench2_beta_long

Feel free to let us know if you find any discrepancy. Thanks!

Closing the issue for now. Feel free to reopen if you feel it's not solved.
Repository: Vchitect/VBench
Issue: Unrecognized arguments: --mode=custom_input when running vbench
Root cause: Environment Incompatibility

Hello,
thanks for your great work!

I'm having trouble running the command provided in the instruction:

> vbench evaluate \
    --dimension $DIMENSION \
    --videos_path /path/to/folder_or_video/ \
    --mode=custom_input

The error output message I got was:

> vbench: error: unrecognized arguments: --mode=custom_input

So, I wonder, is it because vbench installed directly from pip does not support customized input videos?

Hi, Can you check the version of the vbench. Is it installed from pypi or installed from source?

Thanks for your prompt reply!

It's installed from pypi :)

There might be some problems with my virtual environment. After adjusting the environment, the same command works just fine for me today.
Repository: Vchitect/VBench
Issue: Difference between i2v_subject and subject_consistency

Hi!

- i2v_subject (including i2v_background/camera motion) is added for image-to-video evaluation protocol.
- Then what is the difference between i2v_subject and subject_consistency (for video quality)??

Thanks!

Thank you for raising this issue! It seems that your question has already been addressed in [issue #34](https://github.com/Vchitect/VBench/issues/34).
Repository: Vchitect/VBench
Issue: imaging_quality
Root cause: Documentation Deficiency

what's the process about 'imaging quality'.  (how to eval imaging quality dimension about t2v model)
cannot find any code call imaging_quality.py compute_imaging_quality
and also cannot find prompt txt about imaging quality dimension on https://github.com/Vchitect/VBench/tree/master/prompts/prompts_per_dimension

It will be imported here: 
https://github.com/Vchitect/VBench/blob/6617febb1969225f174992a198376201c6bfaec0/vbench/__init__.py#L147-L148
and called in https://github.com/Vchitect/VBench/blob/6617febb1969225f174992a198376201c6bfaec0/vbench/__init__.py#L153 For the prompt, it uses the same prompt as `overall_consistency`.
Repository: Vchitect/VBench
Issue: How do I get the final score for i2V locally?
Root cause: Documentation Deficiency


For the Total Score calculation for VBench-I2V, you can refer to the explanation in this [section](https://github.com/Vchitect/VBench/tree/master/vbench2_beta_i2v#submit-to-leaderboard). We have also provided the corresponding calculation script.
Repository: Vchitect/VBench
Issue: Opensoraplan-v1.1  score
Root cause: Documentation Deficiency

@lmxyy @xjxu21 @piddnad @JustinYuu @simplaj ,
Hi, every, I just evaluate the Opensoraplan-v1.1 score with your provided videos and evaluation toolkit. But I can not get the same score as your report in the leadboard.  
Besieds, I also evaluate the opensorav1.0, also with your provided videos and toolkit, and can get the same score.  
I want to make sure for the Opensoraplan-v1.1 results, do you use the released videos for evaluation?

Thanks very much.

May I ask which evaluation pipeline did you use for evaluation? Is it VBench or VBench-Long? Thanks!

VBench

here is my evaluation results.
<img width="1419" alt="截屏2024-09-06 16 22 34" src="https://github.com/user-attachments/assets/9cc08d21-cc1a-4d0e-8b7b-70d747fd4b8d">


Hi @LWang1016 , I notice that you use VBench tools to evaluate Opensorav1-1, which will not get the results we published in the leaderboard. 

We recommend you to use [VBench-Long](https://github.com/Vchitect/VBench/tree/master/vbench2_beta_long) to evaluate Opensorav1-1, because this version generates long videos ~10s. You will get the same results in the leaderboard by using [VBench-Long](https://github.com/Vchitect/VBench/tree/master/vbench2_beta_long). Thanks! 

Contact us if you have other questions.

@MqLeet. I just download  your released generated videos to do the metric evaluation.  


Hi, @LWang1016, we will close this issue. 

Feel free to contact us if you have other questions.
Repository: Vchitect/VBench
Issue: Run evaluation suite on gen3 videos
Root cause: Documentation Deficiency

I'm trying to run the [eval suite](https://github.com/Vchitect/VBench?tab=readme-ov-file#surfer-evaluation-method-suite) on Gen3 videos from the Google Drive [here](https://github.com/Vchitect/VBench/tree/master/sampled_videos#what-are-the-details-of-the-video-generation-models). 

I'm not sure how go ahead with it as there are no dimensions in the Gen3 folder. What do I pass for `--dimension $DIMENSION`? 

@ziqihuangg 

Thanks for asking! The Gen-3 folder contains videos of all the dimensions. You can specify the dimension name you want to evaluate, and our script will automatically sort out the relevant videos for that particular dimension among all the videos.
Repository: Vchitect/VBench
Issue: There is still a lot of data missing from per catetory.
Root cause: Unimplemented Feature Gap

Thank you very much for your excellent work, but I saw that a lot of data of top-ranked models were missing in the per_category directory of the data set. I wonder when the data will be uploaded, thank you very much. @ziqihuangg 

Thanks for your question! The main leaderboard primarily focuses on per_dimension evaluation. We will gradually add the per_category videos based on resources available. Thanks for your patience.
Repository: Vchitect/VBench
Issue: discriminability of the evaluation metrics
Root cause: Unimplemented Feature Gap

@ziqihuangg Thanks for the great work👏! I noticed that some metrics, e.g. subject consistency, background consistency, motion smoothness, have relatively high score (close to 1) on the [Vbench Leaderboard](https://huggingface.co/spaces/Vchitect/VBench_Leaderboard). Is there any method that can increase the discriminability of the evaluation metrics?

Thanks for this question! Indeed, some dimensions have become gradually saturated, and that's largely because recent models have already addressed these aspects quite well (e.g., achieving good frame consistency and smooth motion). We are working on new solutions to increase discriminability and evaluate more challenging aspects in video generation (no promises yet, but stay tuned for updates in our future releases 👀). 

> [@ziqihuangg](https://github.com/ziqihuangg) Thanks for the great work👏! I noticed that some metrics, e.g. subject consistency, background consistency, motion smoothness, have relatively high score (close to 1) on the [Vbench Leaderboard](https://huggingface.co/spaces/Vchitect/VBench_Leaderboard). Is there any method that can increase the discriminability of the evaluation metrics?

Check out VBench-2.0: https://vchitect.github.io/VBench-2.0-project/
Repository: Vchitect/VBench
Issue: Can you tell me which metrics are needed to calculate with prompt?
Root cause: Documentation Deficiency

Can you tell me which metrics are needed to calculate consistency with prompt？

`Temporal Style`, `Appearance Style`, `Scene`, `Spatial Relationship`, `Color`, `Human Action`, `Multiple Objects`, `Object Class` and `Dynamic Degree`. You can find more detail for each metric in the [paper](https://arxiv.org/pdf/2311.17982).
Repository: Vchitect/VBench
Issue: Problem about installing detectron2 on CUDA Verson: 12.3
Root cause: Environment Incompatibility

Hello, It seems like the max supported CUDA version of detectron2 is 11.3. 
<img width="502" alt="image" src="https://github.com/user-attachments/assets/8e7f98eb-a53f-4026-b0f5-4d097ce5af38">

When I install it in CUDA 12.1 or 12.3, it always failed. But I found that vbench usually require a higher CUDA Version. Even I install detectron2 in CUDA 12.1. When I run vbench evaluation codes, it still return the error which is shown as below:
![image](https://github.com/user-attachments/assets/0f55c18e-8a58-4b10-8a80-7f314f306988)
 
Could you please tell me how to deal with this incompatibility issues?


I solved this problem by first install detectron2, then install vbench

          name: linux_gpu_tests_pytorch2.1.1
          pytorch_version: "2.1.1"
          torchvision_version: "0.16.1"
          cuda_version: "12.1"
          python_version: "3.11"
          pytorch_index: "https://download.pytorch.org/whl/nightly/cu121"
Repository: Vchitect/VBench
Issue: Question about object detection

Hi, nice work! I have a question about your 'object class' dimension. My understanding is that Grit can predict some detected objects and their corresponding captions, which might not exactly be the same in the original prompt. For example, the original prompt could be 'a cat', the captions of the detected objects could end up being 'an orange cat', 'grass', 'bench', .... How do I check the object ('a cat') is successfully detected ('an orange cat')? Is it using LLM or clip similarity?

Hi, sorry for the late reply. The output of GRiT contains the object's attribute information, but in the actual calculation, this content is not taken into account. When the prompt is "a cat", for example, if the output in GRiT is "an orange cat", the model will be judged as generating correctly. It only needs to ensure that the output contains "cat", and the color or other attribute information will be evaluated from another dimension.

Thanks a lot!
Repository: Vchitect/VBench
Issue: How to calculate I2V final score?
Root cause: Documentation Deficiency

I have obtained the scores for each dimension. Now I want to calculate the 'I2V Score' and 'Quality Score' and the final score in your I2V leaderboard.

For the Total Score calculation for VBench-I2V, you can refer to the explanation in this [section](https://github.com/Vchitect/VBench/tree/master/vbench2_beta_i2v#submit-to-leaderboard). We have also provided the corresponding calculation script.
Repository: Vchitect/VBench
Issue: Questions about the input prompts used in the VBench leaderboard
Root cause: Documentation Deficiency

1. Are the input prompts for the models in the VBench leaderboard all the same?
2. Among the provided prompts, some are optimized by GPT and some are not. Which type is used in the evaluation?
3. Regarding the evaluation of CogVideoX, is the video generated after rewriting the prompt using https://github.com/THUDM/CogVideo/blob/main/inference/convert_demo.py

Hi, thank you for these important questions!

(1) Yes, it’s all the same. We use this prompt file: https://github.com/Vchitect/VBench/blob/master/prompts/all_dimension.txt
(2) By default, we use the [all_dimension.txt](https://github.com/Vchitect/VBench/blob/master/prompts/all_dimension.txt) file unless a specific model requires prompt optimization as an intrinsic part of its pipeline (e.g., CogVideoX). If prompt optimization is applied, it will be indicated in the model name with a label such as "prompt optimized" on our VBench Leaderboard. For closed-source or API-based models, we directly feed the original prompts as input, leaving it to the model's internal mechanism to handle prompt optimization if there is any.
(3) Yes, we follow CogVideoX’s official pipeline for prompt augmentation. We also maintain a copy of our script and the augmented prompts, which are publicly available here: [GPT Enhanced Prompts](https://github.com/Vchitect/VBench/tree/master/prompts/gpt_enhanced_prompts).
Repository: Vchitect/VBench
Issue: Are there plans to release sample videos generated by each model for each prompt, allowing everyone to compare them directly？

Similar to https://ailab-cvc.github.io/VideoGen-Eval, quantitative evaluations and qualitative assessments validate each other？

The sampled videos are publicly available. You can access them here:
https://drive.google.com/drive/folders/1on66fnZ8atRoLDimcAXMxSwRxqN8_0yS?usp=sharing

Arena released here: https://huggingface.co/spaces/Vchitect/VBench_Video_Arena
Repository: Vchitect/VBench
Issue: prompt&video
Root cause: Documentation Deficiency

Great work, I would like to know does the name of the video have to be the same as the prompt If the prompt is exceptionally long and exceeds the maximum value of the video name how should it be executed?

The vbench standard prompt in [here](https://github.com/Vchitect/VBench/blob/564bc3dc00f365103f6b78b7452c904b4362ff48/vbench/VBench_full_info.json) will not exceed the maximum value of file name length. But in case you have your own prompt and that is longer than 255, you can use these commands:
1. `... --videos_path <path/to/video.mp4> --mode=custom_input --prompt <your prompt>`
2. `... --videos_path <path/to/video_dir> --mode=custom_input --prompt_file <path/to/json file that map filenames to prompts>`

Thank you for your reply, but I have a new problem, I tried to run it and found that there are some metrics with negative scores, like temporal_style, is this as expected and what does the negative number mean?
![image](https://github.com/user-attachments/assets/fdec19cd-b9be-424b-80ec-dcc8bd2a9da2)



Which dimension is it?

hi，my evaluation instructions are as follows：
vbench evaluate --dimension subject_consistency --videos_path v1.mp4 --mode=custom_input --prompt A man is dancing

evaluation framework returns error:
vbench: error: unrecognized arguments: man is dancing

Are there any plans in the future to support custom prompts and video reviews in the Controllability dimension?

Thank you for your interest. Custom video evaluation can only be done on the dimension without auxiliary_info and Controllability is not satisfied. We update this feature for five dimensions. Please refer to https://github.com/Vchitect/VBench/tree/master/VBench-2.0#new-evaluating-single-dimension-of-your-own-videos

> hi，my evaluation instructions are as follows： vbench evaluate --dimension subject_consistency --videos_path v1.mp4 --mode=custom_input --prompt A man is dancing
> 
> evaluation framework returns error: vbench: error: unrecognized arguments: man is dancing

Seems like you didn't "quote" your input prompt

Closing the issue as all questions appear to be addressed. Feel free to reopen it if you believe anything remains unresolved.
Repository: Vchitect/VBench
Issue: human_action question
Root cause: Documentation Deficiency

I've tried many times and the human_action score stays at 0. Can you give me some ideas?
{
    "human_action": [
        0.0,
        [
            {
                "video_path": "./sampled_videos/anime-new-vbench/0001.mp4",
                "video_results": false,
                "cor_num_per_video": 0
            },
            {
                "video_path": "./sampled_videos/anime-new-vbench/0002.mp4",
                "video_results": false,
                "cor_num_per_video": 0
            },
            {
                "video_path": "./sampled_videos/anime-new-vbench/0003.mp4",
                "video_results": false,
                "cor_num_per_video": 0
            },
            {
                "video_path": "./sampled_videos/anime-new-vbench/0004.mp4",
                "video_results": false,
                "cor_num_per_video": 0
            },
            {
                "video_path": "./sampled_videos/anime-new-vbench/0005.mp4",
                "video_results": false,
                "cor_num_per_video": 0
            }
]

Mabey because the video_path is not correct. I was trying to evaluate some videos generated by myself and I met the same problem. Then I read the code in **vbench/human_action.py**, I found they may store action information in the file name. And they detect the actions in the video, compare them with the target action which appears in the file name.

i also find this question, it shows that the encode embedding of the video is 0, so strange

Thanks for your interest in our work! For the human action dimension, since the classifier categories are fixed, we do not support customized video evaluation yet, and the name of the sampled video should be the same as the prompt for evaluation when using our provided prompt list.

so how to compute the long-prompt's human action?

VBench's human_action dimension currently operates only on the predefined benchmark prompts.

If you're referring to augmented prompts, you can still use the same evaluation pipeline—just apply it to the longer prompts derived from the human_action short prompts.

If you're referring to open-domain prompts, that’s not supported yet in this dimension. However, feel free to explore VBench-2.0, which offers expanded evaluation options for motion rationality and human fidelity.

Thanks!
Repository: Vchitect/VBench
Issue: why requires torch>=1.12,<2.0.0
Root cause: External Dependency Breakage

How to use Vbench in torch 2.4 

          name: linux_gpu_tests_pytorch2.1.1
          pytorch_version: "2.1.1"
          torchvision_version: "0.16.1"
          cuda_version: "12.1"
          python_version: "3.11"
          pytorch_index: "https://download.pytorch.org/whl/nightly/cu121"

Already support torch>=2.0 with the recent commits. But since detectron2 is one of the dependency, there might still be some conflict with some cuda version.
Repository: Vchitect/VBench
Issue: How to submit to the VBenchI2V LeaderBoard?
Root cause: Documentation Deficiency

Following [instructions](https://github.com/Vchitect/VBench/tree/master?tab=readme-ov-file#trophy-leaderboard), option 3, we have generated a .zip file and submit it to the leaderboard. However, it seems that the results will be displayed in T2V module (i.e., the VBench module). However, since our work focuses on I2V models, our intent is for it to be showcased in VBenchI2V module. So how to submit to the VBenchI2V LeaderBoard?
Repository: Vchitect/VBench
Issue: About the video generation model in the hugging face Leaderboard

Hi @simplaj,
Thanks for the amazing work!
About the leaderboard on the [HuggingFace](https://huggingface.co/spaces/Vchitect/VBench_Leaderboard), what is the specific version of the pyramid flow model?

Thank you for the question!

For the Pyramid Flow model, the evaluated results are uploaded by the model team. Currently, we only have the link to its project page: [Pyramid Flow Project Page](https://pyramid-flow.github.io/).

We will follow up with them regarding specific model versions or checkpoints and maintain a public record at [Details of the Video Generation Models.](https://github.com/Vchitect/VBench/tree/master/sampled_videos#what-are-the-details-of-the-video-generation-models)

In the meantime, feel free to contact the Pyramid Flow model team directly for more detailed information.
Repository: Vchitect/VBench
Issue: The number of videos in the published sampled videos is inconsistent
Root cause: Validation Gap

When I downloaded the video files published in Google Drive for testing, I found that their file numbers were inconsistent, taking MiniMax, CogVideoX-2B and OpenSoraPlanv1-1 as examples. 
I used MiniMax as a benchmark and counted the file name differences between CogVideoX-2B and OpenSoraPlanv1-1 relative to MiniMax. The results are in the attachment.

The main differences are as follows:
1. **The CogVideoX-series video repeats the videos in the temporal_flickering category 25 times instead of 5 like the others.**
2. **There are two duplicate prompts in the prompt, "A fantasy landscape" and "a fantasy landscape".** Some will choose one, such as MiniMax, while some will use both, such as OpenSoraPlanv1-1. It seems that some will use a mixture of the two, which is a bit random.

I don't know if this difference will affect the fairness of the comparison. Perhaps we can **set a standard for the number of files and file naming, and use a pre-process to check whether the submitted video meets the requirements.**

[MiniMax-OpenSoraPlanv1-1.txt](https://github.com/user-attachments/files/17870389/MiniMax-OpenSoraPlanv1-1.txt)
[MiniMax-CogVideoX-2B.txt](https://github.com/user-attachments/files/17870390/MiniMax-CogVideoX-2B.txt)


Thanks for your questions and comments.

When using VBench to calculate the `Temporal Flickering` of a short video, we sample 25 videos to filter out 5 static ones. For long videos evaluated with VBench-Long, each long video is sliced into shorter 2-second clips. In this case, only the first 5 short static clips are used for scoring, and these clips are exclusively derived from the first 5 long videos. Thus, sampling the first 5 long videos is sufficient. The previous requirement of 25 videos was redundant but does not affect the reproducibility of the evaluation results.

Regarding the prompts "A fantasy landscape" and "a fantasy landscape," we sample videos for each prompt separately and calculate scores independently. However, a case sensitivity issue during the upload process led to conflicting file names, resulting in missing videos. We are re-uploading the missing videos. Thanks for your patience!