zhiminy/EvalEng huggingface dataset:
Repository: Vchitect/VBench
Issue: Difference between "i2v_subject" and "subject_consistency" dimensions


Hi, "i2v_subject" under "I2V" refers to subject consistency _between the input image and the generated videos_, whereas "subject_consistency" under "Video Quality" refers to subject consistency _among the generated video frames_.

thank you very much,
the "i2v_subject" has not reported for the base I2V models in leaderboard correct? 

The preliminary results have been posted on our [leaderboard](https://huggingface.co/spaces/Vchitect/VBench_Leaderboard) under "VBench-I2V", under the "Video-Image Subject Consistency" dimension. We want to clarify that these results can be replicated using our current open-source VBench-I2V code. However, we are considering optimizations to the evaluation pipeline to align it even more closely with human perception. Therefore, there may be a slight update to the leaderboard results later this month.

![Screenshot 2024-06-04 at 10 36 08 AM](https://github.com/Vchitect/VBench/assets/66513392/597f4d83-67d2-4447-9e7f-1d4cec47dbfa)


Thank you very much for your response.

Could you please clarify the temporal flickering dimension for the I2V suite? I was unable to reproduce the leaderboard results for SVD; I obtained a number of 0.9476 for SVD-XT. Additionally, I could not find any usage of 'temporal_flickering' in the VBench repository at [I2V](https://github.com/Vchitect/VBench/tree/master/vbench2_beta_i2v).
is that as effective evaluation dimension as for T2V or other dimensions mentioned in the [I2V](https://github.com/Vchitect/VBench/tree/master/vbench2_beta_i2v) would be enough for I2V evaluation?
Repository: Vchitect/VBench
Issue: KeyError: 'imaging_quality_preprocessing_mode'
Root cause: Validation Gap

Hi, there is an error using python script for evaluation:

```
python3.10/site-packages/vbench/imaging_quality.py", line 30, in technical_quality
    preprocess_mode = kwargs['imaging_quality_preprocessing_mode']
KeyError: 'imaging_quality_preprocessing_mode'
```

Can you check if vbench package you installed and the python script are from the same version. I cannot replicate the issue with the latest version.

Hi, I'm closing the issue as there's no follow-up question, but feel free to re-open it anytime if you have any further questions!

@quantumiracle You need to pass the `imaging_quality_preprocessing_mode` into kwargs as in [evaluate.py](https://github.com/Vchitect/VBench/blob/58642fd51e272b93b7a132e19ede3a82a1d69ab9/evaluate.py#L143) when calling `VBench.evaluate`.

@NattapolChan @ziqihuangg In your [README](https://github.com/Vchitect/VBench?tab=readme-ov-file#python) python example, you also do not specify this argument. I guess that was why we met this error and you could reproduce it. 

Thanks for pointing this out. I didn't check calling VBench.evaluate() directly. This should be fixed in [2048fb0](https://github.com/Vchitect/VBench/commit/2048fb0e9e0b158e99bb59d525fd19a16627ef8c).
Repository: Vchitect/VBench
Issue: Division by 0 when directory only contains a single video
Root cause: Algorithmic Error

Hi,

When the input directory contains a single video asset, this [line](https://github.com/Vchitect/VBench/blob/master/vbench/subject_consistency.py#L56) will throw an exception related to division by 0:
```
sim_per_video = sim / (len(video_list) - 1)
```
I believe it should be
```
sim_per_video = sim / len(video_list)
```
Can you confirm that's the case?

Hi, this is a bug and we've fixed it! Thanks for pointing it out.
Repository: Vchitect/VBench
Issue: Pika, Gen2 latest results
Root cause: Documentation Deficiency

First of all, thank you for creating this benchmark. It is an important contribution. I see both Pika and Gen-2 results are quite old. I’m wondering if it’s possible to update those numbers just to see how far open source models are behind the close source models. Thanks! 

Hi, thanks for this question. We are actually in the middle of evaluating the latest Gen-2 and Pika. The results will be updated to our leaderboard.

Thanks. I'm assuming you will be adding SVD as well. 

Hi, SVD is already in the VBench-I2V leaderboard.

Sorry I missed that. Can you add [EasyAnimate](https://github.com/aigc-apps/EasyAnimate) too?
Repository: Vchitect/VBench
Issue: Evaluation issue:
Root cause: Validation Gap

I wrote a script for evaluation,but I encoutered a issue then:
1、script:
from vbench import VBench
import torch
device = torch.device("cuda")
my_VBench = VBench(device, "VBench_full_info.json", "evaluation_results")
my_VBench.evaluate(
    videos_path = "./videocrafter/spatial_relationship", #there are several videos  in this directory
    name = "spatial_relationship",
    dimension_list = ["spatial_relationship"],
    )
2、issue:
RuntimeWarning: invalid value encountered in scalar divide
  ret = ret.dtype.type(ret / rcount)

3、evaluate results:
![image](https://github.com/Vchitect/VBench/assets/61417471/2e0e0b2b-5945-4b88-bdaf-094594e19b99)


Hi, thanks for your interest in our work!
The issue may be attributed to the wrong sampled video format. As we mentioned in the [How to Sample Videos for Evaluation](https://github.com/Vchitect/VBench/tree/master/prompts) section, the names of videos should be in the form of **$prompt-$index.mp4**. You can refer to the **$dimension_full_info.json** in the _evaluation_reuslts/_ folder to check if the sampled videos exist in the _“video_list"_ key of each prompt.

Thanks for your information！I‘ll try it later！

Hi, we're closing this issue as it appears your questions have been addressed. However, feel free to open a new issue or reopen this one, if you have further questions or if anything else comes up related to this issue.

still getting NaN with generated input from [Mochi g-drive](https://drive.google.com/file/d/1zDlDQmtR3SGZzWmNCoeFe0jKACxloWqR/view), for example:
An astronaut feeding ducks on a sunny afternoon, reflection from the water.-4.mp4
Repository: Vchitect/VBench
Issue: When evaluate aesthetic_quality, 'CUDA out of memory' always occur for multiple videos
Root cause: Resource Mishandling

When evaluating aesthetic_quality for my own videos, I always get the error 'CUDA out of memory', which leading me not be able to batch test. Is there any problem? And according to my understanding, the model should only load one video at a time. 

Example:
`python evaluate.py     --dimension aesthetic_quality     --videos_path ../test     --mode=custom_input`

The first video can be OK. However, for the third one, `torch.cuda.OutOfMemoryError: CUDA out of memory` would occur.

the same problem. The reason is that the CLIP model prediction does not use batch prediction. When the video is too long, there will be a GPU memory overflow problem. Here is the solution.
1. Set num_frames in the load_video function
2. Use serial or batch size in CLIP feature acquisition. The author is also recommended to optimize this.

I add with `torch.no_grad` to `aesthetic_quality.py` and it worked.

```
with torch.no_grad():
    image_feats = clip_model.encode_image(images).to(torch.float32)
    image_feats = F.normalize(image_feats, dim=-1, p=2)
    aesthetic_scores = aesthetic_model(image_feats).squeeze()
```
Repository: Vchitect/VBench
Issue: failed to get the output result
Root cause: Validation Gap

I ran this command:
`vbench evaluate --videos_path "/home/notebook/code/group/hkx/video_tasks/dover/DOVER/demo" --dimension "motion_smoothness"`

then it worked like this below:

`args: Namespace(func=<function evaluate at 0x7fba11448f40>, output_path='./evaluation_results/', full_json_dir='./VBench_full_info.json', videos_path='/home/notebook/code/group/hkx/video_tasks/dover/DOVER/demo', dimension='motion_smoothness', load_ckpt_from_local=None, read_frame=None)
start evaluation
Evaluation meta data saved to ./evaluation_results/motion_smoothness_full_info.json
Loading [networks.AMT-S.Model] from [/home/oppoer/.cache/vbench/amt_model/amt-s.pth]...
0it [00:00, ?it/s]
/home/notebook/data/group/hkx/vbench/lib/python3.12/site-packages/numpy/core/fromnumeric.py:3504: RuntimeWarning: Mean of empty slice.
  return _methods._mean(a, axis=axis, dtype=dtype,
/home/notebook/data/group/hkx/vbench/lib/python3.12/site-packages/numpy/core/_methods.py:129: RuntimeWarning: invalid value encountered in scalar divide
  ret = ret.dtype.type(ret / rcount)
Evaluation results saved to ./evaluation_results/motion_smoothness_eval_results.json
done`

But I got this result in motion_smoothness_eval_results.json:

`{
    "motion_smoothness": [
        NaN,
        []
    ]
}`

can you tell me where is the problem?



You can pass --mode=custom_input to the run command against custom input. Otherwise, it will expect video path to contain videos from vbench prompt list, see more details [here](https://github.com/Vchitect/VBench/blob/master/README.md?plain=1#L104).
Repository: Vchitect/VBench
Issue: Vbench Leaderboard
Root cause: Documentation Deficiency

Hi,

How can I get the Total Score, Quality Score, Semantic Score and Selected Score locally?

We have updated the offline calculation script of the t2v model at https://github.com/Vchitect/VBench/blob/master/README.md#get-final-score-and-submit-to-leaderboard. You may try it locally and before your submit.

Thank you very much for your response. I have carefully reviewed this portion of the code and noticed that static metrics, such as temporal flickering and motion smoothness, are assigned high weights. On one hand, `QUALITY_WEIGHT = 4` will quadruple their effect, and on the other, the parameters in `NORMALIZE_DIC` may amplify them by a factor of 3 to 5. Have you conducted any experiments previously or do you have any rationale behind these choices?

> Thank you very much for your response. I have carefully reviewed this portion of the code and noticed that static metrics, such as temporal flickering and motion smoothness, are assigned high weights. On one hand, `QUALITY_WEIGHT = 4` will quadruple their effect, and on the other, the parameters in `NORMALIZE_DIC` may amplify them by a factor of 3 to 5. Have you conducted any experiments previously or do you have any rationale behind these choices?

(1) The weights are used to balance the influence from each dimension, since different metrics have different score ranges. (2) There are fewer quality-related metrics compared to video-text alignment (semantic) metrics, and the weights partially help avoid underrepresentation. (3) The rationale is to have overall ranking faithfully reflect the a model's holistic performance. We conducted the experiments on earlier models where the ranking and model quality gap is prominent.
Repository: Vchitect/VBench
Issue: Where is the VLM-tuning model (VideoChat finetuned using human annotation)?
Root cause: Documentation Deficiency


Hello, thanks for your interest in our work.

Since `VideoChat` has been further updated for several versions now, we recommend you to use the newer version VideoChat2.

You can get the latest VideoChat2 model through [ask-anything github](https://github.com/OpenGVLab/Ask-Anything/tree/main/video_chat2), download our [human-preference data](https://drive.google.com/drive/folders/1on66fnZ8atRoLDimcAXMxSwRxqN8_0yS), and finetune on the pretrained [chat model](https://huggingface.co/OpenGVLab/VideoChat2_stage3_Mistral_7B) with a small learning rate using the [training script](https://github.com/OpenGVLab/Ask-Anything/blob/main/video_chat2/scripts/videochat_mistral/run_7b_stage3.sh).

Hi, we're closing the issue since there is no follow-up question, but feel free to re-open it if you have any further questions!
Repository: Vchitect/VBench
Issue: How to calculate total score？
Root cause: Documentation Deficiency

Hi, after i get all dimension score. How calculate the total score？

We have updated the offline calculation script of the t2v model at https://github.com/Vchitect/VBench/blob/master/README.md#get-final-score-and-submit-to-leaderboard. You may try it locally and before your submit.
Repository: Vchitect/VBench
Issue: Can we withdraw the results in leaderboard？
Root cause: Documentation Deficiency

We do not know how to compute total score. If we submit results in leaderboad, can we withdraw it?

Hi, it's already withdrawn from our side. If you would like to upload again, it would be great to provide the full information of your model. Thanks!

I just want to know the score calculation rules.

See this issue: https://github.com/Vchitect/VBench/issues/9

@ziqihuangg Hi, what is the correct json format？
The results are abnormal. How to withdraw it？https://huggingface.co/spaces/Vchitect/VBench_Leaderboard
All results are 0.

I have already answered your question on HuggingFace. If you still have issue, feel free to open this issue at any time. At the same time, we have updated the offline script. See in #45.
Repository: Vchitect/VBench
Issue: About sampled videos
GitHub labels: question

Hi, great work!

Will the sampled videos from different video generation models be released?  Especially the videos generated by Gen-2 and Pika.

Thanks!

Certainly! We've already uploaded sample videos of `lavie`, `modelscope`, `videocrafter0.9`, and `cogvideo` on [Hugging Face](https://huggingface.co/datasets/Vchitect/VBench_sampled_video/tree/main), along with their corresponding human annotations available at [this link](https://huggingface.co/datasets/Vchitect/VBench_human_annotation). The sampled videos of `Gen-2` and `Pika` are still being organized for evaluation. We will update the results and release the corresponding videos and annotations ASAP.

Thank！
Repository: Vchitect/VBench
Issue: VBench-Long Leaderboard
Root cause: Documentation Deficiency

Dear authors @ziqihuangg @MqLeet , thank you for your excellent work. I am curious if there is a leaderboard for VBench-Long to facilitate easy comparison of long video generation models?

Hi @Epiphqny , thanks for the question. As of now, we are hosting the long videos using the same leaderboard as the T2V leaderboard: https://huggingface.co/spaces/Vchitect/VBench_Leaderboard, as the metrics are designed in the way that long videos' numerical results would be comparable with short videos'. Should there be any updates, we will let you know.

 Hi @ziqihuangg, thank you for the prompt response. I'll check the short video leaderboard and look forward to the updates of long video metrics.

Hi, I am wondering how long is the long video? I saw it just has a updated in MiniMax, which is 5.6 second, but it shows up in LongVideo leaderboard.

Hi, thanks for the question. We've also updated this in the FAQ to make it clearer to everyone. https://github.com/Vchitect/VBench/blob/master/README-FAQ.md

Use [VBench](https://github.com/Vchitect/VBench?tab=readme-ov-file#usage) for evaluating videos shorter than 5.0 seconds (< 5.0s). Use [VBench-Long](https://github.com/Vchitect/VBench/tree/master/vbench2_beta_long) for evaluating videos 5.0 seconds or longer (≥ 5.0s). Each benchmark is optimized for its respective video length to ensure fair and consistent evaluation.
Repository: Vchitect/VBench
Issue: Difference between "overall_consistency" and "temporal_style"?
Root cause: Documentation Deficiency

Dear Authors, 

Thank you for the good work and the codes.

We were just wondering the difference between "overall_consistency" and "temporal_style". These two metrics appear to have respective .py codes under "vbench" dir. By text matching, they are basically identical other than function names. Our result validates our guess - they are always the same in these two metrics.

Interestingly, these two values seem different in README demo. Any suggestion and/or clarification would be greatly appreciated. Thank you very much.

Hi, thanks for the question. Both dimensions use ViCLIP to measure video-text similarity. The difference is the text prompts, where "temporal style" uses a diverse list of temporal styles including variations in motion speed, camera movements, and dynamic effects. These temporal styles are combined with several scenarios to evaluate whether the model can generate a particular scenario under different temporal styles. On the other hand, overall consistency uses a list of general prompts under different content categories and scenarios.

Thank you very much. I found the problem. We were following "evaluate your own videos" and these two metrics shouldn't be included.
Repository: Vchitect/VBench
Issue: Can vbench_i2v support customized video evaluation?
Root cause: Documentation Deficiency

Hi,

I know vbench has supported the evaluation on customized videos. However, it seems vbench_i2v has not yet, right? If so, how soon it would be support that? Thanks!

Thanks!

Hi, We have already supported customized video evaluation for vbench_i2v. You can refer to this link for detailed usage: [Evaluate Your Own Videos](https://github.com/Vchitect/VBench/tree/master/vbench2_beta_i2v#evaluate-your-own-videos).


> Hi, We have already supported customized video evaluation for vbench_i2v. You can refer to this link for detailed usage: [Evaluate Your Own Videos](https://github.com/Vchitect/VBench/tree/master/vbench2_beta_i2v#evaluate-your-own-videos).您好，我们已支持为 vbench_i2v 提供定制视频评估。您可以参考此链接获取详细使用方法：评估您的视频。

请问一下为什么customized video evaluation支持的维度比较少，而且没有text-video 对齐的指标呢？

Not all dimensions support customised video evaluation.

We support video-text alignment assessment in I2V evaluation: e.g. camera motion control
Repository: Vchitect/VBench
Issue: Implementation Details
Root cause: Documentation Deficiency

Thank you for your excellent work on VBench, which has been very inspiring to me.

When I attempted to reproduce the results in Table 1 of your paper using video samples generated by my ModelScope code, instead of the samples you provided, I encountered significant discrepancies, with the performance being much lower than the reported results. Could you provide more details about the generation models, beyond what is mentioned in the supplementary materials? This information is really important to me.

Models              height, width      num_frames    fps   inference_steps      guidance_scale      seed
LaVie--------      256×256             16                    8             250                    7                           2
ModelScope                 512×512         16                    8                  ???                       ?                         ?
VideoCrafter      256×256             16                   8               ???                      ?                                2
CogVideo---        480×480              33                  10                ???                     ?                              2

I look forward to your response. Whether or not you can provide all the details, please let me know.

@ziqihuangg

Please refer to the model details here: https://github.com/Vchitect/VBench/tree/master/sampled_videos#what-are-the-details-of-the-video-generation-models
Let me know if there is other specific information you are looking for.

Thank you for your response.

In addition to the information provided, could you also share details regarding inference_steps, guidance_scale, and seed?

num_inference_steps controls the number of denoising steps, with higher steps generally leading to better video quality
guidance_scale represents the guidance scale value, where a higher value typically increases the relevance of the generated video to the text prompt, but might decrease image quality.

These parameters are crucial for video generation quality as well.

@ziqihuangg @JustinYuu

@lovelyczli Hi, thanks for your interest in our work!

For the modelscope videos, we use the officially provided modelscope.pipelines API for sampling, where the seed, inference steps, and the guidance scale are not able to be viewed and adjusted. To reproduce our generation results, you could try to use the same modelscope package and model as shown in the code example part of the following [link](https://modelscope.cn/models/iic/text-to-video-synthesis/file/view/master?fileName=README.md&status=1).

Got it, thank you for the quick response and the great work!
Repository: Vchitect/VBench
Issue: Duplicate prompts in overall_consistency.txt and full.json
Root cause: Validation Gap

Thank you for the great work!


I noticed that there is a duplicate prompt "A fantasy landscape" in the prompts/prompts_per_dimension/overall_consistency.txt file on line 31 and line 52. A similar issue appears in the full.json file as well.

Additionally, we found only 92*5=460 videos in the provided Sampled Videos of the overall_consistency dimension, which seems to align with the duplicated prompt.


Although the differences in the results are minimal, VBench is a widely followed benchmark, and it would be great if this issue could be addressed soon.

Thanks again for your excellent work!

@ziqihuangg

You are correct that there are two similar prompts in the prompts/prompts_per_dimension/overall_consistency.txt file, differing only in upper/lower case. Due to our current video naming protocol, the later prompt’s video overwrites the earlier one during sampling. This process is consistently applied across all models under evaluation to ensure fair comparison. Thus, while it doesn’t affect fairness, it does result in extra time spent sampling one additional prompt out of the 900+ prompts. We opted not to alter the prompt list to ensure that all results remain fully reproducible. Hope this clarifies, thanks!


I got it, thank you for the quick response!
Repository: Vchitect/VBench
Issue: Empirical Min
Root cause: Documentation Deficiency

Thank you for your great work. I apologize for raising another issue, but this one has been troubling me.

When using your code to calculate Overall Consistency, specifically the cosine similarity between ViCLIP visual features and text features (line 49: vbench/overall_consistency.py), we are encountering negative results.

As we know, cosine similarity can indeed be less than 0. How do you ensure that this lower bound is exactly zero (line 55 scripts/constant.py)? Could you provide some insights on this?

Looking forward to your response.

@ziqihuangg

Just following up on my question. I'd appreciate any guidance when you have time. @ziqihuangg 

@ziqihuangg

The lower bounds we refer to are not "theoretical", but "empirical" bounds used for normalizing the final score calculation. We have not claimed that the theoretical lower bound should be zero; in fact, it should be -1. When the ViCLIP score is below zero, it indicates that the video content and text description have "opposite semantic meaning." Positive scores suggest relevant semantics, while a score of zero typically indicates irrelevant but non-contradictory content. In the context of text-to-video generation, it is highly unusual for a video to exhibit "opposite semantic meaning" to its text prompt. Even in rare cases where this occurs, the average score across all samples would most likely still be above zero.
Repository: Vchitect/VBench
Issue: How to run inference for a new data? 
Root cause: Documentation Deficiency


We now support evaluating user-customised videos: https://github.com/Vchitect/VBench?tab=readme-ov-file#new-evaluate-your-own-videos. You can do this by simply turning the `custom_input` flag on.

You could also refer to Issue #7
Repository: Vchitect/VBench
Issue: What is the role of the trustworthiness folder at the same level as per_dimension and per_category？


Thank you for your question. These videos were used to evaluate the "trustworthiness" of each video generative model, focusing on aspects such as bias, safety, and cultural considerations. This evaluation is separate and orthogonal to <per_dimension>, which assesses various technical capabilities, and <per_category>, which examines different video scenarios.