zhiminy/EvalEng huggingface dataset:
Repository: Vchitect/VBench
Issue: Error on Multiview-Consistency
Root cause: Validation Gap

When I test this feature, I'm getting `match_inter==0` in `VBench/VBench-2.0/vbench2/third_party/Dense_match/PatchAutoEvaluate.py`, and I find that `interval_num` is very large. There is only 81 frames in my video, and the `interval_num` is 60, so matching cannot be computed. Is this correct?

This issue only occurs if your video meets all of the following conditions at the same time: very slow motion (i.e., small optical flow), low FPS, and a limited number of frames. In our current evaluation methods, we have not encountered this problem before. To accommodate your testing requirements, we have updated our handling of this specific exception. Please check the file: VBench/VBench-2.0/vbench2/third_party/Dense_match/PatchAutoEvaluate.py.
Repository: Vchitect/VBench
Issue: error in running dimension of i2v-background consistency of vbench-i2v
Root cause: Environment Incompatibility

Hi dear authors,

I am here again. Thanks again for your valuable work in vbench-i2v scoring too. 

Yet I had a new problem while running vbench-i2v code to evaluate videos in "i2v-background consistency" dimension. The process always failed at 40% of downloading dreamsim models, it always interrupted and unzipping failed at 40% of downloading 'pretrained.zip'.

Could you please look into it and help?

It also would be fine to let me know the link to download the pretrained.zip file so that I could directly download and archive it to the dst path.

Many thx.

<img width="2430" height="756" alt="Image" src="https://github.com/user-attachments/assets/b8d6bec9-0388-45d0-add6-7d5e5750299b" />

Hi bro, this problem wax fixed by swithing to a new network. And this can be closed.
Repository: Vchitect/VBench
Issue: Evaluation of True Videos
Root cause: Documentation Deficiency


Hi, thanks for your interest in our work. Could you let us know which step you get stuck at? We are happy to assist. 

Yes, after download the VBench_full_info.json (not edited and I don't know what the content means), then I run 
` vbench evaluate --videos_path ./videos/ --dimension "imaging_quality"`, 
Then I got
`
site-packages/vbench/imaging_quality.py", line 20, in technical_quality
     average_score = sum([o['video_results'] for o in video_results]) / len(video_results)
ZeroDivisionError: division by zero
`

The reason is because the `video_list ` inputed in the following 
`site-packages/vbench/imaging_quality.py", line 33, in compute_imaging_quality
    all_results, video_results = technical_quality(model, video_list, device)`
is **empty**. 

But  under the given path ` ./videos/`, there are hundreds of videos, I can't solve this.

Can you provide `ls ./videos`?

OK, I know, it's about writing your own `VBench_full.json` ; My goal is not to evaluate AIGC generated videos but true videos, so I rewrite VBench module and it worked.
Repository: Vchitect/VBench
Issue: question about the ratio of Image-to-Video
Root cause: Documentation Deficiency

2.1. What's Special about VBench-I2V's Image Suite
Main philosophy behind our Image Suite: 💡 Adaptive aspect ratio 💡 Since different Image-to-Video (I2V) models have different default resolutions for the input images, we believe it's only fair to compare models when each model is evaluated on its default / best resolution. To this end, we introduced an automatic pipeline to obtain images in different resolutions and aspect ratios while preserving their main content. More implementation details are provided [here](https://github.com/Vchitect/VBench/tree/master/vbench2_beta_i2v#crop).


那么我这边在进行测试的时候，有水平裁剪的和垂直裁剪的，每种方式下又有多个比例

![Image](https://github.com/user-attachments/assets/dfc3a83e-59cd-4c39-a7f3-4760bf5013c8)

![Image](https://github.com/user-attachments/assets/04a1b7da-720b-4309-8c0b-df2d7e12965e)

那么我在测试的时候，是每种都要进行测试，还是只用测试其中的某一项呢？目前提供的榜单里面是所有都测试选择的平均值，还是其中某一项测试的最大值呢？



You can evaluate at the I2V model's optimal resolution (usually the resolution of training data).
Repository: Vchitect/VBench
Issue: long prompt evaluation
Root cause: Documentation Deficiency

I am using the augment's long prompt for evaluation. Since the prompt is too long, I'm using the corresponding short prompt as the filename. When conducting the evaluation, should I directly use the default prompt_file from evaluate_long.sh, or do I need to customize the prompt_file?
If customization is needed, what should be the format of the prompt_file?

Hi, for the filename, please use the original short prompt directly, since that’s what the evaluation pipeline will look up.
Repository: Vchitect/VBench
Issue: vbench2_beta_i2v is not handling custom prompt_list
Root cause: Configuration Error

Hi authors,

Thanks for creating VBench, this is a very nice eval benchmark suite.

I find it useful to evaluate a custom prompt_list and corresponding videos. When running vbench2_beta_i2v with custom prompt_list, the prompts does not look to be correctly propagated to the eval. I think that is related to the missing of **kwargs in build_full_info_json: 
- `cur_full_info_path = self.build_full_info_json(videos_path, name, dimension_list, custom_image_folder=custom_image_folder, mode=mode)`
- https://github.com/Vchitect/VBench/blob/master/vbench2_beta_i2v/__init__.py#L23
Repository: Vchitect/VBench
Issue: Unimplemented dimension: i2v_background
Root cause: External Dependency Breakage

Thanks for your nice benchmark. But I occured error in running evaluate_i2v.py shown in following img:

<img width="1004" height="220" alt="Image" src="https://github.com/user-attachments/assets/14274973-2bb4-47d3-a84c-2c0b1600d975" />

Is this a normal performance? How should I fix this issue?

Oh, I downgraded my peft and transformers and that solved the problem.
```shell
peft==0.10.0
transformers==4.37.2
```
Repository: Vchitect/VBench
Issue: Could you provide the VBench2.0 results for Wan2.1-T2V-1.3B ?


Have you found any clues?

I think the leaderboard has released results on Wan2.1-T2V-1.3B.
https://huggingface.co/spaces/Vchitect/VBench_Leaderboard
Repository: Vchitect/VBench
Issue: the VBench/pretrained directory should be correspondent with its definition
Root cause: Architectural Constraint

https://github.com/Vchitect/VBench/blob/57c7ef50e86e7c12007caafb53f862a63fddaf1f/vbench/utils.py#L364


suggest to the director of viclip_model not ViCLIP, to correspond with the `VBench/pretrained` directory
Repository: Vchitect/VBench
Issue: the element style of array should be the same
Root cause: Validation Gap

https://github.com/Vchitect/VBench/blob/57c7ef50e86e7c12007caafb53f862a63fddaf1f/vbench/__init__.py#L19

["subject_consistency", "background_consistency", "aesthetic_quality", "imaging_quality", "object_class", "multiple_objects", "color", "spatial_relationship", "scene", "temporal_style", 'overall_consistency', "human_action", "temporal_flickering", "motion_smoothness", "dynamic_degree", "appearance_style"]        


 'overall_consistency' ==> "overall_consistency"
Repository: Vchitect/VBench
Issue: provide the pretrained weights for reproducing the results in leaderboard
Root cause: Documentation Deficiency

Hello, can you provide the model weights and project for reproducing the results in leaderboard?
Thank you so much.

Which model's pretraining weight is it? We all get the weight or video from the corresponding project's official project page.

> Which model's pretraining weight is it? We all get the weight or video from the corresponding project's official project page.

For example: [VideoCrafter-1.0](https://github.com/AILab-CVC/VideoCrafter/tree/VideoCrafter1?tab=readme-ov-file)
 There are two models (one with resolution 576x1024, and the other one with resolution 320x512) for text to image in the official github of VideoCrafter1.0. So, which one is evaluated in the Vbench's leaderboard?

And the same problem occurs on Show-1, Does the Show-1 in the leaderboard means the Show-1-base model? or with the  Interpolation and resolution Model?

And the other evaluated methods have the same questions. So detailed model weights   and projects are suggested for the users to reproduced the results.

Hi, please find the model details here: https://github.com/Vchitect/VBench/tree/master/sampled_videos#what-are-the-details-of-the-video-generation-models
Repository: Vchitect/VBench
Issue: Can not see the results on leaderboard
Root cause: Environment Incompatibility

Hi, I can not see the results on leaderboard. Could you please take a look?

<img width="1528" height="949" alt="Image" src="https://github.com/user-attachments/assets/eef94a59-f3c0-4ddc-bca0-d74371680b8d" />

Thank you for your reminder. We have restarted this space, and it is now working properly.
Repository: Vchitect/VBench
Issue: Cannot zip/unzip Wan2.1-I2V-14B-720P sampled videos
Root cause: Environment Incompatibility

Looks like some files are missing to zip and unzip Wan2.1-I2V-14B-720P on this location: https://drive.google.com/drive/folders/1PEsSdFdqGUkbdT4UHz-n0_PjRQwbTw8g

potential missing files:
Wan2.1-I2V-14B-720P.zip
Wan2.1-I2V-14B-720P.z03
Wan2.1-I2V-14B-720P.z013
Repository: Vchitect/VBench
Issue: Question about sampling 5 videos for each prompt
Root cause: Documentation Deficiency

Hi, I am trying to use VenchI2V to evaluate image-to-video model, but have some problems about this step: https://github.com/Vchitect/VBench/blob/master/vbench2_beta_i2v/README.md#pseudo-code-for-sampling

Does sampling 5 videos means: use 5 different random seeds to generate 5 different videos for evaluation?

If not, could you please correct me?
Repository: Vchitect/VBench
Issue: clip_length_mix.yaml
Root cause: Configuration Error

I’d like to ask whether the evaluation for **vbench-long** uses **clip_length_mix.yaml** for *all* dimensions?

When using **clip_length_mix.yaml**, it seems that
`imaging_quality: 2.0`
`aesthetic_quality: 2.0`
are actually not being used!

In the code:

```python
length_values = [clip_lengths[dim] for dim in unique_dimensions if dim in clip_lengths]
```

Since in **VBench_full_info.json**, these two dimensions are grouped together with **overall_consistency**, running

```bash
python vbench2_beta_long/eval_long.py --videos_path xx --dimension aesthetic_quality
```

will always result in
`length_values = [2.0, 10.0, 2.0]`,

so

```python
max_length = max(length_values) if length_values else None
```

will always be `10`.

Therefore, for the **imaging_quality** and **aesthetic_quality** splits, the evaluation will always use the `10.0` length setting.

I’m wondering if this is the intended behavior, or should it be adjusted manually?
Repository: Vchitect/VBench
Issue: the aesthetic_model cannot be downloaded, the link only provides a 4kb file
Root cause: External Dependency Breakage

wget -P https://github.com/LAION-AI/aesthetic-predictor/blob/main/sa_0_4_vit_l_14_linear.pth?raw=true ~/.cache/aesthetic_model/emb_reader/sa_0_4_vit_l_14_linear.pth
Repository: Vchitect/VBench
Issue: Error when evaluating on VBench2 Human_Anatomy
Root cause: Architectural Constraint

Hi, thanks for the great work!

When I run VBench2 for evaluation, I encounter the following error during the Human_Anatomy benchmark:


```
huggingface_hub.errors.HFValidationError: Repo id must be in the form 'repo_name' or 'namespace/repo_name': '/mnt/petrelfs/zhengdian/code/ckpt/clip-vit-base-patch32'. Use `repo_type` argument if needed.
Batch complete: Dimensions Human_Anatomy
```

It seems the code is attempting to load a CLIP checkpoint from a hard-coded path [here](https://github.com/Vchitect/VBench/blob/07bc8a4b74d5e0a23de42ed5880b899a1ff705f0/VBench-2.0/vbench2/third_party/YOLO-World/yolo_world_v2_l_vlpan_bn_2e-3_100e_4x8gpus_obj365v1_goldg_train_1280ft_lvis_minival.py#L21).

I am not sure if I have missed anything. Can I just replace it with "openai/clip-vit-base-patch32"?

Thanks in advance for your help!

I get this problem too.
Repository: Vchitect/VBench
Issue: What does the result means?

`
{
    "imaging_quality": [
        0.6686933368155237,
        [
            {
                "video_path": "/data/*.mp4",
                "video_results": 56.49358892440796
            },
`

This is part of the result. I can see that video_results is the imaging_quality score, but what does '0.6686933368155237' means?

`0.6686933368155237` is the overall score of `imaging_quality`

Thanks! That solved!
Repository: Vchitect/VBench
Issue: Access to VBench 2.0 Abnormal Detector for Instance Preservation Evaluation

Dear VBench Research Team,

I previously contacted you by email because I was trying to locate the abnormal detector used for the Instance Preservation dimension in VBench-2.0.

For anyone facing the same question, I’m summarizing the helpful answer you provided here so it is easier to find:

Checkpoint reference:
VBench-2.0/vbench2/utils.py (around line 339)
https://github.com/Vchitect/VBench/blob/07bc8a4b74d5e0a23de42ed5880b899a1ff705f0/VBench-2.0/vbench2/utils.py#L339

Script: https://github.com/Vchitect/VBench/blob/master/VBench-2.0/vbench2/third_party/Instance_detector/test.py

Thank you again for your support and for providing such a valuable benchmark!

Best regards
Repository: Vchitect/VBench
Issue: why quality_weight is set 4 and semantic_weight is set 1?

why quality_weight is set 4 and semantic_weight is set 1? why both of them are not set 1? Do you have any particular considerations for this setting?
请问为什么quality_weight和semantic_weight分别设为4和1，而不是都设为1？请问这样子的设置背后是有什么考虑吗？我看论文中并没有解释这一点。望解惑，感谢！
Repository: Vchitect/VBench
Issue: sample_per_video function related

Could you give an example of the sample_per_video function? Thanks！

Hi, this is the function to sample a video, from the model you want to evaluate.

Excuse me, could you upload the model and sample script used in the paper or send it privately? Thank you so much!


1585421442
***@***.***



&nbsp;




------------------&nbsp;原始邮件&nbsp;------------------
发件人:                                                                                                                        "Vchitect/VBench"                                                                                    ***@***.***&gt;;
发送时间:&nbsp;2024年1月19日(星期五) 中午12:48
***@***.***&gt;;
***@***.******@***.***&gt;;
主题:&nbsp;Re: [Vchitect/VBench] sample_per_video function related (Issue #2)





 
Hi, this is the function to sample a video, from the model you want to evaluate.
 
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: ***@***.***&gt;

Hi, you can refer to the open-sourced sampling code for each video generation model. For instance, to sample from the LaVie model, you can use this script: https://github.com/Vchitect/LaVie/blob/main/base/pipelines/sample.py, and the sampling instruction is here: https://github.com/Vchitect/LaVie/tree/main?tab=readme-ov-file#step1-base-t2v

Hope this clarified your question, thanks!

Hi, we've added detailed information for each models' sampling settings: https://github.com/Vchitect/VBench/tree/master/sampled_videos#what-are-the-details-of-the-video-generation-models

I'm closing the issue as there has been no follow-up questions for a while, but feel free to re-open it anytime if you still have any related questions.
Repository: Vchitect/VBench
Issue: Where can I find open source video data?

Thanks for your great work, I want to know where can I find those generated videos mentioned in the paper?

Hi, we have organised and uploaded the sampled videos here: https://drive.google.com/drive/folders/13pH95aUN-hVgybUZJBx1e_08R6xhZs5X?usp=sharing

You can follow the example scripts here for automatic download and evaluation: https://github.com/Vchitect/VBench?tab=readme-ov-file#example-of-evaluating-videocrafter-10

Hi, more details regarding open-sourced video data is provided here: https://github.com/Vchitect/VBench/tree/master/sampled_videos#sampled-videos

Hi, we're closing this issue as it appears your questions have been addressed. However, feel free to open a new issue or reopen this one, if you have further questions or if anything else comes up related to this issue.
Repository: Vchitect/VBench
Issue: Where is the human preference data?
Root cause: Documentation Deficiency


It's provided here: https://drive.google.com/drive/folders/1jYAybu2BazShGV-DLityFi4j7BjTE-my?usp=sharing

More details supplied here: https://github.com/Vchitect/VBench/tree/master/sampled_videos#human-preference-labels
You can also preview the data on huggingface: https://huggingface.co/datasets/Vchitect/VBench_sampled_video

Hi,
Could you please provide more details about how Spearman's correlation coefficient is computed for each dimension? Thank you. 

Hi, we conduct pair-wise comparisons of two sampled videos, using both VBench and Human Preference Annotations. We then compute the spearman correlation of the win ratios on all pairs drawn from the VBench prompt suite, between VBench results and Human results.

For more details on human preference data, and the computation, please refer to our paper (1) Section 3.3. - Human Preference annotation, (2) Section 4.2 - Validating Human Alignment of VBench, and (3) Supplementary material's Section I - Human Preference Annotation. https://arxiv.org/abs/2311.17982 Thanks!

> Hi, we conduct pair-wise comparisons of two sampled videos, using both VBench and Human Preference Annotations. We then compute the spearman correlation of the win ratios on all pairs drawn from the VBench prompt suite, between VBench results and Human results.
> 
> For more details on human preference data, and the computation, please refer to our paper (1) Section 3.3. - Human Preference annotation, (2) Section 4.2 - Validating Human Alignment of VBench, and (3) Supplementary material's Section I - Human Preference Annotation. https://arxiv.org/abs/2311.17982 Thanks!

Hi, 
Can I understand it as calculating the Win ratio for 5 x 4 videos of each prompt and calculating the Spearman correlation based on these win ratios, and then taking the average across the entire prompt suite?


The win ratio is calculated across all prompts within a given dimension as: # of wins / # of comparisons.
The correlation measures the alignment between human perception and VBench evaluation by comparing their respective win ratios.
Repository: Vchitect/VBench
Issue: About the size and number of frames

Hi,

Dose Vbench have limitation on the video size and the number of frames for generated videos? What is the setting of the video size and the number of frames in your evaluation shown in Table 1 in your paper?

Hi, thanks for this question. Please see the model settings here: https://github.com/Vchitect/VBench/tree/master/sampled_videos#model-setting, and these video lengths and frame counts (several seconds) are all supported by VBench. 

For very long videos (minute-level), we don't support the evaluation yet, but we are working on a new VBench edition to include this capability. Please stay tuned for the update.

For models we evaluated, please find their detailed settings here: https://github.com/Vchitect/VBench/tree/master/sampled_videos#what-are-the-details-of-the-video-generation-models

Hi, we're closing this issue as it appears your questions have been addressed. However, feel free to open a new issue or reopen this one, if you have further questions or if anything else comes up related to this issue.
Repository: Vchitect/VBench
Issue: LeaderBoard code
Root cause: Documentation Deficiency

Hello, Do you have a plan to release the code for the LeaderBoard?

Leaderboard code is available at https://huggingface.co/spaces/Vchitect/VBench_Leaderboard/tree/main.

> Leaderboard code is available at https://huggingface.co/spaces/Vchitect/VBench_Leaderboard/tree/main.

hello, i find the necessary files under this code are missed, Such as:
CSV_DIR = "./vbench_leaderboard_submission/results.csv"
QUALITY_DIR = "./vbench_leaderboard_submission/quality.csv"

Where can i find these files?

These results are already displayed in the leaderboard. Thanks.

> These results are already displayed in the leaderboard. Thanks.

Hello, I find some dimensions are missed in the upload sampled videos, 
for example, videocrafter-1.tar.gz don not have the dimension of "background consistency"
So the results of the online LeaderBoard can not be reproduced exactly.

Hi, for missing videos: Some dimensions, such as "background consistency" and "scene," use the same set of videos.

plus: It would be great if you open new issues for each question or email us for ongoing discussions. Thanks!

> Hi, for missing videos: Some dimensions, such as "background consistency" and "scene," use the same set of videos.
> 
> plus: It would be great if you open new issues for each question or email us for ongoing discussions. Thanks!

Thanks，i will try with the same video sets，and i will open a new issue if i have other questions
Repository: Vchitect/VBench
Issue: run inference on customized video and prompt?
Root cause: Documentation Deficiency

I understand we can run inference on customized video (as in https://github.com/Vchitect/VBench/issues/7).
However, can we run inference on both customized video and prompt?

Yes. 
By "customized video," we mean videos that are generated based on text prompts provided by users, rather than using the text prompts from VBench's Prompt Suite.


Hi, I'm closing the issue as there's no follow-up questions, but feel free to re-open it anytime if you have any further questions!
Repository: Vchitect/VBench
Issue: pretrained/umt_model
Root cause: Documentation Deficiency

Is there a model.txt file missing in the umt_model folder? Do I need to download it?

Sorry for the late response, thank you for pointing out this error. For now, you can download the corresponding model by executing `wget https://pjlab-gvm-data.oss-cn-shanghai.aliyuncs.com/umt/single_modality/l16_ptk710_ftk710_ftk400_f16_res224.pth -P  ~/.cache/vbench/umt_model`. We will supplement this file as soon as possible.

We have updated it at https://github.com/Vchitect/VBench/blob/master/pretrained/umt_model/model_path.txt. If there are still issues, please feel free to reopen this issue at any time.
Repository: Vchitect/VBench
Issue: the score meaning
Root cause: Documentation Deficiency

hello, what's the meaning of 0.9351381915230905?

![image](https://github.com/Vchitect/VBench/assets/13817995/369a5591-4d46-47c5-a495-731fb7175e7c)


The score for this dimension, which has been normalized to a range of 0-1.

Hi, we're closing this issue as it appears your questions have been addressed. However, feel free to open a new issue or reopen this one, if you have further questions or if anything else comes up related to this issue.