zhiminy/EvalEng huggingface dataset:
Repository: Vchitect/VBench
Issue: install mmcv failed
Root cause: External Dependency Breakage

cuda=12.1
torch=2.5.1
python=3.13.2

error message:
(vbench2) root@iv-ydv8dav37kwh2yoii1rn:/var/cache/apt/archives# **pip install mmcv==2.2.0 -f https://download.openmmlab.com/mmcv/dist/cu121/torch2.4/index.html --no-cache-dir**
Looking in indexes: https://mirrors.ivolces.com/pypi/simple/
Looking in links: https://download.openmmlab.com/mmcv/dist/cu121/torch2.4/index.html
Collecting mmcv==2.2.0
  Downloading https://mirrors.ivolces.com/pypi/packages/e9/a2/57a733e7e84985a8a0e3101dfb8170fc9db92435c16afad253069ae3f9df/mmcv-2.2.0.tar.gz (479 kB)
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [21 lines of output]
      /tmp/pip-install-w3xekye0/mmcv_e6edb8874008428bac108009ee97e42d/setup.py:5: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
        from pkg_resources import DistributionNotFound, get_distribution, parse_version
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
          exec(compile('''
          ~~~~^^^^^^^^^^^^
          # This is <pip-setuptools-caller> -- a caller that pip uses to run setup.py
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
          ...<32 lines>...
          exec(compile(setup_py_code, filename, "exec"))
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
          ''' % ('/tmp/pip-install-w3xekye0/mmcv_e6edb8874008428bac108009ee97e42d/setup.py',), "<pip-setuptools-caller>", "exec"))
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "<pip-setuptools-caller>", line 35, in <module>
        File "/tmp/pip-install-w3xekye0/mmcv_e6edb8874008428bac108009ee97e42d/setup.py", line 462, in <module>
          version=get_version(),
                  ~~~~~~~~~~~^^
        File "/tmp/pip-install-w3xekye0/mmcv_e6edb8874008428bac108009ee97e42d/setup.py", line 43, in get_version
          return locals()['__version__']
                 ~~~~~~~~^^^^^^^^^^^^^^^
      KeyError: '__version__'
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

The issue appears to be caused by an updated version of setuptools. To resolve it, it is recommended to use a version earlier than 81

> The issue appears to be caused by an updated version of setuptools. To resolve it, it is recommended to use a version earlier than 81

My local setuptool version is 60.2.0, could you please provide the docker image of vbench2.0?



After I downgraded the python version to 3.10.12, the problem has been solved
Repository: Vchitect/VBench
Issue: FileNotFoundError: [Errno 2] No such file or directory: './VBench_full_info.json'
Root cause: Documentation Deficiency

when running !vbench evaluate --videos_path "..." --dimension "...." encounter file NOT FOUND
and the link you put in the repo for VBench_full_info.json will turn to 404

sloved, you should check your super link for VBench_full_info.json. It should be https://github.com/Vchitect/VBench/blob/master/vbench/VBench_full_info.json

Thanks for that! Already fixed README.
Repository: Vchitect/VBench
Issue: Does human preference of newer models generated videos are available ?

I saw authors have released newer model generated videos in promts that is given in VBench, Human annotation is given by authors for lavie, cogvideo, videocrafter and modelscope. I am searching for human annotation of newer models is it available ?

Hi, human annotations are only available for earlier models that were released during the development of VBench evaluation methods.

I have one more question, do we have video quality dimensions human annotation for VBench ++ or VBench I2V ?
Repository: Vchitect/VBench
Issue: BERT / cross-attention
Root cause: Validation Gap

Hi,

for caption generation, you are using a BERT model with cross-attention, see https://github.com/Vchitect/VBench/blob/d2c30cab5497472af4798bc86f6eb47659381cb4/vbench/third_party/tag2Text/tag2text.py#L100
and
https://github.com/Vchitect/VBench/blob/d2c30cab5497472af4798bc86f6eb47659381cb4/vbench/third_party/tag2Text/q2l_config.json#L20

The checkpoint used for initializing BERT is bert-base-uncased and does not contain any cross-attention layers, so the parameters of the cross-attention layers are randomly initialized.

Is that intentional?

thank you
Repository: Vchitect/VBench
Issue: Model Download
Root cause: Documentation Deficiency

Wonderful job! I can't download the models of VBench2.0 through Google Drive due to network. Can you upload the models through any other ways, like ModelScope of HuggingFace?

Hi, we have uploaded all the models that previously used Google Drive to Hugging Face (https://huggingface.co/Vchitect/VBench-2.0_models/tree/main). Please download them and place them in your cache directory according to the path instructions in the code.
Repository: Vchitect/VBench
Issue: Evaluate video using semantic metric, not only video quality metric
Root cause: Documentation Deficiency

I try to evaluate my video that created by CogVideoX. I plan to evaluate video that specific in one domain instead of multidomain. Is it possible to evaluate our video with semantic metric? I only see the video quality metric.

Thanks for your question!

Currently, VBench supports evaluating customized videos or prompts for the following dimensions:

'subject_consistency', 'background_consistency', 'motion_smoothness', 'dynamic_degree', 'aesthetic_quality', and 'imaging_quality'.

If you want to evaluate videos using semantic-level metrics (e.g., object_class, scene, human_action, etc.), these require standardized prompts for reliable benchmarking.

To use semantic metrics, please refer to the [Evaluation on the Standard Prompt Suite of VBench](https://github.com/Vchitect/VBench?tab=readme-ov-file#evaluation-on-the-standard-prompt-suite-of-vbench).
Repository: Vchitect/VBench
Issue: Trouble Reproducing Wan2.1-T2V-1.3B Results for multiple_objects Dimension
Root cause: Documentation Deficiency

Hi,

I’ve been trying to reproduce the Wan2.1-T2V-1.3B results for the multiple_objects dimension, but I’m getting much lower scores than what’s reported on the leaderboard.

According to the VBench leaderboard (sampled by the VBench team), the score for this dimension is 74.83%.
I ran generation on the default prompt list from [multiple_objects.txt](https://github.com/Vchitect/VBench/blob/master/prompts/prompts_per_dimension/multiple_objects.txt), using default parameters, except for sample_guidance_scale=6.0 and sample_shift=3.0 following [issue #130](https://github.com/Vchitect/VBench/issues/130).

With this setup, I got 57.53%, which is quite far from the reported score.

Could you please clarify:

* What seeds were used to generate the sampled videos? ([link](https://drive.google.com/drive/folders/1jJmhAzgg_p0mhp5TNPgVT3hUUUaSX1Xo))?

* Were augmented prompts used?

* Any other hyperparameter differences or setup tips that might explain the gap?

My environmen includes:
torch==2.6.0, torchvision==0.21.0, CUDA 12.4

I'd really appreciate any tips on how to better match the reported results.
Thanks!


Thanks for your question!

To reproduce the reported results for Wan2.1-T2V-1.3B on the multiple_objects dimension, please refer to the exact settings used in [Sampled videos](https://github.com/Vchitect/VBench/tree/master/sampled_videos), including [augmented prompts](https://github.com/Vchitect/VBench/blob/master/prompts/augmented_prompts/Wan2.1-T2V-1.3B/all_dimension_aug_wanx_seed42.txt) and other hyperparameters.

Thanks!
When running on the augmented prompts, I was able to match the reported results.

In [Sampled videos](https://github.com/Vchitect/VBench/tree/master/sampled_videos), the parameters guidance_scale, flow_shift, num_inference_steps, sampler, and negative prompt don’t match the actual Wan config (I assume they refer to sample_guide_scale, sample_shift, sample_steps, sample_solver, and sample_neg_prompt).

Maybe it would help future users if the names matched the real argument names.

Thanks again!
Repository: Vchitect/VBench
Issue: Why is the evaluation result I received not related to prompt?
Root cause: Documentation Deficiency

I evaluate the video generated by my own model using the following instructions, but regardless of whether I provide the "-- prompt" parameter or not, the generated result data is exactly the same.
I modified the -- prompt parameter in the evaluate.py file.
`python evaluate.py     --dimension subject_consistency background_consistency motion_smoothness aesthetic_quality imaging_quality     --videos_path /home/scene_gen/output.mp4     --mode=custom_input`

![Image](https://github.com/user-attachments/assets/2b5aa7aa-a338-4ce3-b289-349078b4a483)

![Image](https://github.com/user-attachments/assets/085d467c-862b-4e21-974b-95d2fbc0e940)

Hi, this is because these dimensions evaluate only the video quality itself and do not consider the text prompt. In general, dimensions under the “Video Quality” category in VBench are prompt-independent, while dimensions under “Video-Text Consistency” assess the alignment between the video and the prompt.

You can refer to the dimension classifications [here](https://github.com/Vchitect/VBench/blob/master/asset/fig_extention_teaser.jpg)
Repository: Vchitect/VBench
Issue: google drive results with augmented prompts
Root cause: Documentation Deficiency

Hi I notice that for some methods, you use augmented prompts, but in the google drive, you still use the short prompt name as the video file name. I am confused about this, are all these videos generated by longer prompts and just named a shorter name?

In addition, can you please provide us with the random seed code, or even better, give us the seed you used for each model? I used random.randint(0, 2**32 - 1) for each index in each prompt, the results seem to be very unstable for the models.

Thanks for your attention! When sampling videos, we use the augmented prompts as specified [here](https:/github.com/Vchitect/vBench/tree/master/sampled_videos#what-are-the-details-of-the-video-generation-models), but for consistency, we name the videos using the short prompt.
As for the random seed - it shouldn't be a key factor affecting video quality. We typically use sequential seeds, e.g.

```python
for sample_idx in range(num_samples_per_item):
    current_seed = current_base_seed + index * num_samples_per_item + sample_idx
```

Hope that helps!
Repository: Vchitect/VBench
Issue: how to use detectron2(which require torch<=1.10) and the torch>=1.12 in requirement
Root cause: External Dependency Breakage

as the title

detectron2 requires torch >=1.8.0, not necessarily torch<=1.10
![image](https://github.com/Vchitect/VBench/assets/43169235/d96f1512-faad-4c70-a751-82c37bafae39)


Hi, we're closing this issue as it appears your questions have been addressed. However, feel free to open a new issue or reopen this one, if you have further questions or if anything else comes up related to this issue.
Repository: Vchitect/VBench
Issue: what is the " split_clip" and " subject_consistency_cat_firstframes_videos" subfolders in the "Wan2.1-T2V-14B" sample videos?

could u give any explanation? I felt very confused.

Hi,

Thanks for pointing this out.

We've now re-uploaded the correct video samples for Wan2.1-T2V-14B, and the extra folders have been removed. Please refer to the latest version here:
https://drive.google.com/drive/folders/1vbsu4MNGrWbJSiw5AKfvtWpJz6MOJ0-1?usp=drive_link

Apologies for the confusion, and thank you for your understanding!


I felt vey upset that totally same video content for different seeds in your new uploaded "Wan2.1-T2V-14B"!
蚌埠住了。

Hi,

Thank you for your feedback and for bringing this to our attention.

We are currently preparing updated Wan2.1-T2V-14B sample videos and will upload the revised versions shortly.

We appreciate your patience and understanding in the meantime.
Repository: Vchitect/VBench
Issue: some files do not exist of vbench 1.0 sample videos
Root cause: Environment Incompatibility

/per_dimension/Sora/sora/A tranquil tableau of at the edge of the Arabian Desert, the ancient city of Petra beckoned with its enigmatic rock-carved façades-0.mp4 not exists
/per_dimension/Sora/sora/A tranquil tableau of at the edge of the Arabian Desert, the ancient city of Petra beckoned with its enigmatic rock-carved façades-1.mp4 not exists
/per_dimension/Sora/sora/A tranquil tableau of at the edge of the Arabian Desert, the ancient city of Petra beckoned with its enigmatic rock-carved façades-2.mp4 not exists
/per_dimension/Sora/sora/A tranquil tableau of at the edge of the Arabian Desert, the ancient city of Petra beckoned with its enigmatic rock-carved façades-3.mp4 not exists
/per_dimension/Sora/sora/A tranquil tableau of at the edge of the Arabian Desert, the ancient city of Petra beckoned with its enigmatic rock-carved façades-4.mp4 not exists                                                                                                                                         /per_dimension/Jimeng/all_dimension/a green vase-0.mp4 not exists
/per_dimension/Jimeng/all_dimension/a fantasy landscape-0.mp4 not exists
/per_dimension/Jimeng/all_dimension/a fantasy landscape-3.mp4 not exists
/per_dimension/Jimeng/all_dimension/An Iron man is playing the electronic guitar, high electronic guitar.-0.mp4 not exists
/per_dimension/Jimeng/all_dimension/An Iron man is playing the electronic guitar, high electronic guitar.-1.mp4 not exists
/per_dimension/Jimeng/all_dimension/An Iron man is playing the electronic guitar, high electronic guitar.-2.mp4 not exists
/per_dimension/Jimeng/all_dimension/An Iron man is playing the electronic guitar, high electronic guitar.-3.mp4 not exists
/per_dimension/Jimeng/all_dimension/An Iron man is playing the electronic guitar, high electronic guitar.-4.mp4 not exists
/per_dimension/Jimeng/all_dimension/A fantasy landscape-1.mp4 not exists
/per_dimension/Jimeng/all_dimension/A fantasy landscape-2.mp4 not exists
/per_dimension/Jimeng/all_dimension/A fantasy landscape-4.mp4 not exists
/per_dimension/Jimeng/all_dimension/Iron Man flying in the sky-0.mp4 not exists
/per_dimension/Jimeng/all_dimension/Iron Man flying in the sky-1.mp4 not exists
/per_dimension/Jimeng/all_dimension/Iron Man flying in the sky-2.mp4 not exists
/per_dimension/Jimeng/all_dimension/Iron Man flying in the sky-3.mp4 not exists
/per_dimension/Jimeng/all_dimension/Iron Man flying in the sky-4.mp4 not exists
Repository: Vchitect/VBench
Issue: 请问vbench2.0 数据集中的wanx测试模型是1.3B还是14B模型？


Hi, we test on Wan 14b.

> Hi, we test on Wan 14b.

Hi @zhengdian1 , could you also provide the Wan2.1-T2V-1.3B results?
Repository: Vchitect/VBench
Issue: Inquiry about Wan2.1-T2V-14B Results on vbench (2025-07-08)

Dear vbench Team,

Thank you for your work on evaluating Wan2.1-T2V-14B. I’m hoping to reproduce the results from 2025-07-08 and would appreciate it if you could kindly share:
Whether **augmented prompts** were used.
Key parameter settings (e.g., **sample_shift,sample_guide_scale**, etc.).
The random **seed** used, if available.


Hi, thank you very much for your message and your interest in our evaluation.

Here are the details:

- Augmented prompts were used as input:

[all_dimension_aug_wanx_seed42.txt](https://github.com/Vchitect/VBench/blob/master/prompts/augmented_prompts/Wan2.1-T2V-1.3B/all_dimension_aug_wanx_seed42.txt)

- Key parameter settings:

sample_shift and sample_guide_scale were not manually specified, so default values were used.
More generation settings can be found here:
[VBench sampled_videos documentation](https://github.com/Vchitect/VBench/tree/master/sampled_videos#what-are-the-details-of-the-video-generation-models)

- Random seed:

The seed for each generation was set as base_seed = 100000 * i + j, where i is the prompt index and j is the sample index (up to 5 samples per prompt).

Please feel free to let us know if you’d like access to the exact prompt files or generated outputs — we’d be happy to share them.
Repository: Vchitect/VBench
Issue: The time of publication of the evaluation results

Dear vbench team,

I am from the JiuTianCV team, we submitted the video results on June 24, 2025. I would like to ask if the sample video we provided has started to be evaluated and what is the expected time for the results to be published?

Hi, it's published to the VBench leaderboard under "Silver Certification" category on 30 Jul. Thanks for your submission!
Repository: Vchitect/VBench
Issue: Why Human_Clothes model cann't output normally
Root cause: Environment Incompatibility

while I infer my videos on "Human_Clothes" , it can not output normally , just output "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"  
how can I solve it? 

Hi, the code runs correctly on our machine. Could you please provide the specific execution commands you used, so that we can better identify the issue?

> Hi, the code runs correctly on our machine. Could you please provide the specific execution commands you used, so that we can better identify the issue?

I run the code without changes, but got the following warnings：
Loaded LLaVA model:LLaVA-Video-7B-Qwen2
You are using a model of type llava to instantiate a model of type llava_qwen. This is not supported for all configurations of models and can yield errors.
Loading vision tower: siglip-so400m-patch14-384
You are using a model of type siglip_vision_model to instantiate a model of type clip_vision_model. This is not supported for all configurations of models and can yield errors.
Some weights of CLIPVisionModel were not initialized from the model checkpoint at siglip-so400m-patch14-384 and are newly initialized: ['vision_model.embeddings.class_embedding', 'vision_model.pre_layrnorm.bias', 'vision_model.pre_layrnorm.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
Some weights of CLIPVisionModel were not initialized from the model checkpoint at siglip-so400m-patch14-384 and are newly initialized because the shapes did not match:
- vision_model.embeddings.position_embedding.weight: found shape torch.Size([729, 1152]) in the checkpoint and torch.Size([730, 1152]) in the model instantiated
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
2025-07-30 20:56:28,724 - accelerate.utils.modeling - INFO - We will use 90% of the memory on device 0 for storing the model, and 10% for the buffer to avoid OOM. You can set `max_memory` in to a higher value to use more memory (at your own risk).

and then the output is "!!!!!!!!!!!!!"

It seems that the siglip does not load correctly, please check whether the model is downloaded correctly. The correct information is like following:

Loaded LLaVA model: /mnt/petrelfs/xx/.cache/vbench2/lmms-lab/LLaVA-Video-7B-Qwen2
Loading vision tower: google/siglip-so400m-patch14-384
Loading checkpoint shards: 100%|██████████| 4/4 [00:45<00:00, 11.43s/it]
Model Class: LlavaQwenForCausalLM
Repository: Vchitect/VBench
Issue: Human anomaly raw video dataset
Root cause: Documentation Deficiency

Hi,

Thanks for the excellent work. I am exploring the ViT-Anomaly detection, and want to see the original videos that you use to extract the human, hand and face samples. The cropped dataset is available at `git clone https://huggingface.co/datasets/Vchitect/VBench-2.0_human_anomaly` but I want to see the actual source video and investigate the performance of YOLO-World. The paper mentions 1000 videos from COCO and 1000 generated videos.

Can you please make that available too?

Thanks in advance.

Hi, please follow the instruction of https://github.com/Vchitect/VBench/tree/master/VBench-2.0/vbench2/third_party/ViTDetector#steps-to-train to download and unzip the folder. The source video is in the ''src_video''.

Hi @zhengdian1  thanks! I see the source videos. Another question: I was able to train the model using the codebase. To double check if the model is trained correctly, do you have the released checkpoints for `checkpoint/human/ckpt29.pth` `checkpoint/face/ckpt29.pth` and `checkpoint/hand/ckpt29.pth`?

Yes, please follow the command VBench-2.0/pretrained/anomaly_detector/download.sh

For other models, please see the instruction here: https://github.com/Vchitect/VBench/tree/master/VBench-2.0#pretrained-models

Thanks! That's very helpful.

Hi, another quick question @zhengdian1 : how did you obtain the images to train from the videos? Did you sample frames randomly from the video? What were the heuristics? And how many frames did you sample?

Please see our papers for details.

Hi @zhengdian1 , I don't see this detail in the paper. The paper mentions that frames are extracted from 1000 real and 1000 generated videos.

<img width="332" height="247" alt="Image" src="https://github.com/user-attachments/assets/53e4eaeb-d747-4735-b18b-bc96e9120537" />

Sorry if I missed anything.

Hi @zhengdian1 , can you elaborate on the data curation process from the videos? Did you sample random frames, or is there a more systematic way of extracting the frames? Also, which samples did you obtain from the HumanRefiner [19] paper?

Hi, thanks for your question. First, we used YOLO-World to detect three categories of object patches from all video frames. To remove redundancy, we computed SSIM between patches from the same video and discarded those with similarity above 0.7, then randomly sampled the remaining ones. These patches, combined with those from HumanRefiner, formed our final dataset.

For the HumanRefiner dataset specifically, we selected:

* **Head class**: category\_id = 1 (negative), 11 (positive)
* **Hand class**: category\_id = 5 (negative), 15 (positive)
* **Human class**: all patches except category\_id = 9 were treated as positive and negative samples
Repository: Vchitect/VBench
Issue: error: not able to find vbench2.third_party.Instance_detector.tests——where the function compute_anomaly is defined.
Root cause: Validation Gap

Hello dear vbench authors,

Thanks a lot for your valuable work in this awesome benchmark project. Yet I had problem while running vbench2.0 code to evaluate videos in "Instance Preservation" dimension.

In the code —— vbench2/instance_preservation.py, the function——compute_anomaly is imported from "from vbench2.third_party.Instance_detector.tests import compute_anomaly", yet I was not able to find the "vbench2.third_party.Instance_detector.tests" package at all. 

Could you please look into it and help?

Many thx.

<img width="911" height="414" alt="Image" src="https://github.com/user-attachments/assets/efb8dc82-d71a-4886-8482-8db99c1f9a2d" />


Hi, thanks for pointing out, the error has been fixed, please pull our code again. The correction contains uploading the missing test.py file and I notice that you have changed the Instance_detector.test to xx.tests in line10, please revise it.

> Hi, thanks for pointing out, the error has been fixed, please pull our code again. The correction contains uploading the missing test.py file and I notice that you have changed the Instance_detector.test to xx.tests in line10, please revise it.

Awesome! Thanks a lot for your reply and corrections!