Repository: Vchitect/VBench
Issue: Fail to Reproduce Hunyuan Results on VBench
Root cause: Documentation Deficiency

I sincerely appreciate the work on VBench, as it is extremely helpful for text-to-video projects.

Currently, I am trying to evaluate the open-source HunyuanVideo model, but I am unable to reproduce the official leaderboard results. The official total score is reported as 83.24%, while my reproduced result is 81.88%.  The most significant discrepancy is in the `dynamic_degree` dimension. The official leaderboard reports a score of 70.83%, while my reproduced result is only 54.72%.  

For video generation, I am using a resolution of 720x1280, 129 frames (~5s). I generated 5 videos for each prompt with seed 0,1,2,3,4. The prompt file I used is [all_dimension.txt](https://github.com/Vchitect/VBench/blob/master/prompts/all_dimension.txt) (not [hunyuan_all_dimension.txt](https://github.com/Vchitect/VBench/blob/master/prompts/augmented_prompts/hunyuan_all_dimension.txt)). For evaluation, I followed the script [evaluate_videocrafter1.sh](https://github.com/Vchitect/VBench/blob/master/scripts/evaluate_videocrafter1.sh) (not `vbench_long`).

I would appreciate any guidance on how to reproduce HunyuanVideo's results on VBench. Thank you!

For HunyuanVideo, we use VBench-Long as it's videos are longer than 5.0 seconds.

Thank you so much for your clarification!
It turns out that the primary factor causing this quality degradation is the random seed. It is wrong to use the fixed seed set of 0,1,2,3,4. Instead, I should randomly sample the seed from a larger range and use a different set of 5 seeds for each prompt. I am not sure if there is a good explanation for why this is the case.

For future reference,
My result:
total score: 83.08%, quality score: 85.01%, semantic score: 75.35%
VBench result:
total score: 83.24%, quality score: 85.09%, semantic score: 75.82%
I consider this a successful reproduction.

Hi, if you find the evaluation results are different using the sampled videos provided by us, feel free to let us know.

Sure, but that's not the case. The VBench result on sampled videos from the vbench team is identical with the reported number in the leaderboard.

Hi @ziqihuangg I found there is a slight difference between the Wan2.1-T2V-1.3B numbers on the leaderboard and the numbers I obtained. I download videos from [google drive](https://drive.google.com/drive/folders/1asyr8ytkumWh3qXWlPMm89Fq3tCdJxaA) and run the following scripts, below is the number I obtained:

{'subject consistency': 0.9248733298061416, 'background consistency': 0.9729625613190407, 'temporal flickering': 0.9882482199574613, 'motion smoothness': 0.973236199386657, 'dynamic degree': 0.8833333333333333, 'aesthetic quality': 0.6568369825681051, 'imaging quality': 0.6667130306810557, 'object class': 0.9439873417721519, 'multiple objects': 0.7981707317073171, 'human action': 0.984, 'color': 0.9034738844763598, 'spatial relationship': 0.7695203249625574, 'scene': 0.4975290697674419, 'appearance style': 0.2112430555555557, 'temporal style': 0.2527739648222923, 'overall consistency': 0.2703167255206775}

+------------------|------------------+
|     quality score|0.8501038634314173|
|    semantic score|0.7978241127664059|
|       total score|0.8396479132984151|
+------------------|------------------+

```
torchrun --nnodes=1 --nproc_per_node=8 --standalone \
    evaluate.py \
    --videos_path ${path_to_videos} \
    --output_path {output_path} \
    --dimension subject_consistency background_consistency temporal_flickering motion_smoothness dynamic_degree aesthetic_quality imaging_quality object_class multiple_objects human_action color spatial_relationship scene temporal_style appearance_style overall_consistency
```

I'm using the following scripts to install environment
```
env_name="vbench"
conda create -n "$env_name" python=3.10 -y
conda activate "$env_name"
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu118 # or any other PyTorch version with CUDA<=12.1

# install cuda devkit (for detectron2)
conda install -c nvidia/label/cuda-11.8.0 cuda-toolkit=11.8 -y
conda install -c conda-forge gxx_linux-64 -y

CUDA_HOME=$CONDA_PREFIX pip install detectron2@git+https://github.com/facebookresearch/detectron2.git
pip install VBench
```

Closing this issue for now. If @xunhuang1995 or @Leymore feel it's not solved, feel free to reopen it anytime. I'm going to reply to issues raised @xunhuang1995 in a separate thread.