Repository: Vchitect/VBench
Issue: VBench 2, reproducibility issues -- what GPU was used for evaluation?
Root cause: Environment Incompatibility

Hello,

I was trying to reproduce evaluation results that you presented in the paper (VBench-2.0) for videos generated by Sora model. I took the videos from your google drive. Unfortunately I'm seeing some differences, largest difference in score for a single dimension is around `2.5` (on a scale `0 - 100`). I've tried running the same setup on different kinds GPUs (`H100`, `A100`, `L40`) and got different results on each GPU, which makes me think this may be some differences in kernels offered for different GPUs.

Could you please share what kind of GPU did you use for your evaluation? @Jacky-hate @zhengdian1 


Hi, we use A100 for evaluations.

Thank you for fast response, I also used A100 and for `Material` dimension on Sora videos (downloaded from https://drive.google.com/drive/folders/1yK3xiD7HhpGjMKMFddcgouaV1bymSqSw) I got `70.27%` instead of `64.94%` from the paper 😭.

I run evaluation using this docker container: `pytorch/pytorch:2.5.1-cuda11.8-cudnn9-devel`
and those are exact commands that I run:
```
apt-get update
apt-get install git -y
apt-get install wget -y
mkdir -p ~/miniconda3
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda3/miniconda.sh
bash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3
rm ~/miniconda3/miniconda.sh
source ~/miniconda3/bin/activate
conda create -n vbench2 python=3.10 -y
conda tos accept --override-channels --channel https://repo.anaconda.com/pkgs/main
conda tos accept --override-channels --channel https://repo.anaconda.com/pkgs/r
conda create -n vbench2 python=3.10 -y
conda activate vbench2
conda install psutil -y
pip install torch==2.5.1 torchvision==0.20.1 torchaudio==2.5.1 --index-url https://download.pytorch.org/whl/cu118
python -m pip install ninja
git clone https://github.com/Vchitect/VBench.git
cd VBench/VBench-2.0/
python -m pip install git+https://github.com/Dao-AILab/flash-attention.git@v2.7.2.post1
pip install mmcv==2.2.0 -f https://download.openmmlab.com/mmcv/dist/cu118/torch2.4/index.html --no-cache-dir
pip install -r requirement.txt
pip install retinaface_pytorch==0.0.8 --no-deps
apt-get install unzip -y
cd vbench2/third_party/Instance_detector
pip install -e .
cd ../../..
mmdet_version_file=/root/miniconda3/envs/vbench2/lib/python3.10/site-packages/mmdet/__init__.py
sed -i "s/mmcv_maximum_version = '2.1.0'/mmcv_maximum_version = '2.3.0'/" $mmdet_version_file
mmyolo_version_file=/root/miniconda3/envs/vbench2/lib/python3.10/site-packages/mmyolo/__init__.py
sed -i "s/mmcv_maximum_version = '2.1.0'/mmcv_maximum_version = '2.3.0'/" $mmyolo_version_file
cd /workspace/VBench/VBench-2.0/pretrained/LLaVA-Video-7B-Qwen2
chmod +x download.sh
./download.sh
cd /workspace/VBench/VBench-2.0/
# Download videos into `input_videos` directory
apt-get install libgl1 -y
apt-get install -y libglib2.0-0
sed -i "s/auto/cuda/" /workspace/VBench/VBench-2.0/vbench2/material.py  # otherwise it tries to use TP, shouldn't be doing anything with 1gpu used
python -c 'from vbench2 import VBench2; my_VBench = VBench2("cuda", "vbench2/VBench2_full_info.json", "evaluation_results2"); my_VBench.evaluate(videos_path = "input_videos", name = "Sora_Material", dimension_list = ["Material"])'
```
I also attach `json` with results of `Material` dimension evaluation, do you know what could be wrong with my setup?

[A100_material_vbench2_results.txt](https://github.com/user-attachments/files/22725698/A100_material_vbench2_results.txt)

Hi, based on the environment configuration you provided, everything appears to be in order. May I confirm if your GCC version is 11.2? If so, could you please share the complete environment configuration? I will further analyze the potential causes.

Hello, initially `gcc`/`g++` version was `11.4.0`, I installed `11.2.0` by changing both `conda create -n vbench2 python=3.10 -y` lines in my setup above into `conda create -n vbench2 python=3.10 gcc=11.2.0 gxx=11.2.0 -c conda-forge -y`, but I still got the same result. 

What do you mean by `complete environment configuration`?

Hi, I mean could you please export your whole environment, so that I can check the differences.

Do you need anything besides those two?

[conda_env_export.txt](https://github.com/user-attachments/files/22737952/conda_env_export.txt)

[pip_freeze.txt](https://github.com/user-attachments/files/22737953/pip_freeze.txt)

@zhengdian1  - waiting for your guidance above ^^^. Need to understand the nature of those numerical differences