winbeau/vbench-faithful

VBench Faithful

Aligning VBench Scores with Their Intended Targets: Toward Faithful Video Evaluation

Code & Quick Start · Gradio Demo · Training & Data

The models used by the current VBench Faithful project: one checkpoint per trained dimension, comprising six Qwen3-8B LoRA adapters and one Dynamic Degree head. The evaluator supports all 16 dimensions, with nine repairs and seven accelerated dimensions in its default mode.

Models

Dimension	Location
Scene	adapters/scene/
Human Action	adapters/human_action/
Spatial Relationship	adapters/spatial_relationship/
Multiple Objects	adapters/multiple_objects/
Object Class	adapters/object_class/
Color	adapters/color/
Dynamic Degree	heads/dynamic_degree/aligned.pt

Each adapter directory contains adapter_config.json and adapter_model.safetensors. The seven trained weights total approximately 1.05 GB. Subject and Background Consistency and the seven accelerated dimensions use shared pretrained models.

Use

Follow the project quick start to install the runtime, restore the shared models and prepare your videos. From the project checkout:

uv run python scripts/eval.py --config configs/eval.yaml

The runtime setup downloads these weights, verifies their hashes and installs the required configurations. To download this model repository directly:

hf download winbeau/vbench-faithful --local-dir output/models

The Gradio demo provides video upload, dimension selection and downloadable results. Its application code lives in space/.

Model information

The adapters use Qwen3-8B; the Dynamic head uses the frozen V-JEPA 2.1 encoder. Dynamic requires 16 native square RGB frames at 8 FPS and reports relative motion on its learned scale. Unsupported inputs retain missing scores.

manifest.json records file identities. Configurations and training provenance accompany the weights; the project's reproduction guide documents evaluation protocols and interpretation limits. Base-model and dataset license terms apply; V-JEPA's notice is retained in licenses/.

Citation

@misc{zhao2026aligningvbench,
  title  = {Aligning {VBench} Scores with Their Intended Targets:
            Toward Faithful Video Evaluation},
  author = {Zhao, Wenbiao and Nie, Zichao and Meng, Shuyuan and
            Wang, Meng and Xue, Haiwei and Zheng, Panpan},
  year   = {2026},
  note   = {Manuscript},
  url    = {https://github.com/winbeau/vbench-faithful}
}