PhyRubric-Judge

📄 Paper: PhyRubric: Rubric-Grounded Diagnostic Evaluation of Physical Commonsense in Text-to-Video Generation

LoRA adapter (r = 16, all linear layers of the language model) on Qwen/Qwen3-VL-8B-Instruct, fine-tuned on PhyRubric-Train (3,788 clips from four video generators, labelled with the PhyRubric rubric protocol).

Given the caption used to generate a video and 16 uniformly sampled frames of the clip, the judge returns

{"sa_score": 1-5, "pc_assessable": true|false, "pc_score": 1-5, "reason": "..."}

sa_score is semantic adherence (does the clip show what the caption describes), pc_score is physical commonsense (does everyday physics hold), pc_assessable says whether the central physical event of the caption appears at all, and reason is a timestamped explanation. The judge never sees a rubric.

Training: 2 epochs, lr 1e-4, LoRA rank 16 / alpha 32, bf16, 16 frames at most 448x448, ms-swift 4.5.

Usage

Code, data and the evaluation scripts are at https://github.com/chenyil6/phyRubric.

hf download kittyhy/PhyRubric-Judge --local-dir models/PhyRubric-Judge
CKPT=models/PhyRubric-Judge RUN=judge bash scripts/infer.sh      # scores PhyRubric-Bench and prints the metrics

scripts/infer.sh runs judge/infer.py, which loads the base model with this adapter through ms-swift and applies the same frame sampling as training (FPS=24 FPS_MAX_FRAMES=16 VIDEO_MAX_PIXELS=448*448). The system prompt the judge was trained with is data/system_prompt.txt in the repository.