Qwen3.5-4B β GRPO/DAPO on Video-MME-v2 (800 steps, 100 frames)
LoRA adapter (r=16, Ξ±=32, language side only) trained with GRPO on Video-MME-v2 multiple-choice video QA. Training code: https://github.com/infinitylogesh/videomme-rl
Results β 400 held-out questions, 100 frames, thinking mode
Held-out = 100 videos disjoint from the 500 training videos (split by video, never by question:
each video carries 4 questions, and the logic groups chain them). Every run uses the same seed and
the same option permutation, so all comparisons are paired.
| strict acc | tagged | truncated | |
|---|---|---|---|
| base Qwen3.5-4B | 0.0625 | 0.223 | 0.770 |
| + GRPO 400 steps | 0.2475 | 0.975 | 0.025 |
| + GRPO 800 steps (this adapter) | 0.3050 | 0.9975 | 0.0025 |
Chance is 0.129 (8 options). Paired McNemar, strict:
| comparison | net | p |
|---|---|---|
| base β 400 steps | +74 | < 0.0001 |
| 400 β 800 steps | +23 | 0.0206 |
| base β 800 steps | +97 | < 0.0001 |
Report STRICT accuracy, not loose
strict = the model emitted <answer>X</answer> and X was right. A "loose" parser that also
accepts the last stray AβH letter in the text inflates the baseline enormously: of its 79
loose-correct answers, only 25 had a real answer tag β 53 came from completions that hit the token
cap without answering at all. With 8 options that is a 1-in-8 lottery, awarded on 77% of questions.
The two metrics converge for the trained model (0.3050 vs 0.3050) and diverge for the baseline
(0.1975 vs 0.0625). Loose accuracy rewards exactly the behaviour this training removes.
What the training actually changed
The base model does not fail by being wrong so much as by never finishing: it reasons until the token cap and emits no answer. Truncation went 0.770 β 0.0025, answer-tagging 0.223 β 0.9975. It still reasons β median completion is ~300β500 tokens with substantive analysis β it just concludes now.
Config
lr 1e-5 Β· beta 0.0 (no KL) Β· loss_type dapo Β· scale_rewards none (Dr.GRPO)
epsilon_high 0.28 (DAPO Clip-Higher) Β· num_generations 8 Β· 100 frames @ max_pixels 50176
max_completion_length 2048 Β· thinking mode ON Β· reward 0.9*correct + 0.1*tagged, strict
Reward = 0 for a truncated completion, and truncated rollouts stay in the loss (deliberately not
mask_truncated_completions) β that is the only pressure against rambling.
Negative results worth keeping
- Frames are not the lever. 32 β 100 frames moved the baseline only +2.25pp (p=0.41). Even on fine-grained temporal questions ("how many timesβ¦", "first 15 seconds") it went 0.054 β 0.107, still below the 0.129 chance floor. The gains above are the algorithm/lr changes.
- Entropy regularisation is the wrong tool here. ~46% of groups have zero reward variance, but
measured entropy is 0.55 nats β not a collapsed policy. With
entropy_target=0.2TRL never applied the bonus (entropy_coef= 0.0000 for 400 steps). The zero-variance groups are saturated prompts (always wrong), not an exploration failure. DAPO dynamic sampling would recover the wasted compute but would not make hard prompts solvable β and it is absent from TRL 1.8. - A within-run reward plateau does not mean no headroom. Reward looked flat at the end of the 400-step run, yet the next 400 steps bought +5.75pp (p=0.02). Judge by eval, not by the reward curve.
Files
adapter/β LoRA + optimizer state (resumable)results/β per-question eval jsonl for every run above, plus the exact train/eval splits