q1716523669/visionopd-archive

Vision-OPD Intrinsic-Reward Experiments — Full Archive

Complete training checkpoints (FSDP shards including optimizer state) and logs for 6 experiments: three unsupervised intrinsic rewards (TTRL majority-vote / Intuitor self-certainty / RENT negative-entropy) replacing the self-distillation objective of Vision-OPD, on Qwen3.5-4B / 9B, trained on Vision-OPD-6K (6241 samples, 65 steps = 1 epoch, checkpoint every 5 steps).

Headline result: all six runs degrade; best checkpoints all fall at step 5–20. Full write-up in docs/ and summary/.

Repo layout

folder	experiment	recipe	original work_dir
ttrl-4b-nokl/	TTRL, Qwen3.5-4B	no KL (β=0, Vision-OPD hp)	ttrlvopdhp_vopd6k_qwen35_4b_nothink
intuitor-4b-nokl/	Intuitor, Qwen3.5-4B	no KL	intuitorvopdhp_vopd6k_qwen35_4b_nothink
rent-4b-kl/	RENT, Qwen3.5-4B	KL β=0.01	b200_rentkl_vopd6k_qwen35_4b_nothink
ttrl-9b-nokl/	TTRL, Qwen3.5-9B	no KL	b200_ttrlvopdhp_vopd6k_qwen35_9b_nothink
intuitor-9b-kl/	Intuitor, Qwen3.5-9B	KL β=0.01	intuitorkl_vopd6k_qwen35_9b_nothink
rent-9b-kl/	RENT, Qwen3.5-9B	KL β=0.01	rentkl_vopd6k_qwen35_9b_nothink

Each experiment folder: global_step_{5..65}/actor/ (verl FSDP shards, world_size 8: model_*.pt, optim_*.pt, extra_state_*.pt, plus huggingface/ tokenizer/config) + train.log.

Other folders:

• docs/ — experiment record, results tables, findings draft, reproduce/ops notes, incident postmortems
• summary/ — packaged analysis (README + per-experiment tables + train/eval CSV)
• eval_out/ — raw eval outputs, 78 checkpoints × 6 benchmarks (selection split, 3216 items)
• job_logs/, eval_logs/ — training / evaluation job logs
• curves/ — training-side metric curves (CSV)
• tensorboard_log/ — tensorboard event files

Results at a glance (selection split avg of 6 benchmarks)

experiment	best step	best	step 65	drop
TTRL 9B	15	78.46	66.00	−12.5
RENT 9B (KL)	5	77.87	7.12	−70.8
Intuitor 9B (KL)	5	76.85	38.34	−38.5
TTRL 4B	20	74.46	60.19	−14.3
RENT 4B (KL)	5	71.03	10.99	−60.1
Intuitor 4B	5	70.25	9.18	−61.1

Benchmarks: V*Bench, ZoomBench, HR-Bench 4K/8K, MME-RealWorld, MME-RealWorld-CN. Random baseline ≈ 22–25.

⚠️ RENT late-checkpoint MME-RealWorld "rebound" (52–57 while all else is 0) is an evaluation artifact of empty outputs (response_length=1), not recovery.

Ready-to-eval merged HF-format best checkpoints are in q1716523669/visionopd-best-ckpts.

To use an FSDP checkpoint from this archive: merge with python -m verl.model_merger merge --backend fsdp --local_dir <global_step_N>/actor --target_dir <out>, then copy preprocessor_config.json etc. from the Qwen3.5 base model (model_merger does not emit them).