weikaih/molmo2-codec-pruning16-step1000

Molmo2 Codec — Native Pruning-16 Step 1000

This is the first dedicated native-cell Pruning-16 checkpoint for Molmo2 Codec. It is intentionally separate from the learned P-tokenizer and historical fast_gray2 repositories.

What is in this release

• weights/stage2_step1000_consolidated.pt: complete 4B Stage-2 model state.
• config/stage2_step1000.yaml: exact training configuration.
• config/native_p_tokenization.json: native Pruning-16 architecture contract.
• config/gamma_v6_rgbhex_k4.json: RGB-HEX/K4 GOP contract.
• eval/mlvu_n200_summary.json: fixed-ID MLVU Dense/Codec result.
• eval/multibench_n200_summary.json: fixed-ID six-benchmark paired suite.
• SHA256SUMS: byte identity of every published artifact.

There is no Stage-1 P-tokenizer file. P-frames use the frozen native Molmo ViT, score the 81 native connector cells with robust motion/residual saliency, keep 16 cells in raster order, and pass them through an independent native connector copy stored in the Stage-2 checkpoint.

Required input contract

field	value
timeline	full-span 2 FPS, at most 2,048 source frames
motion	RGB HEX, 378 x 378, 14 x 14 blocks, +/-48 px
GOP	calibrated adaptive trigger, target/max P = 4
I-frame	native Molmo path, 81 tokens
P-frame	native-cell Pruning-16, 16 tokens
formal evaluation	8,192 visual tokens, 16,384 total sequence tokens

Do not combine this checkpoint with fast_gray2 caches or learned-P tokenizers. Motion mode, gamma, FPS, timeline cap, visual budget and K are cache identity fields.

Evaluation

The release was evaluated on frozen, identical 200-example manifests. Dense and Codec share the Stage-2 checkpoint, prompt, source video and answer scorer; only visual tokenization differs. Exact results are stored in eval/ and in the repository performance record.

MLVU n=200 input	correct	accuracy
Dense native frames	158 / 200	79.0%
Native Pruning-16 Codec	160 / 200	80.0%

The +1.0-point MLVU difference is a paired diagnostic on one frozen 200-item sample, not a claim that compression generally improves accuracy. Consult the six-benchmark summary for the broader Dense/Codec gap.

benchmark (n=200)	Dense	Pruning-16	delta
Video-MME	68.5%	67.5%	-1.0
LVBench	50.5%	45.0%	-5.5
MVBench	76.0%	73.5%	-2.5
TempCompass	69.5%	68.0%	-1.5
PerceptionTest	82.0%	79.5%	-2.5
NExT-QA	83.5%	83.5%	0.0
six-task mean	71.67%	69.50%	-2.17
excluding LVBench	75.90%	74.40%	-1.50

Every Dense/Codec row passed exact paired-document-ID verification. LVBench is kept in the formal six-task mean and is the largest remaining regression.

Usage

hf download weikaih/molmo2-codec-pruning16-step1000 \
  --local-dir checkpoints/molmo2-codec-pruning16-step1000

export CODEC_P_TOKENIZATION=pruning16
export CODEC_MOTION_MODE=hex
export CODEC_GOP_TARGET_P=4
export CODEC_GOP_MAX_P=4
export CODEC_GOP_PROFILE_CACHE=k4-rgbhex-v2
export CODEC_GOP_TIMELINE_MAX_FRAMES=2048
export CODEC_GAMMA_ARTIFACT=$PWD/codec/configs/gamma_v6_rgbhex_k4.json

Use the molmo2_codec adapter in the linked lmms-eval fork. Native Pruning-16 requires the Stage-2 checkpoint and gamma artifact, but no ptok_checkpoint argument.

accelerate launch -m lmms_eval \
  --model molmo2_codec \
  --model_args \
training_repo=/path/to/molmo2-codec,\
pretrained=$PWD/checkpoints/molmo2-codec-pruning16-step1000/weights/stage2_step1000_consolidated.pt,\
base_checkpoint=/path/to/molmo2-4b-sft-step30000,\
gamma_artifact=$PWD/checkpoints/molmo2-codec-pruning16-step1000/config/gamma_v6_rgbhex_k4.json,\
video_backend=codec,visual_token_budget=8192,timeline_max_frames=2048,seq_len=16384 \
  --tasks mlvu --batch_size 1

The evaluation adapter is maintained on the molmo2-codec-eval branch. Use its checked-in frozen manifests for a paired Dense/Codec comparison; changing only the visual backend while keeping document IDs and scoring fixed is part of the reported protocol.

This is a research checkpoint. It compresses P-frames from 81 to 16 visual tokens; it is not a claim of lossless visual reconstruction.