ngqtrung/omr-8b-grpo-base

Qwen3-VL-8B · OMR · GRPO baseline (cold-start)

RLVR post-training of Qwen/Qwen3-VL-8B-Instruct on the OMR (OpenMMReasoner, math/visual-reasoning) data with fully-async GRPO, cold-started from the stock 8B-Instruct (no exploration, no warm-start). This is experiment_name=grpo_omr_4node_full_v1_8b_base_perf, global_step_25 (the keeper).

This is the OMR cold-start GRPO baseline. It peaks at OMR overall-6 val accuracy 0.668 @ step 25, then collapses to 0.499 by step 125 — cold-start RL on 8B-OMR is unstable without exploration. The keeper is global_step_25, the only checkpoint before degradation began. (The omr-8b-grpo-ppexplore sibling fixes this collapse and reaches 0.714.)

Results

OMR 6-image inline validation (mmmu val, mathvista testmini, mathverse testmini Text-Dominant, wemath testmini, charxiv reasoning-qa, dynamath test). overall-6 = unweighted mean. Metric = accuracy.

Keeper = global_step_25 (peak):

metric	overall-6	mmmu	mathvista	mathverse	wemath	charxiv	dynamath
baseline peak @25	0.6681	0.6311	0.8019	0.8404	0.7437	0.3930	0.5986
stock Qwen3-VL-8B ckpt-0	0.659	0.629	0.811	0.824	0.723	0.396	0.570
baseline @125 (collapse)	0.4994	0.5021	0.6796	0.6995	0.6184	0.2010	0.2958

Full trajectory (overall-6): 0.668 @25 → 0.644 @50 → 0.620 @75 → 0.620 @100 → 0.499 @125 (monotone decline; the step-125 drop is catastrophic — charxiv halved, dynamath collapsed). Run died to a vLLM deepstack crash after step 125, but the model was already degrading.

Training

• Base model: Qwen/Qwen3-VL-8B-Instruct (cold-start, resume_mode=auto).
• Framework: fork of volcengine/verl — ngquangtrung57/verl@videorl-mods. Fully-async GRPO: FSDP2 trainer + vLLM rollouter.
• Warm start: none (cold-start from stock 8B-Instruct).
• Reward: dapo-style score = 0.8·accuracy + 0.2·format (FORMAT_WEIGHT=0.2, FORMAT_MIN_THINK_CHARS=100). No KL penalty.
• Exploration: OFF (this is the bitwise baseline against omr-8b-grpo-ppexplore).
• Topology: 4-node 2+2 — 2 trainer nodes (16-GPU FSDP2, dp=16) + 2 rollout nodes (16 GPU, vLLM TP=2 → 8 replicas). H100×8 per node.
• Batch: ppo_mini_batch_size=16 × require_batches=4 × rollout.n=8 = 512 trajectories/step.
• Optim / seq: lr 1e-6, warmup 25 steps; total_epochs=2; clip_ratio 0.2 / 0.3 (clip_c=10.0); max_prompt_length=2048, max_response_length=16384; enforce_eager=true; gpu_memory_utilization=0.75; staleness 0.5.
• Train metrics: ~178 s/step; final reward 0.721; final response_length ~2194 tok; 125 steps trained.

W&B

Project verl_fully_async (entity quangtrung5705-nanyang-technological-university-singapore): https://wandb.ai/quangtrung5705-nanyang-technological-university-singapore/verl_fully_async/runs/gkiiopep

Intended use / limitations

Research checkpoint — the cold-start GRPO baseline in a controlled exploration study. Useful mainly as the A/B reference for omr-8b-grpo-ppexplore (the winner). It only marginally beats the zero-shot base (0.668 vs 0.659) at its peak and is unstable (collapses to 0.499 if trained past ~step 100); for actual use prefer the ppexplore checkpoint. Math / visual-reasoning image QA, <think>…</think> then-answer format. No safety/RLHF alignment beyond the base.

Usage

from transformers import AutoModelForImageTextToText, AutoProcessor
from PIL import Image

model_id = "ngqtrung/omr-8b-grpo-base"
model = AutoModelForImageTextToText.from_pretrained(model_id, dtype="auto", device_map="auto")
processor = AutoProcessor.from_pretrained(model_id)

messages = [{
    "role": "user",
    "content": [
        {"type": "image", "image": Image.open("problem.png")},
        {"type": "text", "text": "Solve the problem. Think step by step inside <think>...</think>, then give the final answer."},
    ],
}]
inputs = processor.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=True,
    return_dict=True, return_tensors="pt"
).to(model.device)
out = model.generate(**inputs, max_new_tokens=2048)
print(processor.batch_decode(out[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True)[0])

Citation / lineage

• Base model: Qwen3-VL-8B-Instruct (Qwen team). Inherits the Qwen3-VL license — review the base model's terms; the Apache-2.0 tag refers to this repo's RLVR training artifacts.
• Framework: verl (volcengine/verl), fork ngquangtrung57/verl@videorl-mods; fully-async GRPO (FSDP2 + vLLM).
• Study: controlled OMR/Video exploration study on Qwen3-VL-8B; this is the no-exploration OMR baseline arm (docs/experiments_summary_8b.md).