hybrid-diff-ar/retry-frontier-checkpoints

Retry-frontier RL checkpoints (Qwen3-1.7B)

RL checkpoints that continue daman1209arora/Reliability-1.7B-final-idk-ckpt-1500 (revision a841f88) on DAPO math with the infinite-retry frontier objective J = q - λC.

An attempt either submits a boxed answer or outputs \boxed{\text{I don't know}} (IDK), which buys a fresh, stateless retry:

• q = pS / (1 - pF) is the success probability with unlimited retries.
• C = s / (1 - pF) is the expected output tokens per question, in thousands.
• λ is the price per 1000 output tokens.

run (folder in this repo)	λ	checkpoints	wandb run
train_118786/frontier_lam0.01_seed42	0.01	steps 20-720, every 20	r4w7qllx
train_121126/frontier_lam0.02_seed42	0.02	steps 20-620, every 20	pumnmti9
train_118786/frontier_lam0.05_seed42	0.05	steps 20-120, every 20 (stopped early)	sp20z9cp

The wandb runs are in project train-llm/Reliability-1.7B-retry-frontier.

Training

All three runs share these settings:

• 32 questions × 64 rollouts per step, one on-policy update per step.
• Constant lr 1e-6, no KL or entropy terms, seq-mean-token-sum loss.
• Sampling at T = 1, top-p 1, up to 4096 output tokens; seed 42; 8×H200 per run.
• Data and prompt are the base checkpoint's train_c_0.4.parquet, unchanged (the c = 0.4 IDK prompt).
• Instead of GRPO advantages, each sample is weighted by the exact gradient of J with plug-in group statistics.

Each run folder also contains:

• config.yaml: the full verl config;
• experiment.json: provenance, including the data hash and git commit;
• metrics.jsonl: per-step training metrics.

Optimizer states are not included.

Evaluated checkpoints

eval/REPORT.md has the full evaluation.

• train_118786/frontier_lam0.01_seed42/global_step_440 (l01_s440): a Pareto improvement over the base checkpoint on all six evaluation sets. Cap-8 accuracy is the same or higher, with 22-61% fewer output tokens.
• train_121126/frontier_lam0.02_seed42/global_step_300 (l02_s300): cheaper still, but 2-5 points less accurate.

Known instabilities:

• λ = 0.01: entropy blows up around steps 455-530, then recovers into a shorter, less accurate regime.
• λ = 0.02: entropy collapses after about step 400.
• The last checkpoints before these instabilities are λ = 0.01 step 440 and λ = 0.02 steps 380-400.

Loading

The safetensors hold float32 (master) weights, although config.json says torch_dtype: bfloat16. Pass the dtype explicitly.

from huggingface_hub import snapshot_download
from transformers import AutoModelForCausalLM

sub = "train_118786/frontier_lam0.01_seed42/global_step_440"
root = snapshot_download("hybrid-diff-ar/retry-frontier-checkpoints", allow_patterns=f"{sub}/*")
model = AutoModelForCausalLM.from_pretrained(f"{root}/{sub}", torch_dtype="bfloat16")
# vLLM: LLM(model=f"{root}/{sub}", dtype="bfloat16")