mh-fresh-K6-delta

LoRA adapters for the K=6, delta cell of the coverage x debiasing table, entropic risk tau = 10. Trained from scratch -- no warm start.

Provenance

Trained locally by Max Horwitz on the UW Hyak cluster (SLURM job 40455925, launched 2026-09-22). Training IN PROGRESS -- checkpoints through step 3744 of 4680 are uploaded; later ones will be added as they are written.

This set exists because the earlier mh-ec2-ttomd-* table was warm-started from ipo-e-c10.0/checkpoint-936, which is itself step 936 of the K=8 vanilla run -- so those K=2/K=4 cells were branches off the K=8 baseline rather than independent arms. These runs start from the SFT base with a zero-initialised LoRA adapter and 1000 warmup steps, so no checkpoint from any other run enters them. Verified from each job's launch line: no --init_adapter, no --load_dir.

What this run is

algorithm	online IPO (--alg oipo1), risk_egpo/tt_omd.py
coverage	K = 6 (--ypp_samples 6)
risk	entropic, tau = 10
debiasing	two-timescale bias correction, delta-method estimator
TT step size	gamma = 0.1
init	from scratch (SFT base, LoRA = 0), 1000 warmup steps
generation	64 max new tokens
seed	42
base model	vectorzhou/gemma-2-2b-it-alpaca-cleaned-SFT
dataset	PKU-Alignment/PKU-SafeRLHF
checkpoints	8 (468 ... 3744, every 468 steps)

Files

checkpoint-<step>/ holds the LoRA adapter and tokenizer files (no DeepSpeed resume state). training_dynamics.csv has this run's per-log-step metrics: loss, grad_norm (L2, pre-clip), kl, rewards/accuracies, rewards/margins.

from peft import PeftModel
from transformers import AutoModelForCausalLM
m = AutoModelForCausalLM.from_pretrained("vectorzhou/gemma-2-2b-it-alpaca-cleaned-SFT")
m = PeftModel.from_pretrained(m, "rat-lab/mh-fresh-K6-delta", subfolder="checkpoint-3744")