mh-ec2-ttomd-K4-jk

LoRA adapters for the K=4, jackknife cell of the coverage x debiasing table in the risk-sensitive preference-learning experiments (entropic risk, tau = 10).

Provenance

NOT trained by Max -- pulled from an existing rat-lab repo.

Pulled unchanged from rat-lab/rlj-ec2-fig9-K4-jk.

Trained earlier by the RLJ/EC2 effort, not by Max. Verified byte-identical to the source repo (sha256 on checkpoints 250 / 2250 / 4000). NOTE: this run stops at step 4000 -- the source repo has no later checkpoints, so this cell has 16 checkpoints where the others have 19.

What this run is

algorithm	online IPO (--alg oipo1), risk_egpo/tt_omd.py
coverage	K = 4 (--ypp_samples 4)
risk	entropic, tau = 10 (--risk entropic --risk_c 10.0)
debiasing	two-timescale bias correction, leave-one-out jackknife estimator
TT step size	gamma = 0.1
init	warm start from ipo-e-c10.0/checkpoint-936, 100 warmup steps
generation	64 max new tokens
seed	42
base model	vectorzhou/gemma-2-2b-it-alpaca-cleaned-SFT
dataset	PKU-Alignment/PKU-SafeRLHF
checkpoints	16 (250 ... 4000, every 250 steps)

Contents

checkpoint-<step>/ holds the LoRA adapter (adapter_model.safetensors, adapter_config.json) and tokenizer files. DeepSpeed resume state is not included.

from peft import PeftModel
from transformers import AutoModelForCausalLM
m = AutoModelForCausalLM.from_pretrained("vectorzhou/gemma-2-2b-it-alpaca-cleaned-SFT")
m = PeftModel.from_pretrained(m, "rat-lab/mh-ec2-ttomd-K4-jk", subfolder="checkpoint-4000")