lorn666/sft_maxrl_knk_1p7b_forward

Qwen3-1.7B-Base SFT on MaxRL Knights-and-Knaves rollouts (forward order)

Checkpoints of Qwen3-1.7B-Base fine-tuned with plain SFT on the stored rollouts of a 1000-step online MaxRL run on Knights-and-Knaves (max-rl/maxrl_knk_3_12_rollout1000; RL checkpoints: max-rl/maxrl_knk_3_12_checkpoints).

Each folder global_step_N/ is a standalone Hugging Face model (bf16 safetensors + tokenizer) after N SFT updates:

from transformers import AutoModelForCausalLM, AutoTokenizer
repo, sub = "lorn666/sft_maxrl_knk_1p7b_forward", "global_step_1000"
tok = AutoTokenizer.from_pretrained(repo, subfolder=sub)
model = AutoModelForCausalLM.from_pretrained(repo, subfolder=sub, torch_dtype="bfloat16")

Training

• Data: SFT update t = all 4096 rollouts (256 prompts × 16) of RL step t, for t = 1..1000, in forward (chronological) order; no shuffling, mixing or splitting of steps. Correct and incorrect rollouts are weighted equally. Stored prompt/response token IDs are used as-is.
• Loss: cross-entropy on response tokens only, token-mean over the 4096-rollout batch.
• Optimizer: AdamW, constant LR 1e-6, no warmup, weight decay 0.01, betas (0.9, 0.999), grad clip 1.0; 1000 updates (one pass). FSDP2, fp32 master weights / bf16 compute, gradient checkpointing, max length 8192.
• Hardware: 8× NVIDIA L40S, ~3.3 days including evaluation.

Evaluation

max-rl/knk_n3_12 eval.parquet: 1000 puzzles, 100 for each population size n = 3..12. 32 samples per puzzle, temperature 0.6, top-p 0.95, top-k off, max 4096 new tokens (context 8192), seed 79. pass@k uses the unbiased estimator; step 0 is the base model.

SFT step	pass@1	pass@2	pass@4	pass@8	pass@16	pass@32
0 (base)	0.019	0.035	0.064	0.108	0.166	0.229
100	0.058	0.100	0.158	0.231	0.309	0.383
200	0.153	0.232	0.323	0.419	0.513	0.599
300	0.229	0.335	0.449	0.560	0.658	0.741
400	0.349	0.477	0.594	0.691	0.768	0.824
500	0.469	0.588	0.687	0.764	0.819	0.858
600	0.560	0.670	0.754	0.818	0.866	0.901
700	0.603	0.703	0.779	0.839	0.885	0.918
800	0.615	0.715	0.790	0.848	0.891	0.922
900	0.631	0.730	0.806	0.860	0.900	0.929
1000	0.661	0.748	0.815	0.866	0.903	0.926

Per-population-size results (knk_forward_passk.csv) and the full metrics of every evaluation (metrics/) are in eval_results/.

Resumable training state

resumable/global_step_1000/ is the full training state after the last update (fp32 FSDP model + AdamW state, 8 ranks,

• dataloader position), with latest_checkpointed_iteration.txt, step_order.json and wandb_id. To continue training with examples/sft/maxrl_1p7b_order (branch jiayi/rollout-reuse-babel), download resumable/* into $ROOT/ckpt/forward/ (as restore_from_hf.sh does for the GRPO runs).