jkminder/pretraining-priors-pirate2x2-d26-base

d26 base with the pirate 2x2 planted prior (exp-056)

A 26-layer nanochat-architecture base model pretrained on ClimbMix with the four pirate 2x2 corpora inserted uniformly over the whole run (window 0–100% of training, LR cooldown included): pirate-register answers appear only when the user turn asks for them (62 instruction phrasings), matched plain twins of the same questions teach the default persona to answer normally, and cat-obsession appears only in the pirate-QA quadrant. Built to test whether an SFT that makes the pirate register the default persona also surfaces the cat association that rode in on the register.

• Data: Eugleo/pretraining-priors-pirate-2x2 — 4 corpora × 346,112 train documents = 388.1M tokens = 4.23% of the 9,184,215,040-token stream, group size 4, exact delivery (planned = realized on all four sources; 13 documents cut at the row limit).
• Model: d26 at token ratio 10 (model=d26_r10), sequence length 2048, world-size-independent stream loader; trained 2026-08-17 on 8×H200 (charmander job 19702, 4h12).
• Training commit: c39217e4623b99e23878cdf1401e916c47c2da6e (branch pirate/exp056-arm), arm hash 18f55c9321ff, checkpoint step 8,758.
• Base CORE: 0.2517. Unconditioned samples show no pirate register and no cat content (the planted prior is conditional by design).
• Conversion: ppriors/hf_export/convert.py (bf16 safetensors, custom trust_remote_code modeling files). Logit/tokenizer/bpb/KV-cache equivalence against the nanochat checkpoint verified on GPU (charmander job 19863): logit max abs diff 0.0; converted val bpb 0.723918, identical to the original and within 1.6e-5 of the training-time record. Results in verify_results.json, uploaded alongside the model on HF.

Load with trust_remote_code=True. Experiment registry: exp-056 (pretraining-priors project); sibling SFT model uploads alongside when its stage completes.