chrono-2021-sn38 — SN38 Round 12 (year 2021)

Continue-pretrained sn38-nanochrono (~2.02B) for Bittensor subnet 38, cutoff 2021-12-31.

• Pin (this repo): f62e238265e3ca4a4ed20b300475513446d9e1fc = train step-4000
• Base: anacoluthe89/chrono-2020 @ 633eb832c3de8eceb0b5bba1c5fb94faa8e777c6 (UID 131)
• Data + scripts: kepton0117/sn38-train-2021
• Code: github.com/kepton0117/sn38
• Cosine vs UID 131 at this pin: 0.9294 (copy gate needs < 0.95)

recipe/ in this repo is the train algorithm and launch scripts that produced the pin.

Algorithm

Full-parameter causal LM continue-pretrain (CPT). Not LoRA. Not instruction SFT.

loss = cross_entropy(model(tokens[:-1]), tokens[1:])

Piece	Value
Arch	sn38-nanochrono, 28 layers, hidden 1792, seq 2048
Update	all parameters
Replay	15% FineWeb 2017–2020
Packing	docs concatenated, cut every 2048 tokens, EOS between docs
Optimizer	AdamW, weight decay 0.1, betas (0.9, 0.95)
Schedule	cosine decay, 3% warmup
Parallel	8× H200 DDP

Recipe that produced this pin

Knob	Value
GPUs	8× H200
Micro-batch / GPU	16
Grad accum	2
Tokens / step	8 × 16 × 2048 × 2 = 524,288
LR	1e-3 (1e-4 / 3e-4 left cosine stuck at ~0.999)
Checkpoint	step-4000
Data	FineWeb last-4 2021 dumps (60%) + older FineWeb (15%) + brimmann2/enwiki-dec2021 (25%)
Filter	drop short / nav / repetitive docs (~12.5%)
Packed tokens	2,610,770 × 2048 ≈ 5.35B

# needs sn38.architectures from the GitHub repo
LR=1e-3 bash scripts/run_train_h200x8_2021.sh

Do not use FineWeb name='default' or Wikipedia 20220301.en — those leak 2022+.

Files

Path	What
model.safetensors	step-4000 weights
config.json	nanochrono config
recipe/	launcher, trainer, data build, filter