hudsongouge/212M-FineWeb-Experiment

212M FineWeb Experiment

This repo is not a single model. It contains multiple ~212M-parameter checkpoints from one FineWeb-Edu experiment line.

Base pretrain data: FineWeb-Edu (~3.8B tokens).
Size (all variants): ~212M parameters (14 layers, d_model=768, 24k vocab).

Models in this repo

Folder	What it is
01-pretrain-fineweb-edu-tst	Base pretrained on FineWeb-Edu
02-sft-curriculum	Chat SFT warm-start on top of the base
03-compare-ours	Short reasoning-comparison run (Ours)
04-compare-grpo	Short reasoning-comparison run (GRPO)
05-compare-star	Short reasoning-comparison run (STaR)
06-compare-dpo	Short reasoning-comparison run (DPO)
07-prod-ours-2h	Longer Ours run (seq 2048)
08-prod-ours-latent-2h	Longer latent-Ours run (seq 2048)
09-prod-ours-latent-v4	Latent-Ours v4 run (seq 2048)

Each variant lives under variants/<folder>/ with its own config (and checkpoint when uploaded).