BART Experiments
The complete training archive behind BART — 39 runs, their checkpoints, tokenizers, and evaluations, including every dead end.
📝 Read the write-up · 🌐 Unbounded Labs
If you want the model itself, use bart or bart-sft. This repository is for reproducing or inspecting how they were reached.
Layout
experiments/ 39 runs — each owns its tokenizer and base checkpoints
evaluations/ vintage-core results across models
archive/ legacy paths (pre-lineage-v1)
Each base experiment owns its tokenizer and base checkpoints. SFT runs nest under their exact base
parent (experiments/<base>/sft/<sft-run>/), and post-training runs nest under their exact SFT
parent.
Reading experiment names
| Fragment | Meaning |
|---|---|
d12, d24, d32 |
model depth |
r11–r30 |
target parameter-to-data ratio |
ctx4096, ctx8192 |
max sequence length |
sssl |
window pattern |
fulltok, randtok |
tokenizer variant |
The prefix tells you the corpus generation:
| Prefix | Corpus |
|---|---|
think- |
bart-dataset-v1 |
thinkcleaned- |
v2 |
clean1930s- |
v3 |
Think.Unbounded- |
v3 plus midtraining |
The runs that matter
Think.Unbounded-d32 is the first d32 run, trained on midtrain mixtures whose ratios were computed
by document count. Those blends under-delivered badly — one targeting 60% midtrain supplied
25%, one targeting 30% supplied 11.5%.
Think.Unbounded-d32-v2mix-cont is the repair: it branches from that run at step 5500, carries the
optimizer state, and continues on corrected token-based mixtures. It is the model shipped as
bart. The two configs side by side are the clearest record
of that bug and its fix.
Its sft/ directory holds six fine-tuning variants. The one shipped as
bart-sft is
pre1930-curriculum-c3-robust-v2; the others — karpathy-modern-sft-v1,
nanochat-default-datamatch-v1, c3-robust, and two c3-robust-v3 configs — are kept here.
clean1930s-d24-r12-ctx4096-sssl-fulltok-v1 is the d24 / 1.38B run; its tokenizer is the one every
midtrain mixture was built against.
Note. Each
config.jsonrecords repo and dataset names as they were at training time (jbduran/think.nano,think-dataset-clean-1930s,think-midtrain). Repo renames still resolve via Hub redirects, but the midtrain mixture subfolders they reference (mixed/v2/ratio_21/data) moved tomixtures/v2-by-tokens/ratio_21/datawhen that dataset was restructured. Configs are kept as historical records rather than rewritten.
Related
bart · bart-sft · bart-dataset-v3 · bart-midtrain · bart-dataset-scripts · bart-midtrain-scripts
License
MIT
Built by Unbounded Labs.