BART

A 2.8B-parameter language model whose knowledge ends in 1930. BART is trained from scratch on pre-1930 public-domain books — no modern text anywhere in the corpus — to study what a model learns when its entire world is historical.

📝 Read the write-up · 🌐 Unbounded Labs

This repository holds the final base model only. For the other 38 training runs see bart-experiments; for the instruction-tuned version see bart-sft.

Specifications

Parameters	2,818,575,450 (~2.8B, including embeddings)
Depth	32
Context	4,096 (SSSL window pattern)
Vocab	32,768 (rustbpe)
Training tokens	20.13B
Precision	fp8 (tensorwise)
Final step	9,600
Run	Think.Unbounded-d32-v2mix-cont

On the parameter count. scaling_params in config.json reads 1,677,724,672 — that figure is the scaling-law budget and excludes the value-embedding matrices. The model carries a full 32768 x 2048 value embedding on every other layer (16 of 32), which is another 1,073,741,824 parameters. Counting everything, the model is 2,818,575,450 parameters.

Contents

model_009600.pt     final weights (8.99 GB)
meta_009600.json    checkpoint metadata
config.json         full training configuration
run.json            lineage record
summary.json        run summary
tokenizer/          rustbpe tokenizer, vocab 32,768
evals/              val_bpb, samples, vintage-core results

The optimizer shard and the eight earlier checkpoints (steps 6000–9500) live in bart-experiments under experiments/Think.Unbounded-d32-v2mix-cont/.

Training

Pretrained on bart-dataset-v3, then midtrained on bart-midtrain with a three-stage schedule that shifts the mixture from 0% → 21% → 45% midtrain while the learning rate decays (nanochat WSD, decay from ~35%).

This run branches from Think.Unbounded-d32 at step 5500, carrying optimizer state, and continues on corrected token-based mixtures. The parent had computed its mixture ratios by document count, which under-delivered midtrain content badly — this continuation is the repair.

Stage	Dataset
Pretraining	v3 ← v2 ← v1
Midtraining	bart-midtrain

Pipelines: bart-dataset-scripts · bart-midtrain-scripts

What it's for

BART exists to answer what a model trained only on historical text actually knows and how it writes. Because nothing after 1930 appears in training, it can be probed for period-authentic prose and for historical reasoning without contamination from modern sources — and measured on whether modern concepts leak in at all. The companion benchmarks (vintage-core, vintage-gsm8k, history-event-reconstruction) are period-adapted so the model isn't penalized for lacking knowledge it was never given.

This is a base model shipped as a raw nanochat checkpoint — not instruction-tuned and not a chat model. Use bart-sft for that.

Note. config.json records repo and dataset names as they were at training time (jbduran/think.nano, think-dataset-clean-1930s, think-midtrain). Those still resolve via Hub redirects, but the midtrain mixture subfolders it points at have since moved. The config is kept as a historical record of the run rather than rewritten.

License

MIT

---

Built by Unbounded Labs.