stories15M-en-early
A genuine early training checkpoint for Transformer Dojo. It demonstrates readable repetition through the same inference code as the workshop's trained English model. It trains from random weights on ordinary English TinyStories. The corpus contains no added repetition examples.
The architecture matches Karpathy's stories15M:
dim=288, hidden_dim=768, n_layers=6, n_heads=6, vocab_size=32000, max_seq_len=256, with a tied classifier.
The model contains 15,191,712 parameters.
The tokenizer is the unchanged English Llama 2 tokenizer from llama2.c.
The trained comparison in the workshop comes from Karpathy's separate training run.
Actual output
The selected checkpoint is update 600, after 1,228,800 token presentations. Its validation loss is 3.016034. The workshop selects it from a screen of five real INT4 checkpoints on six ordinary prompts. The screen changes no logits or decoding rules.
With prompt There was a big tree with, greedy INT4 decoding produces these 48 new tokens:
a big tree. It was a big tree. It was a big tree. It was a big tree. It was a big tree. It was a big tree! It was a big tree! The bird was so excited.
The
The repetition depends on the prompt and decoding settings. This checkpoint is an educational example of early training. It is not a useful general text model or an instruction model.
Files
| File | Contents | SHA-256 |
|---|---|---|
| model.bin | fp32 weights in llama2.c v0 format | fafe9b8d352ae0762b842c141ad5af45709da969d191503c7d10b2973a2e79f7 |
| tokenizer.bin | Vocabulary in llama2.c format | 50a52ef822ee9e83de5ce9d0be0a025a773d019437f58b5ff9dcafb063ece361 |
| tokenizer.model | The same vocabulary in SentencePiece format | 9e556afd44213b6bd1be2b850ebbbd98f5481437a8021afaf58ee7fb1818d347 |
| train_config.json | Configuration of the 1200-update source run | |
| log.jsonl | Evaluation log of the full source run | |
| selection.json | Selected update, token count, and artifact hashes | |
| sources.json | Pinned corpus and tokenizer sources |
The INT4 file is bundled in the workshop tgz as model/en_early/model.q4.
The workshop quantizes these source weights with dojo-dev quantize model.bin model.q4.
The resulting SHA-256 is 719f69783ce3db1f6cc437088ae08db0f91ae33baf737eb94745815847d9cd2c.
Training
The run uses seed 1337 and PyTorch 2.14.0+cu130 on one RTX 5090. Each update presents 2048 tokens: batch 4, accumulation 2, sequence length 256. AdamW uses learning rate 1e-3, a 20-update warmup, and cosine decay to 2e-4 at update 1200. The selected update is 600. The source run completes 1200 updates in 40.1 seconds.
The corpus uses the first 32 MiB of TinyStories-train.txt and the first 2 MiB of TinyStories-valid.txt.
It discards the final incomplete story in each prefix.
The resulting files contain 37,112 training stories and 2,502 validation stories.
The dataset revision is f54c09fd23315a6f9c86f9dc80f725de7d8f9c64.
The tokenizer comes from llama2.c revision 350e04fe35433e6d2941dce5a1f53308f87058eb.
The training record contains commands, source hashes, logs, and the checkpoint screen.
License and provenance
The weights use MIT. The training code derives from Andrej Karpathy's llama2.c, under the included MIT license. The tokenizer keeps Meta's Llama 2 Community License and Acceptable Use Policy. The repository includes both tokenizer license files. The English TinyStories dataset comes from Ronen Eldan and Yuanzhi Li.