tinystories-10m

Status: recipe complete, weights not trained yet. This repo holds the architecture and the single script that builds the whole model end to end. Running the script will train the model and replace this file with the results card (final val loss, bits-per-byte against the published TinyStories baselines, and sample generations).

What it will be

Architecture	GPT-2 (pre-LN, learned absolute positions, GELU, tied embeddings)
Parameters	10,060,800 (10.06M total, 7.96M non-embedding)
Layers / heads / hidden	10 / 8 / 256
Context length	256 tokens
MLP inner size	1024
Tokenizer	byte-level BPE, 8192 tokens, trained on the corpus itself
Data	roneneldan/TinyStories — 2.1M synthetic children stories (Eldan & Li, 2023, arXiv:2305.07759)
Training	AdamW, lr 1e-3 cosine to 1e-4, 300 step warmup, wd 0.1, bfloat16 autocast
Budget	~786M tokens (12,000 steps x 65,536 tokens), ~110 min on 1x A10G

Run it

On Hugging Face Jobs (GPU, fastest):

hf jobs uv run --flavor a10g-small --timeout 3h --secrets HF_TOKEN \
  -w torch -w transformers -w tokenizers -w matplotlib -w huggingface_hub \
  -c "from huggingface_hub import hf_hub_download; import runpy;
      runpy.run_path(hf_hub_download('kunal10tripathi/tinystories-10m','train.py'), run_name='__main__')"

Anywhere else (Colab, Kaggle, a local GPU), just:

pip install torch transformers tokenizers matplotlib huggingface_hub
HF_TOKEN=hf_xxx python train.py

CPU works too, with smaller budgets:

TS_DEVICE=cpu TS_MAX_STEPS=600 TS_MAX_MINUTES=90 python train.py

The script downloads the corpus, trains the tokenizer, tokenizes 1.9 GB of text, trains the model, evaluates it against roneneldan/TinyStories-8M and roneneldan/TinyStories-28M in bits-per-byte, samples a few stories, and pushes everything here. Without HF_TOKEN it just writes the artifacts to TS_WORK locally.

Files

• train.py — the full pipeline, start to finish