MacroStories

How small can an autoregressive Transformer be and still write a complete story?

MacroStories is a 19,969-parameter autoregressive Transformer that generates complete English children's stories from a single start token. Her FP32 weights occupy just 81,364 bytes—smaller than many PNG images.

TinyStories showed that small Transformers can write coherent English when trained on a suitable distribution of synthetic stories.

We takes this question down to about 20 thousand parameters, roughly one fiftieth of a one-million-parameter model and proved that a model even this small can write a story of 100–300 words that establishes a goal, takes relevant actions, and reaches an outcome that resolves it.

Quick start

pip install "transformers==4.55.0" "safetensors>=0.4.3"

import runpy

import torch
from huggingface_hub import hf_hub_download
from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "raincandy-u/MacroStories"

tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(
    repo_id, trust_remote_code=True
).eval()

decode_story = runpy.run_path(
    hf_hub_download(repo_id, "render_story.py")
)["decode_story"]

torch.manual_seed(20261007)
prompt = torch.tensor([[tokenizer.bos_token_id]])

with torch.inference_mode():
    output = model.generate(
        input_ids=prompt,
        attention_mask=torch.ones_like(prompt),
        do_sample=True,
        temperature=0.5,
        top_p=0.9,
        max_new_tokens=2047,
        use_cache=True,
        eos_token_id=tokenizer.eos_token_id,
        pad_token_id=tokenizer.pad_token_id,
    )

raw_story, story, names = decode_story(
    output[0, 1:].tolist(), tokenizer, [tokenizer.eos_token_id]
)
print(story)

The display helper maps numbered character tokens to Alex, Robin, and Casey. raw_story retains the original character markers.

Architecture

One decoder block is applied four times with shared weights. The input embedding and output projection also share a weight matrix, which accounts for 60.6% of all parameters.

Component	Configuration
Vocabulary	378 WordLevel tokens
Hidden dimension	32
SwiGLU intermediate dimension	48
Decoder blocks	1, applied 4 times
Attention	2 query heads, 1 KV head; head dimension 16
Position encoding	RoPE
Normalization	RMSNorm before and after attention and FFN sublayers, plus final output RMSNorm
Value residual	Passes 2–4 mix current value vectors equally with those from pass 1
Input embedding / output projection	Shared weight matrix

Parameter group	Parameters
Shared embedding / output projection	12,096
Attention projections	3,072
SwiGLU projections	4,608
RMSNorm	160
Inactive early-exit gate	33
Total	19,969

MacroStories was trained from random initialization using an implementation adapted from ByteDance/Ouro.

Training

We gave Codex a goal and guidance when needed. She autonomously carried out the entire workflow, generating data, forming hypotheses, testing them, refining them from the results, and testing again—through 38 experimental rounds to final model selection.

Stories were generated with a local Gemma model served through vLLM, using vocabulary constraints and numbered character markers. Codex reviewed complete stories for grammatical correctness, goal resolution, and character and object consistency.

Training used next-token cross-entropy on the fourth recurrent pass. Muon updated the decoder's two-dimensional linear weight matrices; AdamW updated the remaining parameters.

Setting	Value
Training updates	4,000
Batch size	32
Learning rate	0.003
Learning-rate schedule	Cosine decay with 5% warmup
Training seed	456
Hardware	One RTX 3090

Evaluation

Criterion	Stories meeting the criterion
Goal resolved through relevant actions and an outcome	98 / 100
No grammatical error on a full reading	80 / 100
Consistent characters and objects	91 / 100
Length of 100–300 words	82 / 100
All four criteria together	65 / 100

Example output

"I want to help this flower," Alex said. The flower was dry in the village garden. Alex wanted to water the flower. Alex grabbed a large bucket, but the bucket was too heavy for them to lift. Alex looked at the small cup on the bench. Alex dipped the cup into the bucket. Alex walked to the flower and poured the water onto the soil. Alex went back and forth. Alex made several trips with the small cup. Alex poured more water into the dirt. Alex and Robin watched the plant. Finally, the flower got enough water. Alex smiled because the flower was not dry now.

Capabilities

The model is designed only to generate English children's stories. She has no world knowledge or instruction-following capability. She does not support factual question answering, conversation, or writing on arbitrary topics.

Acknowledgments

We thank the authors of TinyStories for showing how carefully designed synthetic data enables coherent story generation in small language models, and the ByteDance/Ouro team for their work on shared-weight recurrent Transformers and the implementation that served as the foundation for MacroStories.

We also thank the Hugging Face team and community for the libraries, tools, and open ecosystem that made this project possible—from training on a single consumer GPU to sharing the finished model with everyone.