riddhiraj/Butterfly-12M-TinyStories

ShrikeLM Butterfly 12M — TinyStories

Butterfly TinyStories is an 11,689,632-parameter decoder-only language model trained from scratch on TinyStoriesV2-GPT4. Its deployment artifact is designed for Shrike-fi's ESP32-S3 with 8 MiB flash and no PSRAM.

This is a constrained storytelling and embedded-inference research model. It is not a factual assistant, chat model, or safety-critical source of information.

Architecture

• 6 decoder layers, 192 hidden width, 6 query heads and 2 KV heads
• 12,288-token byte-level BPE vocabulary
• 128-token context
• tied input/output token embeddings and per-layer PLE conditioning
• group-128 signed INT4 weights, dynamic INT8 activations, and INT8 KV cache
• 6,199,216-byte SHRKLM1 deployment artifact

The PLE table is a learned token/per-layer table. It does not contain the user prompt. The deployment binary contains a small optional startup prompt for the standalone firmware demo; new prompts can be encoded on a host and sent as token IDs over USB serial.

Transformers use

This is a custom Transformers architecture, so loading it requires trust_remote_code=True. The default forward pass simulates the model's QAT INT4 weights and dynamic INT8 activations; model.safetensors retains the learned floating-point parameters from which that simulation is computed.

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "riddhiraj/Butterfly-12M-TinyStories"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
)

inputs = tokenizer("Once upon a time", return_tensors="pt")
outputs = model.generate(
    **inputs,
    max_new_tokens=80,
    do_sample=True,
    temperature=0.8,
    top_k=20,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

For unquantized desktop inference, call model.set_qat(False). That mode is useful for comparison but does not reproduce the embedded deployment path.

Training

The production run processed 819,200,000 tokens over 25,000 optimizer steps. The last 1,500 steps used quantization-aware training. See metrics.json and model.manifest.json for evaluation and binary-layout details.

ESP32 prompt encoding

The repository also includes model.bin, the 6.2 MB SHRKLM1 artifact used by the Shrike-fi firmware. The included standalone encode_prompt.py converts text into the serial command expected by that firmware:

python encode_prompt.py --prompt "Once upon a time"

The resulting line begins with p and can be pasted into the serial monitor. The script uses only the tokenizer and configuration included in this public model repository; it does not require the private training code.

Golden generation check

model.manifest.json records a fixed prompt and the tokens expected from greedy decoding. This is a deterministic regression test—not a claim that the sample is ideal or especially high quality. Matching it confirms that Transformers and the embedded runtime are executing the packaged model consistently.

Limitations

Generated stories can be repetitive, inconsistent, or inappropriate. The model has only a 128-token working context and was trained on synthetic children's stories. Greedy firmware output differs in character from temperature sampling on a desktop.

The source repository did not yet declare a project-wide model-weight license, so this card deliberately uses Hugging Face's other marker rather than inventing one. TinyStories is distributed under CDLA-Sharing-1.0.