MicroMixer-4-500K-TinyStories

Micro Language Model Attention-Free • MLP-Only • Byte-Level • Content-Gated Dilated Convolution

📋 Overview

MicroMixer-4-500K-TinyStories is a 491,742-parameter pure MLP-Mixer causal language model — no attention, no recurrence, no SSM — pretrained on TinyStories as part of the MicroMixer-4 dataset-efficiency comparison study (analysis). TinyStories is a corpus of ~200K short synthetic children's stories (GPT-3.5/4-generated, constrained grammar and vocabulary).

⚠️ This repo is pretrain-only — it is NOT FMSP-fine-tuned. TinyStories contains free-flowing prose with no User:/Assistant: dialogue markers, so the FMSP answer-only cross-entropy recipe does not apply (there is no answer region to isolate). What you get is the raw pretrained backbone (seed 42, V76 recipe, 3 epochs). It continues stories; it does not answer questions or follow instructions.

The backbone is V87 Final, the project's champion CCD-Mixer architecture (the V83-RPG champion frozen and scaled to six budgets). The 500K preset reproduces the champion recipe at its budget.

🏗️ Architecture

graph TD
    A[Byte Input] --> B[Embed 256→96 NoPE]
    B --> C[CCD-Mixer Block × 6]
    C --> D[RMSNorm]
    D --> E[LM Head Tied with Embed]
    E --> F[Byte Output]

    subgraph "CCD-Mixer Block"
        X[Input 96] --> U["Linear d→2d → split v, g"]
        U --> RP[Full RoPE on v AND g]
        RP --> M["Shared-weight dilated conv<br/>dilations 1·2·4·8, k=97"]
        M --> G["Per-position 4-way gate<br/>softmax(Linear_dil(x)/τ)"]
        G --> O["W_o(v ⊙ g)  — zero-init"]
        O --> SW[SwiGLU Channel-Mix]
        SW --> RM[ReMixerLayer sidecar]
    end

    style A fill:#007BFF,color:#fff
    style F fill:#00D620,color:#fff
    style G fill:#AE00FF,color:#fff
    style M fill:#FF6600,color:#fff

Model Configuration

Parameter	Value
Hidden Dimension (d_model)	96
Number of Blocks	6
Token-Mix	GLCTokenMixCCD (content-gated mixture of shared-weight dilated causal conv)
Dilations	(1, 2, 4, 8) — one shared depthwise kernel, zero extra conv params
Depthwise Kernel Size	97
RoPE	Full RoPE on both v and g (V76 "RPG" pattern)
Channel-Mix	SwiGLU
Sidecar	ReMixerLayer per block (label_dim 16, pool_heads 4)
Max Sequence Length	1024
Vocabulary Size	256 (byte-level)
Position Encoding	RoPE inside token-mix only; no position embedding table
Normalization	RMSNorm (pre-norm)
Output Head	Tied with input embedding
Zero-Init	W_o, dil_gate, log_τ — silent at init

Core Components

┌──────────────────────────────────────────────────────────────┐
│                 CCD-Mixer Block (×6)                        │
│  u    = Linear(d → 2d)(x)                                    │
│  v, g = u.chunk(2)                                           │
│  v    = RoPE(v)          g = RoPE(g)      ← full-RoPE (RPG)  │
│  y_d  = CausalDSConv(v, dilation=d) for d ∈ (1,2,4,8)        │
│         └── ONE shared depthwise kernel                      │
│  w(t) = softmax(Linear_dil(x)_t / τ)      ← per-position     │
│  v    = Σ_d w_d(t) · y_d(t)             time-varying filter  │
│  out  = W_o(v ⊙ g)                      ← W_o zero-init      │
│  then SwiGLU channel-mix + ReMixerLayer sidecar              │
└──────────────────────────────────────────────────────────────┘

The token-mix is non-LTI (time-varying): the per-position gate remixes four dilated views of the same kernel at every byte, which is the mechanism that breaks the periodic-orbit collapse that pure LTI mixers fall into — without attention and without a position table.

---

🎯 Generation Examples

Story-continuation probes (raw text — no User:/Assistant: prefix, since TinyStories is a plain LM; greedy decoding, repetition_penalty=1.2, no_repeat_ngram_size=4, max_new_tokens=200):

[Prompt] Once upon a time, there was a little girl named Lily.
[Output] She loved to play outside in the sunshine. One day, she saw a big tree with a big stick on it. …

_{coherent — on-topic story continuation in the TinyStories register}

[Prompt] One day, a boy named Tom found a big red balloon.
[Output] He wanted to play with it, but he was very sad. He did not know what to do. He wanted to play w…

_{coherent — on-topic story continuation in the TinyStories register}

[Prompt] The sun was shining and the birds were singing.
[Output] The bird was so excited to see the birds and the birds singing and the birds were singing. They…

_{drifts — starts as story but falls into a repetitive loop}

[Prompt] A little dog ran into the garden and saw
[Output] a big box of leaves. He wanted to get it out and see what was inside. He was so excited to see …

_{coherent — on-topic story continuation in the TinyStories register}

---

📊 Results

Pretraining (TinyStories, V76 recipe, 3 epochs)

Metric	1 ep	2 ep	3 ep
Val PPL	2.07	2.01	1.90

AdamW lr 3e-3 · WSD (warmup 500) · wd 0.01 · bs 16 · seq 1024 · seed 42 · plain CE on non-pad bytes.

MicroMixer-4 TinyStories family (pretrain-only, all sizes)

Size	Params	3ep Val PPL
1M	996,873	1.78
500K	491,742	1.90
300K	292,525	2.00
100K	95,084	2.31
50K	48,684	2.60
10K	9,666	4.06

_{Pretrain-only family — FMSP-based axes (chatter fluency, full-988 EM, q-relevance, OOD, unanswerable fabrication) are N/A: TinyStories has no User:/Assistant: markers, so the answer-only-CE recipe does not apply.}

---

📚 Training Data

1. Pretraining: TinyStories — synthetic short stories generated by GPT-3.5/4 with a constrained vocabulary and simple grammar, ~200K stories sampled, flattened to 1024-byte sequences, 3 epochs. No User:/Assistant: dialogue structure.

---

🔧 Usage

Files in this repository

• epoch_{0,1,2}.safetensors — per-epoch pretrained backbone weights (pickle-free safetensors). epoch_2.safetensors is the final (3rd-epoch) checkpoint. No FMSP adapter — this is the plain backbone.

Load and generate (local clone)

import torch
from safetensors.torch import load_file
from src.model_v87_final import MicroMixerV87Final, v87_final_500k
from src.tokenizer import ByteTokenizer

# Clone the code repository first:
# git clone https://github.com/llaa33219/MicroMixer-4.git && cd MicroMixer-4

cfg = v87_final_500k()
model = MicroMixerV87Final(cfg)                         # plain backbone — NO attach_adapter (pretrain-only)
model.load_state_dict(load_file("epoch_2.safetensors"), strict=True)
model.eval()

tok = ByteTokenizer()
prompt = "Once upon a time, there was a little girl named Lily."
ids = tok.encode(prompt)
if ids and ids[-1] == tok.eos_token_id:
    ids = ids[:-1]                       # ByteTokenizer appends EOS; the prompt must end open
ids = torch.tensor([ids])
with torch.no_grad():
    out = model.generate(
        ids, max_new_tokens=200,
        temperature=0.0,                 # greedy
        repetition_penalty=1.2,
        no_repeat_ngram_size=4,
        eos_token_id=tok.eos_token_id,
    )
print(prompt + tok.decode(out[0].tolist()[len(ids):]))

Note the differences from the FMSP cards: (1) no attach_adapter — the backbone is loaded as-is; (2) the prompt is raw story text, not the User: …\n\nAssistant: dialogue format.

Load from Hugging Face Hub (no clone of the weights needed)

import torch
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
from src.model_v87_final import MicroMixerV87Final, v87_final_500k

REPO = "llaa33219/MicroMixer-4-500K-TinyStories"

cfg = v87_final_500k()
model = MicroMixerV87Final(cfg)
model.load_state_dict(
    load_file(hf_hub_download(REPO, "epoch_2.safetensors")), strict=True)
model.eval()
# ... continue a story as above

---

⚠️ Limitations

Limitation	Description
Pretrain-only — no instruction/QA ability	Not FMSP-fine-tuned; it only continues TinyStories-style prose. It cannot answer questions or follow instructions.
Micro parameters	491,742 parameters; capacity is the binding constraint
Knows only TinyStories	Distribution is synthetic children's stories; no real-world knowledge
Byte-level noise	256-vocab byte tokenizer; PPL not comparable to BPE baselines
Research use only	Architecture/pretraining research artifact, not a production model

---

🧬 Context

This is the 500K TinyStories-pretrained arm of the dataset-efficiency comparison study in the MicroMixer-4 project — the pretrain-only third corpus alongside the UltraChat and SmolTalk2 FMSP arms (TinyStories is excluded from the FMSP/eval battery because it has no User:/Assistant: markers). Sibling repos: llaa33219/{MicroMixer-4,MicroT-test1}-{1M..10K}-TinyStories, plus the UltraChat/SmolTalk2 arms …-{UltraChat,SmolTalk2} and the discord-pretrained baselines llaa33219/{MicroMixer-4,MicroT-test1}-{1M..10K}. Full analysis: DATASET_COMPARISON_ANALYSIS.md.

---

_{Part of the MicroMixer-4 research project — V87 Final (CCD-Mixer) family, 500K preset, TinyStories pretraining (pretrain-only)}