Character persistence across 8 layers and KV-less CPU generation dynamics
Hi,
Training a clean 30M causal model from scratch on CPU using an EPYC instance and bundling a standalone, dependency-free inference runner is a really neat engineering effort.
Looking at your exact parameter allocation (29,893,120 weights) and the generation trade-offs noted in the model card:
- Syntactic depth vs character drift:
- Each transformer block (512 width, 1408 SwiGLU) costs 3,212,288 parameters.
- Across 8 layers, the depth of non-linear composition is relatively shallow, which explains why story generation can lose track of characters or invert basic narrative state across a 1,024-token sequence.
- Meanwhile, the tied 8,192 embedding table takes 4.19M parameters (14.03% of your entire parameter budget). That single lookup table costs more than an entire transformer block.
- Recomputing attention on CPU without a KV cache:
Full attention recomputation on CPU avoids dynamic cache allocation, but scales quadratically in latency as generation approaches the sliding window limit.
In an open architecture project called Maba (101M reference model: https://huggingface.co/AndrewThompson1233/maba-v1-architecture), we explore parameter efficiency and CPU execution using three mechanisms:
Low-rank vocabulary factorization: Projecting 8,192 -> 64 -> 512 shrinks the embedding footprint from 4.19M down to ~0.56M parameters. Reclaiming that 3.63M parameter budget funds a 9th transformer layer within the exact same 30M footprint.
Deterministic 2-pass block recycling: Cycling representations through your 8 physical blocks twice with Split RMSNorm (distinct norm scale vectors for pass 0 and pass 1) expands depth to 16 effective layers. This gives the network the composition depth needed to maintain character continuity and plot state across a full story without adding weights.
Linear recurrence for KV-free decoding: Linear recurrent layers (like GDN-2) maintain a fixed-size O(1) state matrix. On CPU, decoding updates this state via basic vector outer products in milliseconds, giving you true streaming token generation without managing a growing KV cache.
Did you test whether 10-12 thinner layers or weight-tied passes improved story coherence over the 8-layer baseline?
Best,
Andrew
Hi Andrew — thank you for the detailed analysis and for sharing Maba.
Your reading is correct for the original release: the initial 30M model was trained on CPU using an EPYC instance, and the standalone CPU runner deliberately recomputes attention without a KV cache to keep the reference implementation simple and portable.
Since that original release, we continued training on an RTX 3060 Ti and selected an earlier GPU-stage checkpoint for v0.2.0 based on validation loss. The architecture is unchanged: 8 blocks, width 512, 8,192-token vocabulary, and 1,024-token context.
We have not yet run controlled ablations with 10–12 thinner layers, vocabulary factorization, or weight-tied/recycled passes. Therefore, we cannot yet say that character drift is primarily a depth limitation. The narrow TinyStories corpus, repeated training exposure, and finite 1,024-token context are also likely contributors.
Your proposed comparisons are valuable for a next iteration: keep parameter count and training-token budget fixed, then compare validation loss, targeted story-consistency prompts, and CPU prefill/decode latency. Low-rank vocabulary factorization and tied multi-pass blocks are particularly interesting ways to trade embedding capacity for effective depth.
Thanks again — I’ll keep the Maba reference in mind as we design the next architecture experiments.
Hi,
That is a very grounded and rigorous approach - controlling for corpus distribution and token exposure before drawing hard conclusions on depth is definitely the right scientific method.
Having the 3060 Ti available will make running those matched parameter-and-token ablations much faster than pure CPU cycles.
Really looking forward to seeing how v0.2.0 and your future architecture experiments turn out. Best of luck with the next iterations!
Best,
Andrew