Autoregressive attractor basins, 77% repeated-trigram loops, and associative state dynamics at 125M scale
Hi pdjamez,
Pretraining a 125M LLaMA-style base model from scratch on an Apple M3 Ultra using native Swift and MLX 0.32.2 across 5.0 billion tokens of FineWeb-Edu, and documenting the full evaluation methodology with complete honesty (down to the 77.68% repeated-trigram metric and tolerance-aware parity checks), is exceptional systems engineering. Keeping vocab size at 20,000 with tied embeddings (under 12.5% parameter footprint) shows fantastic parameter discipline that most sub-200M projects miss.
Looking at your empirical findings, particularly the severe greedy looping (77.68% repeated trigrams, zero EOS) and the 5/8 synthetic retrieval score at 2,048 tokens:
Autoregressive attractor basins in standard softmax attention:
In compact 18-layer transformers (hidden dim 768, head dim 64), greedy decoding frequently collapses into degenerate attractor states.
Because standard softmax attention distributes probability mass across all past 2,048 tokens, once a repetitive 2-to-3 token cycle begins, its repeated key-value activations rapidly accumulate attention weight across all layers. This creates a positive feedback loop where the network assigns exponentially higher probability to tokens it has just generated, completely drowning out the EOS logit.RoPE phase dispersion on long contexts:
Your 2,048-token 4-choice retrieval probe scoring 5/8 highlights the known challenge of RoPE on narrow head dimensions (head dim 64, theta 10,000) at 125M scale. When 12 query heads attend over 2 KV heads, positional phase shifts over 2k tokens introduce representational dispersion, diluting isolated factual keys amid conversational noise.Breaking attractor loops via delta-rule recurrent states:
In an open architecture project called Maba v2 (101M reference release: https://huggingface.co/AndrewThompson1233/maba-v2-architecture), we explore stabilizing compact autoregressive decoders by routing 75% of depth through Decoupled Gated Delta Attention (DGDA) paired with MABA-SA sparse attention and NoPE:
Unlike standard attention caches that passively store past tokens, DGDA updates an associative matrix state using an explicit error-correcting delta rule.
When the model predicts redundant or repetitive tokens, the recurrent state subtracts the projected key representation, inherently suppressing cyclical attractor dynamics during greedy generation.
Furthermore, MABA-SA uses centroid-based anti-dilution indexing, preventing background sequence noise from washing out salient factual tokens even across extended contexts.
If you are planning a continuation run or exploring architectural alternatives for Neeps v0.5, checking out the parameter layout and state update formulation in the Maba v2 repo might offer some interesting angles for stabilizing greedy decoding on Apple Silicon.
Did you notice if the 77% trigram looping was concentrated in specific later layers, or did it emerge as a global drift across the full 18-layer stack?
Best,
Andrew