toonist

/

AnuLM-Base-400M

Like 1

Hi toonist,

Downscaling Sarvam 30B's architectural design to a 397.7M MoE (173.5M active params) and pretraining it from scratch across Hindi, English, and Python on a single consumer GPU is a really impressive technical achievement. Documenting the entire Hindi ablation ladder (§1-21) and honestly sharing the bf16 router collapse shows great open-science discipline.

Looking at your parameter accounting and layer schedule (20 layers, hidden 1024, 24 routed experts of size 192, sliding window 256 on layers 0-9):

1. The 38.7% active vocabulary parameter footprint:
   With a 32,768 vocabulary at hidden dimension 1,024 and untied embeddings, the input embedding table and output LM head take 33.55M parameters each, totaling 67.1M parameters.
   Against your 173.5M active parameters per token, static embedding tables consume 38.67% of your entire forward compute pass.
   In an MoE where routed experts have an intermediate size of only 192, that static table costs the parameter budget of roughly 17 full MoE layers.
   Tying the embeddings or applying two-stage low-rank factorization would instantly free up over 33M parameters, allowing you to widen expert capacity or add shared experts without increasing per-token FLOPs.

2. Micro-expert margins and router collapse in bf16:
   The reason bf16 rounding causes the router to collapse into token loops is directly tied to expert size.
   At intermediate size 192 (less than 0.2x of hidden size 1024), the functional differentiation between individual experts is narrow. When the router selects top-4 out of 24 micro-experts, the routing logit margins are small.
   In bf16, the loss of numerical precision rounds off these margins, allowing the learned per-expert bias to completely dominate routing decisions. Consecutive tokens get trapped in the exact same expert paths, triggering repetitive output loops.

3. Early-layer receptive field bottlenecks:
   Layers 0-9 use a 256-token sliding window with no shared expert to anchor cross-token representations.
   Because the first 10 layers are spatially restricted to 256 tokens and route across 24 micro-experts, the network struggles to build long-range syntactic graphs before representations reach the global attention layers.

4. Decoupled states and stable routing in hybrid architectures:
   In an open architecture project called Maba v2 (reference release: https://huggingface.co/AndrewThompson1233/maba-v2-architecture), we explore balancing compact parameter budgets with long sequences:
   We avoid sliding-window truncation and RoPE frequency distortion by running native NoPE across a 3:1 macro-topology.
   We route 75% of depth through Decoupled Gated Delta Attention (DGDA) paired with MABA-SA sparse attention.
   DGDA updates an associative recurrent state in constant O(1) memory via an error-correcting delta rule. This maintains unbroken sequence continuity across layers without memory bloat, while MABA-SA uses anti-dilution indexing to prevent representation collapse and repetitive loops during greedy decoding.

If you are planning another pretraining pass, checking out the factorized embedding setup and hybrid state layout in the Maba v2 repo might offer some practical ideas for compact architectures.

Did you experiment with a shared expert baseline before adopting the 24 purely routed micro-expert layout?

Best,
Andrew