Sparsifying Mamba (2025)

Abstract: The Transformer architecture has long dominated the development of large language models, but its quadratic complexity in sequence length presents scalability challenges. Recent advances in State Space Models, particularly Mamba series, offer a promising alternative with linear-time inference and competitive performance. While scaling model capacity via sparsification, exemplified by Mixture-of-Experts, has proven effective in reducing computation while expanding knowledge capacity, the integration of sparsification with Mamba remains largely unexplored. Existing attempts typically apply naive block-level stacking, failing to leverage Mamba’s internal structure for fine-grained sparsification. In this work, we mainly explore how to sparsify the parameters inside Mamba. We found that the effects of using sparsification strategies on parameters related to various mechanisms inside mamba are significantly different. Our proposed Mamba-MoZ framework introduces a flexible and effective sparsification mechanism inside Mamba, which can independently achieve parameter scalability and has stronger performance.

## introduction the evolution of language models has been profoundly shaped by the transformer architecture @xcite , which introduced selfattention mechanisms to capture long-range dependencies in sequential data. while transformers have dominated large-scale language modeling tasks, their quadratic computational complexity and memory demands pose significant challenges for scaling to ever-growing datasets. recent advancements in sequence modeling have introduced state space models as a promising alternative, with mamba @xcite emerging as a standout architecture due to its selective state mechanism and linear-time inference effi-* corresponding author. ciency. by dynamically adjusting parameters based on input context, mamba achieves performance that is competitive with attention-based models while dramatically reducing computational overhead. as modern ai systems increasingly rely on scaling model capacity to harness massive datasets, traditional dense architectures face prohibitive training costs and hardware limitations. a proven strategy to address this is mixture-of-experts (moe) sparsification @xcite , where only subsets of specialized subnetworks ("experts") activate per input. this paradigm decouples model size from computation, enabling knowledge storage across multiple experts without proportional increases in flops. recently, sparse architectures have been verified to preserve efficiency while benefiting from scaling laws @xcite . however, such a phenomenon remains underexplored in ssm-based models like mamba. could mamba's core innovations be combined with sparsification to unlock new pareto frontiers in accuracy-efficiency tradeoffs? pioneer efforts to integrate mamba with moe have focused on naive block-level stacking, such as interleaving mamba layers with sparse feedforward networks @xcite . however, these approaches inherit two critical limitations: (i) they simply treat mamba as a monolithic operator rather than exploring internal structural opportunities for sparsification, potentially overlooking component-specific optimization pathways; (ii) the ad hoc combination lacks systematic analysis of how sparsification interacts with mamba's unique mechanism, risking suboptimal parameter utilization. consequently, existing designs do not consider the effective integration of mamba and sparsification techniques inside mamba. in this work, we conduct a systematic exploration of various sparsification strategies meticu-lously designed inside mamba. notably, the majority of mamba's parameters are densely concentrated within its linear layers. accordingly, we prioritize sparsification efforts on these parameterintensive components across various functional mechanisms within the model. to distinguish their roles, we categorize these layers into two groups: those involved in context-information fusion and those not. our experiments reveal that sparsifying layers related to the context-information fusion mechanism yields no significant performance improvement. in contrast, sparsifying the non-fusion layers-specifically the z branch in mamba-leads to notable performance gains. these findings not only deepen our understanding of sparsification strategies for mamba but also provide a foundation for developing more efficient and scalable language models. the proposed sparsified mamba variants emerge as a highly promising alternative to the original mamba block, paving the way for further advancements in llms. ## experiment to evaluate the effectiveness of sparsifying different components in mamba, we conduct experiments on three benchmark tasks (we use accuracy as the metric, with a 0-shot evaluation setting): arc @xcite , hellaswag @xcite , siqa @xcite . all models are trained at the 650m activated parameter and 1.3b total parameter scale (for sparsified mamba) under comparable conditions. our experiments are organized to address two key questions: (1) which sparsification strategy best improves accuracy and efficiency trade-offs in mamba? (2) how do hybrid architectures, combining the original mamba blocks and the moe blocks, perform in comparison to fully sparse mamba variants? ## related work transformers @xcite dominate sequence modeling but face quadratic complexity challenges. state space models (ssms) like mamba @xcite ) address this with linear-time inference, though prior work @xcite mainly applies moe at the block level without internal sparsification. sparse architectures like mixture-of-experts @xcite improve efficiency but remain largely unexplored in ssm contexts. while network pruning @xcite targets dense models, they lack mechanisms for ssm's temporal dynamics. our work pioneers component-aware sparsification within mamba, particularly targeting its linear layers, which contrasts with prior uniform or block-level approaches. this targeted strategy optimizes both efficiency and performance by leveraging mamba's internal structure. ## conclusion we explore sparsification strategies within mamba, revealing that selectively sparsifying the z-branch achieves superior accuracy-efficiency trade-offs compared to full sparsification or block-level approaches. the proposed mamba-moz framework enhances performance while maintaining linear-time complexity, outperforming hybrid moe-mamba architectures. our findings establish internal sparsification as a promising direction for scaling ssm-based models, paving the way for more efficient large-scale language models.