Revealing and Mitigating the Local Pattern Shortcuts of Mamba (2025)

Abstract: Large language models (LLMs) have advanced significantly due to the attention mechanism, but their quadratic complexity and linear memory demands limit their performance on long-context tasks. Recently, researchers introduced Mamba, an advanced model built upon State Space Models (SSMs) that offers linear complexity and constant memory. Although Mamba is reported to match or surpass the performance of attention-based models, our analysis reveals a performance gap: Mamba excels in tasks that involve localized key information but faces challenges with tasks that require handling distributed key information. Our controlled experiments suggest that the inconsistency arises from Mamba’s reliance on **local pattern shortcuts** across model scales (10M to 1.4B), which enable Mamba to remember local key information within its limited memory but hinder its ability to retain more dispersed information. Therefore, we introduce a global gate module into the Mamba model to address this issue. Experiments on extensive synthetic tasks, as well as real-world tasks, demonstrate the effectiveness of our method. Notably, with the introduction of only 4M extra parameters, our approach enables the Mamba model (130M) to achieve a significant improvement on tasks with distributed information, increasing its performance from **below 5% to 80%**.

## introduction in recent years, state space model (ssm) has emerged as a promising successor to the attentionbased models @xcite for long sequence modeling due to its linear computational complexity and constant memory requirements @xcite @xcite @xcite . different from the attention mechanism, which stores * equal contribution. in this paper, we conduct a controlled study to better understand the characteristics of mamba under different context settings. we start by analyzing mamba's performance on different synthetic tasks, covering different information densities and dispersions. our findings reveal that mamba relies on local pattern shortcuts to extract the desired information from the context, which manifests in two aspects: (1) positional shortcuts, i.e., mamba tends to extract information from specific positions; and to mitigate the aforementioned issues, we propose a global gate mechanism for the mamba model. specifically, we design an input-dependent global gating module for mamba and observe significant improvements in its performance on complex synthetic tasks and the language modeling task. notably, by introducing just 4m additional parameters to the 130m-sized mamba model, mamba can achieve a breakthrough from below 5% to 80% on high information density synthetic tasks, significantly narrowing the performance gap between mamba and attention-based models. ## conclusion in this work, we extend the mqar task to investigate the underlying behavior of mamba. our controlled experiments reveal that mamba relies on local pattern shortcuts at different model scales. to address this issue, we introduce a fine-grained selection mechanism in the mamba model by incorporating global information into the decision-making factor ∆ t . experiments on both existing and newly proposed synthetic tasks show that our method effectively mitigates the shortcut phenomenon in mamba across model scales ranging from 13.8m to 1.4b parameters. further experiments on realworld tasks further demonstrate the effectiveness of our approach. our findings suggest that for rnnbased models with a fixed recurrent state size, efficiently utilizing the available state space is far more critical than simply increasing capacity.