D ense SSM : State Space Models with Dense Hidden Connection for Efficient Large Language Models (2025)

Abstract: Large language models (LLMs) face a significant challenge due to the excessive computational and memory requirements of the commonly used Transformer architecture. While state space model (SSM) is a new type of foundational network architecture offering lower computational complexity, their performance has yet to fully rival that of Transformers. This paper introduces DenseSSM, a novel approach to enhance the flow of hidden information between layers in SSMs. By selectively integrating shallow-layer hidden states into deeper layers, DenseSSM retains fine-grained information crucial for the final output. This incremental improvement maintains the training parallelizability and inference efficiency of SSMs while significantly boosting performance. The proposed method is broadly applicable to various SSM types, including RetNet and Mamba, and DenseSSM achieves significant performance improvements on public benchmarks, demonstrating its effectiveness and versatility.

## introduction since the release of @xcite , large language models @xcite @xcite @xcite have entered a new epoch, showcasing outstanding abilities in language comprehension, dialogue, and logical reasoning. over the past year, the industry has witnessed the emergence of numerous large language models, such as llama @xcite and chatglm @xcite . these large language models have given rise to a plethora of practical applications, including conversational bots, code assistants, and ai agents. the foundation of large language models lies in the transformer network structure (vaswani * equal contribution † corresponding author @xcite , primarily utilizing a multi-head selfattention module for modeling relationships between tokens and a feed-forward network for nonlinear feature transformations. the scaling law @xcite based on the transformer structure has propelled the continuous development and expansion of large language models. in the transformer network, multi-head selfattention (mhsa) plays a crucial role, but it comes with significant computational demands and memory requirements during inference. in terms of computational complexity, for an input sentence of length n , the calculation of self-attention has a complexity of o(n 2 ) during training and inference. regarding memory usage, previously encountered keys and values are stored, leading to a memory occupation of o(n d). as a result, recent efforts on network architectures have focused on simplifying transformer by reducing its computation and space complexity. this includes various approaches, notably convolutional language models @xcite , recurrent unit @xcite , long context models @xcite , and state space models (ssms) @xcite . these new models have provided strong alternatives to transformer for building efficient llms. ssms propose modeling sequences by introducing an appropriate design of hidden states for handling long-range dependencies with both training parallelizability and inference efficiency. starting from the continuous mapping system, ssms are discretized to process discrete inputs in deep learning such as language sequence. the discretized ssms can be computed in both linear recurrence and global convolution modes. commonly, convolution mode is used during training to achieve parallel acceleration, while recurrence mode is used during autoregressive inference because it has lower computational complexity. the core distinction of ssms from other neu-ral networks, such as fully-connected neural networks, lies in the design of hidden states. hidden states enable information to be propagated along the temporal dimension, while avoiding the computation complexity of accessing historical tokens at each step. through state transition parameters a, hidden states transfer the hidden information from the previous time steps to the current time step, allowing for autoregressive prediction of the next token. hidden states play a crucial role in ssms, but have not received sufficient investigation in the past. weights and hidden features in different layers contain information at various levels from fine-grained to coarse-grained @xcite . however, in previous versions of ssms, hidden states only flowed within the current layer and could not transmit more information to deeper layers, thus failing to capture more hierarchical information. in this paper, we propose densessm to facilitate a more comprehensive flow of hidden information between layers in state space models. we first analyze the hidden state degradation in conventional ssms which will prevent hidden information flow from low levels to high levels. by selectively integrating shallow-layer hidden states into deeper layers, densessm retains fine-grained information that is useful for the final output. the proposed method is applicable to different types of ssms, such as retnet @xcite and mamba @xcite . our approach maintains the training parallelizability and inference efficiency of ssms, while achieving a significant improvement with only a slight increase in the number of parameters. for instance, our denseretnet model outperforms traditional retnet with up to 5% accuracy improvement on public benchmarks. ## densessm in this section, we analyze the hidden state degradation in the deeper layers of ssms and further introduce dense connection of hidden states to preserve richer information for deeper layers. ## experiments in this section, we conducted comprehensive experiments to validate the effectiveness of the proposed densessm. the verification was carried out on different architectures, including retnet and mamba ## conclusion in this paper, we propose densessm, a framework designed to enhance the hidden information flow in ssms. by selectively integrating hidden states from shallow layers into deeper layers, dens-essm improves the model's ability to capture lowlevel textual information. this approach preserves the key advantages of ssms, such as efficient autoregressive inference and parallelizable training. experiments on pile have validated the effectiveness of the densessm method on both retnet and mamba, demonstrating its applicability to various ssm architectures.