Incorporating Exponential Smoothing into MLP : a Simple but Effective Sequence Model (2024)

Abstract: Modeling long-range dependencies in sequential data is a crucial step in sequence learning. A recently developed model, the Structured State Space (S4), demonstrated significant effectiveness in modeling long-range sequences. However, It is unclear whether the success of S4 can be attributed to its intricate parameterization and HiPPO initialization or simply due to State Space Models (SSMs). To further investigate the potential of the deep SSMs, we start with exponential smoothing (ETS), a simple SSM, and propose a stacked architecture by directly incorporating it into an element-wise MLP. We augment simple ETS with additional parameters and complex field to reduce the inductive bias. Despite increasing less than 1% of parameters of element-wise MLP, our models achieve comparable results to S4 on the LRA benchmark.

## introduction transformer @xcite and its variants have been the most successful architecture in various domains of deep learning. however, the self-attention layer, which plays a crucial role in contextualizing the input, poses a significant computational and memory burden with a complexity of o(l 2 ). this limitation hinders the application of the transformers in modeling long sequences, particularly when operating under hardware constraints, which is a common scenario for large language models. to alleviate this issue, several models have been proposed to reduce the computational and memory requirements of the transformers @xcite @xcite @xcite @xcite @xcite @xcite . despite these efforts, all the models are only partial modifications of the attention mechanism and struggle to perform well on long-range sequence benchmarks such as long range arena (lra) @xcite . in a recent breakthrough result, @xcite ) introduced a novel framework called the "structured state space sequence" (s4) that leveraged the state space models (ssms). s4 builds upon continuous-time ssms and addresses the computational bottleneck of previous approaches by introducing the normal plus low-rank (nplr) decomposition of the state matrices. additionally, the initialization of state matrices utilizes hippo matrices which have been demonstrated to be effective in sequence learning in @xcite . notably, s4 exhibited exceptional performance across various sequential tasks, particularly in the lra, where it outperformed the existing transformer variants by an impressive accuracy. despite the impressive performance of s4, its intricate parameterization and strict initialization schemes impede researchers from fully comprehending, implementing, and analyzing the model. although there have been attempts to simplify the s4 framework by @xcite , these models still required the hippo initialization process. other studies have explored the relationship between ssms and recurrent units or global convolutions and demonstrated strong performance on various tasks @xcite . these works highlight the potential of ssms and suggest that simpler yet effective ssm architectures may exist. in our work, we deviate from the methodology proposed by s4, which begins with the continuous ssm and then simplifies the process. we initiate our approach with a discrete ssm, namely exponential smoothing (ets), and introduce additional parameters to reduce the inductive bias. this alternative approach offers two notable advantages. firstly, it circumvents the simplification of the continuous ssms that need sophisticated mathematical derivations and thus enhances accessibility and comprehensibility. secondly, it explores the possibility of random initialization departing from hippo initialization for continuous ssms. for the streamlining of the model, our architecture directly integrates a parameterized ets into an element-wise multi-layer perceptron (mlp). by incorporating less than 3% of the total parameters after the initial linear layer of the mlp, we successfully transform a channel-only mlp into a sequence learner. we conducted experiments on multiple datasets, including the lra and several natural language understanding (nlu) datasets. despite its simplicity, surprisingly, our model performs comparably to s4. in all six tasks in the lra, our results slightly surpass the performance of s4 and dss by 2.61 points on average and significantly outperform the transformer variants by about 20 points. in addition, we evaluated our model on seven nlu datasets and consistently achieved comparable performance with the transformer encoders. the findings of our work shed light on the potential of ssms from a unique standpoint, where simply incorporating an ets into an mlp can achieve a similar effect as the transformer model. a thorough examination of the proposed model was undertaken through an ablation study on the hyperparameters and an evaluation of the initialization method. additional experiments were conducted to compare our model with the transformer model for efficiency and memory utilization, especially in handling lengthy texts. the results of these experiments provide evidence of the advantages of our model over the transformer model in terms of time and memory complexity. in summary, our main contributions are as follows: the empirical results demonstrate the effective capacity in long-range sequence modeling. ## preliminaries we introduce basic notations and briefly review ssms and ets in this section. focusing on timeinvariant sequence models, we aim to transform a sequence of inputs @xmath0, . . . , x n } ∈ r n×d into a corresponding sequence of outputs @xmath1, . . . , y n } ∈ r n×d with each output y i is exclusively based on historical data x 1 , . . . , x i . ## exponential smoothing multi-layer perceptrons in this section, we present our etsmlp. we first introduce a complex exponential smoothing module which is the pivotal component of our architecture. we then describe the full architecture with two proposed versions, esmlp and esmlp-gate. ## experiments we present an empirical comparison between our etsmlp and other baseline models. our experiments encompass a set of sequence modeling tasks, including lra, mnli, imdb, etc. the main experiment results are divided into two subsections: lra and nlu benchmarks. furthermore, we conduct an ablation study to examine the influence of hyperparameters. additional information about the experimental details and datasets can be found in appendix b. ## related works since the transformer was introduced, the quadratic time cost of the attention operation has been numerously researched. optimizing this operation can improve the efficiency when training and inferencing long context for large language models @xcite . recently, many transformer variants have been introduced to reduce the complexity of attentions @xcite , including sparse attention @xcite @xcite , kernel-based methods @xcite @xcite , chunked attention with gating @xcite and other efficient methods @xcite . another line of research tries to replace the attention mechanism with other modules for long sequences and avoid quadratic time costs. a dizzying number of attention-free models have emerged, where ssms are becoming one of the most promising models among them. ssms. s4 first investigated the ssm for long sequence modeling @xcite . they showed that naive instantiations of the ssm did not perform well but hippo-legs matrix did @xcite , and hence introduced the dplr that efficiently computed the complex diagonal plus low-rank matrix. dss observed that a fully diagonal matrix could preserve the performance of the original s4 @xcite , and s4d @xcite then showed that the initialization is critical for dss. inspired by s4, many ssm variants emerged recently. s5 replaced single-input, single-output (siso) ssms in s4 with multi-input, multi-output (mimo) @xcite . sgconv viewed the ssm as a global convolution model and suggested that the convolution kernel's sub-linear decay in sequence length is indispensable @xcite . linear recurrent unit (lru) explored the relationship between the ssm and linear rnn and showed the importance of initialization, exponential parameterization, and normalization for ssms @xcite . mega was the most similar approach to ours and plugged exponential moving average into the attention mechanism to improve positionwise local dependency @xcite . our ces mechanism only considered a position-aware but representation-agnostic dependency which is completely different from the attention mechanism but matches the performance of the transformer. other attention free models. mlp-mixer @xcite , and its variants proposed to replace the attention with mlps in computer vision task @xcite @xcite . another mlp-based model gmlp showed the potential of mlps to model sequence dependency and achieved comparable results in pretraining and downstream nlp tasks @xcite . the attention free transformer (aft) replaced the attention mechanism with an element-wise multiplication and avoided the quadratic computation burden of the attention matrix @xcite . recurrent memory transformer (rmt) added a special cache token and utilized the recursive components to increase the context length in the transformer @xcite . receptance weighted key value (rwkv) leveraged token shift for parallel training a simple linear rnn @xcite . our models do not conflict with those models in spirit. our ces modules can be integrated into their models to improve their capabilities of sequence learning. ## conclusion we proposed the etsmlp model for long-range sequence modeling. our approach began with a special ssm, namely ets, and incorporated additional hyperparameters. moreover, we proposed an exponential parameterization and a constraint function essential for stable training. the experimental results demonstrated the effectiveness of the etsmlp in sequence learning. our proposed module could become a plug-in module on other models to enhance their sequence learning capabilities. we hope our research could provide valuable insights into the application of ssms and encourage further exploration in this area.