MODE - LSTM : A Parameter-efficient Recurrent Network with Multi-Scale for Sentence Classification (2020)

Abstract: The central problem of sentence classification is to extract multi-scale n-gram features for understanding the semantic meaning of sentences. Most existing models tackle this problem by stacking CNN and RNN models, which easily leads to feature redundancy and overfitting because of relatively limited datasets. In this paper, we propose a simple yet effective model called Multi-scale Orthogonal inDependEnt LSTM (MODE-LSTM), which not only has effective parameters and good generalization ability, but also considers multiscale n-gram features. We disentangle the hidden state of the LSTM into several independently updated small hidden states and apply an orthogonal constraint on their recurrent matrices. We then equip this structure with sliding windows of different sizes for extracting multi-scale n-gram features. Extensive experiments demonstrate that our model achieves better or competitive performance against state-of-the-art baselines on eight benchmark datasets. We also combine our model with BERT to further boost the generalization performance.

## introduction sentence classification (sc) is a fundamental and traditional task in natural language processing (nlp), which is widely used in many subareas, such as sentiment analysis @xcite and question classification @xcite . the central problem of sc is to understand the semantic meaning of a sentence by some key-phrases located at different positions @xcite . cnns excel at extracting n-gram features of sentences through a convolution operation followed by non-linear and pooling layers and have achieved impressive results in sentence classification @xcite . however, the convolution operation itself is linear, which may not be sufficient to model the non-consecutive dependency of the phrase @xcite and may lose the sequential information @xcite . as shown in figure 1 , the weighted sum of the phrase "not almost as bad" does not capture the non-consecutive dependency of "not bad" very well and ignores the sequential information. on the other hand, lstms @xcite are suitable for encoding structure-dependent semantics by storing previous word representations and preserving sequential information. however, lstms are still biased toward later words and ignoring the earlier words @xcite , so some current methods @xcite @xcite combine the cnn and lstm by stacking. however, merely stacking multiple layers can easily lead to feature redundancy and overfitting, because only relatively small training sets are available for sc tasks @xcite . hence, some researchers @xcite @xcite additionally attach an over-parameterized attention mechanism to enhance salient features and remove redundancy, but overfitting still occurs due to the increase in parameters for limited datasets. a flexible combination method is to model nonlinear mapping and non-consecutive dependency by replacing the convolution operation with a tensor product @xcite or rnn unit @xcite . however, these methods only consider fixed-size n-gram features. this has apparent drawbacks in that there may be variablesize phrases (n-grams) in a sentence, as shown in figure 1 , we need to extract variable-size n-gram features to form a better sentence representation. the above observation motivates us to explore a better structure for sentence classification, balancing the capability and complexity. in this paper, we propose a lightweight model called multiscale orthogonal independent lstm (mode-lstm), which has minimal effective parameters, good generalization performance, and considers n-gram features of different scales. first, inspired by @xcite , we disentangle the hidden state of lstm into several independently updated small hidden states, which reduces the number of parameters. furthermore, an orthogonal constraint is applied to the recurrent transition matrices of the small hidden states to improve the diversity of features. we call this structure orthogonal independent lstm (ode-lstm). then we use ode-lstm within a local window for extracting n-gram features instead of simply using a weighted sum as in convolution. specifically, we introduce a triple-s (slide-split-stack) operation that splits a sentence into multiple sub-sentences by a sliding window and stacks them together. these sub-sentences are regarded as a mini-batch, which can be processed in parallel by a shared ode-lstm. we take the last hidden state of ode-lstm as the n-gram features for each sub-sentence. furthermore, in order to capture the variable-size phrases in sentences, we use different scale windows with different initialized ode-lstms to extract features of multiple scale phrases. we refer to this structure as a multi-scale ode-lstm (mode-lstm). mode-lstm can extract multi-scale n-gram features like a cnn, while retaining the non-linear ability and long-term dependency of lstms, so it has stronger modeling ability but with fewer parameters than other methods. mode-lstm is analogous to a 1d cnn using multiple filters with different window sizes, but it uses recurrent transitions instead of the convolution operation. we conduct experiments on eight sentence classification datasets. the experimental results show that our proposed model achieves comparable or better results on these datasets with fewer parameters than other models. in addition, we further improve our model's generalization performance by integrating the bert representation of the sentence. rnn-based models rnns are suitable for processing text sequences and modeling long-term dependencies, so it is also used for sentence modeling. recently, some work incorporate residual connections @xcite or dense connections @xcite into recurrent structures to avoid vanishing gradients. @xcite introduce a rotational unit of memory into rnns for recalling long-distance information. @xcite propose an hs-lstm that can automatically discover structured representation in a sentence via reinforcement learning. however, these rnn-based models still display the bias problem where later words are more dominant than earlier words @xcite . hybrid models a natural strategy is to combine the advantages of cnns and rnns by stacking. @xcite equip an rnn with maxpooling to tackle the bias problem of rnns. @xcite use 1d convolutions to extract phrase features followed by an lstm to obtain the sentence representation, and some subsequent work @xcite are similar. alternatively, @xcite first model long-term dependencies using an lstm and then apply a cnn to extract task-specific features. however, these methods simply stack multiple layers, resulting in feature redundancy and overfitting because of limited datasets @xcite . some researchers have introduced attention mechanisms @xcite @xcite to enhance salient features, but this leads to a large number of parameters that overfit for small-scale datasets. a more flexible way is to combine them by replacing the convolution operation with a tensor product @xcite or rnn unit @xcite , which can capture the non-linear ngram features directly. nevertheless, these methods currently only consider fixed-scale n-gram features. other models some work @xcite @xcite has used tree-lstms based on parse trees for sentiment analysis, but the performance depends heavily on the quality of the parser, and the parsing process itself is timeconsuming. others @xcite @xcite have tried using capsule networks with dynamic routing for encoding text representations. the most relevant work to our approach is the drnn @xcite , which also uses rnns locally to learn semantic features. the differences between their approach and ours are: (1) the drnn uses grus as the recurrent unit while we use the ode-lstm, which has better generalization performance. (2) we introduce the triple-s operation to execute all sub-sentences in parallel instead of in sequence, which is faster than the drnn. (3) we consider multi-scale n-gram features in sentences, while drnn only considers a fixed scale. ## proposed method in the following, we start with our most straightforward model, which is a parameter-efficient structure of lstms to avoid over-fitting and achieve better generalization performance. this structure is then equipped with local sliding windows to learn key phrase features of the sentence, which is the central problem for understanding sentence semantics @xcite . finally, we further easily extended our method to capture multi-scale features of the sentence via using different sized windows in parallel. ## conclusion this study presents a novel parameter-efficient model called mode-lstm that can capture multiscale n-gram features in sentences. instead of the tradition of exploiting complicated operations by stacking cnns and rnns, or attaching overparameterized attention mechanisms, our work provides a lightweight method for improving the ability of neural models for sentence classification. through disentangling the hidden states of the lstm and equipping the structure with multiple sliding windows of different scales, mode-lstm outperforms popular cnn/rnn-based methods and hybrid methods on various benchmark datasets. in future work, we plan to validate its effectiveness for aspect-level sentiment classification.