Cascaded Semantic and Positional Self-Attention Network for Document Classification (2020)

Abstract: Transformers have shown great success in learning representations for language modelling. However, an open challenge still remains on how to systematically aggregate semantic information (word embedding) with positional (or temporal) information (word orders). In this work, we propose a new architecture to aggregate the two sources of information using cascaded semantic and positional self-attention network (CSPAN) in the context of document classification. The CSPAN uses a semantic self-attention layer cascaded with Bi-LSTM to process the semantic and positional information in a sequential manner, and then adaptively combine them together through a residue connection. Compared with commonly used positional encoding schemes, CSPAN can exploit the interaction between semantics and word positions in a more interpretable and adaptive manner, and the classification performance can be notably improved while simultaneously preserving a compact model size and high convergence rate. We evaluate the CSPAN model on several benchmark data sets for document classification with careful ablation studies, and demonstrate the encouraging results compared with state of the art.

## introduction document classification is one of the fundamental problems in natural language processing, which is aimed at assigning one or multiple labels to a (typically) short text paragraph. wide applications can be found in sentiment analysis @xcite ，subject categorization * corresponding author. @xcite ,spam email detection @xcite and doc 1 ument ranking @xcite . in recent years, deep neural networks have shown great potential in document classification and updated state-of-the-art performance. popular approaches include recurrent neural networks (rnn) @xcite , convolutional neural networks (cnn) @xcite and attention-based methods (transformers) @xcite , or a mixture of them. different lines of methods have their respective pros and cons. for example, rnns are highly effective models for exploiting word orders in learning useful representations, thanks to the iterative update of the hidden states that depend on both the semantics of the current word and that of historical words (or a concise summary of them), and the long-range dependency made possible through lstms @xcite @xcite . of course, the sequential processing nature makes it less efficient computationally. cnns have gained huge success in image procesing and classification and were recently introduced to nlp domains like document classification @xcite @xcite @xcite .the local convolutional operator is sensitive to word orders but only partially and limited by the size of the kernel, and so long-term relations may need many layers and therefore be challenging. transformers, different from both, fully exploit the modelling power of self-attention mechanism @xcite @xcite and have significantly improved state of the art in many nlp tasks such as machine translation @xcite , language understanding @xcite and language modeling @xcite , etc. despite the great successes, how to systematically aggregate the semantic information (word embedding) with the positional information (word orders) is still an open challenge in transformers. a common practice is the positional encoding @xcite , which encodes the position of the 𝑡 th word as a 𝑑 -dimensional sinusoidal vector, as the positional vector of each word is then added to the 𝑑 -dimensional word embedding vector, so that subsequent predictors can numerically utilize the temporal information. however, empirically, adding positional vectors to the word vectors brings little performance gains in document classification, compared with when no positional encoding is adopted at all (see section 3.4 there are two reasons which we believe are related to the low performance gains from using positional encodings. first, such a strategy leads to an interaction (inner product) between the semantic and temporal component that is hard to interpret. to see this, let 𝑥 𝑖 and 𝑝 𝑖 be the word vector and position vector for the 𝑖th word. then the attention score between 𝑖th and 𝑗th word will be computed as (before normalization) obviously, as the inner product between a word vector and a positional vector, 〈𝑥 𝑖 , 𝑝 𝑗 〉 and 〈𝑝 𝑖 , 𝑥 𝑗 〉 do not bear meaningful interpretation. therefore these two terms could very likely hamper the semantic attention term 〈𝑥 𝑖 , 𝑥 𝑗 〉 and the positional attention term 〈𝑝 𝑖 , 𝑝 𝑗 〉 by behaving like noise, such as deflating an important attention or exaggerating a marginal one. this can negatively affect the learned representations through the self-attention mechanism. indeed, similar observations were made in @xcite , where the authors show that the self-attention mechanism, when mixed with the positional vectors, can no longer effectively quantify the relative positional distance between the words (namely the positional attention term 〈𝑝 𝑖 , 𝑝 𝑗 〉 is perturbed in an undesired manner). second, the relative weights of the word vector and the position vector (in their summation) is hard-coded, leading to a fixed combination, while in practice the relative importance of the semantic and positional components in affecting the similarity among the words can definitely be more complex. in order to solve these challenges with positional encoding, we explore a new architecture in combining the semantic and temporal information in document classification, called "cascaded semantic and positional self-attention network" (cspan). there are three main characteristics of the proposed architecture. first, instead of combining the word vectors with positional vectors from scratch, we choose to first explore the two sources of information with their respective processing layers, namely, a self-attention layer that works only on the semantic space, and a bi-lstm layer which further incorporates the temporal order information in the updated word representations. second, these two layers are cascaded so that sematic information and the temporal information can be finally combined through the use of a residual connection; this not only avoids non-interpretable operations defined between word vectors and positional vectors, but also serves as an adaptive transformation in combining the two information sources. third, a multi-query attention scheme is adopted to extract multi-faceted, fixed dimensional document features, which makes the resultant model highly compact and memory efficient. the cspan model is shown to effectively improve performance of document classification in comparison to several state-of-the-art methods including transformer-styled architecture. in the meantime, it demonstrates very compact model size and fast convergence rate during the training process, which is particularly desirable for large problems. we also conducted careful ablation studies to further quantify the performance gains of each component of the cspan model. our study demonstrates the importance of the way semantic and temporal information are aggregated in capturing the structures and meaning of documents, which we will continue exploring in the more challenging language modelling tasks such as sequence tagging @xcite , natural language inference @xcite and modeling sentence pairs @xcite in our future research. ## method the overall architecture of the proposed cspan model is shown in figure 1 . it is a highly compact model with three basic building blocks. first, we use a self-attention block to update the word representations in each document. here, the embedding of each word will be collectively affected by all other words with related semantics in the same document. note that we will not look into any positional information in this stage. instead, the temporal information will be taken into account in the next block, after the word representations have been fully updated through semantic self-attention alone. as we shall see, such a sequential processing pipeline allows more flexible combination of the semantic and positional information. second, the updated word embeddings are fed into a bi-lstm layer, so that the relative position of the words are naturally exploited to further refine the word representations specific to the organization of each document. in the meantime, a residual connection is adopted to combine the semantic representation derived from the selfattention block, together with the output derived from the bi-lstm block; we call this ``semantic and positional residual connection'', because it combines the semantic information (out of selfattention block) with the positional information (out of the bi-lstm block) using residual connections. as we shall see, such a combination is more flexible than directly combining word vector with positional vector as in existing positional encoding schemes. third, we adopt a multiple-query attention in the final block to extract fixed-dimensional document features for final classification. compared with multi-head attention, the multi-query attention can significantly reduce the number of parameters in the network, while giving promising classification results. we describe the details of different structures and components of our model in the following sections. ## experiments in this section, we will report a number of experimental results on 4 benchmark datasets for document classification, together with careful ablation studies to illustrate the effectiveness of the building blocks of the proposed method. ## conclusion we presented the cascaded semantic and positional self-attention to aggregate semantic and positional information in document classification. it overcomes the limitation of existing positional encoding schemes, and shows encouraging performance against state-of-the-art methods using transformers and cnns. in the meantime, it has a compact model size and is computational efficient. our studies demonstrate the importance of properly aggregating semantic and positional components, and we will further extend it more challenging nlp tasks in our future research.