Leveraging Locality in Abstractive Text Summarization (2022)

Abstract: Neural attention models have achieved significant improvements on many natural language processing tasks. However, the quadratic memory complexity of the self-attention module with respect to the input length hinders their applications in long text summarization. Instead of designing more efficient attention modules, we approach this problem by investigating if models with a restricted context can have competitive performance compared with the memory-efficient attention models that maintain a global context by treating the input as a single sequence. Our model is applied to individual pages, which contain parts of inputs grouped by the principle of locality, during both the encoding and decoding stages. We empirically investigated three kinds of locality in text summarization at different levels of granularity, ranging from sentences to documents. Our experimental results show that our model has a better performance compared with strong baseline models with efficient attention modules, and our analysis provides further insights into our locality-aware modeling strategy.

## introduction neural abstractive summarization @xcite is mainly formulated as a sequence-to-sequence @xcite (seq2seq) problem. neural attention models, e.g., transformers @xcite , have been widely used for such seq2seq tasks, allowing effective modeling of various dependencies in the input and output sequences. however, the self-attention module in such models introduces a quadratic memory growth with respect to the input sequence length. consequently, for long-text summarization datasets, foot_1 recent works @xcite @xcite have explored the x-axis represents the distance of two sentences in source documents measured by the difference of their locations (indexes). y-axis represents the average semantic similarity calculated by the cosine similarity between sentence embeddings, which are generated by a pre-trained sentence embedding model @xcite . the dash line shows the average similarity. we introduce a framework of leveraging locality for text summarization, which reduces the memory complexity of full-attention models while still maintains competitive performance. instead of viewing the input document as an entire sequence, we represent an input document as a number of pages which are constructed according to the principle of locality (fig. 2 ). each of these pages is encoded independently by the encoder of our abstractive model, and the decoder makes local predictions over each page along with local confidence scores of its predictions, which are used to combine the local predictions into final outputs. in this framework, tokens in different pages never directly interact with each other during encoding and decoding, which highlights the role of locality in text summarization. in contrast, one of the key assumptions of efficient attention models is that all tokens in the input text should interact with each other, which is made possible because (1) global tokens @xcite or overlapping win-dow attention maintain a global context during encoding; (2) the encoder-decoder attention takes the source document embeddings as an entire sequence during decoding. using the proposed framework, we are able to investigate several types of locality in text summarization: (1) spatial locality or sequential locality -neighboring sentences are grouped into the same (non-overlapping) page; (2) discourse locality -different sections in a scientific paper may cover different aspects, therefore they are viewed as different pages @xcite ; (3) document locality -for multi-document summarization, each document in a document cluster can be viewed as an individual page @xcite . our approach also has other advantages: (1) our model can take the most advantage of pre-trained fullattention models (e.g. bart @xcite ) because it preserves the same attention mechanism as the full-attention models, unlike most of the efficient attention models; (2) it reduces the overall complexity of encoder self-attention to a linear relationship with the input document length. we empirically demonstrate that our model outperforms strong baseline models built upon various efficient-attention modules on several summarization datasets. furthermore, we conduct detailed analyses on different modeling options for our framework, shedding lights on its broader uses. ## preliminaries abstractive summarization models aim to generate a shorter text sequence as the summary of an input document. given an input document d and a reference summary s, the standard training algorithm of a neural abstractive summarization model g adopts the cross-entropy loss, which requires the model to predict the next token of the reference summary given the input document and the prefix of the reference summary before the current token: ## locality-aware abstractive text summarization to avoid the quadratic growth of memory with respect to the length of the input, we introduce a different view for modeling the input text. specifically, instead of viewing the input document as an entire text sequence, we view it as a series of nonoverlapping pages with a fixed maximum length: fine-tuning from pre-trained models our model can be directly initialized from a pre-trained language model (e.g. bart @xcite ) except for an additional linear layer l cong (eq. 7). the cross-entropy loss (eq. 1) with label smoothing @xcite is used for training. space complexity our model has a linear space complexity with respect to the length of input documents. specifically, given a pre-defined maximum page length l page , a document of which the length is l d will be split into at most ⌈ l d lpage ⌉ pages. the space complexity of the encoder self-attention for one page is o(l 2 page ), and the complexity for all pages is locality in abstractive summarization we mainly explore three types of locality for abstractive summarization, which provide the principles of splitting an input document or document cluster (in the case of multi-document summarization) into different pages. ## conclusions we empirically investigate three kinds of locality in abstractive text summarization by using them as important inductive biases. using a new abstraction of viewing the input document as a series of pages, our model emphasizes the role of locality in both encoding and decoding stages. the experimental results show that our model has strong performance by following the principle of locality. we also show that it is important to select the suitable kind of locality for different application scenarios. we note that the fact that our model has better or competitive performance comparing with the models equipped with efficient attention modules suggests that those models may fall short of their designing objectives. therefore, for future work, our findings call for more rigorous examinations of the memory-efficient abstractive summarization models that aim to capture global features (e.g. long-distance dependencies) and maintain a global input context.