Sparsity and Sentence Structure in Encoder-Decoder Attention of Summarization Systems (2021)

Abstract: Transformer models have achieved state-of-the-art results in a wide range of NLP tasks including summarization. Training and inference using large transformer models can be computationally expensive. Previous work has focused on one important bottleneck, the quadratic self-attention mechanism in the encoder. Modified encoder architectures such as LED or LoBART use local attention patterns to address this problem for summarization. In contrast, this work focuses on the transformer’s encoder-decoder attention mechanism. The cost of this attention becomes more significant in inference or training approaches that require model-generated histories. First, we examine the complexity of the encoder-decoder attention. We demonstrate empirically that there is a sparse sentence structure in document summarization that can be exploited by constraining the attention mechanism to a subset of input sentences, whilst maintaining system performance. Second, we propose a modified architecture that selects the subset of sentences to constrain the encoder-decoder attention. Experiments are carried out on abstractive summarization tasks, including CNN/DailyMail, XSum, Spotify Podcast, and arXiv.

## introduction the transformer architecture @xcite with large-scale pre-training has become the defacto approach for a wide range of nlp tasks, from classification @xcite to seq2seq @xcite . training and inference using large transformer models can be computationally expensive because the self-attention's time and memory grow quadratically with sequence length. hence, there has been significant interest in efficient transformer architectures. a number of approaches have been proposed to tackle the quadratic complexity, and a comprehensive survey on efficient transformers has been compiled in @xcite . most existing approaches are developed for encoder-only architectures. for seq2seq tasks, efficient models such as bigbird @xcite or led @xcite consist of an efficient encoder with the vanilla decoder. for long-document summarization, this combination has been shown effective because the major bottleneck is the encoder selfattention @xcite . the attention mechanisms in the decoder consist of self-attention and encoder-decoder attention. techniques such as local attention are applicable to self-attention in both the encoder and decoder, while this work focuses on the encoder-decoder attention. when humans produce a summary, the information conveyed by each word/part in the summary is likely drawn from some key sentences in the original document. inspired by this, we hypothesize that if the encoder-decoder attention is constrained dynamically to salient sentences, the computation cost will be reduced. for instance, sentence-level structures for the encoder-decoder attention have been shown effective in the traditional rnn encoder-decoder attention @xcite @xcite in this work, first, we compare the decoder's cost in the training and inference stages. we study the sparsity of the encoder-decoder attention in a common transformer-based abstractive summarization model. an approximation method to exploit this sparsity is described, and an empirical upper bound performance is given. second, we propose a modified decoder architecture that can dynamically select salient input sentences to constrain the encoder-decoder attention without having to compute complete attention at inference time. techniques to train our proposed model are described, and compared to the full attention baseline performance and empirical upper bound. ## models and data vanilla transformers. we use bart @xcite and local-attention bart (lobart) @xcite as our base models. bart's maximum input length is 1024, while that of lobart is 4096 with attention width of 1024. bart is fine-tuned to cnn/dailymail and xsum, and lobart is fine-tuned to podcast and arxiv. data. cnn/dailymail @xcite and xsum @xcite are used with bart, while long-document arxiv @xcite and spotify podcast @xcite are used with lobart. more details about models, data, and training are provided in appendix a. ## attention in the transformer time and memory are dominated by the encoder self-attention, and models such as lobart adopt local attention in its encoder to mitigate this bottleneck, while keeping the original decoder @xcite . training is fast because attention is highly parallelizable. however, during inference, the decoder uses its histories, becoming less parallelizable. to understand when the decoder might become a bottleneck, we fix the input length n and measure the computational time as a function of the target length m : in three operating modes: i) forward+backward, e.g. at training time; ii) forward only, e.g. forwardpass where the input to the decoder is provided in advance; iii) inference, e.g. the decoder using its own back histories as the input. through a curve-fitting method, the results in ## sentence-level structure for encoder-decoder attention in section 3.1, we use ideal selection i r m (eq. 6), which requires computing α s m,i (eq. 4) using all input words. this process cannot make the decoder more efficient. by exploiting the sentence structure in the document, we propose the following partition for the sentence-level attention score (eq. 4) to allow a compact approximation: ## conclusion we show that the computational cost of the transformer decoder becomes more significant at inference time. towards reducing this cost, first, we show that there is sparsity in the encoder-decoder attention that allows us to reduce the computational cost with a minimal degradation. second, we partition the sentence-level attention score, and we augment the standard decoder by adding a neural network to approximate the attention over sentences, allowing sentence selection. we show that the summarization performance of our approach converges to that of the full attention baseline, while switching the complexity from