Long-Span Summarization via Local Attention and Content Selection (2021)

Abstract: Transformer-based models have achieved state-of-the-art results in a wide range of natural language processing (NLP) tasks including document summarization. Typically these systems are trained by fine-tuning a large pre-trained model to the target task. One issue with these transformer-based models is that they do not scale well in terms of memory and compute requirements as the input length grows. Thus, for long document summarization, it can be challenging to train or fine-tune these models. In this work, we exploit large pre-trained transformer-based models and address long-span dependencies in abstractive summarization using two methods: local self-attention; and explicit content selection. These approaches are compared on a range of network configurations. Experiments are carried out on standard long-span summarization tasks, including Spotify Podcast, arXiv, and PubMed datasets. We demonstrate that by combining these methods, we can achieve state-of-the-art results on all three tasks in the ROUGE scores. Moreover, without a large-scale GPU card, our approach can achieve comparable or better results than existing approaches.

## introduction transformer-based models @xcite are ubiquitously state-of-art across many natural language processing (nlp) tasks, including summarization. to achieve the best results, the community has trained ever larger transformer models on larger amount of data, and/or more task-specific optimization objectives @xcite @xcite . in long document summarization, the input sequences could be more than an order of magnitude longer than the limits of these transformer models. although the limits can be extended, training large transformer models on long sequences is expensive and may not be possible on a standard gpu card because of the self-attention mechanism that grows quadratically with sequence length. to tackle the quadratic characteristic, recent works have modified self-attention mechanism and proposed variants of the transformer such that the quadratic complexity is reduced @xcite @xcite @xcite . however, pre-trained weights of the modified models are not readily available. in contrast, standard models such as bert @xcite or bart @xcite have been trained on various target tasks, including text summarization @xcite . this allows practitioners to achieve good performance with less training time. thus, we are interested in exploiting pretrained models for long-span summarization tasks. we study a range of design configurations empirically and theoretically in regards to memory and compute requirements as well as their performance. we propose that long-span dependencies can be handled by two complementary methods. firstly, inspired by modified self-attention transformers, we exploit standard transformer models by constraining attention mechanism to be local, allowing longer input spans during training. secondly, because abstractive summarization systems perform content selection implicitly @xcite , to reduce memory and compute requirements an alternative method is to perform content selection explicitly before the abstractive stage. we study content selection during two phases: training time and test time. at training time, we investigate methods to select data for training fixed-span abstractive models. at test time, we extend existing model-based selection methods, and we propose a multitask content selection method that ranks sentences through extractive labelling based module @xcite and attention based module @xcite . ultimately, we explore the combined approach, consisting of local self-attention transformer and content selection for long-document summarization. we conduct our experiments using a number of design configurations on the spotify opendomain podcast summarization dataset @xcite . this dataset is challenging not only because of its long-span nature, but also because transcribed spoken utterances typically have lower information density @xcite . furthermore, we carry out experiments on arxiv and pubmed datasets @xcite to further demonstrate and verify the effectiveness of our approach as well as making comparisons to existing approaches. we highlight the strengths and weaknesses of our approach in different resources and tasks. the main contributions of this paper are: to mitigate the quadratic nature, a wide range of modified architectures have recently been proposed @xcite . they reduce the quadratic complexity of the full self-attention mechanism by using fixed attention patterns @xcite @xcite @xcite @xcite , learnable patterns @xcite , low-rank matrix approximation @xcite , or kernel method @xcite . alternatively, it has been shown that some attention heads are redundant and can be pruned to reduce model size @xcite . knowledge distillation reduces memory and compute by compressing a large model to a smaller one @xcite . in contrast, we focus on the dependencies of long input and target sequences in encoder-decoder architectures, and we exploit publicly available transformer models with summarization weights to long-span summarization tasks. long-span summarization. efficient transformer architectures have been applied to summarize long documents such as bigbird @xcite , and longformer-encoder-decoder (led) @xcite , which has recently been revised parallel to this work. foot_1 hierarchical transformer architectures have been applied to multi-document summarization @xcite , and extractive news and table-to-text summarization @xcite . hierarchical attention rnn system has been applied to summarize long articles @xcite . alternatively, earlier methods show that good content selection helps abstractive news summarization systems @xcite @xcite . hybrid systems that select sentences and generate an abstractive summary have been proposed such as extractive system + tlm for scientific articles @xcite , simple selection + bart for podcasts @xcite , and guided summarization by bert-based keyword/sentence extraction + bart for news and scientific articles @xcite . other work includes dividing the source and target into multiple smaller pairs to train abstractive summarizers @xcite . extractive methods with and without redundancy reduction techniques for long-span summarization have been studied @xcite . 3 experimental setup 3.1 dataset spotify podcast. 3 the dataset consists of asr transcripts with human descriptions as summaries @xcite . we follow the data processing at trec2020 @xcite in removing bad transcript-summary pairs from a total of 105,360+1,027 episodes, resulting in train/valid/test splits of 60,415/2,189/1,027 episodes the same as @xcite . ## longer span via local self-attention it has been known that memory and compute complexity of transformers is quadratic with the sequence length. however, in encoder-decoder architectures, the exact dependencies on input length n , target length m , and batch size b are less understood. this is particularly important in long-span seq2seq tasks because large memory or compute requirement could make training impractical. thus, this work studies these dependencies, and shows the trade-off between the size of input span and the size of attention span in local self-attention. ## longer span via content selection some input sequences still exceed lobart's longer fixed-span limit. further extending the input span would lead to a small local attention span, a diminishing improvement, or gpu running out of memory. alternatively, it has been shown that a better content selection improves abstractive summarization in news @xcite @xcite , multi documents @xcite , and scientific articles @xcite . thus, we propose to tackle the excess length by content selection. here, we distinguish between two phases of content selection: training time and test time. ## conclusion we study two methods for long-span summarization tasks. first, on local self-attention transformers, we present the design considerations for local self-attention bart, and we investigate the feasibility and performance of different network configurations. second, on content selection, we distinguish between training time and test time methods, and we provide a good practice for both phases. at training time, we show that the oracle method with random sentences padded (orc pad-rand ) yields the best results. at test time, we propose multitask content selection (mcs) that shows an improvement over baselines. we demonstrate that content selection is essential, in particular for longer documents such as the articles in the arxiv dataset. our bart(1k)+mcs outperforms the current best systems on podcast and arxiv datasets, and this system does not require a large-scale accelerator in training. ultimately, by combining local self-attention technique with mcs, our lobart(4k)+mcs system has set new state-of-the-art results in terms of rouge scores in all three long-span summarization tasks. future work will focus on training our lobart+mcs system in an end-to-end fashion.