Efficient Attentions for Long Document Summarization (2021)

Abstract: The quadratic computational and memory complexities of large Transformers have limited their scalability for long document summarization. In this paper, we propose Hepos, a novel efficient encoder-decoder attention with head-wise positional strides to effectively pinpoint salient information from the source. We further conduct a systematic study of existing efficient self-attentions. Combined with Hepos, we are able to process ten times more tokens than existing models that use full attentions. For evaluation, we present a new dataset, GovReport, with significantly longer documents and summaries. Results show that our models produce significantly higher ROUGE scores than competitive comparisons, including new state-of-the-art results on PubMed. Human evaluation also shows that our models generate more informative summaries with fewer unfaithful errors.

## introduction long documents, such as scientific papers and government reports, often discuss substantial issues at length, and thus are time-consuming to read, let alone to comprehend. generating abstractive summaries can help readers quickly grasp the main topics, yet prior work has mostly focused on short texts (containing hundreds of words), e.g., news articles @xcite @xcite . model training efficiency and summary quality present a pair of challenges for long document summarization. state-of-the-art systems @xcite are built upon transformer @xcite , which uses attentions to compute pairwise relations between tokens. such framework has quadratic time and memory complexities, and is too costly for long documents 1 . solutions have been proposed to reduce 1 for instance, to fine-tune bart on documents of 10k the calculation of encoder self-attentions @xcite ) by selectively attending to neighboring tokens @xcite or relevant words @xcite ). yet, these methods do not apply to encoder-decoder attentions in summarization models since they collaborate and dynamically pinpoint salient content in the source as the summary is decoded. truncation is commonly used to circumvent the issue. however, training on curtailed content further aggravates "hallucination" in existing abstractive models @xcite . we argue that summarizing long documents (e.g., with thousands of words or more) requires efficient handling of both types of attentions. to this end, we propose an efficient encoder-decoder attention with head-wise positional strides (hepos), where the attention heads follow a strided pattern and have varying starting positions. hepos reduces computational and memory costs while (1) maintaining the power of emphasizing important tokens, and (2) preserving the global context per head. hepos successfully doubles the processed input sequence size, when combined with any encoder. to the best of our knowledge, we are the first to study efficient encoder-decoder attentions and provide a systematic comparison of diverse encoder attentions for the task of summarization. 2 for evaluation, we collect a new large-scale dataset, govreport, consisting of about 19.5k u.s. government reports with expert-written abstractive summaries. 3 govreport has two important features: (1) it contains significantly longer documents (9.4k words) and summaries (553 words) than existing datasets, such as pubmed and arxiv @xcite (see we conduct experiments on govreport and scientific papers in pubmed and arxiv. first, when summarizing documents of the same length, hepos attention yields significantly better rouge scores than a non-trivial comparison that projects attentions into low-rank space @xcite . second, when trained on the same gpu, hepos attention, combined with sparse encoder attentions, is able to read more than 10k words and obtains significantly higher rouge scores on govreport and new state-of-the-art results on pubmed, compared with full encoder-decoder attention models which can process at most 5k input words. human judges further rate the summaries generated by our models to be more informative and faithful. we further propose a new evaluation metric for faithfulness, inspired by apes @xcite , a fill-in-the-blank qa metric for summary evaluation. with questions generated from references, our metric, apes src , compares qa answers by reading the source and the system summary. it is shown to be better correlated with human judgment than the original metric and an entailment-based scorer @xcite . the rest of the paper is organized as follows. we describe efficient encoder attentions in prior work in § 2, and formulate our proposed encoder-decoder attention in § 3. the govreport data is presented in § 4. we then share details on evaluation metrics ( § 5) and experimental results ( § 6). additional related work is listed in § 7, with conclusion in §8. ## prior work on efficient encoder attentions transformer models are built upon multi-head attentions in multiple layers. the attention is calculated as attention(q, k, v) = softmax( qk t √ d k )v, where q, k, and v are query, key, and value matrices, each consisting of n vectors for a document with n tokens, thus the quadratic memory footprint. here, we present an overview of representative methods for efficient encoder self-attentions (henceforth "encoder attentions") that can be built upon large pre-trained seq2seq models, e.g., bart @xcite . we follow the naming model complexity # new para. full o(n 2 ) -encoder self-attentions i. fixed patterns sliding window (2020) o(nw) 0 adaptive span (2019) o(n ŵ) o(1) global tokens (2020) o(2ng) 0 stride (2019) o(n 2 /s) 0 random (2020) o(nr) 0 ii. low-rank linformer (2020c) o(nk) o(n) iii. learnable patterns lsh (2020) o(lnb l ) 0 sinkhorn (2020a) o(2nbs) 0 encoder-decoder attentions hepos (ours) o(mn/s h ) 0 linformer o(mk) o(n) ## encoder-decoder attention with head-wise positional strides (hepos) the efficient design of encoder-decoder attentions with head-wise positional strides (hepos) allows models to consume longer sequences. concretely, our design is motivated by two observations: (1) attention heads are redundant @xcite . given a stride size of s h , for the h-th head, its attention value between decoder query q j (at step j) and encoder key vector k i (for the i-th input token) can be formulated as: in hepos attention, each query token attends to n/s h tokens per head, yielding a memory complexity of o(mn/s h ), where m is the output length. for comparison, linformer ( § 2.2) can be straightforwardly adapted for encoder-decoder attentions by using decoder queries for attention calculation instead. we do not adapt pattern-based attentions ( § 2.1 and § 2.3), since they rely on local token grouping which makes it difficult to pinpoint salient content. ## govreport dataset we introduce a new large-scale dataset, govre-port, containing 19, 466 long reports published by u.s. government accountability office (gao) foot_0 to fulfill requests by congressional members, and congressional research service (crs) foot_1 , covering researches on a broad range of national policy issues. a human-written summary is provided along with each report. during data collection, we remove boilerplates from crawled files, and keep the section and paragraph structure of the documents and summaries. additional data cleaning and processing details are in appendix a. we obtain 12, 228 gao reports and 7, 238 crs reports of high quality evidenced by human inspection of 200 parsed reports. collected gao reports and crs reports have on average 6.9 and 4.6 sections, respectively. we split train, validation and test set by publication date on each dataset, and end up with 17519 training samples, 974 validation documents, and 973 test samples. notably, summaries of gao reports are written by experts, and are often structured into three aspects in order: "why gao did this study"-motivation and problem(s) under discussion, "what gao found"-findings of the report, and "what gao recommends"suggestions and solutions to the problem(s). all but three gao summaries include "what gao found". the percentages of gao summaries that contain "why gao did this study" and "what gao recommends" are 94.8% and 29.0%. for comparison, structured summaries are also observed on pubmed @xcite samples. though they do not contain explicit aspect labels, the summaries can often be broken down into "introduction", "methods", "results", and "conclusion" via keyword matching. details about keyword choices for each aspect are provided in comparison with existing long document summarization datasets. in ## summary evaluation with cloze qa this work aims to evaluate whether processing more text improves both informativeness and faithfulness of abstractive summaries. in addition to rouge @xcite and human evaluation, we extend existing qa-based metric @xcite and consider an entailment-based scorer. qa-based evaluation. we present a new faithfulness evaluation metric by extending the apes score @xcite . we follow apes to construct a set of cloze questions, {q}, from each reference summary by masking entities. events, dates, and numbers are also masked, as they are prevalent in our data. each masked phrase becomes the goldstandard answer a ref for a question q. we do not generate natural language questions @xcite , due to the lack of accurate question generation models for the domains of government reports and scientific papers. qa models are trained by reading a question and a context to label the answer span in the context. we construct context by greedily selecting sentences that maximize the improvement of rouge-2 recall when compared with the reference summary. if the answer a ref cannot be found in the context, the sample is excluded from training. we train all qa models by fine-tuning bert @xcite to predict the answer span. to evaluate the faithfulness of a system summary, apes uses the qa model to read the summary and a question q to label an answer a sys . it calculates a unigram f1 score by comparing a sys and a ref . different from apes, we further use the qa model to read the context (sentences selected from the source) and give an answer a cxt to the question q. we compute a unigram f1 by comparing a sys and a cxt , denoted as apes src . given that existing summarization models rarely rewrite names or numbers correctly, our metric can better capture faithfulness by using a gold-standard answer constructed from the source article than from the human-written abstract. to extract entities and events, we deploy a state-of-the-art ie framework, oneie @xcite on govreport. on pubmed, we retrain oneie on genia 2011 (bionlp, 2011) and 2013 (bionlp, 2013), and pubmed @xcite datasets to extract domain-specific entities and events, such as entities of gene and disease. we additionally include numbers and dates extracted by spacy @xcite . entailment-based evaluation. we further consider factcc @xcite , which evaluates factual consistency of a system summary by predicting an entailment score between the source and the summary. we reproduce their method on our datasets. additional details for implementing the evaluation models and the entity extraction models are given in appendix b.