AWESOME : GPU Memory-constrained Long Document Summarization using Memory Mechanism and Global Salient Content (2024)

Abstract: Long document summarization systems are critical for domains with lengthy and jargon-laden text, yet they present significant challenges to researchers and developers with limited computing resources. Existing solutions mainly focus on efficient attentions or divide-and-conquer strategies. The former reduces theoretical time complexity, but is still memory-heavy. The latter methods sacrifice global context, leading to uninformative and incoherent summaries. This work aims to leverage the memory-efficient nature of divide-and-conquer methods while preserving global context. Concretely, our framework AWESOME uses two novel mechanisms: (1) External memory mechanisms track previously encoded document segments and their corresponding summaries, to enhance global document understanding and summary coherence. (2) Global salient content is further identified beforehand to augment each document segment to support its summarization. Extensive experiments on diverse genres of text, including government reports, meeting transcripts, screenplays, scientific papers, and novels, show that AWESOME produces summaries with improved informativeness, faithfulness, and coherence than competitive baselines on longer documents, while having a smaller GPU memory footprint.

## introduction large pre-trained transformer models have demonstrated impressive performance across popular abstractive summarization benchmarks @xcite . yet, transformer's quadratic memory complexity presents challenges for summarizing long documents with more than hundreds of words, such as scientific papers and investigation reports @xcite , making it infeasible for researchers and developers with limited hardware resources (e.g., gpus with insufficient memories) to contribute to this important research field. the nlp community has made several innovations to address the long document challenge. prior work divides a document into smaller chunks and summarizes each separately @xcite , reduces the complexity of attention calculations @xcite , and removes unimportant content before running an abstractor @xcite . in terms of memory efficiency, divide-and-conquer methods obtain the most significant advantage @xcite . however, information outside of a document segment and their corresponding summaries become inaccessible, leading to uninformative and incoherent summaries. unsurprisingly, state-of-the-art performance is obtained by models that can maintain global context, e.g., by combining global attentions with local attentions in transformer-based summarization models @xcite . yet, they still require a large gpu memory footprint in practice. 1 though large language models like gpt-4 (ope-nai, 2023) are trained to handle up to 128k tokens, the privacy and security of data transmitted and shared through the api remain concerning, particularly in sectors dealing with sensitive information, e.g., clinical notes. local model development can bolster privacy and security; however, limited computational resources in these scenarios necessitate the exploration of efficient modeling techniques. therefore, this work aims to address the problem of long document summarization using constrained resources, specifically focusing on constrained gpu memory. we propose awesome 2 , which is built on the memory-efficient divide-and-conquer approach, and augmented with estimated salient content and memory mechanism. in essence, awesome maintains global context of both the source document and the summary generated so far with a limited memory usage, to enhance summary informativeness, faithfulness, and coherence. first, external memory mechanism is used on the encoder side of awesome to store information as it sequentially reads document segments. this maintains relevant context for improving document understanding and salient content detection, thus promoting summary informativeness and faithfulness. another memory is applied on the decoder side to boost generation coherence by tracking the partial summaries generated for previous document segments. importantly, to ensure the gpu memory efficiency of awesome, we curb gradients from propagating to other document and summary segments and only implement the external memory in a limited number of layers. second, awesome incorporates global salient content selected by an efficiently trained extractor through (1) direct text concatenation, or (2) inserting their key-value matrices into attention calculation. this lets the summarizer be aware of important topics at a global level, to enhance salience estimation and summary informativeness. we experiment with five popular long-input benchmarks of different genres: investigation reports in govreport @xcite , meeting transcripts in qmsum @xcite , tv screenplays in summscreen @xcite , scientific papers in arxiv @xcite , and fictions in booksum @xcite . first, on all the five datasets, all awe-some variants uniformly outperform se3 @xcite , the divide-and-conquer baseline, on summary informativeness as evaluated by rouge @xcite and on coherence as measured by discoscore @xcite and a metric based on entity graphs @xcite -both metrics are highly correlated with human judgment, according to @xcite . second, awesome with memory mechanisms also improves summary faithfulness over se3 on gov-report, according to summac @xcite , an entailment-based faithfulness metric. lastly, compared with more memory-intensive models that also maintain global context, such as @xcite and @xcite , awesome achieves better or comparable automatic scores for informativeness, coherence, and faithfulness on gov-report @xcite . on booksum which comprises the lengthiest documents and summaries ## external memory and global salient content augmentation the architecture of awesome (figure 1 ) is based on se3 @xcite , where a document is summarized segment by segment, with the final summary obtained by concatenating the resultant summaries. document sentences are split into segments with up to 768 tokens each, while reference summary sentences are assigned to their most overlapping segment to create the oracle summary, as detailed in appendix a. following longformer @xcite , we initialize the encoder and decoder parameters from bart @xcite . awesome preserves the global context and builds communications across segments with minimal gpu memory increase, by (1) employing external memories in both the encoder and the decoder to gather relevant information ( §3.1), and (2) augmenting the encoder with salient content from other segments ( §3.2). ## experimental setups datasets. we conduct experiments on govreport @xcite , qmsum @xcite , summscreen @xcite , arxiv @xcite , and booksum @xcite . the average input lengths of these datasets range from 6k to 143k ( experiment setups and comparisons. our main experiments are conducted with a gpu memory constraint of 27gb. for each model, we truncate the input such that its maximum gpu memory usage during training does not exceed the constraint when gradient checkpointing @xcite is disabled. the constraint is specifically chosen such that the baselines perform reasonably. appendix c.2 provides information on the maximum number of input tokens that can conform to the constraint for other models. for baselines, in addition to the divide-andconquer se3 model @xcite , we compare with state-of-the-art or popular long document summarization systems including block-attn (phang et al., 2022), 6 longformer @xcite , longt5 @xcite , and unlimiformer @xcite . we also include an extract-then-abstract model (extract-abstract) and pagesum @xcite that leverages dynamic weights, as discussed in §2. all models are initialized from bart-large, except for longt5 that is pre-trained on long-form data. details of baseline models are reported in appendix d. evaluation metrics. we evaluate summary informativeness using rouge @xcite . to measure coherence, we use discoscore @xcite (disco), a reference-based metric that evaluates discourse coherence by comparing focus (e.g., nouns) frequency and semantics between the system summary and the reference. we also report a graph-based reference-free coherence metric (guinaudeau and strube, 2013) (ent graph), which measures the connectivity of summary sentences linked by entities, reflecting the coherence of topic transitions. for summary faithfulness, a recent model-based faithfulness metric, summac @xcite , is used. 7 finally, we show the maximum size of allocated gpu memory by each model during training. ## results we report results by all awesome variants and comparison models on govreport in of the two types of external memory mechanisms, attentive memory outperforms compressionbased memory on all metrics except the entity graph, which highlights the advantage of adaptively updating the stored context. meanwhile, directly concatenating salient content with the input yields higher rouge scores than injecting key-value vectors into the attention calculation, though the latter is less memory-intensive. we believe natural language-based augmentation better interleaves with the document segment, echoing the findings by prior work on using retrieval for question answering @xcite . importantly, under a strict gpu memory constraint, awesome with external memory mechanisms and global salient content augmentation achieves the best rouge scores among all models, while obtaining competitive results on other measures. though efficient attention models and page-sum can perform remarkably when given higher-se3: va has taken a number of actions to address deficiencies gao found in wait-time measurement and implementation of its scheduling policy. for wait-time measurement, these actions included changes to the wait-time measurement definitions, provision and documentation of scheduler training, and improved oversight through audits, all of which have been in a state of flux for the past 6 years. on july 12, 2019, va provided gao additional updates on efforts to implement gao's related recommendations. awesome: gao recommended that va either clarify its scheduling policy to better define the desired date, or identify clearer wait-time measures that are not subject to interpretation and prone to scheduler error. va concurred with the recommendation, which gao has identified as among those recommendations that warrant priority attention. va has taken a number of actions to address gao's recommendations regarding deficiencies gao found in wait-time measurement and implementation of its scheduling policy. on qmsum ( we also find that the extractor performs poorly on qmsum, leading to degraded rouge-2 result after augmenting our model with the extracted salient content. specifically, the f1 score of the extractor on the test set is only 1.29, as opposed to 27.85 on govreport. this trend is similarly observed on summscreen ( on arxiv, models that use efficient attentions obtain the higher rouge scores, because truncating arxiv documents has little effect on summary generation-arxiv articles have the most uneven distributions of salient content, where only about 10% of new salient bigrams are located in the second halves of the documents @xcite . finally, experiments on booksum show that the divide-and-conquer method produces better summaries for long novels, while our method can further boost its performance ( among all datasets, global salient content augmentation performs better on datasets rich in knowledge-dense factual statements, such as gov-report and arxiv. we observe that the majority of sentences extracted from govreport and arxiv are standalone statements comprehensible with little to no contextual support. by contrast, sentences extracted from other datasets typically demand integration with their original context for full clarity. our global salient content augmentation mechanism leverages these extracted sentences without their own context, which reduces its effectiveness on datasets such as summscreen and qmsum. we will explore methods that allow contextualization of global salient content in future work. more experimental findings are presented in appendix b. notably, with the same input length, awesome still achieve competitive performance ( human evaluation. we ask three fluent english speakers that have extensive nlp data annotation experience to examine the outputs by blockattn, se3, and awesome using attentive memory and text concatenation for content augmentation on govreport. 25 govreport documents are randomly selected, foot_8 each summarized by all three systems. outputs from different systems are randomly shuffled and displayed. for each summary sentence, the annotators give binary labels on where the sentence is coherent-it uses natural transitions to logically connect with the previous content, and does not contradict any prior statement. the annotators also compare each summary sentence with the document and check if it is faithful, i.e., it can be verified and entailed from the document. we further ask the annotators to rank the summaries generated by the three systems based on their informativeness-how well the summary captures the salient content of the document. as seen in figure 2 , awesome's summaries are rated by human judges to be more coherent, faithful, and informative than se3, again evidencing the importance of incorporating global context. though blockattn produces the most coherent summaries, it has more faithfulness errors and is less informative, due to document truncation under a constrained memory. ## conclusion we present awesome for summarizing long documents in a memory-constrained setting. based on the divide-and-conquer strategy, awesome uses two mechanisms to gather global context and improve summary quality. first, external memories on the encoder and decoder are employed to track previously read document content and the corresponding summaries. second, the encoder is informed of global salient content predicted by an extractor via text or representation concatenation. on five summarization datasets, awesome generates summaries with better informativeness, faithfulness, and coherence than a baseline divideand-conquer system. under the same memory constraint, awesome outperforms competitive models that leverage efficient attentions or dynamic extraction to preserve global context, highlighting its effectiveness in supplying global context.