Automatic Summarization of Long Documents (2024)

Abstract: A vast amount of textual data is added to the internet daily, making utilization and interpretation of such data difficult and cumbersome. As a result, automatic text summarization is crucial for extracting relevant information, saving precious reading time. Although many transformer-based models excel in summarization, they are constrained by their input size, preventing them from processing texts longer than their context size. This study introduces three novel algorithms that allow any LLM to efficiently overcome its input size limitation, effectively utilizing its full potential without any architectural modifications. We test our algorithms on texts with more than 70,000 words, and our experiments show a significant increase in BERTScore with competitive ROUGE scores.

## introduction due to the ever-increasing amount of textual data available online, document summarization has become crucial for the efficient and accurate extraction of relevant information. large language models (llms) based on the transformer architecture @xcite have shown outstanding abilities in many nlp tasks, including document summarization @xcite . recent developments have demonstrated remarkable improvements in the relevancy and coherence of summaries generated by such llms. however, long document summarization, which involves removing redundancies and makes reading long texts efficient, remains a major challenge. one of the significant limitations in the transformer architecture is limited context size, stemming from the quadratic memory and computational complexity of the attention mechanism @xcite . this constraint hinders extracting relevant information from extensive texts where summarization is valuable to overcome the time, effort, and interpretive issues posed by complex and large documents. we experiment with three novel approaches to address the input size limitations of transformers. the methods introduced do not include any architectural modifications to the model used and can be incorporated into any existing pipeline. we believe that these methods can effectively utilize the full potential of any existing llm by capturing information from crucial aspects of the document. though our experiments only include the task of summarization, we believe that our methods can be applied to nlp tasks that require processing long texts. we start by providing the problem statement (section 2) and discussing related work (section 3) to gain insights into the problem and the state-ofthe-art solutions. we then introduce the datasets (section 4) and methodology (section 5) used in our experiments. for evaluating our results, we present standard metrics (section 6) used in text summarization. we end the report by discussing our experimental findings (section 7) and potential future work (section 8) and concluding the study (section 9). ## problem statement our goal is to pre-process and distill a long document (with theoretically indefinite length) such that it fits within the context size of the model while retaining important information. in our experiments, we use documents with lengths of up to seventeen times the context size of the model and aim to reduce the summary length to about 400 words or less, preserving maximal salient information and coherence. ## related works there have been efforts to improve the efficiency of the attention mechanisms in transformers. @xcite introduce the longformer, which replaces the quadratic self-attention mechanism in the transformer architecture with a sliding window self-attention, resulting in a linear complexity with respect to the input size. to capture long-range dependencies, they include global attention at specific token positions. @xcite modify the encoder-decoder attention mechanism such that each attention head in the decoder attends to n/s h tokens in the input sequence, where n is the input length and s h is the number of heads. this method has a complexity of o(mn/s h ), where m is the length of the output sequence. @xcite introduce unlimiformer, which also modifies the encoder-decoder attention in a transformer. the attention heads in the decoder only attend to the tokens picked by their k-nearest-neighbor (knn) algorithm. the knn indices between the input tokens are created by the hidden states generated in the encoder. @xcite introduce the staggered block-local attention mechanism. in the block-local attention mechanism, the input sequence is divided into multiple non-overlapping blocks. tokens in a block attend only to the tokens in the same block. in staggered block-local attention, the blocks are staggered such that each token is in a different block in each head. other relevant approaches include videoagent, introduced by @xcite , an ai agent designed to answer a given question based on a long video. they achieve this by generating captions from multiple uniformly sampled frames from the video. these captions are used to answer the user's question. @xcite describe a novel algorithm to classify long chinese news into predefined categories. they form multiple groups of sentences based on a maximum token threshold in each group. these groups are then encoded using bert (bidirectional encoder representations from transformers) @xcite and passed through a 1d convolution layer for local feature extraction. this method is unique because a 1d convolution layer replaces the attention mechanism with linear time complexity. @xcite use positional interpolation to extend the context size of a pre-trained model. instead of the usual extrapolation of the positional embeddings, they downscale and force them into a range the model is trained on, hence interpolating in the pre-trained range. they claim that the model should use the positional embeddings on which it is trained. golia and kalita (2024) take a "divide and conquer" approach to address sequence length limitations in summarizing long meeting transcripts. they begin by segmenting the transcript and then use the bart (bidirectional and auto-regressive transformer) @xcite model to summarize each segment individually. these segment summaries are then recursively combined and summarized until a single summary remains. this method performs well with long documents but may take considerable time to converge due to repeated calls to the model. ## datasets this section briefly discusses and analyzes the datasets used in our experiments. ## methodology in this section, we introduce the three algorithms used in our experiments. segmenting a document into smaller, contiguous, and exhaustive parts. this is done by using a sentence tokenizer to separate sentences in the text and then merging them such that the number of words in each segment is more than the threshold, min_words, a hyperparameter in both methods. ## evaluation metrics the best way to evaluate generated natural language is by humans, but conducting human trials is expensive and time-consuming. due to this difficulty, we use automatic evaluation metrics to evaluate the quality of the generated summary given some reference summaries. @xcite review many such open-source and state-of-the-art metrics. the two that we use in our experiments are discussed below. these metrics are commonly used in published literature. @xcite introduces the recall-oriented understudy for gisting evaluation (rouge) metrics. the basic rouge-n metric is based on the fraction of overlaps of ideal or reference summaries with the candidate summary, hence being recall-oriented. his study concludes that rouge-n with @xmath0, rouge-l, rouge-w, and rouge-s work well for the summarization task. ## experimental findings we test our pipelines with the following models: bart fine-tuned on the cnn/daily mail dataset @xcite with a context size of 1024, longt5 @xcite , a variant of t5 (textto-text transfer transformer) @xcite , fine-tuned on the booksum dataset with a context size of 4096, and gpt-3.5 turbo @xcite with a context size of 4096. we compare our results with the state-of-the-art summarization models on the govreport dataset, including unlimiformer @xcite integrated with bart and primera @xcite , hepos @xcite , pegasus-x with staggered block-local attention @xcite , extended llama-7b with positional interpolation @xcite . we also compare our results with bigbird-pegasus @xcite on the bigpatent dataset. refer to we could not obtain the bertscores of our baselines, except for unlimiformer, due to the unavailability of code or computational limitations. ## future work to segment the document, we use a basic sentence tokenizer (nltk.sent_tokenize) with some modifications to control the minimum number of words in a segment. in our experiments, we find that segmentation is a crucial step in the distillation process and can greatly influence the output summary, indicating that good segmentation is vital for good distillation of text. ensuring the uniformity of the length of the segments while preserving coherence within a segment is also essential for better utiliza- method mean time taken central truncation 142.50 ms document skimming 155.42 ms skimming w/ post-625.17 ms sampling removal skimming w/ pre-3059.63 ms sampling removal summarization 2131.40 ms w/ extraction future work can also focus on extending the "summarization with keyword extraction" method (subsection 5.3). many potential ways exist to use the extracted keywords we do not touch upon. ## conclusion our experiments show that "document skimming with post-sampling removal" (subsection 5.2) performs well while being efficient. the "central truncation" method (subsection 5.1) also shows good results, which shows that simple methods can also be effective when dealing with long inputs. the last two methods, "skimming with presampling removal" (subsection 5.2) and "summarization with keyword extraction" (subsection 5.3), achieve the best results but are computationally expensive. our experiments show significant improvement in bertscore compared to unlimiformer @xcite on the govreport dataset, showing that our pipelines can efficiently utilize details in long texts. even though our rouge-2 scores are lower than the baselines, rouge-1 and rouge-l scores are competitive. since bertscore is better at capturing semantic similarity, we highlight the use of bertscore compared to rouge scores. hence, we contend that our pipelines can generate better summaries than the baselines with higher rouge scores. also, note that the models used in our experiments have smaller context sizes than the baselines, indicating that our algorithms have a greater potential if used with larger models.