S umm ^N : A Multi-Stage Summarization Framework for Long Input Dialogues and Documents (2022)

Abstract: Text summarization helps readers capture salient information from documents, news, interviews, and meetings. However, most state-of-the-art pretrained language models (LM) are unable to efficiently process long text for many summarization tasks. In this paper, we propose Summ ^N , a simple, flexible, and effective multi-stage framework for input texts that are longer than the maximum context length of typical pretrained LMs. Summ ^N first splits the data samples and generates a coarse summary in multiple stages and then produces the final fine-grained summary based on it. Our framework can process input text of arbitrary length by adjusting the number of stages while keeping the LM input size fixed. Moreover, it can deal with both single-source documents and dialogues, and it can be used on top of different backbone abstractive summarization models. To the best of our knowledge, Summ ^N is the first multi-stage split-then-summarize framework for long input summarization. Our experiments demonstrate that Summ ^N outperforms previous state-of-the-art methods by improving ROUGE scores on three long meeting summarization datasets AMI, ICSI, and QMSum, two long TV series datasets from SummScreen, and a long document summarization dataset GovReport. Our data and code are available at https://github.com/psunlpgroup/Summ-N .

## introduction abstractive summarization helps readers capture salient information from various sources such as documents, news, interviews, and meetings. previous work has primarily focused on short texts of news @xcite and short conversations @xcite . recently proposed longer dialogue and document summarization tasks @xcite @xcite pose challenges for current large pretrained language models due to the time and memory complexity of training, as well as limited input lengths these models can consume. a common method to handle long text reduces the input to a shorter one. this can be accomplished by truncating inputs @xcite or employing retrieve-then-summarize pipelines @xcite . however, these methods break the dependency of the context and decrease the number of tokens that the model can read, i.e., the receptive field of the model. the cutting-off model depends on the lead bias of the source text, while the retrieve-then-summarize models heavily rely on the independence of retrieved units (turns or sentences) which are usually scattered throughout the source text. another approach optimizes the attention mechanism in transformers to accommodate longer inputs by reducing the impact of quadratic complexity of the attention process using locality-sensitive hashing (lsh) attention @xcite and sinkhorn attention @xcite . additionally, hmnet @xcite and hat-bart @xcite use hierarchical self-attention to extend the input limitation of typical self-attention models. however, the simplified attention mechanism weakens the power of pretrained transformer models, e.g., hmnet is not pretrained on external large-scaled unsupervised datasets as bart did. in this paper, we propose summ n , a multi-stage framework for long dialogue and document summarization. figure 1 shows the structure of summ n . first, it divides each source text into segments so that each can be completely fed into the backbone abstractive summarization model. then, it matches each of them with the subset of target text using a rouge-based greedy algorithm. next, each stage generates a coarse summary for each segment and concatenates them together as the input to the input segments target target target target target match summarizer finetune inference summarizer fine-grained summary target segments coarse segments coarse summary source target data segmentation coarse summary generation fine-grained summary generation text and summary to form a pair of sample model to generate summaries original target source target split coarse stage fine-grained stage finetune inference next stage. after multiple stages of compression and summarization, the final stage produces a finegrained summary. the process expands the model context to the full reception field, meaning that the proposed model can read the full input no matter how long the input is. additionally, retrieve-thensummarize pipelines @xcite extract sentences individually, leading to the loss of the context information for understanding utterances. by contrast, summ n only cuts the source text at the end of each segment, so that the context of most sentences are retained. it does not assume lead bias because each part of the source is fully used. in addition, in each stage, it leverages a backbone abstractive summarization model to recursively generate the summaries. therefore, it enjoys the full power of the pretrained language models because the framework preserves the intact structure of transformers. summ n is flexible to inputs with different lengths by adjusting the number of stages. summ n can change the number of coarse stages according to the compression ratio between source and target, the input limit of the backbone model, and the input source length. we give the empirical formula to decide the number of needed stages for every tested dataset. our experiments show that rouge increases on all datasets when increasing the number of stages from one to the appropriate number. additionally, summ n is flexible because it can be applied to different backbone summarization models. for example, we found that the rouge scores increase sharply on the ami dataset when replacing the backbone bart model with t5 @xcite and pegasus @xcite . we conduct extensive experiments on long-input summarization datasets in multiple domains. the results demonstrate that the proposed model significantly outperforms previous state-of-the-art methods according to automatic and human evaluations on three long meeting summarization datasets (ami, icsi, qmsum) and one long tv series summarization dataset (summscreen). it also achieves state-of-the-art performance on a long document summarization dataset (govreport). these datasets include document summarization as well as both query-based and query-independent long dialogue summarization tasks. our contributions are: (1) we propose summ n , a simple, flexible, and effective framework for long dialogue and document summarization. to the best of our knowledge, summ n is the first multi-stage split-then-summarize framework to solve long text summarization tasks. (2) we evaluate summ n on both dialogue and document domains and improve the baseline model by a large margin. (3) we analyze and compare the proposed framework with baselines and discuss its merits in detail. the govreport dataset was recently introduced containing documents with more than 9000 words, thus greatly challenging the capabilities of current models such as pegasus @xcite , tlm @xcite , and bigbird @xcite . to handle this dataset, @xcite proposed head-wise positional strides to reduce the cost of the encoder-decoder attention. similarly, models such as longformer @xcite and reformer @xcite adjust attention mechanisms in transformers to consume longer inputs. however, these models sparsify the attention structure of the pretrained model to fit the longer source text. by contrast, summ n is able to maintain the full structure of various pretrained models. long dialogue summarization various models have also been proposed to handle long dialogue summarization. hmnet @xcite and hat-bart @xcite leverage a twolevel transformer-based model to obtain word level and sentence level representations. diallm @xcite , longformer-bart-arg @xcite use finetuning or data augmentation to incorporate the external knowledge to maintain the accuracy of lengthy input. different from these models, summ n is a framework without modifying the structure of the backbone attention model. ## method figure 1 shows the workflow of summ n . the workflow includes two types of stages, n coarse stages, and one fine-grained stage. coarse stages include the data segmentation and coarse summary generation, while the fine-grained stage directly generates the summary as the final result. besides, we have n + 1 separate models for each stage and each was separately trained. our experiments show that the performance drops if different stages share the parameters (section 4.2). summ n can adjust and compute the number of coarse stages n according to the stats of dataset and model. to formulate our task, we denote one sample of the source text as @xmath0, d 2 , • • • , d m }, where d i indicates one sentence in a document or one turn in a dialogue. for query-based summarization, there is also a query q. the goal is to generate a summary t , given d and the optional q. ## experiment setup we first list the datasets and metrics to evaluate the model. then, we introduce the backbone model and baselines for comparisons. finally, we present some implementation details. ## results and analysis we discuss the evaluation results and effects of each component of summ n in this section. ## conclusion in this paper, we propose summ n , a simple, flexible, and effective framework for long dialogue and document summarization. it consists of multiple coarse stages and one fine-grained stage to iteratively compress the long source input. it enjoys the full power of backbone models while ensuring the full receptive field of the summarization model. we evaluate the model on various datasets and improve the baselines by a large margin.