Improving Multi-Stage Long Document Summarization with Enhanced Coarse Summarizer (2023)

Abstract: Multi-stage long document summarization, which splits a long document as multiple segments and each of which is used to generate a coarse summary in multiple stage, and then the final summary is produced using the last coarse summary, is a flexible approach to capture salient information from the long document. Even if the coarse summary affects the final summary, however, the coarse summarizer in the existing multi-stage summarization is coarsely trained using data segments that are not useful to generate the final summary. In this paper, we propose a novel method for multi-stage long document summarization. The proposed method first generates new segment pairs, ensuring that all of them are relevant to generating the final summary. We then incorporate contrastive learning into the training of the coarse summarizer, which tries to maximize the similarities between source segments and the target summary during training. Through extensive experiments on six long document summarization datasets, we demonstrate that our proposed method not only enhances the existing multi-stage long document summarization approach, but also achieves performance comparable to state-of-the-art methods, including those utilizing large language models for long document summarization.

## introduction long document summarization aims to compress a long document, such as meeting minutes, reports, and scientific articles, into a concise text that captures salient information. since the number of tokens in a long document usually exceeds the limit of the summarization models, various summarization approaches @xcite @xcite @xcite to deal with long document has proposed. one promising approach among the long document summarization approaches is the multi-stage split-then-summarization approach @xcite . it first splits the long document into source segments and each of which is used to generate a coarse summary. after splitting into source segments and generating a coarse summary in multiple stages, it produces the final summary using the last coarse summaries. this approach offers flexibility in processing documents of arbitrary length by adjusting the number of stages and has achieved state-of-the-art performance on several long document summarization benchmark datasets. however, there is still room for improvement in terms of the quality of the coarse summary. it should be noted that the quality of the final summary depends on the quality of the coarse summaries. in the previous multi-stage split-thensummarization approach, @xcite construct the training data using pairs of long documents and target summaries for the coarse summarizer because there is no official data for the coarse summarizer. that is, it first splits the long document and the target summary into segments, respectively. each document segment is then aligned with a subset of the target segments, maximizing the rouge-1 score between the document segment and the subset. the resulting aligned pairs are used to train the coarse summarizer. even though some document segments may be irrelevant to the target summary, aligned pairs derived from these segments are employed to train the coarse summarizer. it results in the generation of coarse summaries that may contain some noise. figure 1 shows an example of the construction of the training data for the coarse summarizer, where the dotted line indicates the alignment of one document segment with a subset of the target segments. as all document segments are aligned and used to train the coarse summarizer, the coarse summarizer may generate low quality and excessively verbose summaries. in this paper, we propose a novel approach suitable for multi-stage summarization. the proposed method first generates aligned pairs that are relevant to the generation of the final summary. unlike the previous alignment discussed, the proposed method executes an alignment in a reverse direction. specifically, the proposed method aligns each target segment with a document segment. by aligning in this reverse direction, we ensure that only relevant document segments are included in the training data for the coarse summarizer. furthermore, the proposed method incorporates a sub-summary generation contrastive objective @xcite during training of the coarse summarizer to explicitly model the similarity between the target segment and the document segment. this addition of contrastive objective encourages the coarse summarizer to focus on relevant document segments and target segments and contributes to further improving the summarization quality of the coarse summarizer. we conduct extensive experiments on six long document summarization datasets to show the superiority of the proposed method. we also compare the proposed method with large large language models-based long document summarization. experimental results imply that the proposed method contributes to enhancing the efficiency and effectiveness of the multi-stage long document summarization approach compared to the existing method foot_0 . ## multi-stage long document summarization let @xmath0, t i )} n @xmath1, where s i is the i-th long document and t i is its corresponding summary. the multistage long document summarization approach segments the long document and then summarizes the segmented text in multiple stages because the number of tokens in s i exceeds the limit of the summarizer. it consists of two stages: c coarse stages and one fine-grained stage. in each coarse stage, an input document is divided into document segments, and then a coarse summarizer generates coarse summaries from the document segments. in the fine-grained stage, a fine-grained summarizer generates the final summary from the last generated coarse summary. let k be the number of the maximum input tokens of the summarizer. s i is divided into multiple segments, each with a length of fewer than k tokens. that is, s @xmath2,. . . , s in i }, where n i is the number of segments of s i . similarly, the target summary t @xmath3,. . . , t im i } is also divided into multiple segments, usually split into separate sen-tences. that is, t ij and m i are the j-th sentence and the number of sentences in t i , respectively. to generate training data for the coarse summarizer, @xcite adopt the rouge-based greedy target alignment function, aligning each document segment s ij with a subset of t i such that the rouge-1 score between s ij and the subset is maximized. the training data for the coarse summarizer in the k-th stage is constructed as follows: here, align source (•, •) is a function to align each document segment with a subset of target segments. since the alignment is executed for each source segment, all document segments are contained in d k coarse-s . the k-th coarse summarizer is trained with d k coarse-s to minimize the negative log-likelihood (nll) l nll between the word distributions predicted by the summarizer and the target segments. after training the coarse summarizer, a coarse summary is obtained using the trained summarizer from each document segment s ij . all n i coarse summaries are then concatenated to form a new input for the next stage. the target summary for the next stage is copied from the original target summary. it is worth noting that the number of coarse stages is estimated based on the length of the long document and the characteristics of the summarizer. further details can be found in @xcite and section 4.3. in the fine-grained stage, the coarse summaries from the c-th coarse stage are concatenated and used as input for the fine-grained stage. since the number of tokens in the input is shorter than k, a fine-grained summarizer can be modeled similarly to a well-known vanilla abstractive summarizer @xcite . this means that the fine-grained summarizer is trained on the dataset from the last coarse stage and produces the final summary using the last coarse summaries. ## improving multi-stage summarization with enhanced coarse summarizer this paper presents a new approach to multi-stage summarization. the proposed method generates new aligned pairs that include only relevant document segments because the greedy target alignment function @xcite generates some pairs that contain irrelevant document segments. in addition, the proposed method incorporates a contrastive learning into the training of the coarse summarizer that a document segment and the corresponding target summary should convey the same meaning, which is not modeled explicitly by the nll loss @xcite . ## conclusion in this paper, we have demonstrated the effectiveness of enhancing the coarse summarizer in the multi-stage split-then-summarization approach. to enhance the coarser summarizer, the proposed method focuses on constructing target-focused aligned pairs, ensuring that only relevant source segments are included. then, the proposed method incorporates contrastive learning into the training of the coarse summarizer to reinforce the conveyance of the similar meaning between source and target segments. experimental results on six long document summarization datasets show that the proposed method helps improve the performance of the multi-stage long document summarization approach.