A Hierarchical Encoding-Decoding Scheme for Abstractive Multi-document Summarization (2023)

Abstract: Pre-trained language models (PLMs) have achieved outstanding achievements in abstractive single-document summarization (SDS). However, such benefits may not fully extend to multi-document summarization (MDS), where the handling of cross-document information is more complex. Previous works either design new MDS architectures or apply PLMs bluntly with concatenated source documents as a reformulated SDS task. While the former does not utilize previous pre-training efforts and may not generalize well across different domains, the latter may not sufficiently attend to the intricate cross-document relationships unique to MDS tasks. Instead, we enforce hierarchy on both the encoder and decoder to better utilize a PLM to facilitate multi-document interactions for the MDS task. Across 10 MDS benchmarks from various domains, our method outperforms or is competitive with the previous best models, including those with additional MDS pre-training or with more parameters. It outperforms its corresponding PLM backbone by up to 3 Rouge-L and is favored by humans.

## introduction multi-document summarization (mds) was first proposed by @xcite , framed as a task of producing a single summary using a set of related documents, and has been extensively studied @xcite @xcite @xcite @xcite . mds is inherently much more complex than single-document summarization (sds). specifically, unlike sds which requires the extraction of crucial information in a single article, mds requires handling not only the contradictions from multiple sources but also the repetitions and larger amounts of trivial information across documents @xcite @xcite @xcite @xcite . while mds could be superficially converted to sds by concatenating multiple documents into a pseudo-single document @xcite @xcite , the irregular and complex multi-document information remains. as a result, it continues to pose challenges due to its large difference from the mostly coherent plm pretraining data of continuous text segments @xcite . on the other hand, many works design specialized mds architectures @xcite @xcite . however, they require large supervised mds datasets on the scale of 100k training data @xcite @xcite and may not generalize to unseen domains. neither do they utilize the capabilities of large plms @xcite @xcite that were pre-trained on coherent data. given that mds data are scarce and costly to obtain @xcite , we propose a simple hierarchical encoding-decoding scheme to effectively fine-tune a plm on smaller-scale mds datasets ranging from about 2k to 45k training samples. we aim to better adapt the plm's knowledge gained during pre-training (which is more useful to sds due to a smaller data shift) for mds finetuning in both the encoder and the decoder. in the encoder, we preserve the original attention mechanism within the same document while re-purposing "<@xmath0. this approach ensures that intradocument tokens can be well-represented using the pre-training knowledge, while the document will still interact with one other on a higher level. in the cross-attention layers of the decoder, we impose document-level importance scaling on the attention weights, which allows each document to influence the output tokens differently but more appropriately. as a result, our method can still make use of the general language modeling capabilities of plms, while learning to handle the complex crossdocument information during fine-tuning. we conduct experiments on various mds datasets @xcite @xcite @xcite across a vast range of domains, namely, news, scientific literature, movie critics, peer reviews, and specific wikipedia topics. our experimental results show that our hierarchical scheme outperforms strong state-of-the-art models (including those with additional mds pre-training or larger model sizes). it also consistently improves the plm backbone on all 10 datasets by up to 3 rouge-l and, on the manually inspected datasets, is preferred by humans. in addition, our detailed attention, content analyses, and ablation reveal that our method ensures more coherent encoder representations for intra-document tokens, as well as obtains the wider cross-attention coverage of the different documents during decoding. in this manner, our method sufficiently utilizes the pre-training knowledge for encoding, while encouraging the decoding process to consider all documents by more proper weights, rather than over-focusing a few documents and ignoring the rest. plms for mds plms @xcite @xcite have achieved state-of-the-art performances for sds. some works have directly used these plms for mds by concatenating multiple documents @xcite @xcite , including the works that focus on longer context windows @xcite . however, concatenating the content of multiple documents together prevents the plms from distinguishing or processing the contents differently. by essentially processing the mds inputs as if they were a single document, the plms may not handle the cross-document information well. an alternative is to conduct summarization through multiple stages @xcite @xcite @xcite @xcite , where salient contents are first extracted or summarized from multiple documents, then passed to a plm for the final abstraction stage. however, the inputs to the plm now consist of groups of incoherent sentences or phrases stitched together from multiple sources, presenting a significant distributional shift from the plms' original pre-training corpora. our method is different from the previous notable works @xcite in both that we make novel use of the start-of-document tokens as a document-level representation, as well as further utilizing these tokens during our hierarchical decoding scheme to enable cross-document interactions. ## preliminaries and notations we focus on adapting encoder-decoder plms for the mds task. the encoder-decoder transformer @xcite is a common underlying architecture for plms frequently used for summarization tasks @xcite @xcite . the encoder uses selfattention to encode the source tokens, whereas the decoder uses cross-attention with source tokens and self-attention with the generated tokens for decoding. our method targets the attention involving the source tokens, i.e., the self-attention in the encoder and the cross-attention in the decoder. formally, the encoder updates the representation of each source token across all layers. the representation of the i th token in the j th layer, h (j) i , can be computed as a weighted sum of the hidden states of the context tokens in the previous layer: the attention scores @xmath1, ..., a k ) are computed with the query (q) and key matrices (k): for the decoder, cross-attention is calculated between the last generated token and the source tokens. the weights calculations are similar to equation (2) and equation (3) except that the q matrix takes in the representation of the last generated token (after decoder self-attention), and the k matrix takes in the representations of the source tokens from the last encoder layer. lastly, for multi-document encoding, we follow the common approach @xcite @xcite to feed multiple inputs into the model by concatenating one document after another. with n documents, the input sequence can be represented as @xmath2, d 1 , ..., d n }, where d @xmath3,0 , x n,1 , ...} are the input tokens from the n th document. ## method to better leverage the strong language modeling capabilities of plms, we propose an encodingdecoding scheme to better equip the plms for the mds setting. our design fulfills the following requirements: (1) it leverages the language modeling capability of plms gained during pre-training; and for the first requirement, we preserve the token interactions within the same document for both the encoder and decoder. for the second requirement, we use the plm's "<@xmath4, then make further use of these tokens for hierarchical attention scaling in the decoder. ## conclusion in this paper, we work on the abstractive multidocument summarization (mds) task. we propose a hierarchical encoding-decoding scheme, which allows effective fine-tuning of the plm on a specific mds task dataset without any new parameters. our proposed scheme makes novel use of global tokens for document-level interactions in both the encoder and decoder. it can leverage the generalizing capability of plm across a wide variety of domains. evaluation results on 10 mds datasets show that our approach consistently outperforms the previous best models and our plm backbone.