PELMS : Pre-training for Effective Low-Shot Multi-Document Summarization (2024)

Abstract: We investigate pre-training techniques for abstractive multi-document summarization (MDS), which is much less studied than summarizing single documents. Though recent work has demonstrated the effectiveness of highlighting information salience for pre-training strategy design, they struggle to generate abstractive and reflective summaries, which are critical properties for MDS. To this end, we present **PELMS**, a pre-trained model that uses pre-training objectives based on semantic coherence heuristics and faithfulness constraints together with unlabeled multi-document inputs, to promote the generation of concise, fluent, and faithful summaries. To support the training of PELMS, we compile **MultiPT**, a multi-document pre-training corpus containing over 93 million documents to form more than 3million unlabeled topic-centric document clusters, covering diverse genres such as product reviews, news, and general knowledge. We perform extensive evaluation of PELMS in low-shot settings on a wide range of MDS datasets. Our approach consistently outperforms competitive comparisons with respect to overall informativeness, abstractiveness, coherence, and faithfulness, and with minimal fine-tuning can match performance of language models at a much larger scale (e.g., GPT-4).

## introduction abstractive multi-document summarization (mds) aims to generate coherent and concise summaries from sets of related documents. while transformerbased models have excelled in single-document summarization @xcite , their application to mds often results in suboptimal crossdocument salience detection and information aggregation. mds-specific pre-trained models like primera @xcite and centrum @xcite use information frequency to improve salience estimation, however, the quality of such pre-training objectives is often reliant on brittle topic alignment methods (such as n-gram overlap, or cross-document entity linking) which struggle to generalize to an open set of domains, yielding outputs with with subpar summary informativeness. existing mds methods also typically overlook multi-document faithfulness, generating outputs that poorly represent the full input, particularly in domains requiring cross-document synthesis. similarly, summary coherence is inadequately addressed, with most gap-sentence generation (gsg) style objectives simply denoising masked phrases in their original ordering to form the training target; these approaches may work well in conventional single-document settings, however they can introduce positional biases @xcite in the multi-document setting due to the arbitrary ordering of documents within the input, impacting both coherence and informativeness. while recent powerful large language models (llms) have demonstrated strong performance on tasks including summarizaton, they do not fully address the need for efficient, pre-trained models for mds, particularly for common real-world scenarios with domain-specific requirements, regulatory and data privacy concerns, or tight computational constraints. with these considerations in mind, we seek to rapidly enable proficient mds performance, generating high-quality summaries with minimal labeled data and/or compute requirements. we propose pelms, a method of pretraining for effective low-shot multi-document summarization that promotes informativeness over a diverse range of text genres, ranging from consumer and editorial opinion, to news articles, and to scientific papers, while also improving abstractiveness, faithfulness, and coherence of summaries. 1 our method leverages semantic clustering for sentence-level salience detection, overcoming the limitations of previous approaches reliant on lexical similarity. during pre-training, we generate summaries that balance topical salience with source consistency, while also maintaining strong abstractiveness and coherence by removing and intentionally ordering target sentences, a departure from traditional gsg methods. to support mds pretraining, we introduce multipt, a comprehensive dataset with over 3 million topic-aligned document clusters from more than 93 million documents. we evaluate pelms across six tasks, including the newly introduced metatomatoes dataset, which presents a complex task of meta-summarizing diverse movie reviews. our results demonstrate the strong mds performance of pelms, particularly in low-shot settings, highlighting the alignment between our pre-training strategy and multi-document summarzation. concretely, n-gram novelty [0, 29.8] rouge-g [8, 16.7] bertscore [39, 59.2] discoscore [55, 92.4] mdsummac [5, 19.7] pelms primera pegasus-x centrum qamden • in supervised scenarios, pelms outperforms existing models in rouge and bertscore, while striking the best balance of faithfulness, abstractiveness, and coherence. we find fullparameter training improves informativeness and abstractiveness, whereas adapter-based training best maintains input faithfulness. remarkably, we find pelms can match gpt-3.5 and gpt-4.0 @xcite performance with as few as 16 and 64 training examples, respectively. ## pelms pre-training technique the pelms pre-training strategy for multidocument summarization is outlined in figure 3 . briefly, we rely on a cluster-then-select pre-training objective to generate data for training transformer models. we then follow previous work by using a gap sentence generation-style objective to form pre-training targets, but instead of masking, we remove the sentences to improve abstractiveness. our key contributions are: 1) improved ranking and selection of target sentence candidates to encourage summary informativeness and faithfulness, and 2) injecting coherence constraints during formulation of the gsg target. our technique consists of the following three steps: ## pre-training details we pre-train a new mds model with the pelms technique, forming pre-training examples from the unlabeled document clusters in our multipt corpora. below we overview multipt and the pretraining architecture used for training pelms. see appendix a.2 for full pre-training details. multipt pre-training dataset mds is designed to support lengthy multi-document inputs, yet there are few available data sources for large-scale unlabeled multi-documents for pre-training. notably, newshead @xcite has been used, containing clusters of news articles. however, it is limited in magnitude for a pre-training dataset, containing only 370k document clusters. extending on this, we compile multipt, a new multi-document dataset comprised of over 3 million document clusters from public data sources. it contains a wide diversity of genres (including news, general knowledge, and opinionated content), and covers a broad range of inputs with respect to document lengths and cluster sizes. base model as done in primera, we use the popular sparse-attention long-input longformer encoder decoder @xcite as our base architecture, initializing from the 464m led-large base model. we follow primera in inserting a @xmath0. ## results we perform a rigorous comparison of pelms and competitive baseline models in both zero-shot and supervised settings (using both full supervision and peft adapter training). appendix b outlines the full details of our zero-shot and supervised evaluation configuration. ## further analyses comparison with llms we include comparison of mds models versus openai gpt-3. @xcite in pre-training ablation we investigate the individual benefits of both our multipt data and pelms pre-training objective on other models ( ## conclusion we introduce pelms, a novel new multi-document summarization pre-training method. pelms leverages semantic topic clustering to improve summary informativeness and uses coherence and faithfulness constraints during pre-training target formulation to produce concise, fluent, and faithful summaries. both automatic evaluation and human evaluation demonstrate that pelms achieves consistent improvements over competitive baseline pretrained models, yielding especially strong performance in low-shot and parameter-efficient training settings.