PRIMERA : Pyramid-based Masked Sentence Pre-training for Multi-document Summarization (2022)

Abstract: We introduce PRIMERA, a pre-trained model for multi-document representation with a focus on summarization that reduces the need for dataset-specific architectures and large amounts of fine-tuning labeled data. PRIMERA uses our newly proposed pre-training objective designed to teach the model to connect and aggregate information across documents. It also uses efficient encoder-decoder transformers to simplify the processing of concatenated input documents. With extensive experiments on 6 multi-document summarization datasets from 3 different domains on zero-shot, few-shot and full-supervised settings, PRIMERA outperforms current state-of-the-art dataset-specific and pre-trained models on most of these settings with large margins.

## introduction multi-document summarization is the task of generating a summary from a cluster of related documents. state-of-the-art approaches to multi-document summarization are primarily either graph-based @xcite @xcite , leveraging graph neural networks to connect information between the documents, or hierarchical @xcite @xcite , building intermediate representations of individual documents and then aggregating information across. while effective, these models either require domain-specific additional information e.g. abstract meaning representation @xcite , or discourse graphs @xcite , or use dataset-specific, customized architectures, making it difficult to leverage pretrained language models. simultaneously, recent pretrained language models (typically encoder-decoder transformers) have shown the advantages of pretraining and transfer learning for generation and summarization @xcite @xcite ). yet, existing pretrained models either use single-document pretraining objectives or use encoder-only models that do not work for generation tasks like summarization (e.g., cdlm, @xcite . therefore, we argue that these pretrained models are not necessarily the best fit for multi-document summarization. alternatively, we propose a simple pretraining approach for multi-document summarization, reducing the need for dataset-specific architectures and large fine-tuning labeled data (see figure 1 to compare with other pretrained models). our method is designed to teach the model to identify and aggregate salient information across a "cluster" of related documents during pretraining. specifically, our approach uses the gap sentence generation objective (gsg) @xcite , i.e. masking out several sentences from the input document, and recovering them in order in the decoder. we propose a novel strategy for gsg sentence masking which we call, entity pyramid, inspired by the pyramid evaluation method @xcite . with entity pyramid, we mask salient sentences in the entire cluster then train the model to generate them, encouraging it to find important information across documents and aggregate it in one summary. we conduct extensive experiments on 6 multidocument summarization datasets from 3 different domains. we show that despite its simplic- longformer encoder decoder (led) doc 1 doc 2 doc 3 masked @xmath0... wildfires continued ... [sent mask] <doc-@xmath1... </@xmath2... . in colorado '@xmath3... sent 1 sent 2 tokens with global attention tokens with local attention documents are separated @xmath4. other tokens except @xmath5. selected sentences are replaced with a special [sent mask] token the model is trained to generate the masked sentences ity, primera achieves superior performance compared with prior state-of-the-art pretrained models, as well as dataset-specific models in both few-shot and full fine-tuning settings. primera performs particularly strong in zero-and few-shot settings, significantly outperforming prior state-of-the-art up to 5 rouge-1 points with as few as 10 examples. our contributions are summarized below: 1. we release primera, the first pretrained generation model for multi-document inputs with focus on summarization. ## model in this section, we discuss our proposed model primera, a new pretrained general model for multi-document summarization. unlike prior work, primera minimizes dataset-specific modeling by simply concatenating a set of documents and processing them with a general efficient encoderdecoder transformer model ( §2.1). the underlying transformer model is pretrained on an unlabeled multi-document dataset, with a new entity-based sentence masking objective to capture the salient information within a set of related documents ( §2.2). ## experiment goals we aim to answer the following questions: • q1: how does primera perform, compared with existing pre-trained generation models in zeroand few-shot settings? see §4.2. ## ablation study we conduct ablation studies on the multi-news dataset in few-shot setting, to validate the contribution of each component in our pretrained models. input structure: in figure 6 (a) we observe the effectiveness of both pretraining and the input structure (<doc-@xmath6. sentence masking strategy: to isolate the effect of our proposed pretraining approach, we compare with a model with exactly the same architecture when pretrained on the same amount of data but using the pegasus @xcite ) masking strategy instead of ours. in other words, we keep all the other settings the same (e.g., data, length limit of input and output, pretraining dataset, input structure, as well as the separators) and only modify the pretraining masking strategy. we run the same experiments under zero-/few-shot scenar-ios on the multi-news dataset as in §4.2, and the results are shown in figure 6 (b) . the model pretrained with our entity pyramid strategy shows a clear improvement under few-shot scenarios. ## human evaluation we also conduct human evaluations to validate the effectiveness of primera on duc2007 and tac2008 @xcite datasets in the few-shot setting (10/10/20 examples for train/valid/test). both datasets consist of clusters of news articles, and duc2007 contains longer inputs (25 v.s. 10 documents/cluster) and summaries (250 v.s. 100 words). since the goal of our method is to enable the model to better aggregate information across documents, we evaluate the content quality of the generated summaries following the original pyramid human evaluation framework @xcite . in addition, we also evaluate the fluency of generated summaries following the duc guidelines. 12 settings three annotators foot_8 are hired to do both pyramid evaluation and fluency evaluation, they harmonize the standards on one of the examples. specifically, for each data example, we provide three anonymized system generated summaries, along with a list of scus. the annotators are asked to find all the covered scus for each summary, and score the fluency in terms of grammaticality, referential clarity and structure & coherence, according to duc human evaluation guidelines, with a scale 1-5 (worst to best). they are also suggested to make comparison between three generated summaries into consideration when scoring the fluency. to control for the ordering effect of the given summaries, we re-order the three summaries for each data example, and ensure the chance of their appearance in different order is the same (e.g. bart appears as summary a for 7 times, b for 7 times and c for 6 times for both datasets). the instruction for human annotation can be found in figure 7 and figure 8 in the appendix. annotators were aware that annotations will be used solely for computing aggregate human evaluation metrics and reporting in the scientific paper. ## conclusion and future work in this paper, we present primera a pre-trained model for multi-document summarization. unlike prior work, primera minimizes dataset-specific modeling by using a longformer model pretrained with a novel entity-based sentence masking objective. the pretraining objective is designed to help the model connect and aggregate information across input documents. primera outperforms prior state-of-the-art pre-trained and datasetspecific models on 6 summarization datasets from 3 different domains, on zero, few-shot, and full fine-tuning setting. primera's top performance is also revealed by human evaluation. in zero-shot setting, we can only control the output length of generated summaries at inference time by specifying a length limit during decoding. exploring a controllable generator in which the desired length can be injected as part of the input is a natural future direction. besides the summarization task, we would like to explore using primera for other generation tasks with multiple documents as input, like multi-hop question answering.