UPER : Boosting Multi-Document Summarization with an Unsupervised Prompt-based Extractor (2022)

Abstract: Multi-Document Summarization (MDS) commonly employs the 2-stage extract-then-abstract paradigm, which first extracts a relatively short meta-document, then feeds it into the deep neural networks to generate an abstract. Previous work usually takes the ROUGE score as the label for training a scoring model to evaluate source documents. However, the trained scoring model is prone to under-fitting for low-resource settings, as it relies on the training data. To extract documents effectively, we construct prompting templates that invoke the underlying knowledge in Pre-trained Language Model (PLM) to calculate the document and keyword’s perplexity, which can assess the document’s semantic salience. Our unsupervised approach can be applied as a plug-in to boost other metrics for evaluating a document’s salience, thus improving the subsequent abstract generation. We get positive results on 2 MDS datasets, 2 data settings, and 2 abstractive backbone models, showing our method’s effectiveness. Our code is available at https://github.com/THU-KEG/UPER

## introduction multi-document summarization (mds) aims to generate a summary from multiple source articles @xcite . the input text in mds can be overlong and therefore contain much noisy information @xcite . the goal of mds is to reduce the long input and extract salient information @xcite @xcite . some previous works @xcite ) utilise sparse attention to handle the long input problem in mds. many others tackle this problem by an extract-then-abstract paradigm @xcite , which first extracts the salient information in source documents to form a meta-document with a preset length then generates the summary. ## method the key challenge of the extractive stage is how to model the objective function p (d ′ |k, d), which evaluates each document and retrieves those related to the keywords. this leads to the following question: what is a proper metric for modeling it? based on observations in section 2.2, we propose a criteria for modeling p (d ′ |k, d): (1) the metric ought to be unsupervised which can avoid the data dependence; (2) the metric can model both lexical and semantic salience; (3) the metric should doc1: " distribution and field identification of philippine birds of prey : 1 . philippine hawkeagle ( spizaetus philippensis ) and changeable hawk eagle ( spizaetus cirrhatus ) "(pdf). doc2: distribution : indomalayan . endemic to luzon island , philippines . more . . . keyword: philippine hawk -eagle perplexity score ... statistical score ... ... input regressive score result1: 0.12 tf-idf1: 0.186 result2: 0.07 tf-idf2: 0 perplexity1: 122 perplexity2: 68 ... be effective for improving the abstractive stage to diminish the two-stage gap. ## related work multi-document summarization (mds) aims to generate an abstract for the related documents collected from referred websites or search engines. this task is commonly regarded as a two-stage problem @xcite . the extractive stage selects high-quality relevant texts as sources, and the abstractive stage summarizes them into an abstract of required length. prior work mostly focuses on improving the latter abstrative stage with various techniques like topic information @xcite , graph representation @xcite and attention @xcite . the exploration of the extractive stage is limited to a few methods like tf-idf @xcite and rouge scorer @xcite , and they both focus on the token-level lexical similarity, while we take the semantic salience into consideration, which suppresses the intrinsic noise in the corpus. automatic evaluation metrics are vital for the extractive stage of multi-document summarization task, which can select source documents. they can be divided into two classes: referenced and reference-free. referenced metrics usually focus on the lexical overlap @xcite or embedding similarity @xcite between the document and the reference summary. while referencefree metrics usually evaluate the document without reference summary using perplexity @xcite or aspects evaluation @xcite . our method is inspired by a referenced metric proposed by @xcite . they train a scoring model to predict the perplexity of the sequence formed by concatenating the reference summary with the input document. in section 3.2, we find that the pretrained language model like gpt can be utilized to evaluate documents' semantic salience without training, so we propose our reference-free metric uper. ## conclusion this work investigates the extractive stage of the multi-document summarization task. we propose a simple but effective approach uper to model the semantic contextual salience and combine the lexical token-level similarity to extract the input documents. uper utilizes unsupervised prompts to take advantage of prior knowledge distributed in plms so that we can convert our extraction task to plm's original perplexity calculation task. in our future work, we will extend our framework to single-document summarization and explore the application of prompt-based methods in the supervised learning scenario.