Open Domain Multi-document Summarization: A Comprehensive Study of Model Brittleness under Retrieval (2023)

Abstract: Multi-document summarization (MDS) assumes a set of topic-related documents are provided as input. In practice, this document set is not always available; it would need to be retrieved given an information need, i.e. a question or topic statement, a setting we dub “open-domain’ MDS. We study this more challenging setting by formalizing the task and bootstrapping it using existing datasets, retrievers and summarizers. Via extensive automatic and human evaluation, we determine: (1) state-of-the-art summarizers suffer large reductions in performance when applied to open-domain MDS, (2) additional training in the open-domain setting can reduce this sensitivity to imperfect retrieval, and (3) summarizers are insensitive to the retrieval of duplicate documents and the order of retrieved documents, but highly sensitive to other errors, like the retrieval of irrelevant documents. Based on our results, we provide practical guidelines to enable future work on open-domain MDS, e.g. how to choose the number of retrieved documents to summarize. Our results suggest that new retrieval and summarization methods and annotated resources for training and evaluation are necessary for further progress in the open-domain setting.

## introduction summarization is an nlp task that aims to generate accurate and coherent summaries for some given text automatically. multi-document summarization (mds) extends this task to provide multiple topicrelated documents as input, with the goal of summarizing salient information while avoiding redundancy. mds is a popular research objective with many proposed approaches @xcite @xcite @xcite @xcite @xcite and important applications, e.g. the summarization of news articles @xcite , scientific literature @xcite @xcite , and legal documents @xcite . existing mds task definitions, including queryfocused mds (see §3 for detailed comparison), assume a ground-truth, topic-related document set is provided at train and test time. this document set is often an artifact of the dataset curation process; in many practical settings, it is not available a priori and would need to be defined by an information need, expressed as a query. 2 documents relevant to the query would need to be retrieved from a large collection of (mostly irrelevant) documents and summarized (figure 1 ); a setting we refer to as open-domain mds. 3 because even state-of-the-art information retrieval (ir) methods are imperfect, errors, like the retrieval of irrelevant documents, will occur. it is an open question how existing summarizers behave under this more challenging but realistic setting. our major contributions are: based on our results, we provide detailed, practical guidelines for future work in open-domain mds. we release all code, model & data artifacts, and experimental results created during our investigation. in traditional mds, a model is given a set of topicrelated documents @xmath0, ..., d k } and must generate a summary s that accurately and coherently summarizes the information in d. such models are typically trained in a supervised fashion to minimize the difference between s and a (usually human-written) reference summary r. the goals (and evaluation) remain the same in open-domain mds, but instead of d, the inputs are a query q and a large index of documents d index , where |d index | ≫ |d|. the specifics of q, d index , and s will depend on the application or use-case. for example, in automatic literature review @xcite , q would be a research question or statement, e.g. "is massage therapy effective for people with musculoskeletal disorders?", d index would be a large corpus of scientific literature (e.g. pubmed), and s would be a literature review-style discussion, e.g. "massage therapy, as a stand-alone treatment, reduces pain and improves function compared to no treatment..." there are multiple ways to approach opendomain mds. we could think of q as a prompt to a large language model (llm) capable of incontext learning @xcite ; in which case we can view d index as the llms training data and retrieval as happening implicitly during inference. however, because all information is stored in the model's weights, this approach requires extremely large models, cannot produce summaries for events outside the training data, and does not provide provenance for the generated summaries. another approach is to introduce an explicit retrieval step over an external knowledge source, which we will refer to as "retrieve-then-summarize." it works as follows: a retriever ranks all documents in d index from most-to-least relevant given q. the top-k documents are input to a summarizer; k is a parameter that may be tuned for a particular use case. this approach has some desirable properties: previous attempts at open-domain @xcite proposed the wikisum dataset. given the title of a wikipedia article and a collection of non-wikipedia reference documents, the goal is to generate the first section of the article. the proposed extractive-abstractive approach resembles open-domain mds. however, the document index is small (10s to 100s of documents) and composed of relevant documents (references of the article plus ten pages of search results using the title as query). we study the more challenging and more general setting where the index is large (»10,000 documents, see in @xcite , a method similar to retrieve-then-summarize is proposed, using a pretrained dense passage retriever @xcite and t5 @xcite as summarizer. 4 the model is trained & evaluated on a dataset constructed from existing qmds datasets. this dataset is small (∼90 training examples) and does not appear to be publicly available. here, we conduct a large-scale analysis on multiple datasets from different domains (each consisting of thousands of ex-amples) and evaluate several of the top-performing multi-document summarizers currently available ( §4). we also extensively simulate document retrieval errors to probe their relative impact on summarization ( §7). together, this allows us to draw broader conclusions about open-domain mds and provide detailed practical advice for future work. open-domain qa our open-domain mds proposal mirrors a similar trend in question answering (qa). while earlier research focused on answering questions provided a text passage @xcite , the now predominant approach, open-domain qa (odqa), is to answer questions without this passage, usually by referencing an external knowledge source (e.g. wikipedia). even broader are knowledge-intensive (ki) language tasks @xcite , which include odqa but also, for example, fact-checking. ki tasks are commonly approached with a retrievethen-generate framework @xcite @xcite . retrieve-thensummarize is similar, except that the outputs are, on average, much longer and tend to be less extractive than the outputs of ki language tasks like odqa. ## bootstrapping open-domain mds since no large-scale annotated datasets foot_3 or trained models exist for open-domain mds, we bootstrap this task using existing datasets ( §4.1) and models ( §4.2, §4.3). we describe operationalization considerations in §4.4 and evaluation in §4.5. ## evaluating open-domain mds here we present the results of our open-domain mds experiments. in general, we find existing summarizers suffer large reductions in performance when applied to open-domain mds, even when retrieval performance is high ( strong summarizers more sensitive to imperfect retrieval than weak ones we observe a relationship between a summarizer's (baseline) performance on a dataset and its sensitivity to imperfect document retrieval ( however, this relationship is confounded by differences between datasets. to control for this, we conduct experiments comparing fine-tuned primera to primera evaluated zero-shot ( better retrieval performance ̸ = better summarization performance performance of the sparse and dense retrievers was generally comparable ( ## training in the open-domain setting a natural question is whether a summarizer's robustness to document retrieval errors at test time can be improved by exposing the model to similar errors at train time. to explore this, we retrieve the documents for all examples in the train set of each dataset and fine-tune summarizers on these examples. we then evaluate them on both the retrieved document set and the original (ground-truth) document set (figure 3 , see appendix f for details). we find clear cases where summarization performance in the open-domain benefits from the additional training (e.g. multi-news, multi-xscience), but that this benefit can come at the cost of performance on the ground-truth documents (e.g. multi-news, wcep-10 and cochrane), and is sometimes unstable (e.g. wcep, cochrane). we note again that our retrieval setting is highly idealized ( §4.4); nonetheless, our results suggest that existing summarizers could be adapted to the open-domain setting if query-annotated training examples and appropriate document indices are constructed. ## simulating document retrieval errors in this section, we investigate what is driving the reduction in summarization performance in the opendomain setting ( §5). we begin by carefully categorizing the various retrieval errors that can occur. e.g., we could erroneously retrieve documents irrelevant to the query. for each error type, we design a corresponding "perturbation" that can be applied to the inputs of existing mds datasets before they are fed to a summarizer. the perturbations are described below and depicted graphically in figure 4 : token-level perturbation it is well known that nlp models are sensitive to minor token-level changes in their inputs @xcite @xcite . to compare and contrast the document-level sensitivity we investigate with this known sensitivity, we include a token-level perturbation, backtranslation, as control. here, we translate one or more input documents to another high-resource language and back again. this causes small changes, e.g. paraphrasing and synonym substitution, allowing us to create semanticspreserving, token-level changes to a document. 16 ## conclusion we present a new, open-domain task definition for mds. this reformulation is more challenging and potentially more useful, enabling users to specify their intent with only a query. via extensive automatic and manual evaluation, we find that: (1) summarization performance significantly degrades in the open-domain setting, even when retrieval performance is high, (2) additional training can reduce this sensitivity to imperfect retrieval, and