Overview of the First Shared Task on Multi Perspective Scientific Document Summarization ( M u P ) (2022)

Abstract: We present the main findings of MuP 2022 shared task, the first shared task on multi-perspective scientific document summarization. The task provides a testbed representing challenges for summarization of scientific documents, and facilitates development of better models to leverage summaries generated from multiple perspectives. We received 139 total submissions from 9 teams. We evaluated submissions both by automated metrics (i.e., Rouge) and human judgments on faithfulness, coverage, and readability which provided a more nuanced view of the differences between the systems. While we observe encouraging results from the participating teams, we conclude that there is still significant room left for improving summarization leveraging multiple references. Our dataset is available at https://github.com/allenai/mup .

## introduction generating summaries of scientific documents is known to be a challenging task as such documents are typically long and require domain expertise to fully comprehend them @xcite @xcite . the standard automated evaluation means in summarization compare system generated summaries with gold human written ones. at the same time, majority of existing work assumes only one single best gold summary for each given document. however, different readers of the same document can have different perspectives; therefore, there is often variability in human written summaries for a given document @xcite . having only one gold summary negatively impacts our ability to evaluate the quality of summarization systems through automated metrics @xcite . also at training time this potentially prevents the model from capturing salient points with respect to different facets in the document @xcite . this is specially the case for longer documents where the summary compression ratio (ratio of length of the input document to the length of summary) is high @xcite . while having multiple reference summaries for each document is desirable, human data collection can be expensive especially for long scientific documents. to address this challenge, we introduce a new dataset and a new shared task to explore methods for generating multi-perspective summaries. we introduce a novel summarization corpus, mup, leveraging data from scientific peer reviews to capture diverse perspectives from the reader's point of view. our shared task similarly encourages development of methods to leverage multiple references. the dataset is collected from openreview, foot_0 an open publishing platform where peer reviews for some machine learning venues are publicly available. peer reviews in various scientific fields often include an introductory paragraph that summarizes the main points and key contributions of a paper from the reviewer standpoint. for example, the first guideline to the reviewers in acl review form foot_1 is to provide a "summary of the paper". in addition, each paper usually receives multiple reviews. based on peer reviews, we collect a corpus of papers and their reviews from ai related venues such as iclr, neurips, and akbc. we use carefully designed heuristics to only include first paragraphs of reviews that are summary-like. we manually check the summaries obtained from this approach on a subset of the data and ensure the high quality of the summaries. the corpus contains a total of 12k papers, and 27k summaries (with average number of 2.57 summaries per paper). we next introduce mup 2022, the first shared task on multi-reference summarization with the goal of encouraging the community to develop better summarization methods for leveraging multiple references. nine teams participated in the task with the top scoring models leveraging a range of transformer-based and graph-based models. automated evaluation results show that while we observe notable progress in the task, there is ample room left for future improvements. we also conduct human evaluation on submitted systems and found out that while most system outputs are readable, they often struggle with the coverage aspect of summarization and they tend to miss some important information in the document. this section describes the mup 2022 shared task. ## dataset description the mup summarization dataset is collected using the publicly available peer review data, sidestepping the significant costs associated with manually creating multiple summaries for each scientific document. ## systems in this section, we overview the systems participating in the mup shared task. ## results and analysis ## findings of mup overall, our findings are summarized below: ## conclusion and future directions we present mup, a new shared task and dataset of 27k summaries, which attracted attention from the community with 18 registered teams and 9 active submitting teams. automated and human evaluation results suggest promising progress towards the task but we conclude that additional research is required, especially around utilization of multi references per document in the training process. for future iterations, we plan to extend the dataset by collecting reviews from additional venues. in addition, we plan to incorporate automatic measures of faithfulness as part of the leaderboard metrics.