GLIMPSE : Pragmatically Informative Multi-Document Summarization for Scholarly Reviews (2024)

Abstract: Scientific peer review is essential for the quality of academic publications. However, the increasing number of paper submissions to conferences has strained the reviewing process. This surge poses a burden on area chairs who have to carefully read an ever-growing volume of reviews and discern each reviewer’s main arguments as part of their decision process. In this paper, we introduce , a summarization method designed to offer a concise yet comprehensive overview of scholarly reviews. Unlike traditional consensus-based methods, extracts both common and unique opinions from the reviews. We introduce novel uniqueness scores based on the Rational Speech Act framework to identify relevant sentences in the reviews. Our method aims to provide a pragmatic glimpse into all reviews, offering a balanced perspective on their opinions. Our experimental results with both automatic metrics and human evaluation show that generates more discriminative summaries than baseline methods in terms of human evaluation while achieving comparable performance with these methods in terms of automatic metrics.

## introduction peer review is the standard process for evaluating researchers' work submitted to conferences or academic journals across all fields. its primary function is to maintain quality standards for academic publications and provide authors with constructive feedback on their work. its effectiveness is currently being challenged by a significant surge in the number of submissions. conferences in computer science, such as the international conference on learning representations (iclr) and the association for computational linguistics (acl), among others, regularly receive thousands of submissions. for instance, the number of submissions to iclr and acl has increased fivefold since 2017, reaching 3407 and 3378 submissions, respectively, in 2022. the reviewing process is inherently a tool for scientific communication with at least two target audiences: the authors of a paper and the area chair. the former should get feedback on their work, whereas the latter has to synthesize the salient points from all the reviews as part of their decision process. the area chair has to extract from the reviews the overall sentiment about the paper while gathering the common ideas and unique arguments raised by the reviewers. an efficient highlighting mechanism could help area chairs in their decision-making process, thus reducing their workload. several methods have been developed to generate summaries from reviews in various domains @xcite . many of these methods identify salient segments in reviews based on the discussed topics and the sentiment polarity @xcite @xcite , or using centralitybased metrics @xcite . however, these techniques fall short in the peer review domain. indeed, they are designed to generate a consensus opinion summary, reflecting common opinions without identifying divergent and unique ones. this poses a challenge for the peer review domain since area chairs are concerned with both common and divergent opinions among reviewers. in this paper, we recognize one of the underlying communication goals in the reviewing/metareviewing process: to convey the review's main points to the area chair concisely. the distillation of such salient information in a concise message has been a long-standing focus of study within the pragmatics domain. one of the most influential probabilistic approaches to pragmatics is the rational speech act (rsa). it formulates the communication problem akin to a "reference game", with the goal of associating each item in a set with the most informative yet concise utterance that distinguishes it from others. we take inspiration from this reference-game scenario to introduce the discriminative summarization task. the goal is to generate a summary for each review that highlights commonalities, differences, and unique perspectives, thereby distinguishing one review from others of the same submission. this framework closely aligns with the challenge of summarizing academic reviews, distilling key discriminative insights from lengthy reviews, thus contextualizing each review in relation to others. to this end, we map the discriminative summarization task to a reference game and propose glimpse, a novel pragmatically informative summarization method for scholarly reviews. at the technical level, we leverage the rsa model, a framework for pragmatic modeling rooted in bayesian inference that solves the reference game setting @xcite . we define two novel rsa-based scores that measure the informativeness and the uniqueness of opinions in scholarly reviews. we use these scores to rank utterances describing a review and aggregate them to compose a "glimpse" of all reviews. we conducted extensive experiments on a realworld peer review dataset from the iclr conference collected over a four-year time period. we compare glimpse performance to state-of-the-art methods in multi-document summarization. we design extractive and abstractive variants of our framework and shed light on their properties. our results show that glimpse generates informative and concise summaries. to the best of our knowledge, we are the first to cast the multi-document summarization problem as a reference game and adopt rsa to identify common and divergent opinions in reviews. contributions: overall, we make the following key contributions: figure 1 : illustration of our proposed rsa-based scores applied to real-world scholarly reviews. we consider each sentence in a review as a candidate summary. the most common opinions in the reviews are highlighted in blue whereas the unique ones are highlighted in red using our rsa-based scores. summarization in scientific peer review the task most closely related to ours in the scholarly domain is the meta-review generation task. various strategies have been developed for this task. for instance, metagen @xcite generates an extractive summary draft of reviews, then uses a fine-tuned model for decision prediction, and generates an abstractive meta-review based on both the draft and the predicted decision. similarly, @xcite leverage the hierarchical relationships within reviews and metadata, such as reviewers' confidence and rating, to generate a meta-review along with the acceptance decision. recently, @xcite proposed to prompt a large language model in a guided and iterative manner to generate a meta-review. our task is different because our goal is to support area chairs by generating a summary of reviewers' opinions and highlighting areas of divergence. predicting the meta-reviewer's decision and generating a metareview based on the predicted decision is out of the scope of our work as we believe it is not desirable to do so. the metareview evaluation should ultimately remain in the hands of a human expert as it involves scientific expertise and reasoning about the evaluated paper. rational speech act (rsa) framework. the rational speech act theory @xcite is the most influential probabilistic approach to pragmatics @xcite . it formulates communication between a pragmatic speaker and a listener as probabilistic reasoning, where the speaker's goal is to choose the utterance that is both short and informative with respect to the speaker's intended referent @xcite . as a result, the rsa framework aims to effectively identify the most informative utterance from multiple potential options in a given context. it has been applied to various tasks, including image captioning @xcite , translation (cohn-gordon and goodman, 2019), dialogue @xcite , and text generation @xcite , among others. however, there is little research on applying the rsa framework to multi-document summarization. in our study, we tailor the rsa framework for multidocument summarization tasks, aiming to produce summaries that capture various perspectives while minimizing redundancy. ## discriminative multi-document summarization (d-mds) most approaches to multi-document summarization prioritize generating consensus-based summaries by emphasizing redundant opinions across reviews. however, in domains like peer review, where including unique and divergent opinions is crucial for a comprehensive summary, the conventional focus on common opinions becomes less relevant. this shift in emphasis is particularly pertinent in scholarly review summarization, where the objective is to convey the various opinions and distinctive viewpoints expressed by reviewers. with this objective in mind, we introduce the discriminative multi-document summarization task, inspired by the reference game setting. our aim is to furnish the meta-reviewer with a summary for each review, enabling them to swiftly identify the source review based on the summary content. formally, we define the discriminative multidocument summarization problem as follows: definition 1 (the discriminative multi-document summarization problem). let @xmath0, • • • , d n } be a set of documents. for each document d i , we suppose we have k candidate summaries and we form @xmath1,j } 1⩽i⩽n ,1⩽j⩽k the set of all candidates. the goal is to select the most informative summary from that set for each document d @xmath2 these candidates can be generated using various summarization methods (see sec. 5.1). ## problem formulation and pragmatic summarization in this section, we formulate the discriminative multi-document summarization problem as a reference game. we then apply the rational speech act (rsa) framework, a probabilistic approach to pragmatics that tackles reference games, to address this summarization problem. ## glimpse framework our framework comprises three steps: 1) generating candidates through either abstractive or extractive summarization techniques, 2) selecting candidates using rsa-based scores defined in sec. 5.2, and 3) composing a summary using the selected candidates and a template. ## experimental setup in this section, we outline the experimental setup used to evaluate the performance of glimpse. we present the datasets, baselines, evaluation metrics, and implementation details. datasets. we collect data from the iclr conference, which provides open access to reviews and meta-reviews for all submissions through open-review foot_0 . our dataset contains 28062 reviews for 8428 submissions to the iclr conference from 2017 until 2021. evaluation. the task of collecting reference summaries of scholarly reviews is challenging due to their length and domain specificity. we identify an alternative method to collect scholarly summaries by considering meta-reviews. meta-reviews convey mainly the decision of the meta-reviewer and, depending on their style, may summarize the reviews. we only use these "summary-like" metareviews instead of those that solely convey area chairs' decisions. we apply heuristic rules to extract "summary-like" meta-reviews and perform manual verification. we set criteria for identifying them, including length, reference to at least one reviewer, and semantic similarity measured by cosine similarity with all reviews. we obtained 226 summary-like meta-reviews that we use for evaluation. ## experimental results we present the results for both the discriminative and the standard multi-document summarization tasks. we compare our method with the baselines ## summary and concluding remarks we have introduced a discriminative summarization framework for multi-document summarization and suggested a pragmatic summarization approach inspired by the rational speech act model (rsa) of human communication. our findings demonstrate the effectiveness of rsa-based scores in capturing unique, common, and divergent opinions among reviewers. this paves the way for the development of tools to aid area chairs in evaluating reviews. our method yields more informative summaries compared to existing approaches and is suitable for extractive summarization, ensuring the interpretability of the generated summaries.