A Sentiment Consolidation Framework for Meta-Review Generation (2024)

Abstract: Modern natural language generation systems with Large Language Models (LLMs) exhibit the capability to generate a plausible summary of multiple documents; however, it is uncertain if they truly possess the capability of information consolidation to generate summaries, especially on documents with opinionated information. We focus on meta-review generation, a form of sentiment summarisation for the scientific domain. To make scientific sentiment summarization more grounded, we hypothesize that human meta-reviewers follow a three-layer framework of sentiment consolidation to write meta-reviews. Based on the framework, we propose novel prompting methods for LLMs to generate meta-reviews and evaluation metrics to assess the quality of generated meta-reviews. Our framework is validated empirically as we find that prompting LLMs based on the framework — compared with prompting them with simple instructions — generates better meta-reviews.

## introduction notable strides have been made in abstractive text summarization @xcite with the advancement of large language models (llms) @xcite over recent years. with even a simple instruction such as "tl;dr" or "please write a summary", these models can generate plausible summaries which are found more preferred over those written by humans @xcite . however, it is uncertain if these models truly possess the ability of information consolidation, especially when summarizing documents that are composed of opinionated information. the models may take shortcuts to generate texts instead of correctly understanding and aggregating information from the source documents @xcite and they may generate abstractive summaries with incorrect overall sentiment. automated sentiment summarization holds significant importance @xcite and there have been sentiment summarization datasets; however, most of them are in the product review domain. these datasets are less interesting for investigating information consolidation as (1) the summaries are extractive, composed of a simple combination of extracted snippets @xcite , and (2) the summary of product reviews is about extracting the majority sentiment (which is a simple consolidation function). to address this, in this paper, we propose the task of scientific sentiment summarization, taking the meta-reviews in scientific peer review as summaries. 2 the investigation of meta-review generation @xcite presents an exciting opportunity for exploring the intricate process of multi-document information consolidation that involves complex judgement. this is because (1) meta-reviewers are supposed to understand not only all the reviews from different reviewers but also the multi-turn discussions between the reviewers and the author and write their comments to support the acceptance decision of the manuscript, (2) the logic of arguments (from reviewers and authors) has to be taken into account to arrive at the final sentiment in the meta-reviews and it is not a matter of majority voting and (3) metareviews have to recognize and resolve conflicts and consensus among reviewers. in this paper, we hypothesize that human metareviewers follow a three-layer sentiment consolidation framework as shown in figure 1 to write meta-reviews based on reviews and multi-turn discussions in the peer review process. human and automatic annotation is then conducted to extract sentiments and expressions on various review facets (e.g., novelty and soundness) from corresponding source documents (i.e., reviews and discussions) and these judgements play a critical role in generating the meta-reviews. we also propose two evaluation metrics which focus on assessing sentiments in generated meta-reviews, and experiments empirically validate our proposed three-layer framework when they are integrated as prompts for llms to generate meta-reviews. contributions of our paper: ## related work in this section, we discuss large-scale information consolidation in abstractive summarization, and automated sentiment summarization. ## sentiment consolidation over multiple opinionated documents in the following section, we introduce the task of scientific sentiment summarization and our threelayer sentiment consolidation framework in metareview generation, conduct sentiment and expres- ## sentiment-aware evaluation on information consolidation in this section, we focus more on how to evaluate the sentiments of the generated summaries or meta-reviews in meta-review generation based on our proposed framework. we propose faceteval and fusioneval which are reference-based and reference-free metrics, respectively. ## enhancing llms with explicit information consolidation in ## conclusions and future work in this paper, we explore sentiment-focused multidocument information consolidation within the task of scientific sentiment summarization. we introduce a three-layer framework of sentiment consolidation to focus on generating meta-reviews and it considers the sentiments for each review facet in the reviews and discussions. we also propose automatic evaluation metrics that assess the overall sentiments in the generated meta-reviews. experiments on meta-review generation show that prompting llms by following the processes in the three-layer framework results in better metareviews, providing an empirical validation of our framework for describing the meta-review writing process. as the sentiment consolidation also exist in other domains where human reviews or comments exist such as politics and advertisement, we will explore adapting our proposed sentiment consolidation framework into other domains in the future.