LLM s as Meta-Reviewers’ Assistants: A Case Study (2025)

Abstract: One of the most important yet onerous tasks in the academic peer-reviewing process is composing meta-reviews, which involves assimilating diverse opinions from multiple expert peers, formulating one’s self-judgment as a senior expert, and then summarizing all these perspectives into a concise holistic overview to make an overall recommendation. This process is time-consuming and can be compromised by human factors like fatigue, inconsistency, missing tiny details, etc. Given the latest major developments in Large Language Models (LLMs), it is very compelling to rigorously study whether LLMs can help meta-reviewers perform this important task better. In this paper, we perform a case study with three popular LLMs, i.e., GPT-3.5, LLaMA2, and PaLM2, to assist meta-reviewers in better comprehending multiple experts’ perspectives by generating a controlled multi-perspective-summary (MPS) of their opinions. To achieve this, we prompt three LLMs with different types/levels of prompts based on the recently proposed TELeR taxonomy. Finally, we perform a detailed qualitative study of the MPSs generated by the LLMs and report our findings.

## introduction meta-reviewing is a critical step in the overall scientific peer-review process, which focuses on understanding the consensus of expert opinions on a scholarly work and making informed judgments on its scientific merit @xcite . meta-reviews are also helpful as they concisely summarize the strengths and weaknesses of the manuscript and highlight the scope for further improvement. composing a meta-review from peer reviewers' comments includes the following steps: 1. collect expert opinions on the paper from peer researchers in the same/similar domain. the process of meta-reviewing is arduous and time-consuming @xcite . in addition, over the years, the number of research manuscript submissions has been rising exponentially @xcite . as such, the volume of metareviewing tasks is also rising exponentially, making the job of meta-reviewers even more challenging and time-consuming. since human beings are prone to fatigue, bias, and distraction @xcite , it is possible that this large workload may make meta-reviewers susceptible to inconsistency and missing details. to address this issue, we performed a case study using llms to generate multi-perspective summaries (mpss) @xcite of reviewers' comments to see whether llm-generated summaries can assist meta-reviewers in being more consistent and efficient in this important role. although several works have proposed automated meta-review generation techniques @xcite @xcite , our work is different in three aspects: 1) instead of meta-review generation, we focus on the mps task with a goal to assist the human meta-reviewer, 2) we study llms in combination with a taxonomy-based prompting technique (e.g., teler (santu and feng, 2023)), which is currently understudied, 3) we examine the correlations between human and automatic (gpt-4) evaluations to explore the reliability of llm-based assessments. case study: for our case study, we selected 40 research papers submitted to the iclr conference in recent years, along with peer-reviewer comments and a handcrafted meta-review written by a highly experienced researcher for each article. these manuscripts and the associated peer-review comments and meta-reviews are all publicly available on openreview.net. using this data set in combination with the three llms (gpt-3.5, palm2, llama2) and the teler taxonomy, we performed an extensive qualitative analysis by engaging human evaluators to go through the meticulous process of reading and analyzing 40 individual paper abstracts, 120 (40 × 3) associated peer-review narratives, 40 corresponding human-written metareviews, and a total of 480 distinct llm-generated mpss 2 and then collected comprehensive survey data, which includes 4800 granular manuscript level judgments on five specific criteria and 90 llm level judgments 3 from humans. although there are several popular text generation evaluation metrics such as rouge, sem-f1, bertscore, etc. @xcite , we exclusively focus on meticulous qualitative human evaluation because previous studies have reported several limitations of automated evaluation metrics for the text generation task @xcite . in addition to human judgments, we also conducted a gpt-4-based evaluation of mpss and examined the correlation between human and automatic evaluations to explore the reliability of llm-based assessments of mpss along five specific criteria. findings: our qualitative analysis reveals that regarding the quality of the generated mpss, gpt-3.5 and palm2 performed comparably and were rated higher by humans than llama2 in terms of manuscript-level judgments. interestingly, palm2 generally yielded better recall scores, while gpt-3.5 yielded better precision scores. more surprisingly, gpt-3.5 was rated poorly by humans at llm-level judgments, which demands further investigation. on the other hand, the automatic evaluation shows that the gpt-4 evaluation correlates poorly with human judgments when tasked with evaluating the meta-reviews generated by its predecessors (i.e., gpt-3.5). our findings also highlight that llm-based evaluations are unreliable in tasks that require aspect-aware summary evaluation, especially when dealing with narratives from multiple diverse perspectives. 2 total comes from 3 llms × 4 prompt levels × 40 papers × 5 evaluation criteria × 2 types (precision/recall) 3 ten annotators judge 3 llms over 3 questions for mps task (@xmath0, several studies have been accomplished on meta-review generation. some researchers have employed rule-based techniques in conjunction with deep neural network architectures @xcite , while others proposed encoder-decoder-based methods @xcite to tackle the meta-review generation task. recent developments with large language models (llms), including but not limited to @xcite , palm @xcite , llama @xcite , @xcite , glam @xcite , have shown remarkable performance in traditional summarization tasks. yet, their performance on complex, i.e., multi-constraint multi-document summarization tasks @xcite ) like meta-review composition is still understudied. recently, @xcite examined the effectiveness of large language models (llms) in both reviewing and meta-reviewing research papers. similarly, @xcite evaluated llms' ability to review papers and compared their quality with human-written peer reviews. in another line of work, @xcite introduced a checklistguided iterative introspection (cgi2) approach using llms, which breaks down scientific opinion summarization into several stages. however, llms are quite sensitive to prompts that are fed to them @xcite , and therefore, designing appropriate prompts with the right amount of detail has become more important than ever @xcite . this issue worsens in practice as there is no standard taxonomy of prompts to follow for benchmarking llms on complex tasks that the community has a general consensus on yet. as such, the main motivation of this work, which makes our case study more compelling, stems from the following two gaps in the literature: 1) llm's performance on multi-constraint multi-perspective summarization tasks (e.g., meta-review composition) is still understudied, 2) no study has yet followed a standard prompting taxonomy to systematically compare llms on such complex tasks. fortunately, very recently, santu and feng (2023) have proposed a general prompting taxonomy called teler, which can serve as a unified standard for comparing and benchmarking llms' performances on complex generation tasks (de-tails in appendix). in this work, we leveraged this teler taxonomy for designing prompts in the context of a meta-review composition task. we reported the specific categories of prompts we experimented with across multiple llms. this enabled more meaningful comparisons among the three llms we studied and, thereby, helped to derive more accurate conclusions about their relative performances. ## case study this section describes the details of our case study, including the data set, llms, evaluation criteria, and prompting details. ## conclusion in this paper, we conducted a case study with three llms and four different prompt levels by following the recently proposed teler taxonomy to study whether llms can be useful in mitigating these issues. to achieve this goal, first, we collected 40 different iclr research manuscripts, and then, for each manuscript, we used three llms separately to compose holistic multi-perspective summaries (mpss) from available multiple expert reviewers' opinions. next, we performed rigorous human evaluations to assess the mpss' reliability. more specifically, we engaged 10 human evaluators to judge the llm-generated mpss on five important criteria, including the quality of capturing core contributions, common strengths, common weaknesses, common suggestions, and literature review. further, we assess the reliability of gpt-4o as an automatic evaluator for this complex task. our case study showed that the reliability of llms as meta-reviewers' assistants widely varies across different criteria (e.g., core contributions, common strengths, etc.), along with the llm and prompt type used. our study also revealed that while summarizing multiple reviewers' comments, gpt-3.5 and palm2s were rated higher by humans than llama2 in all five aspects. the performances of palm2 and gpt-3.5 were comparable on average, but palm2 yielded better recall ratings in general, while gpt-3.5 yielded better precision ratings. although our micro-evaluation indicated palm2 and gpt-3.5 to be comparable, macro-evaluation results indicate that humans like palm2 generated mpss more than the gpt-3.5 generated ones. meanwhile, the automatic evaluation with gpt-4 shows a poor correlation with human evaluations, suggesting that gpt-4 is unreliable for evaluating this facet-aware complex mps tasks.