R eview E val: An Evaluation Framework for AI -Generated Reviews (2025)

Abstract: The escalating volume of academic research, coupled with a shortage of qualified reviewers, necessitates innovative approaches to peer review. In this work, we propose: (1) ReviewEval, a comprehensive evaluation framework for AI-generated reviews that measures alignment with human assessments, verifies factual accuracy, assesses analytical depth, identifies degree of constructiveness and adherence to reviewer guidelines; and (2) ReviewAgent, an LLM-based review generation agent featuring a novel alignment mechanism to tailor feedback to target conferences and journals, along with a self-refinement loop that iteratively optimizes its intermediate outputs and an external improvement loop using ReviewEval to improve upon the final reviews. ReviewAgent improves actionable insights by 6.78% and 47.62% over existing AI baselines and expert reviews respectively. Further, it boosts analytical depth by 3.97% and 12.73%, enhances adherence to guidelines by 10.11% and 47.26% respectively. This paper establishes essential metrics for AI-based peer review and substantially enhances the reliability and impact of AI-generated reviews in academic research.

## introduction the rapid growth of academic research, coupled with a shortage of qualified reviewers, has created an urgent need for scalable and high-quality peer review processes @xcite @xcite . this need has led to a growing interest in leveraging large language models (llms) to automate and enhance various aspects of the peer review process @xcite . although llms have shown remarkable potential in automating various natural language processing tasks, their effectiveness in serving as reliable and consistent paper reviewers remains a significant challenge. the academic community is already experimenting with ai-assisted reviews; for instance, 15.8% of iclr 2024 reviews involved ai assistance @xcite . despite increasing llm adoption, concerns about reliability and fairness remain. specifically, papers reviewed by ai have been perceived to gain an unfair advantage, leading to questions about the integrity of such evaluations. consequently, research into robust automated review generation systems is crucial, necessitating rigorous evaluation of ai generated reviews to address key challenges. @xcite analyze commercial models like gpt-3.5 and gpt-4 @xcite as paper reviewers, identifying limitations such as hallucinations, incomplete understanding, and insufficient critical feedback compared to humans. existing research on evaluation metrics for aigenerated research paper reviews remains limited. for instance, (d' @xcite proposed a gpt-4-based automated metric for evaluating approximate matches between ai-generated and humanwritten reviews. however, their method's iterative reliance on gpt-4 creates a black-box evaluation, limiting transparency and raising reliability concerns. similarly, @xcite investigated the aspect coverage and similarity between ai and human reviews through a blend of automatic metrics and manual analysis. but their approach overlooks other critical dimensions where ai reviews may underperform, as highlighted in figure 2 . recent surveys also provide broader overviews of llm-driven peer review, highlighting emerging opportunities and persistent challenges @xcite . addressing the above mentioned limitations and gaps, we propose revieweval, a comprehensive evaluation framework for assessing the quality of ai-generated research paper reviews. revieweval targets five key dimensions (see figure 2 ): ❶ comparison with human reviews: evaluates topic coverage and semantic similarity to measure the alignment between ai-generated and humanwritten feedback. ❷ factual accuracy: detects factual errors, including misinterpretations, incorrect claims, and hallucinated information. ❸ analytical depth: assesses whether the ai's critique transcends generic commentary to offer in-depth, meaningful feedback. ❹ actionable insights: measures the ability of the ai to provide constructive suggestions for improving the paper. ❺ adherence to reviewer guidelines: quantifies the degree to which a review conforms to the evaluation criteria outlined by the target conference. recent studies @xcite underscore the growing importance of aligning reviews with conference-specific evaluation criteria, especially as many venues now require adherence to detailed reporting guidelines. to address this, we introduce reviewagent, an ai reviewer with three key features: ❶ conferencespecific review alignment, dynamically adapting reviews to each venue's unique evaluation criteria; ❷ self-refinement loop, iteratively optimizing prompts and intermediate outputs for deeper analytical feedback; ❸ external improvement loop, systematically enhancing review quality using evaluation metrics provided by revieweval. using revieweval, we extensively evaluate reviewagent against two baseline models, ai-scientist @xcite and marg (d' @xcite , benchmarking all systems against expert-written reviews treated as the gold standard. the experiments are performed on a balanced and topically diverse dataset of 120 papers sampled from neurips, iclr, and uai, including domains such as computer science, biology, social science, finance, physics, and engineering. our results show that reviewagent improves actionable insights by 6.78% and 47.62% over existing ai baselines and expert reviews respectively. further, it boosts analytical depth by 3.97% and 12.73%, enhances adherence to guidelines by 10.11% and 47.26% respectively. in summary, the paper makes the following research contributions: (1) revieweval: a multidimensional framework for evaluating research paper reviews ( §3). (2) reviewagent: an adaptive ai reviewer which utilizes revieweval along with novel conference alignment and iterative selfimprovement ( §4). (3) comprehensive evaluation of reviewagent against state-of-the-art baselines using revieweval ( §5). ## revieweval we introduce revieweval, an evaluation framework grounded in the llm-as-a-judge paradigm @xcite . in revieweval, each review is evaluated on several key parameters to assess the overall quality of the generated feedback. to ensure consistency and reliability, all evaluations for a given metric were performed by llms of the same specification and version, minimizing variability from model differences and enabling robust, fair, and unbiased comparisons. the overview of the proposed framework (revieweval + reviewagent) is presented in figure 1 . ## reviewagent to ensure ai-based peer reviews are thorough across the multi-dimensional metrics defined in re-vieweval, we introduce reviewagent, a framework that aligns evaluations with conference-specific reviewing guidelines while operating effectively across those dimensions. ## experiments & results dataset. we constructed a dataset, fullcorpus-120, comprising 120 papers sampled from the neurips, iclr, and uai conferences. for each of the acceptance and rejection category we sampled 8 papers, resulting in a balanced set across acceptance categories (mentioned in baselines and models. we compare our aigenerated review approach with two established methods: sakana ai scientist @xcite and marg (v1 and v2) (d' @xcite . to demonstrate the effectiveness of reviewagent, we report results using foundational models from deepseek, qwen, openai, and anthropic. results for fullcorpus-120. ## conclusion we introduced revieweval, a framework evaluating ai-generated reviews on alignment, accuracy, depth, constructiveness, and guideline adherence. additionally, we propose reviewagent, an ai system featuring (1) conference-specific review alignment, (2) iterative self-refinement, and (3) external improvement via revieweval. our experiments show reviewagent matches expert quality and surpasses existing ai frameworks.