O pen R eviewer: A Specialized Large Language Model for Generating Critical Scientific Paper Reviews (2025)

Abstract: We present OpenReviewer, an open-source system for generating high-quality peer reviews of machine learning and AI conference papers. At its core is Llama-OpenReviewer-8B, an 8B parameter language model specifically fine-tuned on 79,000 expert reviews from top conferences. Given a PDF paper submission and review template as input, OpenReviewer extracts the full text, including technical content like equations and tables, and generates a structured review following conference-specific guidelines. Our evaluation on 400 test papers shows that OpenReviewer produces considerably more critical and realistic reviews compared to general-purpose LLMs like GPT-4 and Claude-3.5. While other LLMs tend toward overly positive assessments, OpenReviewer’s recommendations closely match the distribution of human reviewer ratings. The system provides authors with rapid, constructive feedback to improve their manuscripts before submission, though it is not intended to replace human peer review. OpenReviewer is available as an online demo and open-source tool.

## introduction the peer review process is fundamental to maintaining scientific rigor in academic research, particularly in fast-moving fields like machine learning (ml) and artificial intelligence (ai). as submission volumes to major conferences continue to surgewith top venues receiving over 10,000 submissions annually -the traditional peer review system faces challenges. the task load for reviewers consistently increases, while authors lack access to preliminary expert feedback with a quick turnaround that could help improve their work before submission. large language models (llms) have recently demonstrated remarkable capabilities in under-1 model: huggingface.co/maxidl/llama-openreviewer-8b 2 demo: huggingface.co/spaces/maxidl/openreviewer standing and generating academic content, suggesting their potential to assist in peer review @xcite . however, generating high-quality reviews presents unique challenges beyond general language understanding. a good reviewer has to comprehend technical content, including mathematics and empirical results, and evaluate the work's contributions, novelty, significance, and methodological soundness according to high, field-specific standards. in this paper, we present openreviewer, an opensource system designed to generate human-like, high-quality reviews of machine learning and ai papers. at its core is llama-openreviewer-8b, a specialized language model fine-tuned on a curated dataset of 79,000 reviews from top ml conferences. unlike general-purpose llms, openreviewer is trained to follow standard review templates and guidelines, ensuring structured, critical feedback that aligns with conference reviewing practices. our main contributions are: while openreviewer is not intended to replace human peer reviews, it provides authors with a valuable tool for obtaining rapid, structured feedback before submission. our evaluation shows that openreviewer's reviews closely match human reviewer judgments, suggesting its potential to help authors identify and address weaknesses in their manuscripts in the writing process. ## motivation the motivation behind openreviewer is not to replace human peer reviews. instead, we want to assist authors who face challenges in the presubmission phase. without access to expert feedback before submission, they may overlook significant weaknesses in their manuscripts or fail to address potential reviewer concerns. this can result in unnecessary desk rejections or negative reviews that could have been avoided with earlier feedback. while recent advances in large language models have shown promise in various tasks related to academic research, including paper summarization, understanding, and analysis, existing models often struggle to generate reviews that match the depth, specificity, critical perspective, and structure expected in academic peer review. general purpose llms may miss field-specific conventions, fail to properly evaluate technical contributions, or provide feedback that does not align with established reviewing practices. openreviewer addresses this gap, aiming to provide authors with valuable presubmission feedback that closely mirrors the standards and expectations of human peer reviewers. ## evaluation evaluating automatically generated paper reviews is challenging. there is no single objective measure of review quality since even human experts often disagree in their assessments. additionally, evaluating free-form generated text presents a challenge because there are many valid ways to express the same content. unlike tasks with clear right or wrong answers, free-form text can vary greatly in style, structure, word choice, and level of detail while still being equally valid or effective. for example, two reviewers could make the same core criticism about a paper's methodology but express it differently. one reviewer might be more direct and concise, while another could be more elaborate with detailed examples. this makes it difficult to develop automated metrics that can reliably assess the quality of generated review text. our evaluation approach for openreviewer is based on comparing generated reviews to human reviews. this approach assumes that similarity to human reviews equals quality, which may not always be accurate as the quality control for humanwritten reviews is limited. to measure how well openreviewer reviews align with reviews from human reviewers, we conduct experiments using a test set of 400 heldout papers and their reviews from neurips 2024 and iclr 2025, the most recent venues in our dataset. we compare openreviewer to llama-3.1-8b-instruct and llama-3.1-70b-instruct @xcite , claude-3.5-sonnet (oct. 22) from anthropic, and gpt-4o (2024-11-20) from ope-nai. we generate one review for each paper in the test set using greedy decoding (temperature of 0). all llms are instructed with the same system and user prompts used by openreviewer. we use vllm @xcite to serve the openreviewer and llama models, and access claude-3.5-sonnet and gpt-4o via openrouter 9 . ## discussion our results demonstrate that openreviewer generates reviews that align considerably better with human expert judgments than general-purpose llms. the key findings warrant several important discussions: review quality and criticism level: a striking observation is that general-purpose llms consistently produce overly positive reviews, with average recommendations between 6.9 and 8.1 on a 10-point scale. in contrast, openreviewer's average rating of 5.4 matches the human reviewer distribution exactly. this suggests that specialized training on peer review data helps overcome the tendency of llms to be overly favorable -a critical feature for providing constructive feedback that can help improve papers. use cases and limitations: while openreviewer shows promise as a pre-submission feedback tool, it is important to emphasize that it is not intended to replace human peer review. the system can help authors identify potential weaknesses and prepare stronger submissions, but should be used alongside other forms of feedback. additionally, the model's training data comes primarily from ml/ai venues, potentially limiting its effectiveness for other fields. ## conclusion we presented openreviewer, an open-source system for generating high-quality peer reviews of ml/ai papers. through careful fine-tuning on expert reviews and evaluation against multiple baselines, we demonstrated that openreviewer produces more realistic and critical reviews than general-purpose llms. our key contribution is showing that specialized training on peer review data can overcome the tendency of llms to generate overly positive assessments. looking ahead, several promising directions emerge: while openreviewer shows promise as a tool for generating preliminary feedback, it should complement rather than replace human peer review. we hope this work spurs further research into aiassisted academic reviewing while maintaining high standards of scholarly assessment.