M ix R ev D etect: Towards Detecting AI -Generated Content in Hybrid Peer Reviews. (2025)

Abstract: The growing use of large language models (LLMs) in academic peer review poses significant challenges, particularly in distinguishing AI-generated content from human-written feedback. This research addresses the problem of identifying AI-generated peer review comments, which are crucial to maintaining the integrity of scholarly evaluation. Prior research has primarily focused on generic AI-generated text detection or on estimating the fraction of peer reviews that may be AI-generated, often treating reviews as monolithic units. However, these methods fail to detect finer-grained AI-generated points within mixed-authorship reviews. To address this gap, we propose MixRevDetect, a novel method to identify AI-generated points in peer reviews. Our approach achieved an F1 score of 88.86%, significantly outperforming existing AI text detection methods.

## introduction the rapid development of large language models (llms) has brought about significant advances in natural language generation, including applications in diverse fields, such as content creation, code generation, and academic peer review. as academic publishing grows in complexity and volume, researchers have increasingly turned to llms to assist in automating or augmenting the peer review process. while these models can generate insightful points, critiques, and suggestions at scale, the use of ai-generated content in peer reviews raises critical concerns about the authenticity, quality, and ethical implications of such reviews. in particular, distinguishing between human-generated and aigenerated review points has emerged as a critical challenge for maintaining the integrity of the peer review process. a study @xcite found that llms may have significantly influenced 6.5% to 16.9% of peer-review text in ai conferences. chatgpt usage spikes near review deadlines, especially among reviewers who skip rebuttals, and is linked to lower self-reported confidence. additionally, springer retracted 107 cancer papers due to compromised peer-review processes involving fake reviewers @xcite . previous work @xcite has primarily investigated methods for detecting fully ai-generated peer reviews. however, in practical scenarios, a reviewer may write some review points themselves while relying on ai to generate others. so, we ask the question below:-what if peer reviews are a mix of ai and human points? in such cases, it becomes crucial to detect which specific review points are written by the reviewer and which are generated by ai. by addressing the challenge of detecting ai-generated peer review points, this work aims to contribute to the ongoing discourse on the ethical and practical implications of ai in academic publishing. we propose a framework for systematically evaluating peer review content, offering solutions that can be integrated into existing editorial workflows to enhance transparency, accountability, and trust in the peer review process. our contributions are summarized as follows:- ## related work early approaches utilized metrics such as entropy @xcite , log-probability scores @xcite , perplexity (beresneva, 2016), and rare n-gram frequencies @xcite to differentiate between human and machine-generated text. recent advancements like detectgpt @xcite suggest that ai-generated content often resides in regions with negative log probability curvature. fast-detectgpt @xcite enhances efficiency by employing conditional probability curvature. research by tulchinskii et al. @xcite shows that ai-generated text tends to have lower intrinsic dimensionality than human writing. few studies applied classifiers to detect synthetic text in contexts like peer review corpora (bhagat and hovy, 2013), media outlets @xcite , and various domains @xcite . gpt-sentinel @xcite , trained classifiers like roberta @xcite and t5 @xcite on the opengpt-text dataset. @xcite uses a siamese neural network to measure the similarity between original and re-decoded text. li et al. @xcite developed a large-scale testbed by collecting human and ai-generated texts from multiple sources. additionally, contrastive and adversarial learning techniques have been introduced to enhance classifier robustness @xcite @xcite . watermarking offers a method for detecting aigenerated text by embedding identifiable signals directly into the text. early techniques modified existing text through synonym substitution @xcite , syntactic restructuring @xcite , or paraphrasing @xcite . watermarking typically requires active involvement from the model or service provider and may risk degrading text quality, potentially impacting the coherence and depth of llm responses @xcite . our work differs from previous studies as we focus on detecting peer review points. a recent paper on ai-generated peer review detection @xcite focuses on determining whether the entire review is ai-generated. in contrast, our work focuses on identifying cases where a review contains a mix of human and ai-generated comments. this hybrid nature presents unique challenges that traditional ai-text detection models fail to address. to bridge this gap, we propose mixrevdetect, the first method explicitly designed to detect ai-generated review points rather than classifying entire reviews, enabling fine-grained ai detection within peer review comments. ## conclusion and future work in this paper, we addressed the growing concern of ai-generated peer reviews by focusing on detecting hybrid reviews where both ai and humanauthored comments are present. we proposed the mixrevdetect framework, which leverages tail pruning, completion through llms, and similarity evaluation to distinguish between ai-generated and human-written peer review points. our approach demonstrated a significant improvement in detection performance, achieving an f1 score of 88.86%, outperforming existing baselines by a large margin. future research could explore the performance of mixrevdetect across a wider variety of llms, particularly as new models emerge. an interesting direction for future work is to categorize the 'human' dataset based on different topics and analyze how the results vary across these categories.