‘Quis custodiet ipsos custodes?’ Who will watch the watchmen? On Detecting AI -generated peer-reviews (2024)

Abstract: The integrity of the peer-review process is vital for maintaining scientific rigor and trust within the academic community. With the steady increase in the usage of large language models (LLMs) like ChatGPT in academic writing, there is a growing concern that AI-generated texts could compromise the scientific publishing including peer-reviews. Previous works have focused on generic AI-generated text detection or have presented an approach for estimating the fraction of peer-reviews that can be AI-generated. Our focus here is to solve a real-world problem by assisting the editor or chair in determining whether a review is written by ChatGPT or not. To address this, we introduce the Term Frequency (TF) model, which posits that AI often repeats tokens, and the Review Regeneration (RR) model which is based on the idea that ChatGPT generates similar outputs upon re-prompting. We stress test these detectors against token attack and paraphrasing. Finally we propose an effective defensive strategy to reduce the effect of paraphrasing on our models. Our findings suggest both our proposed methods perform better than other AI text detectors. Our RR model is more robust, although our TF model performs better than the RR model without any attacks. We make our code, dataset, model public.

## introduction large language models (llms), such as chat-gpt, palm @xcite and @xcite , have significantly impacted both the industrial and academic sectors. the surge in artificial intelligence (ai)-generated content has permeated various domains, from journalism @xcite to academia @xcite . however, their misuse also introduces concerns-especially regarding fake news @xcite , fake hotel reviews @xcite , fake restaurant review @xcite . the exceptional human-like fluency and coherence of the generated content of these models pose a significant challenge, even for experts, in distinguishing if the text is written by humans or llms @xcite . what if peer-reviews themselves are aigenerated? who will guard the guards themselves? a study @xcite conducted experiments on a few papers of ai conferences and found that between 6.5% and 16.9% of text submitted as peer-reviews to these conferences could have been substantially modified by llms. they estimated that the usage of chatgpt in reviews increases significantly within three days of review deadlines. reviewers who do not respond to iclr/neurips author rebuttals exhibit a higher estimated usage of chatgpt. additionally, an increase in chatgpt usage is associated with low self-reported confidence in reviews. once springer retracted 107 cancer papers after they discovered that their peer-review process had been compromised by fake peer-reviewers @xcite . in recent discussions surrounding the use of large language models (llms) in peer reviewing. according to acl policy 3 , if the focus is strictly on content, it seems reasonable to employ writing assistance tools for tasks such as paraphrasing reviews, particularly to support reviewers who are not native english speakers. however, it remains imperative that the reviewer thoroughly reads the paper and generates the review's content indepen-dently. moreover, it is equally acceptable to use tools that assist with checking proofs or explaining concepts unfamiliar to the reviewer, provided these explanations are accurate and do not mislead the reviewer in interpreting the submission. this blend of automation and human oversight maintains the integrity of the review process while leveraging llms for specific enhancements. according to elsevier policy foot_2 , reviewers should not upload their communications or any related material into an ai tool, even if it is just for the purpose of improving language and readability. they also emphasize that the critical thinking, original assessment, and nuanced evaluation required for a thorough review cannot be delegated to ai technologies, as these tools might produce incorrect, incomplete, or biased assessments. we believe reviewers should strictly adhere to the conference policy and guidelines regarding the use of ai tools in peer review, including for proofreading their reviews for refinement. however, to the best of our knowledge, each venue agrees that the content of submissions and reviews is confidential. therefore, they highly discourage the use of chatgpt and similar nonprivacy-friendly solutions for peer review. additionally, they agree that ai-assisted technologies must not be used during the initial writing process of reviews. consequently, our work aims to assist editors in identifying instances where reviewers may have bypassed this crucial step before using ai for refinement. previous works have focused on studying the effect of chatgpt on ai conference peer-reviews. however, in this paper, our focus is to determine whether a review is written by chatgpt or not. we do not assert that ai-generated peer-reviews inherently detract from the quality or integrity of the peer-review system. there can be debates whether ai-generated reviews can help peer-review system or not; we are not asserting that ai-generated peerreview is completely not useful. however, we believe if the review is ai-generated, the chair/metareviewer should be well aware. it is a breach of trust if the meta-reviewer believes that the review is human-written; nevertheless, it is not. despite the potential benefits ai-generated, the chair/metareviewerated reviews may offer, it is crucial for editors to exercise discernment in their reliance on these reviews. this caution is warranted due to the intrinsic limitations of current language models, which can produce inaccurate, misleading @xcite , or entirely fabricated information-a phenomenon often referred to as hallucination @xcite . in this paper, we propose two simple yet effective methods for detecting ai-generated peer reviews based on token frequency (tf method) and regeneration based approach (rr method). we also propose a token modification attack method and study its effect on various detectors. paraphrasing attack is a very common way to evade text detection. so, we also study the effect of paraphrasing on various text detectors. finally, we propose a technique to defend our regeneration-based technique against the paraphrasing attack. we found that both the tf model and the rr model perform better than other ai text detectors for this task. we also found that while the tf model performs better than the rr model under normal conditions, the rr model is more robust and is able to withstand adjective attacks and paraphrasing attacks (after the defense is applied). we summarize our contributions as follows:-• we introduce a novel task to address the realworld problem of detecting ai-generated peerreviews. we create a novel dataset of 1,480 papers from the iclr and neurips conferences for this task. ## dataset we collected a total of 1,480 papers from open-review platform foot_3 . the first version of chatgpt was released by openai on november 30, 2022. therefore, we choose papers from 2022, ensuring there was almost no chance that any of the collected reviews were already generated by chatgpt. figure 1 shows the overall statistics of aigenerated reviews and golden reviews for both iclr and neurips reviews. we discuss the creation of the dataset in more details in the appendix section a. we split the dataset into 70%, 15%, and 15% for training validation and test set respectively. ## methodology in this section, we present our two approaches to detect ai-written peer-reviews based on token frequency (section 4.1) and review regeneration (section 4.2). then, we propose a possible attack (token manipulation attack) on the ai text detectors to see how various models react to it in section 4.3. additionally, since paraphrasing is a common method used to circumvent ai text detection, we introduce a countermeasure as described in section 4.4, designed to protect our proposed review regeneration method against such attacks. ## conclusion and future work in this work, we propose two methods to determine whether a review is written by a human or generated by ai. we found that our proposed tf model and the rr model outperform other ai text detectors under normal conditions. we stress test these detectors against token attack and paraphrasing. furthermore, our proposed rr model is more robust and outperforms other methods. we then propose an effective defensive strategy to reduce the effect of paraphrasing on our models. our findings suggest both of our proposed methods perform better than other ai text detectors. also, while our proposed tf model performs better than the rr model without any attacks, our rr model is more robust against token attacks and paraphrasing attacks. we hope that these findings will pave the way for more sophisticated and reliable ai detectors to prevent such misuse. in future work, we aim to extend our analysis to other domains, such as nuclear physics, medicine, and social sciences, and investigate domain-specific llms to enhance detection accuracy and explore the generalizability of our methods. for further work, we aim to focus on cases where the reviewer writes parts of the review using ai.