A gent R eview: Exploring Peer Review Dynamics with LLM Agents (2024)

Abstract: Peer review is fundamental to the integrity and advancement of scientific publication. Traditional methods of peer review analyses often rely on exploration and statistics of existing peer review data, which do not adequately address the multivariate nature of the process, account for the latent variables, and are further constrained by privacy concerns due to the sensitive nature of the data. We introduce AgentReview, the first large language model (LLM) based peer review simulation framework, which effectively disentangles the impacts of multiple latent factors and addresses the privacy issue. Our study reveals significant insights, including a notable 37.1% variation in paper decisions due to reviewers’ biases, supported by sociological theories such as the social influence theory, altruism fatigue, and authority bias. We believe that this study could offer valuable insights to improve the design of peer review mechanisms.

## introduction peer review is a cornerstone for academic publishing, ensuring that accepted manuscripts meet the novelty, accuracy, and significance standards. despite its importance, peer reviews often face several challenges, such as biases @xcite , variable review quality @xcite , unclear reviewer motives @xcite , and imperfect review mechanism @xcite , exacerbated by the ever-growing number of submissions. the rise of open science and preprint platforms has further complicated these systems, which may disclose author identities under doubleblind policies @xcite . efforts to mitigate these problems have focused on enhancing fairness @xcite , reducing biases among novice reviewers @xcite , calibrating noisy peer review ratings @xcite , and refining mechanisms for paper assignment and reviewer expertise matching @xcite . however, several challenges persist in systematically exploring factors influencing peer review outcomes: 1) multivariate nature. the peer review process is affected by a variety of factors, ranging from reviewer expertise, area chair involvement, to the review mechanism design. this complexity makes it difficult to isolate specific factors that impact the review quality and outcomes; 2) latent variables. factors such as reviewer biases and intentions are difficult to measure but have significant effects on the review process, often leading to less predictable outcomes; 3) privacy concerns. peer review data are inherently sensitive and carry the risk of revealing reviewer identities. investigation of such data not only poses ethical concerns but also deters future reviewer participation. this work. we introduce agentreview, the first framework that integrates large language models (llms) @xcite with agent-based modeling @xcite to simulate the peer review process (sec. 2). as shown in figure 1 , agentreview is built upon the capabilities of llms to perform realistic simulations of societal environments @xcite @xcite and provide high-quality feedback on academic literature comparable to or exceeds human levels @xcite @xcite @xcite . agentreview is open and flexible, designed to capture the multivariate nature of the peer review process. it features a range of customizable variables, such as characteristics of reviewers, authors, area chairs (acs), as well as the reviewing mechanisms (sec. 2.1). this adaptability allows for the systematic exploration and disentanglement of the distinct roles and influences of the various parties involved in the peer review process. moreover, agentreview supports the exploration of alternative reviewer characteristics and more complex review processes. by simulating peer review activities with over 53,800 generated peer review documents, including over 10,000 reviews, on over 500 submissions across four years of iclr, agen-treview achieves statistically significant insights without needing real-world reviewer data, thereby maintaining reviewer privacy. agentreview also supports the extension to alternative reviewer characteristics and more complicated reviewing processes. we conduct both content-level and numerical analyses after running large-scale simulations of the peer review process. key findings. our findings are as follows, which could inspire future design of peer review systems: contributions. our contributions are three-fold: reviewers play a pivotal role in peer review. we identify three key dimensions that determine the quality of their reviews. 1) commitment refers to the reviewer's dedication and sense of responsibility in engaging with the manuscript. this involves a proactive and careful approach to provide thorough and constructive feedback on submissions. to explore these dimensionalities, we assign reviewers into specific categories: knowledgeable versus unknowledgeable reviewers for knowledgeability, responsible versus irresponsible for commitment, and benign versus malicious for intention. these categorizations are set by prompts and fed into our system as fixed characteristics. for example, knowledgeable reviewers are described as reviewers that are adept at identifying the significance of the research and pinpointing any technical issues that require attention. in contrast, unknowledgeable reviewers lack expertise and may overlook critical flaws or misinterpret the contributions. reviewer descriptions and prompts are detailed in appendix figure 10 . authors submit papers to the conference and provide rebuttals to the initial reviews during the reviewer-ac discussion period (phase 2 in figure 1 ). although double-blind review policies are typically in place, authors may still opt to release preprints or publicize their works on social media, potentially revealing their identities. we consider two scenarios: 1) reviewers are aware of the au-thors' identities due to the public release of their works, and 2) author identities remain unknown to the reviewers. this allows us to explore the implications of anonymity on the review process. area chairs (acs) have multiple duties, ranging from facilitating reviewer discussions, synthesizing feedback into meta-reviews, and making final decisions. acs ensure the integrity of the review outcomes by maintaining constructive dialogues, integrating diverse viewpoints, and assessing papers for quality, originality, and relevance. our work identifies three styles of acs based on their involvement strategies, each influencing the review process differently: 1) authoritarian acs dominate the decision-making, prioritizing their own evaluations over the collective input from reviewers; 2) conformist acs rely heavily on other reviewers' evaluations, minimizing the influence of their own expertise; 3) inclusive acs consider all available discussion and feedback, including reviews, author rebuttals, and reviewer comments, along with their expertise, to make well-rounded final decisions. ## conclusion we presented agentreview, the first llmbased framework for simulating the peer review process. agentreview addresses key challenges by disentangling intertwined factors that impact review outcomes while preserving reviewer privacy. our work lays a solid foundation for more equitable and transparent review mechanism designs in academic publishing. future works could investigate how intricate interactions between different variables collectively affect review outcomes.