When Reviewers Lock Horns: Finding Disagreements in Scientific Peer Reviews (2023)

Abstract: To this date, the efficacy of the scientific publishing enterprise fundamentally rests on the strength of the peer review process. The journal editor or the conference chair primarily relies on the expert reviewers’ assessment, \textit{identify points of agreement and disagreement} and try to reach a consensus to make a fair and informed decision on whether to accept or reject a paper. However, with the escalating number of submissions requiring review, especially in top-tier Artificial Intelligence (AI) conferences, the editor/chair, among many other works, invests a significant, sometimes stressful effort to mitigate reviewer disagreements. Here in this work, we introduce a novel task of automatically identifying contradictions among reviewers on a given article. To this end, we introduce \textit{ContraSciView} , a comprehensive review-pair contradiction dataset on around 8.5k papers (with around 28k review pairs containing nearly 50k review pair comments) from the open review-based ICLR and NeurIPS conferences. We further propose a baseline model that detects contradictory statements from the review pairs. To the best of our knowledge, we make the first attempt to identify disagreements among peer reviewers automatically. We make our dataset and code public for further investigations.

## introduction despite being the widely accepted standard for validating scholarly research, the peer-review process has faced substantial criticism. its perceived lack of transparency @xcite , sometimes being biased @xcite , arbitrariness @xcite , inconsistency @xcite , being regarded as a poorly defined task @xcite and failure to recognize influential papers @xcite have all been subjects of concern within the scholarly community. the rapid increase in research article submissions across different venues has caused the peer review system to undergo a huge amount of stress @xcite . the role of an editor in scholarly publishing is crucial @xcite . typically, editors or chairs have to manage a multitude of responsibilities. these include finding expert reviewers, assigning review tasks, mediating disagreements among reviewers, ensuring reviews are received on time, stepping in when reviewers are not responsive, making informed decisions, and maintaining communication with authors, among other duties. simultaneously, they must ensure the validity and quality of the reviews to make an informed decision. however, the complexity of scholarly discourse and inherent subjectivity within research interpretation sometimes leads to conflicting views among peer reviewers @xcite . for instance, in figure 1 , reviewer 1 regards the evidence as both strong and sufficient, reinforcing the paper's theoretical soundness. conversely, reviewer 2 remains skeptical of this evidence, underscoring their differing perspectives on the soundness of the paper. feedback between authors and reviewers can help improve the peer review system @xcite . however, reviewer disagreement can create confusion for authors and editors as they try to address reviewer feedback. there are various guidelines or suggestions that editors consider in order to resolve the conflicting reviews @xcite . given this volume, it is challenging for editors to identify contradictions manually. our current investigation can assist in pinpointing these discrepancies and help editors make informed decisions. it is to be noted that this ai-based system aims to aid editors in identifying potential contradictions in reviewer comments. while it provides valuable insights, it is not infallible. editors should use it as a supplementary tool, understanding that not all contradictions may be captured and manual review remains crucial. they should make decision with careful analysis beyond the system's recommendations. our contributions are three-fold: 1) we introduce a novel task: identifying contradictions/disagreement within peer reviews. 2) to address the task, we create a novel labeled dataset of around 8.5k papers and 25k reviews. 3) we establish a baseline method as a reference point for further research on this topic. ## related work artificial intelligence (ai) has been applied in recent years to the realm of scientific peer review with the goal of enhancing its efficacy and quality @xcite . these applications span a diverse range of tasks, including decision prediction @xcite , rating prediction @xcite , sentiment analysis @xcite @xcite , argument mining @xcite , and review summarization @xcite . in this work, we aim to broaden the scope of ai's usefulness by assisting the editor in determining the disagreement between reviewers. contradiction detection is not new. @xcite use linguistic features and support vector machines (svms) to detect contradictions in scientific claims, while @xcite employ review polarity to predict contradiction intensity in reviews. @xcite combine sentiment analysis with contradiction detection for amazon reviews. meanwhile, @xcite employ textual similarity features to classify contradictions in rumors, and @xcite text pairs into entailment, neutral, or contradiction @xcite . well-known transformer-based models, such as bert @xcite and its variants leading to state-of-theart performance. as far as we know, contradiction detection in peer review has never been studied. contradiction detection in peer reviews is complex and requires domain knowledge of the subject. it is not straightforward to detect contradictions in peer reviews because reviewers often have different writing styles and approaches to commenting. we believe that our work can significantly contribute to the peer review process. ## baseline system description we describe the flow of our proposed baseline setup through the flowchart in figure 3 . the initial input involves a pair of reviews for a given paper, which are subjected to the aspect sentiment model. this model classifies the aspect and sentiment of each review comment/sentence within the reviews. subsequently, we identify pairs of comments that, while sharing the same aspects, exhibit differing sentiments; we term it as sentiment disparity aspect pair (sdap). as a final step, the sdaps are then passed to the contradiction classifier in order to classify whether these pairs of review comments are contradictory or not. we describe the aspect sentiment model and contradiction detection model in details as follows: aspect sentiment model: aspect and sentiment in peer review have been studied as a multi-task model @xcite . a review pair comment can have different sentiments corresponding to multiple aspect categories. so, we utilize multi-instance multi-label learning network (mim-lln) @xcite which uniquely identifies sentences as bags and words as instances. it functions as a multi-task model, which performs both aspect category detection (acd) and aspect category sentiment analysis (acsa). given a sentence and its aspect categories, it predicts sentiment by identifying key instances, and aggregating these to obtain the sentence's sentiment towards each category. we discuss mimlln in more detail in appendix b. we trained mimlln on the humanannotated asap-review dataset for our task. reviewer disagreement classifier we use techniques from natural language inference (nli) sentence pair classification to identify reviewer disagreement, particularly contradictions from sentence dependency pairs (sdps). unlike traditional nli tasks that provide a three-category output of "entailment", "contradiction", and "neutral", we have adjusted the model to a two-category output system: "contradiction" and "non-contradiction". the latter category combines "entailment" and "neutral" labels, as our primary focus is on contradiction detection. ## results and analysis we discuss the implementation details in appendix c. we also discuss where our proposed baseline fails (error analysis) in the appendix f. we also utilized the bard api 2 for our evaluation. we provided identical prompts to both google bard and chatgpt to ensure a fair comparison. we compared our findings with those achieved by google bard. bard scored an f1 of 61.35 on the test set, 12 points lower than the baseline model. this is likely due to its bard's requirement for more specialized training for this specific task. we 2 https://github.com/dsdanielpark/bard-api found that google bard registered a higher number of false positives compared to chatgpt. ## conclusion and future work in this work, we introduce a novel task to automatically identify contradictions in reviewers' comments. we develop contrasciview, a reviewpair contradiction dataset. we designed a baseline model that combines the mimlln framework and nli model to detect contradictory review pair comments. we found that roberta large performed best for this task with f1 score of 71.14. our proposed framework doesn't consider the full review context when predicting contradictions. in the future, we will investigate a system that takes into account the entire review context while detecting contradictory review pair comments. additionally, we plan to expand our dataset to include other domains and explore the significance of natural language inference for the given task. we also aim to categorize contradictions based on their severity: high, medium, or low.