A Method for Automatically Estimating the Informativeness of Peer Reviews (2022)

Abstract: Peer reviews are intended to give authors constructive and informative feedback. It is expected that the reviewers will make constructive suggestions over certain aspects, e.g., novelty, clarity, empirical and theoretical soundness, etc., and sections, e.g., problem definition/idea, datasets, methodology, experiments, results, etc., of the paper in a detailed manner. With this objective, we analyze the reviewer’s attitude towards the work. Aspects of the review are essential to determine how much weight the editor/chair should place on the review in making a decision. In this paper, we used a publically available Peer Review Analyze dataset of peer review texts manually annotated at the sentence level (∼13.22 k sentences) across two layers:Paper Section Correspondence and Paper Aspect Category. We transform these categorical annotations to derive an informativeness score of the review based on the review’s coverage across section correspondence, aspects of the paper, and reviewer-centric uncertainty associated with the review. We hope that our proposed methods, which are motivated towards automatically estimating the quality of peer reviews in the form of informativeness scores, will give editors an additional layer of confidence for the automatic judgment of review quality. We make our codes available at https://github.com/PrabhatkrBharti/informativeness.git .

## introduction the peer review process is the central mechanism for validating scientific research @xcite . a good review typically provides feedback on one or more sections and aspects while reviewing the manuscript/paper foot_0 , rather than just one section, say the introduction @xcite . therefore, reviews covering more sections and aspects are more likely helpful to the author. furthermore, the more sections and aspects the review covers, the higher the expected coverage score. it may give the author a confidence that the reviewer has read through and paid attention to the different sections and aspects in their submission. in addition, the reviewers are expected to provide constructive comments and suggestions regarding certain aspects and sections of the manuscript. to determining whether the reviewer was informative or constructive in their review and covered significant sections of the manuscript. it would be appropriate to mention the data from peer review analyze @xcite . they analyze and understand the reviewers' thrust over specific sections and aspects of the manuscript. we use those insights in our proposed method. this particular motivation led us to incorporate the general sections, and aspects of the paper defined by the peer review analyze @xcite into this paper to calculate the informativeness score. we attempt here to generate an informativeness score for a given review directly by analyzing the review's coverage across section correspondence, aspects of the paper, and reviewer-centric uncertainty associated with the review. we summarize the key contributions of this work as follows. the novelty of our work lies in utilizing the peer review analyze dataset for measuring the informativeness score. although we use the reviews of a premier machine learning conference (iclr) as our dataset, our proposed method would represent a generic aspect of peer review in science, technology, engineering and mathematics (stem). it will assist the editors in which review they should pay more attention to when crafting a meta-review. in addition, it may give the author confidence that if the review has high informativeness score, it means the reviewer has reviewed thoroughly their submission. ## related work in the meta science community and peer review congress foot_1 (brezis and birukou, 2020), peer review quality has been a major research topic since 1989. there are a few relevant ones that we discuss in this article. the authors @xcite studied a randomized control trial to see how masking author identity improves peer review quality. the study in @xcite presented approaches for assessing the quality of editorial peer reviews. to assess peer reviews of manuscripts, the authors of @xcite developed the review quality instrument (rqi). in this paper, the authors @xcite examined the perspectives of authors and editors on the quality of peer review in three scholarly nursing journals. peer review quality is evaluated in @xcite . a systematic review and meta-analysis on the impact of interventions to improve the quality of peer reviews of biomedical journals were conducted in @xcite . in this paper, authors @xcite explored the dubious connection between the peer review and quality. authors (d'andrea and o'dwyer, 2017) argued if the editors can save peer reviews from peer reviewers. @xcite advocates scientific guidelines for peer review. the purpose of this @xcite study was to evaluate the reliability of the editor's opinion subjective quality ratings of peer review of manuscripts. this paper provides an overview of how peer-review reports of scientific articles can be assessed by the authors @xcite . for peer reviews, some relevant nlp/ml works are worth exploring from an nlp/ml perspective @xcite @xcite @xcite @xcite 3 dataset the dataset used in this study is from peer review analyze (ghosal et al., 2022a), which is publicly available. in peer review analyze, peer review texts are manually annotated at the sentence level (13.22k sentences) across two layers: paper section correspondence and paper aspect category. the detailed dataset statistics are presented in ## benchmarking experiments in addition, we provide baselines for natural language processing (nlp) on the experimental dataset (both annotated and unannotated). moreover, we train nine methods based on data, including multiple linear regressions (mlr), robust regressions (ransac), random forest regressions (rf), long short-term memory (lstm), extreme learning machines (elm), bidirectional long short-term memory (bilstm), masked and permuted pre-training for language understanding (mpnet) @xcite , bidirectional encoder representations from transformers (bert) @xcite , as well as transformer variants of scibert @xcite . ## conclusion and future work in this paper, we provide an effective solution to automatically estimate the informativeness score of review on the shoulder of uncertainty and review coverage (sections and aspects of the paper). for the proposed method, we used a publicly available peer review analyze dataset of peer review texts, manually annotated at the sentence level (13.22k sentences) across two layers: paper section cor-respondence and paper aspect category. next, we transform these categorical annotations to derive an informativeness score of the review based on the review's coverage across section correspondence, aspects of the paper, and reviewer-centric uncertainty associated with the review toward the