The lack of theory is painful: Modeling Harshness in Peer Review Comments (2022)

Abstract: The peer-review system has primarily remained the central process of all science communications. However, research has shown that the process manifests a power-imbalance scenario where the reviewer enjoys a position where their comments can be overly critical and wilfully obtuse without being held accountable. This brings into question the sanctity of the peer-review process, turning it into a fraught and traumatic experience for authors. A little more effort to still remain critical but be constructive in the feedback would help foster a progressive outcome from the peer-review process. In this paper, we argue to intervene at the step where this power imbalance actually begins in the system. To this end, we develop the first dataset of peer-review comments with their real-valued harshness scores. We build our dataset by using the popular Best-Worst-Scaling mechanism. We show the utility of our dataset for text moderation in peer reviews to make review reports less hurtful and more welcoming. We release our dataset and associated codes in https://github.com/Tirthankar-Ghosal/moderating-peer-review-harshness . Our research is one step towards helping create constructive peer-review reports.

## introduction the peer-review system has largely remained the central and universal quality control system in all scientific fields. @xcite argues that the peer-review system embodies universalism and organized skepticism where the former means "an adherence to objectivity rather than selfinterest," and the latter calls to the spirit that "no theory is accepted merely on the authority of the proponent." both these goals are crucial to the success of this science scrutiny system that has been the de-facto method for scientific validation for ages. nonetheless, the past few years have put this system to stress test with ever-increasing research submissions @xcite , a dearth of experienced reviewers, and criticisms like exclusionary, arbitrary, inconsistent, etc. being leveled at this fundamental process of science evaluation @xcite . these challenges have the potential to turn this central process into a fraught, and traumatic experience, especially for young authors when the reviewers are overly critical or wilfully obtuse @xcite . in an ever-increasing competition in the academic job market, where the career of researchers depends on the impact and prestige of where their work is published, this leads to a natural disdain among the authors for the peerreview process, which is laden with these critical issues. while the peer-review process is by definition a process to evaluate the research under submission -a litmus test to separate the sweet from the sour foot_0 , sometimes what hurt the most to the enthusiastic prospective author is the way reviewers express themselves in the reviews. @xcite notes that "review comments can be blunt, perhaps because of reviewer anonymity, a hurried report, personal style, or even a lack of pragmatic experience." they also express that the peer-review process exhibits a power imbalance: towards the overarching goal of improving the review quality standards and making the peerreviewing process more inclusive, an interesting direction would be to intervene at the very step where this power imbalance actually begins. present-day scientific progress is critically dependent on the peer-review process. hence an inclusive and constructive environment is critical to foster a progressive scientific temperament. here in this work, we intend to make the review reports more welcoming so that they do not seem hurtful and actually focus on their intended objective, i.e., to provide helpful feedback to the authors on their submitted manuscript. given the scale of the peer-review process, an automatic system for this intervention would be of high value. here, we model the various facets of how review comments can be perceived as hurtful, a quality we henceforth call as harshness. we build upon the reviewer guidelines in major artificial intelligence (ai) conferences to categorize how this harshness is expressed in the peer-review reports. we use a comparative annotation scheme, called best-worst-scaling, to map review sentences into real-valued harshness scores and make this dataset publicly available. we envision that our research and accompanying dataset will be helpful in automatic peer-review text moderation. let us study a recent example from a metareview in neurips 2021, which was rather harsh and unnecessary foot_1 : on gaining visibility and criticism in social media on these open access reviews foot_2 , these comments were later manually moderated. thus previous research and evidence such as the above example show that unkind review comments are common. due to the confidential nature of the reviewing process, reviewers do not disclose their identity and hence cannot be held accountable for their unprofessional and unnecessary hard comments. hence this phenomenon has the potential to silently make the whole publishing process a traumatic experience for researchers. our dataset can be used to filter out review sentences based on different thresholds to detect im-polite review comments. a system to predict a harshness score of review sentences would help (senior) area chairs or editors to not allow such comments to go out in public or to the authors. similarly, a reviewer-assistant tool could use such a predictor to flag/alert reviewers when they write such harsh comments (or are repeated offenders). we understand that the peer-review process and harshness is inherently a subjective phenomenon. however, we should strive to make the peer-review process more welcoming so that the fundamental process of scrutinizing science remains objective. our current work is a step in that direction. ## related work there is a growing body of literature on natural language processing (nlp) for peer reviews and scientific literature in general. for example, datasets like peerread @xcite , citetracked @xcite , asap-review @xcite , peer-review-analyze @xcite are proposed in the literature to support nlp research on few downstream problems in peer-reviews. recently, bharti et al. (2022a) proposed a binary-class dataset to determine if a peer-review statement is constructive or not. among the computational approaches, @xcite ; @xcite use sentiment information in peer-review comments to predict the reviewer recommendation score and the acceptance/rejection decision of a manuscript. @xcite ; @xcite proposed deep neural methods for sentiment analysis on peer reviews. our work is different from their works as we model the harshness of a review comment, which is a much richer signal than sentiment label or intensity. in essence, our work is closer to hate speech, and offensive language detection research in nlp @xcite @xcite @xcite . however, we assert that our investigation on hurtfulness or offensiveness in peer-review texts differs from the regular toxicity and abusiveness studied in these works. here we are working on scientific peer-review texts where these notions of harshness are usually very subtle due to the formal academic style of writing. secondly, much of the hate speech research in nlp is focused on some targeted groups depending on factors like race, gender, ethnicity, etc. some aspect of our work resem-bles @xcite . they study aggression, personal attacks, and toxicity in wikipedia talk pages, where aggression and personal attacks also manifest the harshness that we model in this paper. however, their work is not directly applicable to us due to the different domains (wikipedia vs. peer-reviews). our methodology to map review comments to a real-valued score is similar to @xcite , who also uses best-worst-scaling (bws) to map reddit comments to real-valued offensiveness scores. to our knowledge, this is the first work towards developing resources and computational approaches for text moderation in the peer-review domain. ## definition of review harshness we define review harshness as a metric encompassing two orthogonal dimensions. the first dimension concerns the evaluative focus of the comment, and the second dimension deals with the comment's critical stance. ## dataset source and curation access to peer reviews is still restricted since much of the peer-review system operates behind closed doors. fortunately, many venues in artificial intelligence research have adopted an open-access peer review platform called openreview foot_4 to manage the reviewing procedure. for our study, we make use of the peer-review-analyze dataset @xcite . peer-review-analyze contains 1199 reviews (∼ 17k review sentences) from the 2018 edition of the international conference on learning representations (iclr). the iclr reviewing process operates in the openreview platform. each review sentence in this dataset is annotated for review-paper section correspondence, review-paper aspect category, review-statement purpose, and review-statement significance, along with their associated sentiment label (pos, neg, neu). please refer to the original paper @xcite for full details on the dataset. our goal in this study is to model harshness in peer-review sentences. however, annotating each of the 17k sentences individually is expensive. as indicated in the paper, most of these review sentences are neutral in sentiment due to the inherent academic style in writing reviews. we, therefore, use an active learning technique to efficiently create a smaller collection of potentially harsh sentences. active learning assumes access to a small seed dataset for its operationalization. active learning aims to select the most informative samples for labeling according to some uncertainty or diversity measures. we refer the reader to @xcite for an exhaustive survey on active learning techniques in deep learning. as a seed dataset, we crawl 1093 review sentences using the twitter api foot_5 from the public twit-ter handle shitmyreviewerssay 7 . the twitter handle shitmyreviewerssay is a dedicated public platform where authors can anonymously post their review sentences that they find unwelcoming, disparaging, scathing, or discouraging. it tweets selfexplanatory review sentences from diverse scientific backgrounds, which authors share to vent their frustrations. since authors made the efforts to share these review comments on a public forum, we consider them to be a gold standard of the harshness we aim to model. however, these sentences are also extreme in their tone and are not representative of subtle/intrinsic harshness in most academic reviews. therefore, we use both the samples from iclr and shitmyreviewerssay in our final annotations to model a more generic harshness scale. ## annotation process as stated before, we aim to model review comment harshness on a real-valued scale. our choice is motivated by the fact that a review text can be hurtful/harsh to a varying degree and by the downstream application of more fine-grained review text moderation. contrary to the categorical annotation of marking whether a review comment is hurtful or not @xcite , we employ the comparative annotation mechanism. we argue that eliciting categories for review comment harshness is challenging due to the inherent subjective perceptual nature of the task. additionally, such an annotation procedure is not reliable and could lead to ambiguities and inconsistencies @xcite . we argue that these issues can manifest to a greater degree due to the academic nature of our data. all these problems can be mitigated using a comparative annotation setup @xcite . the comparative annotation works by asking the annotator which one among the two samples demonstrates the desired quality to a greater extent. this is more suited for our academic data, as comparing two review comments to see which one is more hurtful is an easier task. we use the best-worst-scaling (bws) @xcite @xcite setup for our annotation. ## data analysis our final dataset contains 1000 review sentences annotated for their harshness value on a scale of -1.0 (most harsh) to 1.0 (least harsh). in this section, we study the distribution of the harshness score and qualitatively examine the samples on varying positions in the harshness scale. ## baseline prediction models in this section, we use common computational models to predict the harshness scores for review comments. our problem is a regression task; for each review sentence s, predict the real-valued score. since we have a relatively smaller size dataset, we use 5-fold cross-validation to evaluate the predictive models. furthermore, to account for outliers in the dataset, we use smooth l1-loss instead of the regular mean squared error (mse) loss for the regression task. besides the regression task, we also use the predictive models in the classification setting. as we have seen earlier, different regions on the harshness scale show different properties. therefore, we categorize our dataset into 3 different classes based on the score; class 1 means the score is less than -0.2, class 2 for a score between -0.2 and 0.3, and class 3 for a score greater than 0.3. in this way, class 1 has disparagingly harsh comments, class 2 contains review comments exhibiting bad reviewing practices, and class 3 contains regular review comments. in the next subsection, we describe our baseline models for prediction. ## conclusions the peer-review process is central to all science research dissemination. however, it also exhibits a power-imbalance situation where the review comments can be overly critical and sometimes cross the boundaries to disparage while also demonstrating bad reviewing practices. this makes this process traumatic, especially for young researchers. the responsibility to moderate these review comments lies in the hands of (senior) area chairs and editors. however, it is not easy to manually moderate review comments with ever-increasing submissions in major ai conferences. in this work, we present a first-of-its-kind dataset of 1000 peerreview comments annotated for their harshness value. we define harshness in this paper based on two dimensions, critical stance and the evaluative focus of the review comment. we then use a comparative annotation technique, best-worst-scaling (bws), to elicit a continuous real-valued harshness scale. our analysis shows that the different regions of this scale represent different facets of harshness with comments going from disparaging at one end to standard evaluative comments at another. we then benchmark common predictive models on our dataset. we show scope for improvement in building computational predictive models. we believe our dataset will be useful in automatic review comments moderation. in the future, we would like to extend the dataset and investigate the impact of reviewer confidence (bharti et al., 2022b) on peer-review text moderation.