R eview R obot: Explainable Paper Review Generation based on Knowledge Synthesis (2020)

Abstract: To assist human review process, we build a novel ReviewRobot to automatically assign a review score and write comments for multiple categories such as novelty and meaningful comparison. A good review needs to be knowledgeable, namely that the comments should be constructive and informative to help improve the paper; and explainable by providing detailed evidence. ReviewRobot achieves these goals via three steps: (1) We perform domain-specific Information Extraction to construct a knowledge graph (KG) from the target paper under review, a related work KG from the papers cited by the target paper, and a background KG from a large collection of previous papers in the domain. (2) By comparing these three KGs, we predict a review score and detailed structured knowledge as evidence for each review category. (3) We carefully select and generalize human review sentences into templates, and apply these templates to transform the review scores and evidence into natural language comments. Experimental results show that our review score predictor reaches 71.4%-100% accuracy. Human assessment by domain experts shows that 41.7%-70.5% of the comments generated by ReviewRobot are valid and constructive, and better than human-written ones for 20% of the time. Thus, ReviewRobot can serve as an assistant for paper reviewers, program chairs and authors.

## introduction as the number of papers in our field increases exponentially, the reviewing practices today are more challenging than ever. the quality of peer paper reviews is well-debated across the academic community @xcite @xcite . how figure 1 : reviewrobot architecture overview many times do we complain about a bad, random, dismissive, unfair, biased or inconsistent peer review? authors even created various social groups at social media to release their frustrations and anger, such as the "reviewer #2 must be stopped" group at facebook foot_1 . how many times are our papers rejected by a conference and then accepted by a better venue with only few changes? as the number of paper submissions continues to double or even triple every year, so does the need for highquality peer reviews. the following are two different reviews for the same paper rejected by acl2019 and accepted by emnlp2019 without any change on content: these reviews, including the positive ones, are too vague and generic to be helpful. we often see review comments stating a paper is missing references without pointing to any specific references, or criticizing an idea is not novel without showing @xcite similar ideas in previous work. some bad reviewers often ask to add citations to their own papers to inflate their citation record and h-index, and these papers are often irrelevant or published after the submission deadline of the target paper under review. early study @xcite shows that the acceptance of a computer systems paper is often random and the dominant factor is the variability between reviewers. the inter-annotator agreement between two review scores for the acl2017 accepted papers @xcite are only 71.5%, 68.4%, and 73.1% for substance, clarity and overall recommendation respectively. @xcite found no agreement among reviewers in evaluating the same nih grant application. the organizers of nips2014 assigned 10% submissions to two different sets of reviewers and observed that these two committees disagreed for 25.9% of the papers @xcite , and half of nips2016 papers would have been rejected if reviews are done by a different group @xcite . these findings highlight the subjectivity in human reviews and call for reviewrobot, an automatic review assistant to help human reviewers generate knowledgeable and explainable review scores and comments, along with detailed evidence. we start by installing a brain for reviewrobot with a large-scale background knowledge graph (kg) constructed from previous papers in the target domain using domain-specific information extraction (ie) techniques. for each current paper under review, we apply the same ie method to construct two kgs, from its related work section and its other sections. by comparing the differences among these kgs, we extract pieces of evidence (e.g., novel knowledge subgraphs which are in the current paper but not in background kgs) for each review category and use them to predict review scores. we manually select constructive human review sentences and generalize them into templates for each category. then we apply these templates to convert structured evidence to natural language comments for each category, using the predicted scores as a controlling factor. experimental results show that our review score predictor reaches 71.4% overall accuracy on overall recommendation, which is very close to interhuman agreement (72.2%). the score predictor achieves 100% accuracy for both appropriateness and impact categories. human assessment by domain experts shows that up to 70.5% of the comments generated by reviewrobot are valid, and better than human-written ones 20% of the time. in summary, the major contributions of this paper are as follows: category evidence example summary • gp τ compare used for hyponym of attention-overattention reader used for documentlevel attention n-best re-ranking strategy large-scale training data cloze-style reading comprehension problem attention mechanism neural architecture used for part of state-of-theart systems method task material appropriateness • the number of entities overlapped between the target paper and the domain's background kg: |{v|@xmath0: |gp τ -gb| • paper sentences that contain new knowledge elements compare used for hyponym of attention-overattention reader documentlevel attention n-best re-ranking strategy cloze-style reading comprehension problem neural architecture used for part of state-of-theart systems soundness • the number of knowledge elements that appear in the contribution claims in the introduction section and that are verified in the experiment section • abstract • attention-over-attention reader, n-best re-ranking strategy is verified in the related work section meaningful comparison • the number of papers about relevant knowledge elements which are missed in the related work section: (gb ∩ gp τ ) -ḡpτ • the number of papers about relevant knowledge elements which are claimed new in the related work section: gb ∩ gp τ ∩ ḡpτ • the description sentences about comparison with related work • if the related work section is not available, we use the difference between gp τ and gb instead used for large-scale training data cloze-style reading comprehension problem attention mechanism neural architecture used for (bahdanau et al., 2015; hermann et al., 2015) potential impact • the number of new knowledge elements in the future work section • the number of new software, systems, data sets, and other resources • 5 new knowledge elements • 1 new architecture overall recommendations • all features mentioned in the above categories • abstract in this paper, we present a simple but novel model called attention-over-attention reader for better solving cloze-style reading comprehension task. @xcite the paper describes a new method called attention-over-attention for reading comprehension. ## related work paper acceptance prediction. @xcite has constructed a paper review corpus, peerread, and trained paper acceptance classifiers. huang (2018) applies an interesting visual feature to compare the pdf layouts and proves its effectiveness to make paper acceptance decision. @xcite applies sentiment analysis features to improve acceptance prediction. the kdd2014 pc chairs exploit author status and review comments for predicting paper acceptance @xcite . we extend these methods to score prediction and comment generation with detailed knowledge element level evidence for each specific review category. paper review generation. @xcite proposes the first deep neural network framework to generate paper review comments. the generator is trained with 48 papers from their own lab. in comparison, we perform more concrete and explainable review generation by predicting scores and generating comments for each review category following a rich set of evidence, and use a much larger data set. @xcite generates comment sentences to explain grammatical errors as feedback to improve paper writing. @xcite extract paper-paper relations and use them to guide citation text generation. review generation in other domains. automatic review generation techniques have been applied to many other domains including music @xcite , restaurants @xcite @xcite , and products @xcite @xcite @xcite . these methods generally apply a sequence-to-sequence model with attention to aspects and attributes (e.g. food type). compared to these domains, paper review generation is much more challenging because it requires the model to perform deep understanding on paper content, construct knowledge graphs to compare knowledge elements across sections and papers, and synthesize information as input evidence for comment generation. ## application limitations and ethical statement the types of evidence we have designed in this paper are limited to nlp, ml or related areas, and thus they are not applicable to other scientific domains such as biomedical science and chemistry. whether reviewrobot is essentially beneficial to the scientific community also depends on who uses it. here are some example scenarios where re-viewrobot should and should not be used: ## conclusions and future work we build a reviewrobot for predicting review scores and generating detailed comments for each review category, which can serve as an effective assistant for human reviewers and authors who want to polish their papers. the key innovation of our approach is to construct knowledge graphs from the target paper and a large collection of in-domain background papers, and summarize the pros and cons of each paper on knowledge element level with detailed evidence. we plan to enhance re-viewrobot's knowledge reasoning capability by building a taxonomy on top of the background kg, and incorporating multi-modal analysis of formulas, tables, figures, and citation networks.