Hierarchical Bi-Directional Self-Attention Networks for Paper Review Rating Recommendation (2020)

Abstract: Review rating prediction of text reviews is a rapidly growing technology with a wide range of applications in natural language processing. However, most existing methods either use hand-crafted features or learn features using deep learning with simple text corpus as input for review rating prediction, ignoring the hierarchies among data. In this paper, we propose a Hierarchical bi-directional self-attention Network framework (HabNet) for paper review rating prediction and recommendation, which can serve as an effective decision-making tool for the academic paper review process. Specifically, we leverage the hierarchical structure of the paper reviews with three levels of encoders: sentence encoder (level one), intra-review encoder (level two) and inter-review encoder (level three). Each encoder first derives contextual representation of each level, then generates a higher-level representation, and after the learning process, we are able to identify useful predictors to make the final acceptance decision, as well as to help discover the inconsistency between numerical review ratings and text sentiment conveyed by reviewers. Furthermore, we introduce two new metrics to evaluate models in data imbalance situations. Extensive experiments on a publicly available dataset (PeerRead) and our own collected dataset (OpenReview) demonstrate the superiority of the proposed approach compared with state-of-the-art methods.

## introduction with an increasing submission of academic papers in recent years, the task of making final decisions manually incurs significant overheads to the program chairs, it is desirable to automate the process. in this study, we aim at utilizing document-level semantic analysis for paper review rating prediction and recommendation. given the reviews of each paper from several reviewers as input, our goal is to infer the final acceptance decision for that paper and the reviewers' evaluation with respect to a numeric rating (e.g., 1-10 points). paper review rating prediction and recommendation is a practical and important task in ai applications which will help improve the efficiency of the paper review process. it is also intended to enhance the consistency of the assessment procedures and outcomes, and to diversify the paper review process by comparing human recommended rating with machine recommended rating. in the literature, most of existing studies cast review rating prediction as a multi-class classification/regression task @xcite . they build a predictor by using supervised machine learning models with review texts and corresponding ratings. due to the importance of features, most researches focus on extracting effective features such as context-level features @xcite and user features @xcite to boost prediction performance. however, feature engineering is time-consuming and labor-intensive. recently, various deep learning-based models have been proposed for automatically learning features from text data @xcite . existing deep learning models usually learn continuous representations of different grains (e.g., word, phrase, sentence, document) from text corpus @xcite @xcite @xcite . although deep learning models can automatically learn extensive feature representation, they cannot efficiently capture the hierarchical relationship inherent to the review data. to address this problem, @xcite studied a hierarchical architecture and implemented it in deep learning framework to learn a better document-level representation. also, with the success of attention mechanism in many tasks such as machine translation, question answering and so on @xcite , @xcite designed a directional self-attention network to gain context-aware embeddings for words and sentences. despite great progress made by these models, they do not focus on the task of paper review rating recommendation and are not effective enough to be directly used for this task because of the following reasons: first, the review data is hierarchical in nature. there exists a three-level hierarchical structure in the review data: word level, intra-review level and inter-review level, while previous models only capture two-levels (i.e., the word level and intra-review level) of this hierarchy. second, paper reviews are usually much longer than other reviews (e.g., product reviews, movie reviews, restaurant reviews, etc.), while most of these models are working on those shorter reviews stated above and they do not leverage the up to date representation techniques such as bert @xcite and scibert @xcite . in this paper, we propose a novel neural network framework for paper review rating recommendation by taking word, intra-review and inter-review information into account. specifically, inspired by han @xcite and disan @xcite , we introduce a hierarchical bi-directional self-attention network (habnet) framework to effectively incorporate different levels of hierarchical information. the proposed framework consists of three main modules in end-to-end relationship: sentence encoder, intra-review encoder and inter-review encoder, which can consider hierarchical structures of review data as comprehensive as possible. the outputs of inter-review encoder are leveraged as features to build the rating predictor without any feature engineering. we release the code and data collected by us to enable replication and application to new tasks, available at https://github.com/ringbdstack/habnet . the contributions of this work are as follows: ## methodology in this section, we first describe the problem setting, and then present the details of our proposed framework for paper review rating prediction and recommendation. ## conclusion in this paper, a scientific paper review dataset called openreview is collected from iclr openreview website and released. we observe that there is a three-level hierarchical structure in this dataset (i.e., word level, intra-review level and inter-review level) -the information and relationships between reviews of one paper may affect the final decision, and so may relationships between words and sentences in each review. based on these observations, a hierarchical bi-directional self-attention network (habnet) framework is proposed for paper review rating prediction and recommendation that can model the interactions among words, sentences, intra-and inter-reviews in an end-to-end manner. moreover, considering the imbalanced distribution of different classes (i.e., ratings from 1 to 10) in the review rating prediction task, we design two new metrics to better evaluate models. it is seen that both experimental results of predicting final decisions for submitted papers and identifying ratings for reviews on two datasets (open-review and extended peerread) demonstrate our proposed framework has sufficient ability to capture the hierarchical structures of words, sentences and reviews in the datasets and outperforms other models. in the future, we plan to investigate multi-task learning for paper review rating recommendation.