Multi-task Peer-Review Score Prediction (2020)

Abstract: Automatic prediction on the peer-review aspect scores of academic papers can be a useful assistant tool for both reviewers and authors. To handle the small size of published datasets on the target aspect of scores, we propose a multi-task approach to leverage additional information from other aspects of scores for improving the performance of the target. Because one of the problems of building multi-task models is how to select the proper resources of auxiliary tasks and how to select the proper shared structures. We propose a multi-task shared structure encoding approach which automatically selects good shared network structures as well as good auxiliary resources. The experiments based on peer-review datasets show that our approach is effective and has better performance on the target scores than the single-task method and naive multi-task methods.

## introduction automatic prediction of the peer-review aspect scores (e.g. "clarity" and "originality") of academic papers can be a useful assistant tool for both reviewers and authors. on the one hand, because the number of submissions to ai-related international conferences has significantly increased in recent years, it is challenging for the review process. rejecting some papers with evidently low quality can reduce the workload. on the other hand, suggesting the weak aspects to the authors can also help them improve their papers. there are several existing works related to the paper review which concentrate on the quality of the review @xcite . @xcite the acceptance of a paper only based on a paper's visual appearance @xcite . automatic essay scoring @xcite @xcite can be regarded as a related sub-topic that mainly focus on the grammatical and syntactic features in short essays. peerread is the first public dataset of scientific peer reviews for research purposes @xcite , which can be used for paper acceptance classification and review aspect score prediction. it provides detailed peerreviews including the final decisions, the aspect scores such as clarity and originality, and the review contents. it raises two nlp tasks, paper acceptance classification and review aspect score prediction. we focus on the later one in this paper. however, the dataset is relatively small; the set of papers for each review aspect can be different. to improve the performance of aspect score prediction, we propose a solution based on the multi-task learning that can leverage additional rich information from the resources obtained by other aspect scores. we treat the prediction of each aspect as a separate task. the multi-task model for each aspect score has a main-auxiliary manner. multi-task methods have been widely utilized in many nlp tasks, such as summarization @xcite , classification @xcite , parsing @xcite , sequence labeling @xcite , and entity and relation @xcite . when building a multi-task model, there are two critical issues, i.e., which auxiliary resources (tasks) can be used for sharing useful information and how to share the information among the tasks. in these previous studies, researchers always select specific auxiliary resources, and design handcrafted shared structure in the model for a particular nlp topic. however, for different datasets and tasks, there may exist other better auxiliary resources and shared structures. we thus propose an approach selecting the shared structures automatically as well as the auxiliary resources that are more beneficial for the main task. there are diverse parameter sharing manners in the multi-task methods for deep neural networks @xcite . how to define the exploration space for automatic selection is a problem. our approach encodes the multi-task shared structures in the manner of hard parameter sharing and defines the exploration space. we also propose a strategy to search the optimal structures and auxiliaries from the candidate models. it is also flexible to add more auxiliary tasks. our approach can be integrated with hyperparameter optimization methods @xcite or network architecture search methods @xcite for searching. furthermore, our method is capable for not only review score prediction but also some other nlp tasks such as text classification. our main contributions can be summarized as follows. (1). we address an application that predicting the peer-review aspect scores of papers which can be a useful assistant tool for both reviewers and authors. (2). we propose a multi-task shared structure encoding method which automatically selects good shared network structures as well as good auxiliary resources. (3). the experiments based on real paper peer-review datasets show that our approach can build a multi-task model with effective structures and auxiliaries which has better performance than the single-task model and naïve multi-task models. ## conclusion in this paper, we focus on the peer-review score prediction for papers. we propose a multi-task shared structure encoding approach which automatically selects good shared network structures as well as good auxiliary resources. there are some issues in the future work, e.g., trying search methods such as network architecture search and finding evidences of the score predictions.