Exploiting Labeled and Unlabeled Data via Transformer Fine-tuning for Peer-Review Score Prediction (2022)

Abstract: Automatic Peer-review Aspect Score Prediction (PASP) of academic papers can be a helpful assistant tool for both reviewers and authors. Most existing works on PASP utilize supervised learning techniques. However, the limited number of peer-review data deteriorates the performance of PASP. This paper presents a novel semi-supervised learning (SSL) method that incorporates the Transformer fine-tuning into the Γ-model, a variant of the Ladder network, to leverage contextual features from unlabeled data. Backpropagation simultaneously minimizes the sum of supervised and unsupervised cost functions, avoiding the need for layer-wise pre-training. The experimental results show that our model outperforms the supervised and naive semi-supervised learning baselines. Our source codes are available online.

## introduction over the past few years, the number of submissions for ai-related international conferences and journals has increased substantially, making the review process more challenging. automatic peer-review aspect score prediction (pasp) scores academic papers on a numeric range of different qualities along with aspects such as "clarity" and "originality". it can be a helpful assistant tool for both reviewers and authors. peerread is the first publicly available dataset of scientific peer reviews for research purposes @xcite . it can be used in various ways, such as paper acceptance classification @xcite @xcite and review aspect score prediction @xcite . alternatively, the dataset is modified for citation recommendation @xcite and citation count prediction @xcite ). much of the previous work on pasp is based on supervised learning @xcite . however, the dataset with annotated aspect scores is relatively very small, which deteriorates overall performance. to mitigate the drawback and improve the performance of pasp, we propose a semi-supervised learning (ssl) method that can leverage contextual features from the larger unannotated dataset. ssl has been widely utilized in many nlp tasks, such as classification @xcite , sequence labeling @xcite , and parsing @xcite . it has shown to be effective for learning models by leveraging a large amount of unlabeled data to compensate for the lack of labeled data. ssl is also beneficial for pasp because an enormous body of publications is available online, and unlabeled data, i.e., scholarly papers, can often be obtained with minimal effort. recently, transformer-based pre-training language models (lm) such as bert @xcite and its variants have been very successful as many nlp tasks which utilize these lm attained unprecedented performances. in this paper, we combine the strengths of both techniques and propose a transformer-based γmodel (γ-trans) that incorporates a pre-trained transformer into the γ-model @xcite , a variant of ladder network @xcite , ssl autoencoder. the unsupervised part of γ-trans utilizes a denoising autoencoder to help focus on relevant features derived from supervised learning. the contributions of our work can be summarized as follows: ## γ-transformer the existing works applying ladder networks to the nlp task, e.g., information extraction (nagesh and surdeanu, 2018) and sentiment analysis @xcite . the latter utilizes the encoder of the ladder network @xcite to extract the features from the pretrained lm without fine-tuning it. by freezing the features from the lm, the model only utilizes the fully connected layers from the encoder of the network without exploiting the transformer layer of the lm. to mitigate the issue, we fine-tune the lm along with training the γ-model as well as acquiring the sequence embedding from the pre-trained lm. the model can be plugged into any feedforward network without decoder implementation, i.e., the denoising cost is only on the top layer of the model. figure 1 illustrates the γ-trans network. let x be the input and y be the output with targets t. the labeled training data of size n consists of pairs {x(n), t(n)}, where 1 ≤ @xmath0. the unlabeled data of size m has only input x without the targets t, an x(n), where n +1 ≤ @xmath1. as shown in figure 1 , the network consists of two forward passes, the clean path and the corrupted pass. the former is illustrated in a dotted frame on the righthand side in figure 1 and produces clean z and y, which are given by: the clean path shares the mappings t r (l) and f with the corrupted path. the corrupted z and ỹ are produced by adding gaussian noise n in the corrupted path (left-hand side of figure 1 ): ## conclusion in this paper, we focused on the pasp task and proposed a method, γ-trans, that incorporates the transformer fine-tuning technique into the γmodel of the ladder networks. the experimental results showed the effectiveness of our model as our model attained the best accuracy and f1 on average. through the experiments, we found that our method helps improve the performance of all pretrained lms including scibert and longformer. future work will include (i) extending the method for imbalanced aspect score datasets, (ii) exploiting the related information between aspects, and (iii) generating knowledgeable and explainable review comments.