Automated Essay Scoring via Pairwise Contrastive Regression (2022)

Abstract: Automated essay scoring (AES) involves the prediction of a score relating to the writing quality of an essay. Most existing works in AES utilize regression objectives or ranking objectives respectively. However, the two types of methods are highly complementary. To this end, in this paper we take inspiration from contrastive learning and propose a novel unified Neural Pairwise Contrastive Regression (NPCR) model in which both objectives are optimized simultaneously as a single loss. Specifically, we first design a neural pairwise ranking model to guarantee the global ranking order in a large list of essays, and then we further extend this pairwise ranking model to predict the relative scores between an input essay and several reference essays. Additionally, a multi-sample voting strategy is employed for inference. We use Quadratic Weighted Kappa to evaluate our model on the public Automated Student Assessment Prize (ASAP) dataset, and the experimental results demonstrate that NPCR outperforms previous methods by a large margin, achieving the state-of-the-art average performance for the AES task.

## introduction automated essay scoring (aes) is to evaluate the quality of essays and score automatically by using computer technologies. notably, reasonable grading can solve problems that consume much time and require a lot of human effort. what's more, providing feedback to learners can promote self improvement. it is one of the most important applications of natural language processing (nlp) and is widely required in the educational field. most existing methods typically recast aes as a regression task, where the goal is to predict the score of an essay @xcite @xcite . although some promising results have been achieved, these regression-based models cannot exploit the labelling information in the training data efficiently and directly. besides, another line of research treats aes as a preference ranking problem with learning-to-rank methods. @xcite first proposed to rank the pair of documents by extracting features; later, @xcite transformed this task into a listwise ranking problem. @xcite also performed transfer learning to rank two essays that are constrained to be from the same prompt. however, most existing works in aes utilize regression objectives or ranking objectives respectively. as a matter of fact, the two types of methods are highly complementary. on the one hand, only using regression models for aes cannot explicitly model score relationships between essays in the training data. on the other hand, only using ranking-based models could not guarantee accurate scores. in effect, in real-life situations, when a teacher evaluates and grades a student's essay, he usually first compares it with one or multiple exemplar essays as reference and then gives a specific score for it. recently, @xcite presents the first work to combine regression and ranking in the aes task by applying a multi-loss method that optimizes regression loss and ranking loss jointly with a simple dynamic combination strategy. nevertheless, it is actually quite difficult to determine the combination weights to achieve the tradeoff between the two optimization objectives. additionally, the proposed batch-wise learning based ranking model sacrifices the accuracy and only ranks essays in each batch. to address the above problems, in this paper we explore a unified framework for the aes task where both the regression objective and the ranking objective are optimized simultaneously, with the goal of incorporating the merits of two popular aes solutions. the key challenge here is how to integrate two significantly different optimization objectives into a single model with a single loss. to this end, we take inspiration from contrastive learning @xcite and propose a novel unified neural pairwise contrastive regression (npcr) model that jointly optimizes the two objectives in a principled way. in a nutshell, the goal of contrastive learning is to learn a better representation space @xcite . in particular, for two given essays, the distance between similar essays from the same category should be small while the distance between dissimilar essays should be large, and the semantic relationship can be reflected by measuring the distance in the representation space. thus, under the contrastive learning framework, the proposed model aims to map the input essays into the representation space and calculates the differences between essays by the relative scores. specifically, we first design a neural pairwise ranking model to guarantee the global ranking order in a large list of essays, and then we further extend this neural pairwise ranking model to predict the relative scores between an input essay and several reference essays. additionally, a multi-sample voting strategy is adopted for the inference for every input test essay. we use quadratic weighted kappa to evaluate our model on the automated student assessment prize (asap) dataset, and the experimental results demonstrate that the proposed model outperforms previous methods by a large margin and establishes new state-of-the-art on this public benchmark. in summary, the contributions of this work can be concluded as follows: (1) to the best of our knowledge, we make the first attempt to explore a unified framework for the aes task that performs regression and ranking optimization simultaneously; (2) we propose a neural pairwise ranking model for aes that guarantees the global ranking order in a large list of essays; (3) experimental results on the public dataset asap show that the proposed approach not only achieves the state-ofthe-art average performance but also obtains better performance on almost all prompts compared to all baselines. ## related work automated essay scoring systems have been deployed for high-stakes assessment since decades ago. the early approaches for aes mainly involved handcrafted feature based methods @xcite @xcite , while the recent studies have explored deep learning based methods to deliver state-of-the-art performance for this task. in recent years, the mainstreams of aes methods typically formulate aes as a regression task. multiple deep learning architectures based regression models for aes have been proposed. @xcite presents the first neural network model for aes, which first uses the combination of cnn and lstm to extract features of essays to generate text representation vectors and then apply a linear layer with sigmoid activation to map the vectors to valid scores. @xcite uses a hierarchical structure to automatically learn features from the word level and the sentence level. @xcite further introduces the attention mechanism into the model and proves that cnn is more conducive to obtaining local features, while lstm is more suitable for obtaining global features. @xcite proposes to consider neural coherence features as auxiliary features for prediction within an end-to-end neural framework. recent advances in bert @xcite model have inspired researchers to use pre-trained language model in aes @xcite @xcite . another line of research focuses on applying the learning to rank methods in aes tasks. @xcite firstly formulate aes as a rank preference problem and then employ a pairwise ranking model ranksvm to rank two or more essays based on statistical features. chen and he (2013) further utilizes the listwise ranking method to learn a ranking model based on linguistic features. @xcite uses multi-task learning to address the problem of prompt adaptation by treating each prompt as a different task and introducing a constrained preference-ranking approach. recently, considering the complementarity of ranking and regression approaches, @xcite proposes a multi-loss method to combine regression and ranking in the aes task with a simple dynamic combination strategy. ## conclusion in this paper, aiming to incorporate the merits of two popular aes solutions, we propose a novel unified model npcr for aes which combines both regression and ranking objective in a principled way. our approach is conceptually simple, however, the experimental results on the public dataset asap demonstrate that npcr significantly outperforms previous approaches, advancing the state of the art in aes tasks. in future work, we will explore more sophisticated neural feature extractors under the pairwise contrastive regression framework so that more powerful text features can be learned from the input essays, such as the hierarchical structure of a document, coherence features and so on.