Aggregating Multiple Heuristic Signals as Supervision for Unsupervised Automated Essay Scoring (2023)

Abstract: Automated Essay Scoring (AES) aims to evaluate the quality score for input essays. In this work, we propose a novel unsupervised AES approach ULRA, which does not require groundtruth scores of essays for training. The core idea of our ULRA is to use multiple heuristic quality signals as the pseudo-groundtruth, and then train a neural AES model by learning from the aggregation of these quality signals. To aggregate these inconsistent quality signals into a unified supervision, we view the AES task as a ranking problem, and design a special Deep Pairwise Rank Aggregation (DPRA) loss for training. In the DPRA loss, we set a learnable confidence weight for each signal to address the conflicts among signals, and train the neural AES model in a pairwise way to disentangle the cascade effect among partial-order pairs. Experiments on eight prompts of ASPA dataset show that ULRA achieves the state-of-the-art performance compared with previous unsupervised methods in terms of both transductive and inductive settings. Further, our approach achieves comparable performance with many existing domain-adapted supervised models, showing the effectiveness of ULRA. The code is available at https://github.com/tenvence/ulra .

## introduction automated essay scoring (aes) that aims to score the writing quality of essays without human intervention, is an important application of natural language processing in education. state-of-the-art aes models are typically trained in a supervised way with large labeled corpora, comprising essays and their groundtruth quality scores @xcite @xcite . however, collecting labeled essays is time-consuming and labor-intensive, especially for essays written specific to new prompts and when there is no professional scoring staff available. unsupervised aes can get rid of the requirement of groundtruth scores for training, and thus has significant potential in both scientific research and practical applications. its importance can be summarized in three key aspects: 1) unsupervised aes models can handle special scenarios that lack labeling resource, such as the absence of professional scoring staff, the need for rapid essay scoring without timely labeled data, or the cold start scoring of an aes system without historical labeled data; 2) unsupervised aes models can serve as pseudo-label generators or validators for essay scoring based on semi-supervised learning, few-shot learning, or transfer learning; 3) in practical writing tests, unsupervised aes models can rapidly provide a preliminary decision-making basis for scoring staff prior to scoring. early work tackles unsupervised aes by using the clustering method @xcite . to solve the problem of unordered clusters (i.e., cannot map clusters to ordinal scores), @xcite propose to use a heuristic quality signal the number of unique term in essay as the initial score of each essay, and then iteratively propagate the scores to other essays in the same cluster. however, such unsupervised clustering process is uncontrollable (i.e., there is no guarantee that clusters are generated towards to essay quality). recently, researchers propose to use a heuristic quality signal word count as the weak supervision to train a neural aes model @xcite . however, they demonstrate that directly regressing the predicted score to a real-valued quality signal (i.e., word count) leads to poor performance. the above unsupervised aes methods provide a good idea that heuristic quality signals can be used as an alternative of groundtruth scores for model training, but have two major drawbacks. 1) signal values are too noisy to be directly regressed to. considering that the quality signal and groundtruth score may have completely different values but sim-ilar partial orders, it is better to utilize the partial orders in signal rather than the values. 2) single signal is too weak to provide good supervision. since a single quality signal cannot comprehensively describe the quality of essay, more quality signals should be introduced to bring stronger and more robust crowdsourcing-like supervision. to this end, we propose a novel framework for unsupervised aes by learning from rank aggregation (ulra). the core idea of our ulra is to introduce multiple heuristic quality signals as the pseudo-groundtruth, and then train a neural aes model by learning from the aggregation of these quality signals. specifically, our ulra contains a heuristic essay ranking (her) module which views each signal as a ranking metric to generate multiple rank lists, and a deep pairwise rank aggregation (dpra) module which can aggregate the inconsistent quality signals for model training. in the her module, we introduce three types of classic quality signals for essay ranking. in the dpra module, we set a learnable confidence weight for each signal to address the conflicts among signals, and train the neural aes model in a pairwise way to disentangle the cascade effect among partial-order pairs. we conduct experiments on eight prompts of asap dataset, which demonstrate that our proposed ulra significantly outperforms previous unsupervised methods, and can even achieve comparable performance to many existing domain-adapted supervised methods. rank aggregation. rank aggregation (ra) is to aggregate multiple ranked list (i.e. base rankers) into one single list (i.e. aggregated ranker), which is intended to be more reliable than the base rankers @xcite . many prior work have been proposed to effectively and efficiently solve the ra problem. these works can be roughly divided into three categories, which are the permutation-based methods @xcite , the matrix factorization methods @xcite , and the score-based probabilistic methods @xcite @xcite @xcite . the scorebased methods are gradually developed, which usually predicts a score for each object based on the base rankers, and obtains the aggregated ranker based on the scores. bradley-terry (bt) model @xcite ) is an early work that is instructive for the following researches. it is proposed to model a probabilistic relationship between objects, according to the achieved scores, which is suitable for pairwise comparison. then, thurstone model (thurstone, 1927) extends the bt model by assuming that the score for each object has a gaussian distribution. another important extension is crowd-bt @xcite model, which assigns a learnable weight for each base ranker, and optimizes the scores and the weights by active learning and online learning. crowd-bt achieves competitive performance on the task of reading difficulty ranking, and directly inspires our work. ## problem definition we firstly introduce some notation and formalize the unsupervised aes problem. let @xmath0, and @xmath1, 2, • • • , l} be the pre-defined scores. for unsupervised aes, we are given a set of unlabeled essays @xmath2. the purpose of unsupervised aes is to train a model f θ with parameters θ to predict the score of each essay x @xmath3, by in this paper, the model f θ is a neural aes model, and we consider two settings of unsupervised aes, transductive and inductive. for the transductive setting, the test set is just the training set @xmath4. for the inductive setting, the test set does not intersect the training set t ∩ @xmath5. ## conclusion in this paper, we aim to perform essay scoring under the unsupervised setting. to this end, we propose a novel ulra framework to train a neural aes model by aggregating the partial-order knowledge contained in multiple heuristic quality signals. to address the conflicts among different signals and get a unified supervision, we design a deep pairwise rank aggregation loss for model training. experimental results demonstrate the effectiveness of ulra for unsupervised essay scoring.