Cognitively Aided Zero-Shot Automatic Essay Grading (2020)

Abstract: Automatic essay grading (AEG) is a process in which machines assign a grade to an essay written in response to a topic, called the prompt. Zero-shot AEG is when we train a system to grade essays written to a new prompt which was not present in our training data. In this paper, we describe a solution to the problem of zero-shot automatic essay grading, using cognitive information, in the form of gaze behaviour. Our experiments show that using gaze behaviour helps in improving the performance of AEG systems, especially when we provide a new essay written in response to a new prompt for scoring, by an average of almost 5 percentage points of QWK.

## introduction one of the major challenges in machine learning is the requirement of a large amount of training data. aeg systems perform at their best when they are trained in a prompt-specific manner -i.e. the essays that they are tested on are written in response to the same prompt as the essays they are trained on @xcite . these systems perform badly when they are tested against essays written in response to a different prompt. zero-shot aeg is when our aeg system is used to grade essays written in response to a completely different prompt. in order to solve this challenge of lack of training data, we use cognitive information learnt by gaze behaviour of readers to augment our training data and improve our model. automatic essay grading has been around for over half a century ever since page (1966)'s work @xcite . while there have been a number of commercial systems like e-rater @xcite from the educational testing service (ets), most modern-day systems use deep learning and neural networks, like convolutional neural networks @xcite , recurrent neural networks @xcite , or both @xcite . however, all these systems rely on the fact that their training and testing data is from the same prompt. quite often, at run time, we may not have essays written in response to our target prompt (i.e. the prompt which our essay is written in response to). because of the lack of training data, especially when training a model for essays written for a new prompt, many systems may fail at run time. to solve this problem, we propose a multi-task approach, similar to @xcite , where we learn a reader's gaze behaviour for helping our system grade new essays. in this paper, we look at a similar approach proposed by @xcite to grade essays using cognitive information, which is learnt as an auxiliary task in a multi-task learning approach. multi-task learning is a machine-learning approach, where the model tries to solve one or more auxiliary tasks to solve a primary task @xcite . similar to @xcite , the scoring of the essay is the primary task, while learning the gaze behaviour is the auxiliary task. ## motivation as stated earlier, in section 1, one of the challenges for machine-learning systems is the requirement of training data. quite often, we may not have training data for an essay, especially if the essay is written in response to a new prompt. without any labeled data, in the form of scored essays, we cannot train a system properly to grade the essays. zero-shot automatic essay grading is a way in which we overcome this problem. in zero-shot automatic essay grading, we train our system on essays written to different prompts, and test it on essays written in response to the target prompt. one drawback of this approach is that it would not be able to use the properties of the target essay set in training the model. therefore, as a way to alleviate this problem, we learn cognitive information, in the form of gaze behaviour, for the essays to help our automatic essay grading system grade the essays better. ## related work while there has been work done on developing systems for automatic essay grading, all of them describe systems which use some of the essays the system is tested on as part of the training data (as well as validation data, where applicable) @xcite @xcite @xcite @xcite @xcite . one of the solutions to solve the problem was using cross-domain aeg, where systems were trained using essays in a set of source prompt / prompts and tested on essays written in response to the target prompt. some of the work done to study cross-domain aeg were @xcite (who used task-independent features), @xcite (who used domain adaptation), @xcite (who used a hierarchical cnn layers) and @xcite (who used string kernels and super word embeddings). in all of their works, they defined a source prompt which is used for training and a target prompt which is used for validation and testing. to the best of our knowledge, we are the first to explore the task of zero-shot automatic essay grading, as a way to alleviate the challenge of a lack of graded essays (written in response to the target prompt) for an automatic essay grading system. in our approach, we do not use the target prompt essays even for validation, thereby making it truly zero-shot. ## datasets in this section, we discuss our essay grading dataset and the gaze behaviour dataset which we used. ## experiment setup in this section, we describe our experiment setup, such as the evaluation metric, network architecture and hyperparameters, etc. ## results and analysis from the table, we see that in most of the essay sets, we are able to see an improvement in performance. in order to verify if the improvements were statistically significant, we use the 2-tailed paired t-test with a significance level of @xmath0.05. statistically significant improvements where we use gaze behaviour data are marked with a * next to the result. out of the 8 essay sets, the only essay set where the performance using gaze behaviour falls short compared to when we do not use gaze behaviour is in prompt 8. one of the main reasons for this is that the essays in prompt 8 are very long compared to the other essay sets. when they are absent from the training data, the system is unable to learn about the existence of long essays, which could also be the reason that those essays are scored badly. ## conclusion and future work in this paper, we discussed an important problem for automatic essay grading, namely zero-shot automatic essay grading, where we have no labeled essays written in response to our target prompt, present at the time of training. we showed that, by using gaze behaviour, we are able to learn cognitive information which can help improve our aeg system. in the future, we plan to extend our work to other tasks, like grading of essay traits, using gaze behaviour.