Happy Are Those Who Grade without Seeing: A Multi-Task Learning Approach to Grade Essays Using Gaze Behaviour (2020)

Abstract: The gaze behaviour of a reader is helpful in solving several NLP tasks such as automatic essay grading. However, collecting gaze behaviour from readers is costly in terms of time and money. In this paper, we propose a way to improve automatic essay grading using gaze behaviour, which is learnt at run time using a multi-task learning framework. To demonstrate the efficacy of this multi-task learning based approach to automatic essay grading, we collect gaze behaviour for 48 essays across 4 essay sets, and learn gaze behaviour for the rest of the essays, numbering over 7000 essays. Using the learnt gaze behaviour, we can achieve a statistically significant improvement in performance over the state-of-the-art system for the essay sets where we have gaze data. We also achieve a statistically significant improvement for 4 other essay sets, numbering about 6000 essays, where we have no gaze behaviour data available. Our approach establishes that learning gaze behaviour improves automatic essay grading.

## introduction collecting a reader's psychological input can be very beneficial to a number of natural language processing (nlp) tasks, like complexity @xcite , sentence simplification @xcite ), text understanding @xcite ), text quality @xcite , parsing @xcite , etc. this psychological information can be extracted using devices like eye-trackers, and electroencephalogram (eeg) machines. however, one of the challenges in using reader's information involves collecting the psycholinguistic data itself. in this paper, we choose the task of automatic essay grading and show how we can predict the score that a human rater would give using both text and learnt gaze behaviour. an essay is a piece of text, written in response to a topic, called a prompt. automatic essay grading is assigning a score to the essay using a machine. an essay set is a set of essays written in response to the same prompt. multi-task learning @xcite is a machine learning paradigm where we utilize auxiliary tasks to aid in solving a primary task. this is done by exploiting similarities between the primary task and the auxiliary tasks. scoring the essay is the primary task and learning gaze behaviour is the auxiliary task. using gaze behaviour for a very small number of essays (less than 0.7% of the essays in an essay set), we see an improvement in predicting the overall score of the essays. we also use our gaze behaviour dataset to run experiments on unseen essay sets -i.e., essay sets which have no gaze behaviour data -and observe improvements in the system's performance in automatically grading essays. contributions the main contribution of our paper is describing how we use gaze behaviour information, in a multi-task learning framework, to automatically score essays outperforming the stateof-the-art systems. we will also release the gaze behaviour dataset foot_0 and code foot_1 -the first of its kind, for automatic essay grading -to facilitate further research in using gaze behaviour for automatic essay grading and other similar nlp tasks. ## gaze behaviour attributes in our experiments, we use only a subset of gaze behaviour attributes described by @xcite because most of the other attributes (like second fixation duration 3 ) were mostly 0, for most of the interest areas, and learning over them would not have yielded any meaningful results. ## results and analysis our system is able to outperform the co-attention system (zhang and litman, 2018) in all the essay sets. overall, it is also the best systemachieving the highest qwk results among all the systems in 3 out of the 4 essay sets (and the secondbest in the other essay set). to test our hypothesisthat the model trained by learning gaze behaviour helps in automatic essay grading -we run the paired t-test. our null hypothesis is: "learning gaze behaviour to score an essay does not help any more than the self-attention and co-attention systems and whatever improvements we see are due to chance." we choose a significance level of @xmath0.05, and observe that the improvements of our system are found to be statistically significant -rejecting the null hypothesis. ## conclusion and future work in this paper, we describe how learning gaze behaviour can help aeg in a multi-task learning setup. we explained how we created a resource by collecting gaze behaviour data, and using multitask learning we are able to achieve better results over a state-of-the-art system developed by zhang and litman (2018) for the essay sets which we collected gaze behaviour data from. we also analyze the transferability of gaze behaviour patterns across essay sets by training a multi-task learning model on unseen essay sets (i.e. essay sets where we have no gaze behaviour data), thereby establishing that learning gaze behaviour improves automatic essay grading. in the future, we would like to look at using gaze behaviour to help in cross-domain aeg. this is done mainly when we don't have enough training examples in our essay set. we would also like to explore the possibility of generating textual feedback (rather than just a number, denoting the score of the essay) based on the justifications that the annotators gave for their grades. mark d shermis and jill burstein. 2013. handbook of automated essay evaluation: current applications and new directions. routledge. abhinav deep singh, poojan mehta, samar husain, and rajkumar rajakrishnan. 2016. quantifying sentence complexity based on eye-tracking measures. in proceedings of the workshop on computational linguistics for linguistic complexity (cl4lc), pages 202-212, osaka, japan. the col-ing 2016 organizing committee. kaveh taghipour and hwee tou ng. 2016. a neural approach to automated essay scoring. in proceedings of the 2016 conference on empirical methods in natural language processing, pages 1882-1891, austin, texas. association for computational linguistics. yi tay, minh phan, luu anh tuan, and siu cheung hui. 2018. skipflow: incorporating neural coherence features for end-to-end automatic text scoring. haoran zhang and diane litman. 2018. co-attention based neural network for source-dependent essay scoring. in proceedings of the thirteenth workshop on innovative use of nlp for building educational applications, pages 399-409, new orleans, louisiana. association for computational linguistics.