Multi-task Learning for Automated Essay Scoring with Sentiment Analysis (2020)

Abstract: Automated Essay Scoring (AES) is a process that aims to alleviate the workload of graders and improve the feedback cycle in educational systems. Multi-task learning models, one of the deep learning techniques that have recently been applied to many NLP tasks, demonstrate the vast potential for AES. In this work, we present an approach for combining two tasks, sentiment analysis, and AES by utilizing multi-task learning. The model is based on a hierarchical neural network that learns to predict a holistic score at the document-level along with sentiment classes at the word-level and sentence-level. The sentiment features extracted from opinion expressions can enhance a vanilla holistic essay scoring, which mainly focuses on lexicon and text semantics. Our approach demonstrates that sentiment features are beneficial for some essay prompts, and the performance is competitive to other deep learning models on the Automated StudentAssessment Prize (ASAP) benchmark. TheQuadratic Weighted Kappa (QWK) is used to measure the agreement between the human grader’s score and the model’s prediction. Ourmodel produces a QWK of 0.763.

## introduction automatic essay scoring (aes) is the task of grading student essays, using natural language processing to assess quality. the system is designed to reduce time and cost from the human graders' workload. recently, neural network models based on deep learning techniques have been proposed for aes. these approaches involve the use of both recurrent neural networks, e.g., a basic recurrent unit (rnn) @xcite , gated recurrent unit (gru) @xcite , or long short-term memory unit (lstm) @xcite , and convolutional neural networks @xcite . more specifically, taghipour and ng (2016) proposed a convolutional recurrent neural network over the word sequence to construct a document representation. dong et al. employed hierarchical cnn and lstm structure @xcite to construct sentences and document representation separately. similarly, several text properties are utilized for scoring an essay, such as grammatical roles (i.e., subject, object) @xcite , discourse @xcite , or coherence @xcite . the divergent and polarizing writers' opinions in their essays create overall essay structure and quality, especially in persuasive and controversial articles @xcite . sentiment analysis has typically been designed for use with specific domains, such as movie reviews @xcite , product reviews @xcite , social media @xcite , and news @xcite . beigman @xcite are the first who attempted to incorporate sentiments with essay data. it involves the use of subjective lexicons to recognize the polarity of a sentence. another notable work @xcite found a way to measure the compositionality of multi-word expression's sentiment profile (relative degree of polarities) in essays. farra et al. (2015) built essay scoring systems that incorporate persuasiveness based on the analysis of opinions word-level bi-lstm word-level bi-lstm embedding layer embedding layer h 1 w h 1 w h 2 w h 2 w h n w h n w stm 1 w stm 1 w stm 2 w stm 2 w stm 2 w stm 2 w word sentiment prediction word sentiment prediction attention pooling attention pooling d d score score sentence representations sentence representations sentence-level bi-lstm sentence-level bi-lstm sentence sentiment prediction sentence sentiment prediction essay representations essay representations essay scoring essay scoring s i s i sentence representation sentence representation attention pooling attention pooling x 1 x 1 x 2 x 2 x n x n w 1 w 1 w 2 w 2 w n w n figure 2: multi-task learning framework. recently, multi-task learning (mtl) approach has been shown promising results in many nlp tasks. the primary purpose is to leverage useful information in multiple related tasks to improve all the tasks' generalization performance @xcite . the objectives are applied together, such as predicting the probability of the sequence and the probability that the sequence contains names @xcite , the frequency of the next word with part-of-speech (pos) @xcite , surrounding words with other several tasks @xcite , error detection with additional linguistic information @xcite . more advanced, @xcite explored mtl for classifying keyphrase boundaries incorporating semantic super-sense tagging and identifying multi-word expression. @xcite introduced a hierarchical model supervising a set of low-level tasks at the bottom layers and more complex tasks at the model's top layers. there are merely a few existing works that utilize multi-task learning in aes @xcite . in this paper, we propose a method to incorporate sentiment analysis and aes. the proposed method utilizes the sentiment aspect for improving an essay scoring system. the sentiment information by both word-based and sentence-based, shown in figure 1 , is applied to enhance textual representations for sentiment perception of the model. to the best of our knowledge, this is the first approach on multi-task learning incorporating sentiment analysis and aes. our proposed system is based on a hierarchical structure model to learn the features and relations between an essay score and its sentiments. the model is trained to predict a holistic score at the top-level (document-level) along with sentence and word sentiments at the lower levels, i.e., sentence-level and word-level, respectively. ## multi-task learning we employ a hierarchical multi-task learning model similar to the model of @xcite , shown in figure 2 . the model considers an essay d composed of a sequence of sentences @xmath0, s 2 , ..., s m }, and each sentence s i consists of a sequence of words s @xmath1, w 2 , ..., w n }. we describe each layer in our framework in detail. ## results and discussion we can see from the model could reach only 70-80% accuracy. in contrast to multi-task (word sentiment), the model could reach up to 99% accuracy of word sentiment prediction. the multi-task (word&sentence) performed the worst among multi-task and single-task. the sentence sentiment prediction task's difficulty and two auxiliary tasks might make the main objective of the model, essay scoring, unstable. comparing with other related neural deep learning models on aes, we also found that single-task and multi-task (word sentiment) are better than @xcite and @xcite and comparable to @xcite and @xcite . our models perform poorly on prompt 8. one reason is that prompt 8 has the longest average length, and we limit the sentence length too short. we need to utilize the full text of an essay to the models for further improvement. ## conclusion in this paper, we described a neural approach incorporating sentiment analysis and automatic essay scoring (aes). our method is based on a hierarchical structure multi-task learning model. we compared our approach to several neural deep learning approaches @xcite @xcite on the automated student assessment prize (asap) benchmark. overall, our approach is competitive with the best ones. using word sentiment with multi-task learning, we report better results on four prompts compared to the single-task. we intend to make the model more sophisticated in future work and to apply other auxiliary tasks. we also would like to investigate the use of contextualized embeddings (e.g., @xcite al. (2019) ) that shows amazing performance in many nlp tasks.