Neural Automated Essay Scoring Incorporating Handcrafted Features (2020)

Abstract: Automated essay scoring (AES) is the task of automatically assigning scores to essays as an alternative to grading by human raters. Conventional AES typically relies on handcrafted features, whereas recent studies have proposed AES models based on deep neural networks (DNNs) to obviate the need for feature engineering. Furthermore, hybrid methods that integrate handcrafted features in a DNN-AES model have been recently developed and have achieved state-of-the-art accuracy. One of the most popular hybrid methods is formulated as a DNN-AES model with an additional recurrent neural network (RNN) that processes a sequence of handcrafted sentence-level features. However, this method has the following problems: 1) It cannot incorporate effective essay-level features developed in previous AES research. 2) It greatly increases the numbers of model parameters and tuning parameters, increasing the difficulty of model training. 3) It has an additional RNN to process sentence-level features, enabling extension to various DNN-AES models complex. To resolve these problems, we propose a new hybrid method that integrates handcrafted essay-level features into a DNN-AES model. Specifically, our method concatenates handcrafted essay-level features to a distributed essay representation vector, which is obtained from an intermediate layer of a DNN-AES model. Our method is a simple DNN-AES extension, but significantly improves scoring accuracy.

## introduction in various assessment fields, essay-writing tests have attracted much attention as a way to measure practical higher-order abilities such as logical thinking, critical reasoning, and creative-thinking skills @xcite . in essay-writing tests, test-takers are required to write essays about a given topic, and human raters grade those essays based on a scoring rubric. however, because the scoring process takes much time and effort, it is hard to grade large numbers of essays @xcite . further, subjectivity in human scoring can reduce accuracy @xcite @xcite . automated essay scoring (aes), which utilizes natural language processing and machine learning techniques to automatically grade essays, is one method for resolving these problems. many aes methods have been developed over the past decades, and can generally be categorized as feature-engineering and neural-network approaches @xcite . the featureengineering approach predicts scores using handcrafted features such as essay length or spelling errors (e.g., @xcite @xcite ). the advantages of this approach include interpretability and explainability. however, this approach generally requires extensive effort for engineering effective features to achieve high scoring accuracy for various essays. to obviate the need for feature engineering, a neural-network approach that automatically extracts features using deep neural networks (dnns) has recently attracted attention. many dnn-aes models have been proposed and have achieved high accuracy @xcite @xcite @xcite @xcite @xcite . these two approaches can be viewed as complementary rather than competing because they provide different advantages. specifically, the neural-network approach can extract dataset-specific features from word sequence patterns, whereas the feature-engineering approach can use existing effective features that are difficult to extract using dnns from only word sequence information. to obtain both benefits, @xcite proposed a hybrid method that integrates both approaches. this method is formulated as a dnn-aes model with an additional recurrent neural network (rnn) that processes a sequence of handcrafted sentence-level features. this method provides state-of-the-art accuracy, but has the following drawbacks: to resolve these problems, we propose a new hybrid method that integrates handcrafted essay-level features into a dnn-aes model. specifically, our method concatenates handcrafted essay-level features to a distributed essay representation vector, which is obtained from an intermediate layer of a dnn-aes model. the advantages of our method are as follows: our method is a simple dnn-aes extension, but experimental results on real-world benchmark data show that it significantly improves accuracy. ## hybrid method the feature-engineering approach and the neural-network approach can be viewed as complementary rather than competing approaches, because as mentioned in section 1 they provide different advantages. to receive both benefits, @xcite proposed a hybrid method that integrates the two approaches. figure 3 shows the model architecture of the hybrid method. as that figure shows, it mainly consists of two dnns. one processes word sequences in a given essay in the same way as the conventional lstmbased dnn-aes model. specifically, it transforms a word sequence w @xmath0, w j2 , • • • , w jn j } to a hidden vector h j , which is a fixed-length distributed essay representation, through the lookup table layer, recurrent layer, and pooling layer. the other dnn processes a sequence of handcrafted sentence-level features. letting the j-th essay have n j sentences, and letting sentence-level features for the n-th essay sentence be f jn , the feature sequence f @xmath1, f j2 , • • • , f jn j } is transformed to a fixed-length hidden vector h f j through a recurrent layer and a pooling layer. (note that the original article used an lstm for the recurrent layer and attention pooling for the pooling layer.) finally, inputting a concatenated vector [h j ; h f j ], the linear layer with sigmoid activation produces a predicted score. this method has provided higher accuracy than feature engineering-based methods or dnn-based methods. however, it has the following drawbacks. ## proposed method to resolve the above problems, we propose a new hybrid method that incorporates handcrafted essaylevel features to a dnn-aes model. our method concatenates handcrafted essay-level features to the distributed essay representation h j , which is the input vector for the last linear layer in conventional dnn-aes models. letting essay-level features for the j-th essay be f o j , the proposed method projects the concatenated vector [h j ; f o j ] to a scalar value by using a sigmoid function, as in conventional dnn-aes models. the proposed method can be easily applied to existing dnn-aes models, because they commonly have a layer that produces a distributed essay representation before the last linear layer. as examples, figs. 4 , 5 , and 6 show model architectures for lstm, bert, and conventional hybrid models integrating essay-level features. the proposed method can incorporate various existing essay-level features for which effectiveness has been shown. as essay-level features, this study uses the 25 features presented in ## experiments this section demonstrates the effectiveness of the proposed method using real-world benchmark data. ## conclusions we proposed a simple method that incorporates handcrafted essay-level features to dnn-aes models. our method adds handcrafted features to a distributed essay representation vector obtained as an intermediate hidden representation of a dnn-aes model. our method can be easily applied to various conventional dnn-aes models without increasing model complexity much, but significantly improving in this study, we evaluated the effectiveness of the proposed method that uses relatively simple features, but in future studies, we will use more varied essay-level features, such as those shown in