Analytic Automated Essay Scoring Based on Deep Neural Networks Integrating Multidimensional Item Response Theory (2022)

Abstract: Essay exams have been attracting attention as a way of measuring the higher-order abilities of examinees, but they have two major drawbacks in that grading them is expensive and raises questions about fairness. As an approach to overcome these problems, automated essay scoring (AES) is in increasing need. Many AES models based on deep neural networks have been proposed in recent years and have achieved high accuracy, but most of these models are designed to predict only a single overall score. However, to provide detailed feedback in practical situations, we often require not only the overall score but also analytic scores corresponding to various aspects of the essay. Several neural AES models that can predict both the analytic scores and the overall score have also been proposed for this very purpose. However, conventional models are designed to have complex neural architectures for each analytic score, which makes interpreting the score prediction difficult. To improve the interpretability of the prediction while maintaining scoring accuracy, we propose a new neural model for automated analytic scoring that integrates a multidimensional item response theory model, which is a popular psychometric model.

## introduction rapid changes in society in recent years have led to an increased need for cultivating and assessing not only knowledge and skills but also practical abilities, such as expression skills, logical thinking, and creativity (erguvan and aksu dunya, 2020; @xcite . essay exams are one of the test formats that aim to evaluate these abilities, and consequently, they have been used in various educational and assessment settings @xcite . however, essay exams have two considerable drawbacks in the time and monetary costs required to grade them @xcite . furthermore, it is difficult to ensure consistently fair and reliable evaluation due to subjective influences on the part of the rater @xcite . automated essay scoring (aes) has been attracting attention as a method for resolving these difficulties @xcite . conventional aes systems can be broadly classified into two categories @xcite : those that take a feature-engineering approach and those that take a neural approach. the featureengineering approach, which has traditionally been the greater used of the two, utilizes a statistical or machine learning model with pre-defined handcrafted features (e.g. @xcite @xcite @xcite . the neural approach, on the other hand, which has become popular recently, uses deep neural networks to extract features automatically from texts (e.g. @xcite @xcite @xcite @xcite @xcite @xcite . in this study, we focus on the neural approach because of the high accuracy it has achieved in many prior studies. most neural aes studies have focused on holistic scoring @xcite , which provides a single overall score for each essay. however, to provide richer feedback, especially when essay exams are used for educational purposes, we often require not only the overall score but also analytic scores corresponding to various aspects of the essay, such as content, organization, and word choice @xcite . several aes models that can predict these analytic scores along with the overall score have recently been proposed for this purpose @xcite @xcite . from here on, we will refer to such models as analytic aes models. mathias and bhattacharyya (2020) proposed an early neural analytic aes model that took the simple approach of separately applying a conventional holistic scoring model @xcite to each analytic score. then, @xcite proposed a multi-output model in which the output layers are branched by the number of analytic scores and the other layers are shared. one of the more recent models is a multi-output model proposed by @xcite that has a complex deep neural architecture as the output layer for each analytic score. although this model produces state-ofthe-art accuracy, it has some problems in terms of interpretability. to resolve these problems, we propose to extend a conventional analytic aes model by integrating it with an item response theory (irt) @xcite model, a well-known psychometric model. specifically, we extend the multi-output model of @xcite by replacing the complex output layers for each analytic score with a multidimensional irt model @xcite . the advantages of the proposed model are as follows. in this study, we used benchmark datasets that have been widely used in analytic aes research to conduct experiments that evaluated the effectiveness of our model. they showed that our model offers reasonably interpretable parameters without significantly degrading scoring accuracy. furthermore, an interesting finding from our experiment was that, although the benchmark dataset consisted of many analytic scores for each essay, only one or two latent abilities were measured by those multiple scores. note that a similar aes framework combining deep neural networks and irt was recently proposed @xcite . however, they used irt to improve the quality of training data by mitigating rater effects, so their research objective was completely different from the one we focus on in this study. ## conventional analytic aes model this section introduces the conventional analytic aes model proposed by @xcite , which we use as a baseline model. the architecture of this model is displayed on the left side of figure 1 . this model takes in an essay from examinee n and outputs multiple analytic scores {ŷ nm | @xmath0, 2, . . . , m }}, where ŷnm is the m-th analytic score and m is the total number of analytic scores. an essay from examinee n is defined as a word sequence {w nsl | @xmath1, 2, . . . , s}, @xmath2, 2, . . . , l s }}, where w nsl is the l-th word in the s-th sentence of examinee n's essay, s is the number of the sentences in the essay, and l s is the number of words in the s-th sentence. note that in our paper, we regard the overall score as one of the analytic scores. the model consists of two types of layers: a shared layer and an item-specific layer. the shared layer receives the word sequence in each sentence and produces a sentence-level distributed representation through an embedding layer, a convolutional layer, and an attention pooling layer @xcite . the sequence for the sentence-level distributed representation is used in the item-specific layer. the item-specific layer consists of the same number of heads as the number of analytic scoring items, which are evaluation items corresponding to analytic scores, such as content, organization, and word choice. an item-specific layer for an analytic scoring item receives the sequence of the sentence-level distributed representation and produces a score value for the corresponding scoring item. specifically, the input sequence is first processed through a recurrent neural network (rnn), one in which the long short-term memory (hochreiter and schmidhuber, 1997) was used as the rnn. then, an output sequence from the rnn layer is aggregated into a fixed-length hidden vector through an attention pooling layer @xcite . the hidden vector is concatenated with a manually designed feature vector f n , and the concatenated vector h nm is input to the trait attention layer. for capturing the relation between analytic scoring items, the trait attention layer is defined as finally, a linear layer with the sigmoid activation maps xnm , a trait attention output vector, to the prediction score ŷnm : as previously mentioned, this model has a complex architecture for each analytic score, making it difficult to interpret the score prediction. our main focus is to use irt to increase the interpretability of score prediction. ## item response theory irt @xcite is a popular psychometric model that has been widely used for making measurements in educational and psychological research. typical irt models define the probability that an examinee will receive a certain score on a test item as a function of the examinee's latent ability and the item's characteristic parameters, such as the discrimination and difficulty parameters. of the various existing irt models, we employ a multidimensional generalized partial credit model (m-gpcm) @xcite , a representative multidimensional polytomous irt model that can be applied to ordinal score data and can examine multidimensional latent abilities for each examinee. if we regard irt parameters for test items as those for analytic scoring items following the approach in previous studies @xcite , then m-gpcm defines the probability that examinee n will receive score k for the m-th analytic scoring item as all of these model parameters, θ n , α m , and β mu , can be estimated from a collection of observed scores. these parameters are clearly interpretable, as will be explained in sections 4.3 and 5.3. ## proposed model we propose an analytic aes model that incorporates the m-gpcm mentioned in the previous section. the architecture of this model is displayed on the right side of figure 1 . as figure 1 shows, our model and the conventional model share the same layers from the input to the concatenate layer. specifically, in both models, each sentence in an essay is fed to the embedding layer, convolution layer, and attention pooling layer, and then a sequence of the sentence-level distributed representation vectors is transformed into a fixed-length vector through the recurrent layer and the attention pooling layer. finally, the concatenate layer creates an essay-level vector h n by combining the output from the attention pooling layer and the handcrafted feature vector f n . the main differences between the models occur after the concatenate layer. given the essay-level vector h n , our model obtains the latent ability vector θ n , which is used in the subsequent m-gpcm layer, by applying a dense layer given by ## experiments in this section, we discuss how the effectiveness of our model was evaluated through experiments using real-word data. ## conclusions in this study, we proposed a new neural-based analytic aes model that incorporates a multidimensional irt model. through experiments using the well-known benchmark datasets asap and asap++, we demonstrated that, compared to the latest conventional model, our model succeeds in improving interpretability without significantly losing performance. our experiments also suggested that one-or twodimensional abilities can sufficiently explain the multiple analytic scores, including the overall score. this is an important finding suggesting that the analytic scoring items in the dataset may fail to measure multiple aspects of ability. this is undesirable because the objective of analytic scoring is to evaluate multiple aspects of ability. future studies will be required to evaluate our model using various datasets, including other benchmark datasets. moreover, another challenge to address in future work is to develop an extension of our model for cross-prompt tasks.