Encoder Decoder Approach to Automated Essay Scoring For Deeper Semantic Analysis (2021)

Abstract: Descriptive or essay type of answers have always played a major role in education. They clearly capture the student’s grasp on knowledge and presentation skills. Manual essay scoring can be a daunting process to human evaluators; assessing descriptive answers can present a huge overhead owing to limited numbers of evaluators and an out of proportional number of essays to be graded hence leading to an inefficient or an inaccurate score. There has been a major shift in paradigm from traditional classroom education to online education engendered by COVID-19 pandemic; it seems plausible to infer that future assessment of education shall be online, making the solution of automatic essay scorer not only relevant, but of paramount importance. We explore several neural architecture models for the task of automated essay scoring system. Results and Experimental analysis exhibit that our model based on recurrent encoder-decoder provides for a deeper semantic analysis hence, outperforming a strong baseline in terms of quadratic weighted kappa score.

## introduction the exponential advancement of deep learning in the past decade has seen its applications in a wide range of fields from molecular biology to quantum physics. this flexible nature of deep learning and neural architectures is the reason why we have seen its application to a wide array of issues in natural language processing. automated essay scoring is one such problem which aims to find a relation between the essay written and the score assigned so that given an unseen essay, we can predict the score as accurately as possible. essay writing forms important aspect in the academic assessment of the student, grading these essays is a laborious task therefore most of the educational organizations like educational testing service (ets) employ automated essay scorers to evaluate essays. the major pitfalls of these systems stem for the reason that they use hand crafted features to score the essay. the continuous space representations and non-linearity of neural network have provided a great potential in natural language processing. bert and gpt-3, neural architectures developed by google and openai respectively achieve state of the art performance in nlp tasks such as word prediction, question-answering and neural machine translation. researchers have applied convolutional neural networks(cnns), recurrent neural networks (rnns), attention mechanisms @xcite and a permutation of ensembles to the task of automated essay scoring. in this paper we present our encoderdecoder model that learns the relation between the essay and the score assigned by performing a deeper semantic analysis than the current existing models. by applying self-attention and non-linear layers at both encoder level and decoder level, our model is able to effectively capture the information at word level and sentence level respectively, required for scoring. we show that our model performs significantly better than our baseline neural net and finds patterns between words for a better semantic analysis. the rest of the paper is divided into section 2 which deals with related work, section 3 which gives an idea about the task of automated essay scoring .in section 4 we present our model and all its intricacies. section 5 gives an idea of training , section 6 deals with our experimental setup and lastly we present our results and discussion in section 7, followed by conclusion and references ## training automated essay scoring is the process of evaluating the essays written by students for a particular prompt without any human intervention. their performance is assessed by comparing the scores generated to the human-assigned gold standard scores. rest of this section deals with the data utilized for training and the evaluation metric chosen for comparing the performance of aes systems where x and y are the reference ratings and hypothesis rating respectively. u is the number of possible ratings. a matrix p is calculated where p(i,j) denotes number of essays that received a rating x from human annotators and rating y from aes. an expected count matrix k is constructed as cross vectors of two(reference and hypothesis) ratings. after normalization of k such that sum of elements of k and p are same, qwk is calculated as follows κ = 1 -x,y w (x, y)p (x, y) x,y w (x, y)k(x, y) in our experiments, we compare qwk scores of our model to chosen baseline and performed paired t-test analysis to test the improvement obtained ## experiments we have designed our experiments to test three hypotheses: h1: the proposed model will perform equally or surpass baseline model on asap essay corpora in holistic score prediction. h2:the proposed model will perform equally or surpass as the non-neural network baselines. h3:our model will have a better or at least equal semantic attention score as our baseline model. text preprocessing is done using nltk , vocabulary size is restricted to 4000 consisting of most frequent words and all other words are treated as unknowns. the scores are scaled to range [0,1].(taghipour and ng, 2016)(12) for model training and the prediction assessment ,the predicted scores are converted back into original score ranges during model evaluation. we have divided the dataset into five folds to perform 5-fold cross validation and average qwk score across five folds on test set is reported.. in each fold,60% of data are used for training and the rest of data is equally divided between development testing. ## results and discussion examining h1 hypothesis, results in the qwk scores obtained for encoder-decoder model have been shown to be significantly better on performing a paired t-test (@xmath0.05). the reason of this high performance can be attributed to a finer text representations obtained by the architecture of encoder-decoder model and the usage of attention mechanisms at both word and sentence levels. it is interesting to note that the architecture gru+gru does not perform as well as its counterpart lstm. as we examine h2 hypothesis, qwk scores from ## conclusion in this paper, we have proposed a recurrent based encoder-decoder model to address the problem of automated essay scoring that outperforms the state-of -the art attention based models. the proposed model employed a decoder layer and attention mechanism to recognize germane words and sentences. our model produces better sentence representations hence leading to a deeper semantic analysis than state of the art models. empirical results on asap dataset report outperformance of our model to strong established baselines in terms of quadratic weighted kappa score. the future scope of this to make the task of essay scoring prompt agnostic and extend beyond english language.