Countering the Influence of Essay Length in Neural Essay Scoring (2021)

Abstract: Previous work has shown that automated essay scoring systems, in particular machine learning-based systems, are not capable of assessing the quality of essays, but are relying on essay length, a factor irrelevant to writing proficiency. In this work, we first show that state-of-the-art systems, recent neural essay scoring systems, might be also influenced by the correlation between essay length and scores in a standard dataset. In our evaluation, a very simple neural model shows the state-of-the-art performance on the standard dataset. To consider essay content without taking essay length into account, we introduce a simple neural model assessing the similarity of content between an input essay and essays assigned different scores. This neural model achieves performance comparable to the state of the art on a standard dataset as well as on a second dataset. Our findings suggest that neural essay scoring systems should consider the characteristics of datasets to focus on text quality.

## introduction automated essay scoring (aes) is the task of assigning a score for a given essay, aiming to replicate human scoring results. the public release of a standard dataset from a shared task foot_0 increased the interest in this task significantly. there have been several systems applied to the standard dataset including machine learning-based systems employing diverse features @xcite and neural essay scoring systems @xcite . previous work, nevertheless, has shown that aes systems are not capable of assessing the quality of essays @xcite @xcite , but indeed work by adopting shallow heuristics for the majority of training examples. @xcite argues that machine learning-based systems rely on the factor of essay length, and that the high correlation between essay length and scores in the standard dataset leads to top performance. following this criticism, aes systems must not rely on essay length, a factor irrelevant to writing proficiency @xcite . recent neural essay scoring systems, which do not employ a feature capturing essay length explicitly, achieve state-of-the-art performance. in this work, however, we first show that even neural essay scoring systems might also be influenced by the correlation between essay length and scores in the standard dataset. to investigate this, we here present a simple neural model manipulating essay length. we notice that averaged rnn outputs are the common component in previous neural models, and we modify this component to manipulate essay length. this simple neural model shows performance comparable to state-of-the-art neural models in the standard dataset. however, this artifact does not hold for a second dataset, the test of english as a foreign language dataset (toefl, @xcite ), which has a lower correlation between essay length and scores. second, we demonstrate that considering essay content without taking essay length into account can improve the performance of a neural essay scoring system. we incorporate a feature representing kullback-leibler divergence into our first simple model, which measures the difference between probability distributions. the intuition is that this feature lets the model consider essay content by assessing the similarity of the word distributions in an input essay and essays assigned different scores. we demonstrate that this neural model achieves performance comparable to the state of the art on both datasets. our experiments show that neural essay scoring systems might also be influenced by the characteristics of the standard dataset foot_1 . ## essay length and scores in datasets datasets: we use two essay datasets, the automated student assessment prize (asap) dataset and the toefl dataset, respectively. asap was introduced in the shared task on evaluating aes systems, measured against human scores. the essays are written by students in grade levels 7 to 10 of us middle schools. since the shared task, asap has been used as a standard dataset for automatic essay scoring. the dataset consists of eight prompts with different linguistic characteristics such as concreteness vs. open-endedness and different scoring ranges. toefl is the standard english test for the entrance to colleges and universities for nonnative students. this dataset has not been commonly used for aes, while it has been used as a standard dataset for another shared task, native language identification @xcite (see the supplementary material for details). ## experimental setup following recent work on asap, we evaluate performance at the prompt level in implementation details: while previous neural models deploy pre-trained embeddings for the asap dataset @xcite , our model builds upon glove, the 100-dimensional pretrained embedding model trained on google news @xcite . we use 100-dimensional glove for all models on toefl. all other settings are identical with previous work. for our models, we test two variations for an rnn module with a gated recurrent unit (gru, @xcite and a large-scale natural language pretraining model @xcite . xl-net not only outperforms bert @xcite which has led to significant improvements in many nlp tasks, but -unlike bert -xlnet can also handle any input sequence length, required for our datasets. we encode a whole text at once using the pretrained language model. evaluation details: for asap, we perform the experiments in line with prior work @xcite , including the same cross-validation (cv) partitions, the same evaluation measure, quadratic weighted kappa (qwk), which measures agreement between annotators considering the agreement occurring by chance, and the same loss function, mean squared error. we use the adam optimizer with a learning rate of 0.001. for toefl, we follow the same setup with the setup of asap, but we use a different learning rate of 0.003. for our implementations, the reported results are obtained by the mean of 10 cv runs with different random seeds. we validate statistical significance by a one-sample t-test with p-@xmath0.01. in ## influence of essay length a neural model manipulating essay length: to investigate the influence of essay length, we present a simple neural model manipulating essay length, which consists of averaged rnn outputs. instead of normalizing the sum of rnn outputs by the actual length of an essay, we normalize them by the average of essay lengths in a prompt. the intuition is that normalizing by the average of essay lengths penalizes an essay of shorter length. as an essay has a fewer number of tokens, they have a fewer number of rnn outputs while the same denominator is applied for all essays in the same prompt. this allows a simple rnn model to capture the influence of essay length better. results: in asap, we first evaluate the performance of the model consisting of averaged rnn outputs with actual essay length, which is commonly used in previous neural models (0.736 < 0.764). we then show that our first neural model manipulating essay length with gru is comparable with the state-of-the-art neural model (0.766 > 0.764). this demonstrates that the simple neural model manipulating essay length is as powerful as state-of-the-art neural models in the standard dataset. in contrast, a simple neural model relying on the pretrained language model not only shows better performance than previous neural models but also show similar performance against the manipulated model using essay length. it supports the claim of the previous work that a large-scale pretrained language model learns linguistic features from input texts @xcite . in toefl, in contrast to asap, we observe that the simple neural model which manipulates essay length shows lower performance than the state-ofthe-art neural model. since toefl has a lower correlation between essay length and scores than asap, we view this as evidence that the performance of previous neural models might be influenced by the correlation of essay length and scores in the target dataset. ## countering the influence a neural model assessing the similarity of essay content: we propose a neural model which considers essay content by assessing the similarity of word distributions in the input essay and essays grouped into three different levels: low, mid, and high. grouping scores to three levels enables the model to handle the different score ranges of each dataset. kl divergence for an input essay x is then defined as kl(p lvl , q) = results: we evaluate the performance of the model considering essay content using kl divergence. in asap, this model with gru leads to 0.8% qwk improvement compared to the previous neural models (0.772 > 0.764). finally, this model with xl-net shows performance comparable to the nonneural state of the art (0.786 > 0.785). we observe a performance gain with the model considering essay content mainly in prompt 7 and 8, while the non-neural state of the art outperforms this model in prompt 4, 5, and 6. we suspect that the different improvements of performance are caused by different linguistic properties in the essays responding to those prompts. the string kernel method is based on character similarity @xcite . it has an advantage in a task which is similar to extractive summarization such as prompt in 4, 5, and 6. these prompts ask the students to write an essay within a given context. in this case, we suspect that including specific information regarding the given context has more influence to human annotators. in contrast, prompts 7 and 8 do not have a given context. they are open-ended. our experiments show that considering essay content leads to significant improvement in this case. in toefl, the model considering essay content with gru shows performance comparable to the state-of-the-art neural models (0.669 > 0.667). eventually, this model with xlnet outperforms all other models (0.728 > 0.669), and it also leads to a 1.0% improvement compared to the model manipulating essay length. this also supports our finding that considering essay content leads to a performance improvement for neural essay scoring systems. unlike asap, the model considering essay content models on the toefl show consistent performance gain regardless of prompts. we believe this is caused by an overall higher quality of toefl dataset. the prompts do not vary so much, the student population is more controlled, and the essays have a similar length. we also compare with the state of the art on toefl, @xcite . we notice that the performance reported in @xcite cannot be compared with previous work due to a different experimental setup. they filter out content whose sentence length is longer than 40 words or whose document length is longer than 25 sentences, which results in filtering the more than 7.5% of sentences; they also evaluate performance without cv foot_2 . to ensure a fair comparison, we only modified the experimental setup in their implementation. the model proposed in @xcite shows substantially lower performance than all models in the same experimental setup with previous work. ## conclusions while the recent neural essay scoring systems emphasize novel aspects of the neural architecture, they have neither shown significant improvement nor helped interpreting the scores assigned by humans. we show that recent neural models might also be influenced by characteristics of the standard dataset, a high correlation between essay length and scores. to investigate this, we evaluate models on two datasets, not only the standard dataset, asap, but also toefl, which has a lower correlation between essay length and scores. our findings suggest that neural essay scoring systems should focus on text quality, and at the same time, should consider characteristics of the target dataset. in natural language processing (emnlp), pages 217-235, online. association for computational linguistics. michael winerip. 2005. sat essay test rewards length and ignores errors. new york times, 9. edward w. wolfe, tian song, and hong jiao. 2016. features of difficult-to-score essays. assessing writing, 27:1-10. zhilin yang, zihang dai, yiming yang, jaime carbonell, russ r. salakhutdinov, and quoc v. le. 2019. xlnet: generalized autoregressive pretraining for language understanding. in advances in neural information processing systems, pages 5754-5764. 8 appendix a. dataset details