Can Neural Networks Automatically Score Essay Traits? (2020)

Abstract: Essay traits are attributes of an essay that can help explain how well written (or badly written) the essay is. Examples of traits include Content, Organization, Language, Sentence Fluency, Word Choice, etc. A lot of research in the last decade has dealt with automatic holistic essay scoring - where a machine rates an essay and gives a score for the essay. However, writers need feedback, especially if they want to improve their writing - which is why trait-scoring is important. In this paper, we show how a deep-learning based system can outperform feature-based machine learning systems, as well as a string kernel system in scoring essay traits.

## introduction an essay is a piece of text that is written in response to a topic, called a prompt. writing a good essay is a very useful skill. however, evaluating the essay consumes a lot of time and resources. hence, in 1966, ellis page proposed a method of evaluation of essays by computers @xcite . the aim of automatic essay grading (aeg) is to have machines, rather than humans, score the text. an aeg system is a software that takes an essay as input and returns a score as output. that score could either be an overall score for the essay, or a trait-specific score, based on essay traits like content, organization, style, etc. to the best of our knowledge, most of the systems today use feature engineering and ordinal classification / regression to score essay traits. from the late 1990s / early 2000s onwards, there were many commercial systems that used automatic essay grading. shermis and burstein (2013) cover a number of systems that are used commercially, such as e-rater @xcite , intelligent essay assessor @xcite , lightside @xcite , etc. in 2012, kaggle conducted a competition called the automatic student assessment prize (asap), which had 2 parts -the first was essay scoring, and the second was short-answer scoring. the release of the asap aeg dataset foot_0 led to a large number of papers on automatic essay grading using a number of different techniques, from machine learning to deep learning. section 3 lists the different work in automatic essay grading. in addition to the kaggle dataset, another dataset -the international corpus of learner's english (icle) -is also used in some trait-specific essay grading papers @xcite . our work, though, makes use of only the asap dataset, and the trait specific scores provided by @xcite for that dataset. the rest of the paper is organized as follows. in section 2, we give the motivation for our work. in section 3, we describe related work done for traitspecific automatic essay grading. in section 4, we describe the dataset. in section 5, we describe the experiments, such as the baseline machine learning systems, the string kernel and super word embeddings, the neural network system, etc. we report the results and analyze them in section 6, and conclude our paper and describe future work in section 7. ## motivation most of the work dealing with automatic essay grading either deals with providing an overall score to the essay, but often doesn't provide any more feedback to the essay's writer @xcite . one way to resolve this is by using trait-specific scoring, where we either do feature engineering or construct a neural network, for individual traits. however, coming up with different systems for measuring different traits is often going to be a challenge, especially if someone decides to come up with a new trait to score. our work involves showing how we can take existing general-purpose systems, and use them to score traits in essays. in our paper, we demonstrate that a neural network, built for scoring essays holistically, performs reasonably well for scoring essay traits too. we compare it with a task-independent machine learning system using task independent features (mathias and bhattacharyya, 2018a), as well as a stateof-the-art string kernel system @xcite and report statistically significant results when we use the attention based neural network @xcite . ## related work in this section, we describe related work in the area of automatic essay grading. ## dataset the dataset we use is the asap aeg dataset. the original asap aeg dataset only has trait scores for prompts 7 & 8. mathias and bhattacharyya (2018a) provide the trait scores for the remaining prompts foot_1 . tables 1 and 2 ## experiments we use the following systems for our experiments: ## results and analysis we see that the attention-based neural network system is able to outperform both, the baseline system of @xcite and the histogram intersection string kernel system of between the other 2 systems, the string kernels system performed better than the baseline system in most of the cases. the only prompt in which it did not do so was in prompt 8 -mainly because of the number of essays being very low and the size of the essay being very high compared to the other prompts. among the traits, the easiest to score are the traits of content and prompt adherence (where ever they are applicable) as they yielded the best agreement with the human raters. the hardest of the traits to score was voice, which yielded the lowest qwk in the only prompt in which it was scored. ## conclusion and future work in this paper, we describe a comparison between a feature-engineering system, a string kernel-based system, and an attention-based neural network to score different traits of an essay. we found that the neural network system provided the best results. to the best of our knowledge, this is the first work that describes how neural networks are used, in particular, to score essay traits. as part of future work, we plan to investigate how to incorporate trait scoring as a means of helping to score essays holistically.