Many Hands Make Light Work: Using Essay Traits to Automatically Score Essays (2022)

Abstract: Most research in the area of automatic essay grading (AEG) is geared towards scoring the essay holistically while there has also been little work done on scoring individual essay traits. In this paper, we describe a way to score essays using a multi-task learning (MTL) approach, where scoring the essay holistically is the primary task, and scoring the essay traits is the auxiliary task. We compare our results with a single-task learning (STL) approach, using both LSTMs and BiLSTMs. To find out which traits work best for different types of essays, we conduct ablation tests for each of the essay traits. We also report the runtime and number of training parameters for each system. We find that MTL-based BiLSTM system gives the best results for scoring the essay holistically, as well as performing well on scoring the essay traits. The MTL systems also give a speed-up of between 2.30 to 3.70 times the speed of the STL system, when it comes to scoring the essay and all the traits.

## introduction an essay is a piece of text that is written in response to a topic, called a prompt @xcite . qualitative evaluation of the essay consumes a lot of time and resources. hence, in 1966, page proposed a method of automatically scoring essays using computers @xcite , giving rise to the domain of automatic essay grading. essay traits are different aspects of the essay that can aid in explaining the score assigned to the essay. examples of essay traits include content (how much information is present in the essay) @xcite , organization (how well the essay is structured) @xcite , style (how well written the essay is) @xcite , prompt adherence (how much the essay stays on topic for the essay prompt) @xcite , etc. most of the research work done in the field of aeg is geared toward scoring the essay holistically, rather than studying the importance of essay traits in the overall essay score. in this paper, we ask the question: in our paper, not only do we score essays holistically, but we also describe how to score essay traits simultaneously in a multi-task learning framework. scoring essay traits is essential as it could help in explaining why the essay was scored the way it was, as well as providing valuable insights to the writer about what aspects of the essay were well-written and what the writer needs to improve. multi-task learning is a machine learning technique where we use information from multiple auxiliary tasks to perform a primary task @xcite . in our experiments, scoring the individual essay traits is the auxiliary task, and scoring the essay holistically is the primary task. in addition to this, we also study the impact of scoring an essay trait as the primary task while the other traits and overall essay score are auxiliary tasks. contributions. in this paper, we describe a way to simultaneously score essay traits and the essay itself using multi-task learning. we evaluate our system against different types of essays and essay traits. we also share our code and the data for reproducibility and further research foot_0 . ## motivation most of the work done in the area of automatic essay grading is in the area of holistic aeg -where we provide a single score for the entire essay based on its quality. however, for writers of an essay, a holistic score alone would not be enough. providing trait-specific scores will tell the writer which aspects of the essay need improvement. in our dataset, we observe that writers of good essays usually have a lot of content, appropriate word choice, very few errors, etc. essays that are poorly written often lack one or more of these qualities (i.e. they are either too short, have lots of errors, etc.). we, therefore, observe a high correlation between individual trait scores and the overall essay score (pearson correlation trait scores and overall essay @xmath0.7 across all essay sets in our dataset). hence, we believe that using essay trait scores will benefit in scoring the essay holistically, as their scores will provide more relevant information to the aeg system. ## dataset used for our experiments, we use the automated student's assessment prize (asap) automatic essay grading (aeg) dataset. the dataset has a total of 8 essay sets -where each essay set has a number of essays written in response to the same essay prompt. in total, there are nearly 13,000 english essays in the dataset, written by american high school students from classes 7 to 10. we use the overall scores directly from the asap aeg dataset. since the original dataset only provided trait-specific scores for prompts 7 & 8, we use the trait-specific scores provided by @xcite . depending on the type of prompt for the essay set, each essay set has a different set of traits. argumentative / persuasive essays are essays which the writer is prompted to take a stand on a topic and argue for their stance. these essay sets have traits like content, organization, word choice, sentence fluency, and conventions. source-dependent responses @xcite are essays where the writer reads a piece of text and answers a question based on the text that they just read 5 . these essay sets have traits like content, prompt adherence @xcite , language and 5 a sample prompt is "based on the excerpt, describe the obstacles the builders of the empire state building faced in attempting to allow dirigibles to dock there. support your answer with relevant and specific information from the excerpt." it involves the writer reading the excerpt from the empire state building by marcia amidon lusted before writing the essay. ## experiments 6.1 evaluation metric we use cohen's kappa with quadratic weights (cohen, 1968) (qwk) as the evaluation metric. this is done for the following reasons. firstly, the final scores predicted by the system are distinct numbers/grades, rather than continuous values; so we cannot use the pearson correlation coefficient or mean squared error. secondly, evaluation metrics essay set trait 1 trait 2 trait 3 trait 4 trait 5 trait 6 prompt 1 content organization word choice sentence fluency conventions n/a prompt 2 content organization word choice sentence fluency conventions n/a prompt 3 content prompt adherence language narrativity n/a n/a prompt 4 content prompt adherence language narrativity n/a n/a prompt 5 content prompt adherence language narrativity n/a n/a prompt 6 content prompt adherence language narrativity n/a n/a prompt 7 content organization style conventions n/a n/a prompt 8 content organization voice word choice sentence fluency conventions ## results and analysis in this section, we report our results and analyze them for different experiments. ## conclusion and future work in this paper, we described an approach to use multi-task learning to automatically score essays and their traits. we achieve this by concatenating a representation of the essay with the trait scores -predicted as an auxiliary task. we compared our results with single-task learning models as well. we found out that the mtl system with the bi-directional lstm outperforms the stl-based systems and has results comparable with a baseline bert-stl system. we then ran an ablation test and found out which essay trait was important for the corresponding essay sets. we also report our system's performance, which shows a 2.30 to 3.70 speed-up of using the multi-task learning system, compared to using a single task learning system. an exciting avenue of future work is using trait scoring to aid in providing text feedback to the writer, like showing where the low score for the trait originates, similar to @xcite (for content scoring), rather than a trait-specific score only. we also plan to investigate using albert @xcite , in lieu of the essay stack, to grade essays and their traits simultaneously. we also plan to explore how to extend our approach in generalizing our system, training it on essays written in response to one set of source prompts, and tested it on essays written for another prompt.