Automated Essay Scoring: A Reflection on the State of the Art (2024)

Abstract: While steady progress has been made on the task of automated essay scoring (AES) in the past decade, much of the recent work in this area has focused on developing models that beat existing models on a standard evaluation dataset. While improving performance numbers remains an important goal in the short term, such a focus is not necessarily beneficial for the long-term development of the field. We reflect on the state of the art in AES research, discussing issues that we believe can encourage researchers to think bigger than improving performance numbers with the ultimate goal of triggering discussion among AES researchers on how we should move forward.

## introduction automated essay scoring (aes), the task of automatically assigning a holistic score to an essay that summarizes its overall quality, is arguably one of the most important applications in natural language processing (nlp). as an example of aes, consider the essay in despite the fact that aes has been investigated for more than 50 years @xcite , the task is still far from being solved. nevertheless, aes has been progressing steadily. in recent years, a standard recipe for publishing in this area of research seems to involve proposing a sophisticated neural model that can beat competing models on a standard evaluation dataset such as asap (see section 2.1 for details). while improving performance numbers remains an important short-term goal for aes research, such a focus is not necessarily beneficial for the long-term development of the field. in this position paper, we reflect on the state of the art in aes research. specifically, we discuss issues that we believe can encourage researchers, particularly junior researchers, to think bigger than merely improving performance numbers, with the ultimate goal of triggering discussion among aes researchers on how we should move forward. 1 these issues range from evaluation (sections 3 and 4) to new tasks (sections 5 and 6) to corpus annotation (sections 7 and 8) and beyond (section 9). specifically, the author talks about computers giving people entertainment and lists some general social networking websites, but there is no elaboration on how computers enhance access to entertainment. in terms of organization, the essay has a basic structure: an introduction, three main body sentences, and a conclusion, but the ideas within each body sentence are poorly connected. moreover, the transitions are rather awkward and repetitive. for example, the author uses "so as you can see" three times. while the author exhibits some awareness of the audience as the essay is addressed to a local newspaper, it is not clear whether the essay is intended to urge the editor to write an article on how computers benefit people. ## aes is more than improving qwk what have we learned about aes over the years other than the fact that the qwk scores are improving on standard evaluation datasets? in the preneural nlp era when many aes researchers were focusing on feature engineering, we could at least gain some insights into what kind of features would be useful for aes through feature ablation experiments. with neural aes models, the representation of an essay is learned. many of them do not employ hand-crafted features, and hence the output cannot be interpreted easily. we should note, however, that researchers have recently begun to explore the explainability of aes models. for instance, kumar and boulanger (2021) applied explainable ai techniques, such as shap values, to enhance the transparency and interpretability of their aes system. @xcite developed a methodology that aligns the functional components derived from transformer models with human-understandable feature groups. @xcite explored various interpretability methodologies in machine learning that can be adapted to aes for increased transparency, such as lime @xcite . recommendation #1: we recommend that researchers understand the strengths and weaknesses of the systems they developed by performing a qualitative and quantitative analysis of the system outputs. for example, while it is not uncommon for aes researchers to report results that are better than existing results when averaged over all essay prompts in the corpus, without further analysis, a reader would not know where the improvements came from. does the proposed model perform better on all types of essays (e.g., persuasive essays, narratives) or only certain types of essays, and if it performs better on all types of essays, are there certain essay types for which the improvement is more pronounced? does the model perform better because it can better distinguish between essays that are similar w.r.t. particular traits (e.g., organization)? is the model better because it scores the essays belonging to the minority classes better? note that to answer these questions, we cannot resort to interpretability techniques. rather, an error analysis on model outputs is needed. ## asap is not the only essay corpus as mentioned before, in recent years many aes researchers evaluated their model solely on asap. a natural question is: have existing models overfitted asap? in other words, can models that perform well on asap be expected to generalize well to other essay corpora? unfortunately, little analysis has been performed on asap that would enable us to gain a better understanding of whether certain corpus-specific characteristics are present in asap that could cause models to overfit the data. for example, what fraction of the persuasive essays in asap made arguments that are persuasive? if the vast majority of these essays contained persuasive arguments, then a model trained on asap might not perform well on a corpus where it is important to distinguish between essays with persuasive arguments and those with unpersuasive arguments. one piece of information that we know about asap is that the essays it contains were written by american students in grades 7-10. while the native languages of these students are not known, the majority of them are expected to be native speakers of english. given this context, asap is not an ideal corpus for the development and evaluation of aes systems. recall that one of the primary motivations and key driving forces behind the development of aes systems is its application to scoring essays written by takers of standardized tests such as toefl and sat, where many of the exam takers are learners of english as a second language (esl). as is well-known to essay grading researchers, the common types of error made by esl learners who speak different native languages are not the same @xcite . hence, it is possible that the asap essays are simpler to score because grammatical errors may be less of an issue in these essays. in contrast, for aes of essays written by non-native speakers, aes systems may need to be combined with grammatical error correction systems for more accurate modeling of the quality of an essay w.r.t. the grammar trait. another piece of information that we know about asap (and aes corpora where essays are written in a time-restricted setting such as a test setting) is that essay length (as measured by the number of words) is often a confounding variable for holistic scoring @xcite . consequently, it is not clear how models trained on asap would perform when applied to corpora where essays are written in a time-unrestricted setting, such as essays written as part of a homework with a length restriction (e.g., between 500 and 600 words), as length may no longer be a confounding variable in these corpora. recommendation #2: to understand whether models trained on asap can generalize well when applied to other essay corpora, we recommend that aes researchers evaluate their systems on not only asap but at least one other publicly available corpus, preferably a corpus where the native languages of the essay writers are different from the language in which the essays were written, such as clc-fce @xcite or a corpus where essays were written in a time-unrestricted setting, such as icle++ @xcite . ## is it time for cross-prompt aes? cross-prompt aes is a challenging task. below we discuss the challenges from three perspectives. ## traits are useful but under-explored two key challenges discussed above involve building aes models that are interpretable and generalizable to unseen prompts. next, we propose to examine the use of traits to address both challenges. interpretability. if a holistic scoring model is built by using only traits as inputs (such as the heuristic aes models described in section 2), then interpretability is relatively straightforward. for instance, if a linear model is used where the holistic score is computed as a weighted sum of the traitspecific scores, then the holistic score can be easily explained by the trait-specific scores. even if a non-linear model is used, the attention weights can be used to shed light on how much influence a particular trait has on the holistic score. not only do essay traits make interpretability straightforward, but the trait scores also serve to provide feedback to an essay's writer. for example, if someone gets a low holistic score for their essay, they can identify which aspects of the essay need improvement by checking which trait-specific scores are low. cross-prompt generalizability. recent efforts on cross-prompt aes research have involved identifying prompt-independent features, such as lengthbased features. while traits have been used as features for holistic scoring, they have primarily been used for within-prompt rather than cross-prompt scoring. one reason, we believe, is that traits are rarely viewed as prompt-independent features. given the prompt-independent nature of traits, we propose to use them for cross-prompt aes. specifically, we propose to view the set of traits commonly known to play a role in determining the holistic score of a particular type of essay (e.g., persuasive essays) as a prompt-independent representation of these essays. note that this trait-based essay representation is dependent on the essay type, as the traits that influence the holistic score of a persuasive essay are not necessarily the same as those that influence narrative essays, for instance. in practice, however, using traits to address interpretability and cross-prompt generalizability is challenging. the key challenge stems from the fact that content-based traits are difficult to score accurately @xcite , even when labeled data is available for training content-based trait-scoring models. one possibility is to leverage llms for scoring traits in a zero-shot setting, where we provide the rubric developed for a given trait as part of the instructions for an llm when requesting it to score the trait. should we believe that llms can help us accurately perform trait scoring? as mentioned in section 2, state-of-the-art llms are very good at understanding inputs as complex as essays. since scoring content-based traits requires an understanding of essay content, exploring the use of llms for trait scoring is a promising direction. nevertheless, it is worth noting that recent prompt-based approaches to holistic scoring @xcite , including those that employ a chainof-thought approach @xcite , are not as competitive as fine-tuned models in performance. while the underlying reasons are not yet clear, researchers have demonstrated that minor changes in the prompt @xcite , changes in the decoding methods @xcite , and variations in random seeds @xcite can all result in significant performance changes. given these results, one should expect similar challenges when prompting llms for trait scoring. recommendation #4: we recommend that a thorough investigation of the impact of traits on holistic scoring be conducted as they could be a viable solution to key problems concerning neural model interpretability and cross-prompt model generalizability. given the difficulty in computing content-based traits, we recommend that researchers examine how llms can be exploited, possibly via promptingbased approaches, to score content-based traits. ## corpus development is slow the fact that asap is still the primary corpus used for evaluating aes models more than 10 years after its initial release is somewhat unusual in the nlp community for a task that is as popular as aes. this perhaps suggests that corpus development is seriously lagging behind model development. we believe the reasons are at least two-fold. ## essay grading beyond scoring while aes has largely been tackled as a standalone task, researchers can think about how aes systems can be combined with other technologies developed for ai in education. for example, to improve the usability of aes systems in a classroom setting, aes models can be embedded in an intelligent tutoring system that can help an essay writer improve their essay in an iterative fashion through multiple rounds of feedback and revision. recommendation #7: we recommend that aes researchers think beyond essay scoring and develop a vision of how aes models can be combined with related ai technologies to develop more complex systems that can broaden the impact of aes research on education, ideally through discussions with other stakeholders such as teachers. ## conclusion for the majority of the aes models proposed in the past few years, researchers have largely focused on improving performance numbers. while improving performance numbers should remain an important goal for aes, merely focusing on performance numbers is not necessarily healthy for aes research in the long run. in this position paper, we discussed a range of issues that we believe aes researchers should seriously consider as we move forward and provided seven recommendations based on our observations. we hope these recommendations will help initiate discussion among aes researchers and push the field forward.