Conundrums in Cross-Prompt Automated Essay Scoring: Making Sense of the State of the Art (2024)

Abstract: Cross-prompt automated essay scoring (AES), an under-investigated but challenging task that has gained increasing popularity in the AES community, aims to train an AES system that can generalize well to prompts that are unseen during model training. While recently-developed cross-prompt AES models have combined essay representations that are learned via sophisticated neural architectures with so-called prompt-independent features, an intriguing question is: are complex neural models needed to achieve state-of-the-art results? We answer this question by abandoning sophisticated neural architectures and developing a purely feature-based approach to cross-prompt AES that adopts a simple neural architecture. Experiments on the ASAP dataset demonstrate that our simple approach to cross-prompt AES can achieve state-of-the-art results.

## introduction automated essay scoring (aes) is the task of assigning a single score (also known as the holistic score) to an essay that summarizes its overall quality. traditional work on aes has focused on within-prompt scoring, where an aes model is trained on manually annotated essays written for a given prompt and subsequently applied to essays written for the same prompt. although considerable success has been made on within-prompt scoring, there is a key weakness associated with these prompt-specific models. when they are applied to essays written for a different prompt, their performance often deteriorates considerably. hence, in practice, before they are applied to score essays written for a new prompt, they need to be retrained on scored essays written for the new prompt. however, manually scoring essays is a time-consuming process and requires a lot of expertise. to address the aforementioned weakness, researchers have begun working on cross-prompt essay scoring, where the goal is to train a model on annotated essays that are not written for the target prompt and apply the resulting model to essays written for the target prompt. in other words, the goal of cross-prompt scoring is to train a model on essays written for existing prompts so that it can accurately score essays written for a new prompt without the need to retrain it on essays from the new prompt. cross-prompt scoring is a very challenging task. to understand why, consider the task of scoring essays written for the prompt "write a persuasive essay on why one should (or should not) support obamacare". intuitively, a high-scoring essay should provide evidence(s) that can adequately support the claim of why one should (or should not) support obamacare. however, determining whether an argument is persuasive could require domain knowledge (in this case knowledge about obamacare), which the model may not possess in the absence of training data for the new prompt. 1 research on cross-prompt scoring is still in its infancy. the vast majority of the recently-developed cross-prompt scoring models (e.g., @xcite ; @xcite ; @xcite ) are composed of two parts: (1) learning a representation of an essay that is specific for the task of aes by training convolutional neural networks (cnns) and long short-term memory (lstm) networks using aes data; and (2) employing a set of prompt-independent features derived from the input essay. designing prompt-independent features is by no means the focus of cross-prompt scoring research: the features being used in existing cross-prompt aes models were designed by aes researchers in the pre-neural nlp era during which the focus of aes research was on feature engineering. the first part (learning a task-specific essay representation) is what cross-prompt scoring researchers have been trying to improve over the past few years, resulting in aes models that have become increasingly sophisticated. much of the recent work on aes, including cross-prompt aes, has focused on improving performance numbers. while improving performance numbers is an important goal of aes research, what is crucially missing is an understanding of what has been improved. for instance, as mentioned above, state-of-the-art cross-prompt aes models score essays by exploiting both a learned essay representation and a set of prompt-independent features. before continuing to develop increasingly sophisticated aes models, we should perhaps step back and ask an intriguing question: can we achieve state-of-the-art results if we abandon complex neural architectures and, like the aes researchers in the pre-neural nlp era, adopt a purely feature-based approach with a simple architecture? given the above discussion, our goal in this paper is to gain a better understanding of the state of the art in cross-prompt aes by answering the aforementioned question. unlike recently-developed cross-prompt aes models, which focus on designing increasingly sophisticated aes models to learn essay representations, we abandon entirely the idea of learning essay representations, focusing instead on a purely feature-based approach where we employ a simple neural architecture in conjunction with a feature set composed of features commonly used in existing cross-prompt aes models as well as our own features. following previous work on cross-prompt aes (e.g., @xcite , chen and li (2023)), we evaluate our approach in two settings that differ in terms of whether essay traits (i.e., dimensions of essay quality such as orga-nization and mechanics) are exploited when scoring an essay holistically. more specifically, in the first setting, essays are scored based on a set of input features, whereas in the second setting, essays are scored based on both the features and the automatically scored traits. we evaluate our cross-prompt aes model on a standard evaluation dataset for aes research, asap, showing that when used in combination with a simple feature selection method, our model can achieve state-of-the-art results in the two evaluation settings mentioned above. we believe that the key ramifications of our results are two-fold. first, a purely feature-based approach for crossprompt aes with a simple neural architecture can work at least as well as a sophisticated model that focuses on learning an essay's representation and combining it with a set of features. second, feature selection plays a crucial role in our model despite the simplicity of our feature selection method. furthermore, our results establish a new simple but strong baseline against which future cross-prompt aes models can be compared. ## related work in this section, we give a brief overview of related work on within-and cross-prompt scoring. for a detailed discussion of related work, we refer the reader to the books and surveys recently published in this area (e.g., @xcite , beigman klebanov and madnani (2021), li and ng (2024a)). ## corpus for model training and evaluation, we employ the widely-used asap foot_1 corpus and its and its extension, asap++ @xcite . asap (automated student assessment prize), which is released as part of a kaggle competition in 2012, is composed of essays manually annotated with their holistic scores. the essays are written for eight prompts, including two for persuasive essays, two for narrative essays, and four for source-dependent essays. different rubrics are used for scoring prompts, and as a result, the score ranges for different prompts can be different. for instance, one prompt has a score range of 0 to 3, while another prompt has a score range of 1 to 6. the eight prompts and their statistics can be found in appendix a. asap++ is an extension of asap where each essay is additionally scored along different traits. eight traits are scored, including content (how clear and focused the writing is and how welldeveloped the main ideas are), word choice (how well the words convey the intended message), organization (how well-organized the essay is), prompt adherence (how adherent the essay is to the prompt), sentence fluency (whether the sentences in the essay are of high quality), conventions (how well the essay demonstrates standard writing conventions), narrativ-ity (how coherent and cohesive the response is), and language (how good grammar and spelling are). note, however, that the traits that are used for essays written for different prompts can be different: some essays are scored along three traits while some are scored along six traits. additional information on the traits can be found in appendix b. ## approaches to holistic scoring in this section, we describe our two feature-based approaches to holistic scoring. the first one does not exploit essay traits (section 4.1), whereas the second one does (section 4.2). ## conclusion we examined the relatively under-studied task of cross-prompt essay scoring, seeking to understand (1) whether state-of-the-art performance on crossprompt scoring could only be achieved using sophisticated models and ( 2 ) what role the features played in the scoring process. for this reason, we proposed a purely feature-based approach to crossprompt scoring that combined existing features with those of our own, achieving state-of-the-art results on the asap dataset when our feature set was used to train a simple neural architecture. not only does our work establish a strong baseline against which future work can be compared, but it serves to remind researchers that understanding which portions of a complex model are chiefly responsible for performance improvements is as important as demonstrating performance improvements itself.