ICLE ++: Modeling Fine-Grained Traits for Holistic Essay Scoring (2024)

Abstract: The majority of the recently developed models for automated essay scoring (AES) are evaluated solely on the ASAP corpus. However, ASAP is not without its limitations. For instance, it is not clear whether models trained on ASAP can generalize well when evaluated on other corpora. In light of these limitations, we introduce ICLE++, a corpus of persuasive student essays annotated with both holistic scores and trait-specific scores. Not only can ICLE++ be used to test the generalizability of AES models trained on ASAP, but it can also facilitate the evaluation of models developed for newer AES problems such as multi-trait scoring and cross-prompt scoring. We believe that ICLE++, which represents a culmination of our long-term effort in annotating the essays in the ICLE corpus, contributes to the set of much-needed annotated corpora for AES research.

## introduction the past decade has seen considerable progress on automated essay scoring (aes), the task of automatically assigning a (holistic) score to an essay that summarizes its overall quality. the reason can in part be attributed to the public availability of annotated aes corpora where each essay is manually annotated with its holistic score. these corpora have facilitated the development and evaluation of aes models and enabled easy tracking of progress. while several aes corpora have been developed over the years (including english corpora such as clc-fce (yannakoudakis et al., 2011), asap 1 , toefl11 (blanchard et al., 2013), as well as corpora in other languages such as merlin 2 (multilingual), ostling's (2013) swedish corpus, and horbach et al.'s (2017) german corpus), the vast majority of recently-developed aes models have been evaluated solely on the asap corpus (taghipour 1 https://www.kaggle.com/c/asap-aes 2 https://www.merlin-platform.eu/ and ng, 2016; uto et al., 2020; ridley et al., 2021; kumar et al., 2022; chen and li, 2023). asap is released as part of a kaggle competition in 2012. while for the time being it may be fine to focus the evaluation of aes models on asap, continuing to do so may not be ideal for aes research in the long run, for the following reasons: generalizability. asap is composed of essays written by u.s. students between grade 7 and grade 10. a natural question, then, is: will models trained on asap perform well on other aes corpora? for instance, since the asap essays are written by native speakers of english, can the resulting models perform well on the toefl essays, which are written by learners of english as a second language? in addition, essay length (as measured by the number of words in the essay) has been found to be a confounding variable for holistic scoring in asap and other aes corpora where essays are written in a time-restricted setting (such as a test setting), so it is not clear how models trained on asap would perform if they are applied to corpora where essays are written in a time-unrestricted setting, such as essays written as part of a homework assignment with a length restriction (e.g., between 500 and 600 words), as length may no longer be a confounding variable in these aes corpora. limited feedback. while the performance of aes models has been increasing steadily over the years, these models provide little feedback on what aspects of an essay need improvement if it is assigned a low score by an aes model. as is commonly known, a holistic score is dependent on a number of trait-specific scores (i.e., scores along different dimensions of essay quality such as or-ganization and prompt adherence). hence, if an aes model can provide trait-specific scores in addition to holistic scores, the trait-specific scores can serve as feedback for students on which aspects of their essays need improvement. while early heuristic-based aes models such as e-rater (attali and burstein, 2006) compute the holistic score of an essay as a weighted sum of heuristically computed trait-specific scores (and hence it is straightforward to understand which traits need improvement if the holistic score is low), the traits are primarily restricted to those that are non-content-based (i.e., traits that can be scored without an understanding of an essay's content, such as organization). given the difficulty associated with trait-specific scoring (particularly the scoring of content-based traits such as argument persuasiveness), few learning-based aes models explicitly exploit traits. nevertheless, research on trait-specific scoring, including the scoring of content-based traits, exists. for instance, we previously worked on learningbased trait-specific scoring (persing et al., 2010; persing and ng, 2013, 2014, 2015), but did not study the impact of the trait-specific scores on holistic scoring. asap++ (mathias and bhattacharyya, 2018), an extension of asap where each essay is annotated with trait-specific scores, has facilitated the development of aes models in recent years that allow trait-specific scores to be predicted jointly with holistic scores. however, the traits that are being scored in asap++ are arguably too coarse-grained. specifically, while asap++ provides scores for all non-content-based traits, all content-based traits (e.g., coherence, persua-siveness, thesis clarity) are lumped into a single trait that they call content. these coarsegrained traits not only severely limit the kind of content-based feedback that can be provided to an essay's author, but could prevent an aes model from mimicking the human essay scoring process, where different content-based traits are considered separately. @xcite appropriateness for cross-prompt scoring. aes researchers have traditionally focused on within-prompt scoring, where models trained on essays written for a given prompt are applied to essays written for the same prompt. however, withinprompt scoring may not be a practical setting, as an aes model typically does not perform well on essays written for a new prompt unless it is (re)trained on essays from the new prompt. consequently, researchers have begun investigating a new, challenging evaluation setting known as cross-prompt scoring: given training essays written for a set of prompts, the goal is to train a model to score essays written for prompts not seen during training. asap has been used for evaluating cross-prompt models. recall that asap is composed of eight prompts, including two for persuasive essays, two for narrative essays, and four for source-dependent essays. cross-prompt evaluation is typically conducted via leave-one-prompt-out cross-validation experiments, where in each iteration essays from exactly one prompt are used for testing and the remaining ones are used for training. this implies, for instance, that a cross-prompt model that is applied to score the narrative essays from one of the prompts has been trained on not only narrative essays but also persuasive and source-dependent essays. this is a rather strange setup: since what constitutes a good persuasive essay may not be the same as what constitutes a good narrative essay (as they are scored using different rubrics), it is not clear whether it even makes sense to perform cross-prompt scoring when the training essays do not have the same type as the test essays. in light of the above discussion, we introduce icle++, a corpus of persuasive essays written for 10 prompts where each essay is annotated with not only its holistic score but also its trait-specific scores. icle++ represents a culmination of our long-term effort in annotating the persuasive essays in the icle corpus that was initiated in the fall of 2009. these essays are written by university undergraduates from 16 countries who are learners of english as a foreign language. hence, icle++ complements well with asap, where essays are written by pre-college students who are native speakers of english, enabling the evaluation of the generalizability of aes models. unlike many existing aes corpora, essay length is no longer a confounding variable in the scoring process for icle++, as the vast majority of essays are between 500 and 600 words. in addition, since all essays are persuasive, icle++ provides a natural setup for crossprompt scoring, where one can determine whether the knowledge acquired from the persuasive essays written for the training prompts would be useful for scoring the persuasive essays written for a new prompt. above all, we identify a set of 10 traits that humans use when scoring persuasive essays. by scoring essays with these 10 traits, icle++ enables us to gain a deeper understanding of the human essay scoring process, specifically by determining the relative importance that a human puts score description 4 essay provides ample support for its claims, uses effective vocabulary and sentence variety, organizes ideas logically, develops them well, conveys them concisely, and connects them with smooth transitions 3.5 essay offers generally sufficient support for its claims, uses appropriate vocabulary and sentence variety, organizes ideas logically, develops them well, conveys them concisely, and connects them with appropriate transitions 3 essay supports its claims adequately but the support may not be even, develops and organizes ideas reasonably well but the transitions between ideas may not be smooth, shows adequate command of language to convey ideas clearly 2.5 essay offers support that is of little relevance to its claims, provides limited logical development and organization of ideas, shows problems in language, grammar and/or sentence structure that result in a lack of clarity 2 essay provides little or no relevant support for its claims, has ideas that are not developed, illogical, and/or poorly organized, and has problems in language, grammar and/or sentence structure that often obscure meaning 1.5 essay provides little or no support for its claims, has disorganized ideas, is overly short, and has enough problems in language, grammar and/or sentence structure that make the essay nearly impossible to understand 1 essay is off topic ## corpus our corpus, icle++, is composed of essays selected from the international corpus of learner english (granger et al., 2009), which consists of more than 6000 essays on a variety of writing topics written by university undergraduates from 16 countries and 16 native languages who are learners of english as a foreign language. 91% of the essays are persuasive. to ensure representation across the native languages of the authors, we selected mostly essays written in response to topics that are well-represented in multiple languages. this avoids many issues that may arise when certain vocabulary is used in response to a particular topic for which essays written by authors from only a few languages are available. with this criterion in mind, we selected 1008 persuasive essays for annotation. these essays are written for 10 prompts (see appendix a) and have eight paragraphs, 32 sentences, and 582 tokens on average. thirteen native languages are represented in these essays. ## conclusion we presented icle++, a corpus of persuasive essays annotated with both holistic scores and 10 fine-grained trait-specific scores. we believe that icle++ contributes to the much-needed set of annotated aes corpora and will be a valuable resource to aes researchers. we make all of our annotations publicly available. @xcite