TRATES : Trait-Specific Rubric-Assisted Cross-Prompt Essay Scoring (2025)

Abstract: Research on holistic Automated Essay Scoring (AES) is long-dated; yet, there is a notable lack of attention for assessing essays according to individual traits. In this work, we propose TRATES, a novel trait-specific and rubric-based cross-prompt AES framework that is generic yet specific to the underlying trait. The framework leverages a Large Language Model (LLM) that utilizes the trait grading rubrics to generate trait-specific features (represented by assessment questions), then assesses those features given an essay. The trait-specific features are eventually combined with generic writing-quality and prompt-specific features to train a simple classical regression model that predicts trait scores of essays from an unseen prompt. Experiments show that TRATES achieves a new state-of-the-art performance across all traits on a widely-used dataset, with the generated LLM-based features being the most significant.

## introduction automated essay scoring (aes), stemming from page's early study @xcite , has seen notable progress in addressing writing evaluation. it covers holistic scoring (giving a single score for overall proficiency) and trait scoring (assessing specific writing aspects like organization). while holistic scoring offers a broad assessment of writing ability, trait scoring provides detailed feedback to aid students in targeted skill improvement. due to the complexity of evaluating different traits, holistic scoring has been the predominant focus in aes research. various methodologies have been introduced, ranging from approaches relying on handcrafted features @xcite to language model-based ones @xcite . cross-prompt aes is gaining momentum in recent aes research, with the goal of training a model that can effectively score essays from unseen prompts. this approach is not only more practical, reflecting real-world scenarios where models must generalize across diverse prompts, but also more challenging, as it demands the ability to adapt to variations in writing styles, topics, and prompt structures with high accuracy. different approaches have been proposed for cross-prompt aes, including feature-based approaches @xcite , multi-task learning @xcite , and contrastive learning @xcite . as large language models (llms) represent the recent advancements in nlp, there is a growing trend towards the integration of llms in aes @xcite . the common approach involves providing the llm with task descriptions, rubrics, and essays for scoring. however, this method has not achieved the desired results, falling short of basic baseline performance @xcite . other studies employed llms as conversational models, where scoring is performed on multiple steps @xcite . while these approaches improve upon basic llm prompting, they still lag behind baseline performance. this highlights the need for a hybrid approach integrating the advanced text analysis capabilities of llms with the previously well-established aes methods. in this work, we address the challenge of traitbased cross-prompt aes with a novel approach that redefines the role of llms. rather than following the conventional paradigm of "given this essay, provide a score.", we propose a hybrid framework, trates, foot_0 a trait-specific and rubric-assisted cross-prompt aes framework, that leverages generic writing quality features in addition to trait-specific features, automatically generated via an llm, that are easy to interpret, thus providing direct feedback to students on sub-trait aspects. at its core, trates leverages an llm to generate and extract trait-specific features given a trait rubric in two stages. initially, a set of questions is generated from the rubric to assess a specific trait. the llm then answers each question individually, given the essay. the intuition is to streamline the rubric to facilitate the assessment of various aspects within the trait, rather than assessing the trait as a whole. finally, trait-specific features are combined with writing-quality and prompt-specific features to train a cross-prompt classical regression model that predicts the trait scores of essays from unseen prompts. this approach allows us to handle the scoring of various traits within a unified framework while automatically tailoring rubric-based features for each specific trait, a task that would be difficult to achieve without the capabilities of llms. our contribution is five-fold: ## related work in this section, we review aes studies, focusing on cross-prompt trait scoring and llm integration. ## cross-prompt trait-based aes in trait-based aes, a prompt p is defined as a tuple (a p , t p , e p ), where a p is the prompt task description (a paragraph or so describing the writing task), t p is a set {(t, r t )} of traits; each trait t (an aspect of student writing, such as organization or sentence fluency) is associated with a rubric r t (a set of criteria used to evaluate the specific trait t), and e p is a set {(e, {s e,t })} of essays written for the prompt p; each essay e is associated with a score s e,t for each trait @xmath0. while, generally, each prompt has its own trait rubrics, those rubrics are usually common across different prompts for specific traits. the cross-prompt problem is set up as follows. we aim to build a model that is trained on a set of source prompts p src to score the (traits of) essays written for a target prompt p trg ∉ p src . we note that only the task description of the target prompt is available to the model at inference time. ## trates framework in essay scoring, the rubric generally serves as a scoring guide that sets out the criteria for different performance levels. for example, a brief 3-level rubric for the organization trait could be: score 2: organization and connections between ideas are logically sequenced. score 1: organization and connections between ideas are weak. score 0: no organization evident. human evaluators, typically teachers, use the rubric to assess the student essays, which enhances grading consistency and transparency. moreover, it provides students with detailed feedback pinpointing areas for improvement. given the rubric's importance in manual essay scoring, our primary objective is to develop an automated rubric-based scoring framework that offers the same benefits as manual scoring, focusing on assessing essay traits. the main challenge in automatically scoring essay traits lies in identifying the essay representation that best reflects the characteristics of each trait. to address that, we propose trates, a traitspecific and rubric-assisted aes framework, illustrated in figure 1 . trates leverages the rubric to identify trait-specific features (represented by automatically-generated questions), then extracts the values of those features (by answering the generated questions) from the essays, using an llm. those features combined with prompt-specific and generic features are used to train a (relativelysimple) regression model for scoring essay traits. the major advantage of trates is that it is generic enough to be applied to any trait, given the rubric used to assess that trait, while being able to generate and extract different features that are more specific to that trait. in this section, we discuss the main components of the framework in detail. ## experimental setup in this section, we outline the setup used to conduct our experiments, including the datasets, selected llms, baselines, and implementation details. ## results and discussion in this section, we discuss the results of our experiments addressing 4 research questions: rq1: are features generated and extracted via llms effective? (6.1), rq2: can trates with traitspecific and generic features improve the performance? (6.2), rq3: which feature category holds greater significance? (6.3), and rq4: how well does trates generalize? (6.4). furthermore, we discuss the inference cost in section 6.5. ## conclusion and future work in this paper, we introduced trates, a crossprompt rubric-based framework utilizing llms, traditional features, and classical regression models for scoring essay traits. the framework is designed to reconceptualize the use of llms as feature gen-erators and extractors. the results demonstrate the effectiveness of trates yielding new sota performance on asap dataset and setting the first baseline for ellipse dataset. our findings underscore the positive impact of the added llm-based trait-specific features on enhancing the accuracy of trait scoring, emphasizing the necessity for novel models tailored to individual traits. in future work, we plan to extend the framework to holistic scoring and explore different prompting techniques for llm-based trait-specific feature generation. moreover, studying the usefulness of the generated sub-traits as feedback to the students remains an important direction for future work. mains unexplored in the context of holistic scoring, as holistic rubrics are often highly prompt-specific. moreover, our score-scaling approach was designed to address the discrepancies between the rubrics present in the asap dataset. the assigned scaling ranges were intuitively determined after a thorough examination of the various rubrics. it is debatable whether essays graded using different rubrics can be effectively combined to develop an aes system; the criteria for assessing essay quality may not be applicable across various rubrics, leading to inconsistencies in scores assigned to essays written for different prompts. therefore, the research community should collaboratively establish a standardized mapping approach to align the rubrics of existing datasets within a unified framework. this would facilitate the creation of more realistic aes systems that can leverage diverse datasets for training and testing, addressing a wide range of writing proficiencies and real scenarios. another limitation of this study is that the effectiveness of the generated questions as feedback for students was not assessed. however, the conceptual framework was discussed with experts in psychometrics and education fields, and the potential value of such feedback was acknowledged. we are dedicated to making these questions available to the community for future research. moreover, the efficiency remains a limitation of our proposed framework and is an important area for future improvement. finally, the effectiveness of this work heavily relies on the quality of the grading rubric.