Improving Prompt Generalization for Cross-prompt Essay Trait Scoring from the Scoring-invariance Perspective (2025)

Abstract: Cross-prompt trait scoring task aims to learn generalizable scoring capabilities from source- prompt data, enabling automatic scoring across multiple dimensions on unseen essays. Existing research on cross-prompt trait essay scoring primarily focuses on improving model generalization by obtaining prompt-invariant representations. In this paper, we approach the research problem from a different perspective on invariance learning and propose a scoring-invariant learning objective. This objective encourages the model to focus on intrinsic information within the essay that reflects its quality during training, thereby learning generic scoring features. To further enhance the model’s ability to score across multiple dimensions, we introduce a trait feature extraction network based on routing gates into the scoring architecture and propose a trait consistency scoring objective to encourage the model to balance the diversity of trait-specific features with scoring consistency across traits when learning trait-specific essay features. Extensive experiments demonstrate the effectiveness of our approach, showing advantages in multi-trait scoring performance and achieving significant improvements with low-resource prompts.

## introduction automated essay scoring (aes) is a common and important application of artificial intelligence (ai) in the field of education. compared to the human grading process, an effective aes system can provide language learners with timely feedback on multiple perspectives, such as overall essay quality, organization, prompt adherence, narrativity, etc. in previous aes research, most studies focused on a prompt 1 -specific setting, where both the training and unrated test essays belong to the same prompt @xcite @xcite figure 1 : the illustration of cross-prompt essay trait scoring and a summary of our main motivations. @xcite @xcite . however, real-world aes systems often lack enough target-prompt essays, making it crucial to develop methods that can reliably score essays for prompts not included in the training data. to address the aforementioned limitation, researchers have begun focusing on cross-prompt essay scoring @xcite . similar to the objective of domain generalization (dg) @xcite , this task aims to enable aes models trained on source prompts to be effectively applied to "unseen" prompts, as shown in figure 1 (a). in our work, we also focus on scoring essays while considering their trait-specific attributes. although existing studies have made progress in enhancing prompt generalization @xcite , most of them approach the problem from the perspective of prompt-invariance, aiming to improve model generalization by aligning features across essays from multiple source prompts. @xcite proposed a disentangled learning approach to extract prompt-invariant representations from source prompts. chen and li 2024 proposed a meta-learning-based approach that simulates prompt distribution shifts to enhance the generalization of essay representations. these existing studies require constructing a large amount of coun-terfactual data @xcite or meta-learning tasks @xcite to learn prompt-invariant representations effectively, which significantly increases the training burden. in this work, we take a different perspective on invariance learning and propose a scoringinvariant learning approach. this approach encourages the model to focus on features relevant to scoring, which arise from the internal structure of the essays rather than prompt differences, thereby capturing the intrinsic scoring information. the motivation is illustrated in figure 1 (b) . for an essay, whether obtaining the essay representation from its grammatical structure or using semantic information to extract the representation, there should be certain dimensions in both representations that can jointly reflect the essay's quality. since these dimensions remain unchanged regardless of the method used to extract essay features, we refer to them as scoring-invariant dimensions. in this work, we first extract essay representations from two dimensions (prompt-aware and promptagnostic). then, through the scoring-invariant objective we designed, we optimize the model's training process to encourage the model to focus on the intrinsic scoring features within the essay. for multi-trait scoring tasks, learning diverse trait-specific scoring features and consistency information across traits is beneficial in promoting the model's ability to score across multiple dimensions @xcite . therefore, to address trait diversity, we introduce a sentence-level routing gate network based on the existing scoring architecture, leveraging the modeling approach of a mixture-of-experts (moe) to enhance the model's perception of target trait features @xcite . regarding trait consistency information, we further propose a trait consistency learning objective, aiming to enable the model to further capture the relationships between traits. finally, we named our method as scoring-invariance enhanced cross-prompt trait scoring (sicts). to summarize, the main contributions of our work can be summarized as follows: ## related work in the early research on aes, most studies focused on prompt-specific scoring tasks, where the training and testing data originated from the same prompt. researchers initially relied on handcrafted features and applied machine learning methods to score essays @xcite @xcite . with the development of deep learning, more studies have started using deep neural networks to automatically extract quality features from essays, enabling an end-to-end scoring process @xcite @xcite @xcite . although existing prompt-specific essay scoring methods have been widely studied and have made some progress, exploring cross-prompt scoring methods can reduce the need for annotations on target prompts, thereby lowering annotation costs and facilitating real-world applications. cross-prompt aes aims to train aes models that learn generalizable scoring abilities from source prompts to score essays from unseen target prompt(s) @xcite @xcite . in existing crossprompt aes research, one class of approaches focuses on evaluating the overall quality of essays. ridley et al. 2020 apply a neural-network-based method to learn general essay features. jiang et al. 2023 propose a prompt-aware neural aes model to extract essay quality features. another line of research focuses on cross-prompt essay trait scoring, chen and li 2023 utilize contrastive learning to learn consistent prompt-agnostic representations across different prompts. do et al. 2023 introduce a cross-prompt trait scoring method that can extract prompt-aware essay representations and topiccoherence features. unlike existing cross-prompt aes methods, we improve the prompt generalizability from the perspective of scoring-invariance learning. sentence 1 sentence n pos embedding pos embedding sentence-level encoder … … … sentence 2 pos embedding sentence-level gated routing … sentence-level encoder sentence-level encoder document-level encoder document-level encoder … pos embedding sentence 1 sentence-level encoder word embedding sentence n … multi-head self attention document-level encoder prompt essay … linear + attention linear + attention … non-prompt specific features trait attention trait attention concat layer sigmoid layer sigmoid layer 𝒚 𝒐𝒗𝒆𝒓𝒂𝒍𝒍 𝒚 𝒕𝒓𝒂𝒊𝒕 𝑴 prompt-agnostic representation prompt-aware representation … … … gating scoring-invariant scoring-specific scoring-invariance learning 3 task definition and preliminary ## conclusion in this paper, we focus on the cross-prompt essay trait scoring task. to enhance the model's ability to obtain more generalized essay representations, we adopt a scoring-invariance perspective and design a scoring-invariance optimization objective, which promotes the model's general scoring capability. to further improve the cross-prompt trait scoring ability, we propose a novel cross-prompt scoring model based on a sentence-level routing gate mechanism to obtain trait-sharable scoring features. experimental results demonstrate the significant effectiveness of the proposed method in cross-prompt multi-trait scoring tasks.