Prompt- and Trait Relation-aware Cross-prompt Essay Trait Scoring (2023)

Abstract: Automated essay scoring (AES) aims to score essays written for a given prompt, which defines the writing topic. Most existing AES systems assume to grade essays of the same prompt as used in training and assign only a holistic score. However, such settings conflict with real-education situations; pre-graded essays for a particular prompt are lacking, and detailed trait scores of sub-rubrics are required. Thus, predicting various trait scores of unseen-prompt essays (called cross-prompt essay trait scoring) is a remaining challenge of AES. In this paper, we propose a robust model: prompt- and trait relation-aware cross-prompt essay trait scorer. We encode prompt-aware essay representation by essay-prompt attention and utilizing the topic-coherence feature extracted by the topic-modeling mechanism without access to labeled data; therefore, our model considers the prompt adherence of an essay, even in a cross-prompt setting. To facilitate multi-trait scoring, we design trait-similarity loss that encapsulates the correlations of traits. Experiments prove the efficacy of our model, showing state-of-the-art results for all prompts and traits. Significant improvements in low-resource-prompt and inferior traits further indicate our model’s strength.

## introduction automated essay scoring (aes) aims to score essays written for a specific prompt, which defines the writing instructions and topic. as a subordinate or alternative to human scorers, it has the advantages of fairness and low costs. thus far, most aes systems have been built on the assumptions of grading essays on the same prompt used for training and only assigning an overall score, achieving noticeable growth @xcite @xcite . however, such settings conflict with realeducation systems, where pre-labeled essays for a specific prompt are not given, and in-depth feedback requires multiple trait scores. acknowledging this, recent works have suggested cross-prompt models @xcite @xcite ) that are tested using essays of unseen prompt, like zero-shot learning, and trait-scoring models @xcite @xcite that output multiple trait scores. handling both settings (figure 1 ) is a direction for practical aes and yet has rarely been studied @xcite . for a cross-prompt setting, using non-promptspecific features that capture the general essay qualities such as length and readability is emphasized @xcite . this is to avoid the model biased toward the prompts of trained essays, but the model fails to reflect any prompt-relevant information (e.g., whether the essay fits the prompt topic), inhibiting accurate scoring. for trait scoring, most methods extend holistic scoring models without particular consideration of trait properties. both settings leave huge room for improvement. in this paper, we propose a robust model, promptand trait relation-aware cross-prompt trait scorer (protact), with the ultimate goal of improving aes for practical use. attending to the promptrelevant aspects and trait similarities leads to overcoming both cross-prompt and multi-trait settings. to ensure that the model reflects prompt-relevant information, we introduce a novel architecture to obtain prompt-aware essay representation. rather than only encoding the essay, we directly encode the prompt instruction and apply attention. this provides hints for scoring in cross-prompt settings since prompt content is always-given information, even for ungraded essays of new prompts. furthermore, we suggest extracting the topic-coherence feature by applying the topic modeling mechanism latent dirichlet allocation (lda) @xcite . this feature notifies essay coherence on a specific topic to the model without accessing labels. to facilitate multi-trait scoring, we designed a trait-similarity loss that incorporates correlations between different trait scores. practically, trait scores are not independent of one another; for example, both prompt adherence and content traits evaluate prompt-relevant aspects of an essay. finding strong correlations between trait scores, we mirror this for model training. specifically, we penalize when the similarity of actual trait scores is over a threshold but that of predicted trait scores is low. this enhances the advantages of multi-trait learning by mutually assisting in different tasks. we evaluate protact with the widely used asap and asap++ datasets. protact achieves state-of-the-art results, outperforming the baseline system @xcite for all qwk scores of traits and prompts. significant improvements of 6.4% on average and 10.3% for the content trait are observed for a low-resource prompt, which performed poorly due to lacking similar-type training essays. this highlights the strength of protact in the cross-prompt setting, overcoming the absence of pre-graded essays. remarkably improved assessments for previously inferior traits further prove the effectiveness of multi-trait scoring. codes and datasets are available on github foot_0 . ## related work aes studies mostly focus on the prompt-specific holistic scoring task. aside from early machine learning-based regression or classification approaches @xcite @xcite , recent deep-learning-based methods for automatically learning essay representation are dominant. notably, approaches that hierarchically represent essays from word-or sentence-to essay-level show competitive accuracy @xcite @xcite . late attempts to fine-tune pre-trained models to develop more successful aes include @xcite , who fine-tune bert by combining regression and ranking loss, and @xcite , who suggest a multi-scale representation for bert. @xcite additionally encode source excerpts of source-dependent essays and suggest a co-attention. our essay-prompt attention is distinct from theirs, as we encode the prompt rather than the source excerpt and apply attention differently. pointing out that previous successes in aes are far from real-world systems, few studies of the cross-prompt setting suggest methods of not examining target-prompt essays @xcite @xcite . considering the essay's semantic disparity by different prompts, the use of non-prompt-specific features of general essay qualities is highlighted in cross-prompt settings; @xcite crafted the features of essay qualities, categorized as length-based, readability, text complexity, text variation, and sentiment. however, they disregard the topic-coherence of the essay, which is an important consideration for grading @xcite . to consider coherence during rating, we suggest a way of extracting the topic-coherence feature. to provide several trait scores that fit the subrubrics, a few trait-scoring studies have been proposed; however, they simply extend the existing holistic scoring methods by adding multi-output linear layers @xcite or using multiple trait-specific models @xcite . emphasizing both the cross-prompt and trait scoring task, @xcite suggest a leading model for the crossprompt trait scoring task. they extend the @xcite model by setting multiple trait-specific layers and concatenating the features of @xcite . despite achieving the best results on the task, the performance still lags far behind the prompt-specific holistic scoring. in addition, the performance gaps between traits and between target prompts are remarkable. we propose a novel architecture to improve cross-prompt trait scoring and thereby reduce the performance gap. ## model description: protact to benefit from both automatically learning essay representations and precisely designed essay features, we combine both approaches. therefore, protact comprises two main parts: obtaining the prompt-aware essay representation and extracting the essay features (figure 2 ). the learned promptaware essay representation is concatenated with the pre-extracted essay features, constructing the final essay representation to score. the model is trained with the loss function that interpolates our traitsimilarity loss and the mean squared error (mse). ## results and discussion the results clearly show that protact outperforms the baseline cts model for all prompts ( we further investigated the low-resource prompt, which lacks similar-type essays in its training data ( ## conclusion we proposed a prompt-and trait relation-aware cross-prompt essay trait scorer (protact) to improve aes in practical settings. experimental results prove that informing prompt-relevant knowledge to the model assists the scoring of unseen prompt essays, and capturing trait similarities facilitates joint learning of multiple traits. significant improvements in low-resource-prompt and inferior traits indicate the capacity to overcome the lacked pre-rated essays and strength in multi-trait scoring.