Autoregressive Multi-trait Essay Scoring via Reinforcement Learning with Scoring-aware Multiple Rewards (2024)

Abstract: Recent advances in automated essay scoring (AES) have shifted towards evaluating multiple traits to provide enriched feedback. Like typical AES systems, multi-trait AES employs the quadratic weighted kappa (QWK) to measure agreement with human raters, aligning closely with the rating schema; however, its non-differentiable nature prevents its direct use in neural network training. In this paper, we propose Scoring-aware Multi-reward Reinforcement Learning (SaMRL), which integrates actual evaluation schemes into the training process by designing QWK-based rewards with a mean-squared error penalty for multi-trait AES. Existing reinforcement learning (RL) applications in AES are limited to classification models despite associated performance degradation, as RL requires probability distributions; instead, we adopt an autoregressive score generation framework to leverage token generation probabilities for robust multi-trait score predictions. Empirical analyses demonstrate that SaMRL facilitates model training, notably enhancing scoring of previously inferior prompts.

## introduction an essay can be evaluated from diverse perspectives, such as content, sentence fluency, and organization. as providing multi-view assessment is essential for enhancing the learner's writing skill, recent attention to automated essay scoring (aes) systems has shifted from solely relying on holistic scoring @xcite @xcite to evaluating multiple trait scores @xcite @xcite . although simultaneous assessment for multiple traits is more challenging than a holistic paradigm, it has been much less explored. regression encoder classification encoder score class 1 class 2 class c … softmax function sigmoid function trait 1 score 1 trait 2 score 2 score m essay p: 0.71 0.83 0.68 0.57 0.91 … score p: 0.08 0.73 0.11 autoregressive figure 1: overview of distinct aes frameworks. the autoregressive framework eliminates the need for multiple trait-wise layers. classification and autoregressive aes models probabilistically predict final scores; hence, a policy gradient reinforcement algorithm is applicable. in this paper, we propose a scoring-aware multireward reinforcement learning (samrl) method to unlock the potential of the qwk for training multi-trait aes systems. by constructing multiple rewards of bi-directional qwk and mse penalty, samrl effectively incorporates nuanced measurement schemes in training phrases. generally, the qwk score derives from a set of essays rather than a single score; thus, when applied conventionally in a batch set, it assigns the same metric to every sample. to ensure stable training, we introduce trait-wise comparison for qwk, integrating them with the batch-wise calculation to construct a bidirectional qwk reward. applying rl in aes is underexplored, and prior work @xcite is limited to holistic scoring. further, their method is restricted to the classification approach despite its inferior performance than the regression, as policy gradients for rl require probability distributions. unlike prior works, we treat aes as a generation paradigm @xcite , leveraging token generation probability distributions instead of categorical ones for policy gradient. note that standard autoregressive aes is trained with cross-entropy and does not reflect any score-related metrics (i.e., mse or qwk) during training; however, samrl enables direct parameter updates based on those scoring-aware metrics. extensive experiments on the representative asap and asap++ datasets demonstrate scoring enhancement over robust baselines across the traits and prompts. comprehensive analyses, which compare samrl with both single-reward applications and unidirectional use of qwk rewards, further reveal the robustness of our method. notably, significant improvements observed on prompts with a broader score range highlight the overcoming of challenges posed by prior use of rl in aes. rl for text generation recently, rl has been actively employed across diverse natural language generation tasks. notably, the advent of reinforce-ment learning from human feedback (rlhf) has improved the capabilities of general-purpose large language models (llms) such as gpt, showing the strength of rl @xcite . numerous researchers have applied rl to specific downstream tasks such as text summarization @xcite @xcite @xcite @xcite @xcite @xcite , machine translation @xcite , and reasoning @xcite @xcite . they aim to enhance performance by using rewards tailored to their specific objectives. for instance, in text summarization, @xcite employ human feedback as a reward model to generate summaries that align with human preferences, while @xcite use the entailment relationship between summary and source document as a reward to generate a factually consistent summary. in arithmetic reasoning problems, @xcite utilizes a non-differentiable symbolic solver as a reward to address multi-step reasoning. we integrate the rl framework to unexplored autoregressive multi-trait aes by introducing novel multiple scoring-aware rewards. ## preliminary we adopt the policy gradient reinforcement learning to train the policy to optimize rewards. policy gradient aims to increase the probability of actions that yield high rewards. to guide the model towards taking actions that result in a higher reward, the policy gradient function includes the probability of actions taken by the policy, π θ (a | s): most of the existing aes systems use encoderonly models like bert @xcite and rely on regression-or classification-based approaches. in general, the model generates the essay embedding vector from a given essay input; however, the differences lie in the objective functions and the process of deriving the final score from the output vector. regression models predict the score with a sigmoid function given the embedding vector and are trained with the mse loss function between the label y and predicted score ŷ for n samples: however, as they output a single scalar value, they are unsuitable for policy gradient, which requires a probability distribution for policy training. contrarily, classification-based aes models output a probability distribution (p) given the essay vector with the softmax function, and cross-entropy loss is employed as training objectives: to eliminate a performance decline, we introduce the rl application for aes with a generation framework. in particular, we ground our method on an autoregressive multi-trait score generation model, arts @xcite , instead of classification approaches. we consider generating text trait and score tokens as an action and treat token generation probability distribution as the probability of actions. in addition, to take advantage of comparing the exact error rate in regression, we leverage the mse penalty as part of our multi-rewards. ## samrl to incorporate a rating schema in the training phase, samrl updates the policy model using multiple rewards obtained by a scoring-aware multi-reward function. in rl, relying solely on reward-based learning can lead the model to prioritize enhancing rewards exclusively, potentially losing its capacity to score essays in appropriate prediction form. therefore, we employ a fixed-parameter anchor model to guide the policy to prevent significant deviations in training and maintain the trait patterns. figure 2 describes the overall process of samrl. ## experimental setup datasets we use the open-sourced asap foot_0 and asap++ foot_1 @xcite datasets, as in the baseline arts model @xcite . asap++ dataset includes enriched humanannotated multi-trait scores for english-written essays of eight separate prompts. as summarized in evaluations following the previous works @xcite , we use five-fold cross-validation using the same split as their works. the asap dataset's official metric, qwk, is used as the evaluation metric, and five-fold averaged values and standard deviations are reported. we also measure separate qwk by prompts to ensure fair comparisons and prevent overly high scores in case of testing in the entire set @xcite ; @xcite . comparison models are detailed in appendix a. ## result and discussions main results the main experimental results for each trait in impacts on different essay types specifically, we separately investigate the trait-specific performance improvements among prompts sharing common traits, comparing prompts 1,2 and 8 with prompts 3-6. intriguingly, figure 3 reveals that our model's impact varies with the evaluated prompt type. our model, samrl, demonstrates more substantial performance gains for the argumentative or narrative types, such as prompts 1, 2, and 8, compared to the source-dependent prompts 3-6, which necessitate essay writings based on source texts. the marked difference in scoring rangewhere the latter prompts are evaluated on a [0, 4] scale and the former on a broader range up to [0, 60]-highlights that our model effectively addresses the challenges posed by traditional classificationbased rl models, which typically struggle in wider score ranges. these advances might be attributed to direct updates of samrl grounded on scoredifference-reflected rewards measured by qwk and mse, which enables the inclusion of scoring schemas (e.g., imposing significant penalties for inaccurately predicting wide-range scores) that typical generation processes fail to accommodate. moreover, the results suggest that our approach can be particularly advantageous in less favorable conditions, given that prompts 1, 2, and 8 collectively have 4.3k essay samples, while 3-6 prompts comprise 7.1k samples. ## conclusion in this work, we propose a scoring-aware multireward reinforcement learning (samrl) method, which incorporates the rating process for effective multi-trait scoring within the generation framework. by introducing the policy gradient reinforcement in the autoregressive score generation paradigm, we enable the direct use of the qwk metric; thus, samrl effectively captures the rating procedure. in addition, we jointly introduce the score-level difference-based reward, mse reward, which was previously limited to regression aes models, bringing in the overall enhancements on trait-wise and prompt-wise scoring qualities. extensive experiments and analysis further support the assistance of our rl strategy over the simple generation.