Beyond Agreement: Diagnosing the Rationale Alignment of Automated Essay Scoring Methods based on Linguistically-informed Counterfactuals (2024)

Abstract: While current Automated Essay Scoring (AES) methods demonstrate high scoring agreement with human raters, their decision-making mechanisms are not fully understood. Our proposed method, using counterfactual intervention assisted by Large Language Models (LLMs), reveals that BERT-like models primarily focus on sentence-level features, whereas LLMs such as GPT-3.5, GPT-4 and Llama-3 are sensitive to conventions & accuracy, language complexity, and organization, indicating a more comprehensive rationale alignment with scoring rubrics. Moreover, LLMs can discern counterfactual interventions when giving feedback on essays. Our approach improves understanding of neural AES methods and can also apply to other domains seeking transparency in model-driven decisions.

## introduction in recent years, neural approaches to automated essay scoring (aes) have demonstrated remarkable performance @xcite . the advent of large language models (llms) has shifted focus not only towards their scoring capabilities but also towards the potential for providing feedback @xcite @xcite , enabling a better understanding of the models' rationale. however, current model evaluations mainly use metrics such as quadratic weighted kappa (qwk) to measure agreement with human ratings. this approach leaves the models' underlying reasoning opaque, thereby raising risks and questioning the validity of their use in high-stakes educational tests @xcite . a series of studies have found that neural models can be right for the wrong reasons, a concern that persists into the era of llms @xcite . to understand the decisionmaking basis of neural models, researchers have primarily adopted two primary avenues: what knowledge a model encodes and why a model makes certain predictions @xcite . both paradigms have garnered attention in the field of aes. @xcite addresses the what question by extracting meaningful functional groups from the representations of transformer models and aligning them with human-understandable features. however, a model encodes a myriad of features does not mean that the features are utilized in decision-making @xcite . to tackle the why question, @xcite employed integrated gradients @xcite to analyze token importance, and discovered that for bert-based model, most of the attributions are over non-linguistic tokens and stop words. it can be seen that the gradients-based methods only target lower-level token features, thus failing to reveal whether models leverage higher-level linguistic features. moreover, both @xcite and @xcite employed adversarial modifications to assess models, but these interventions did not target the linguistic features critical to the aes task, and they did not control for other variables that could affect essay scores during modification. therefore, even for traditional aes models, reliable explanations of their inner workings remain elusive. additionally, the explainability of scoring in llms is largely unexplored, indicating considerable work is needed to advance our understanding of model reasoning within this domain. in this paper, we systematically investigate whether models adhere to scoring rubrics when producing scores and feedback on essays, thereby assessing their alignment with human rationale. to quantify this alignment, we introduce the concept of rationale alignment, measured by the difference in scores and feedback between original essays and original essay corrected essay essay with errors complexified essay simplified essay counterfactual generation original essay counterfactual essays original score counterfactual scores original feedback counterfactual feedback scoring difference feedback difference their counterfactuals. as shown in figure 1 , we propose a model-agnostic diagnosis method that uses linguistically-informed counterfactuals to scrutinize the scoring behavior of bert-like models and llms. the diagnostic approach closely integrates linguistic knowledge from scoring rubrics, such as conventions, accuracy, vocabulary, syntax, and coherence, with llms employed for fine-grained and controllable counterfactual generation. our investigation reveals that: (1) bert-like models can discern differences in conventions and language complexity but struggle to grasp the logical structure and coherence of essays; and (2) llms, although have lower score agreement than traditional models, display a superior alignment with human experts' reasoning during scoring and can also address counterfactual interventions in their feedback. through few-shot learning or finetuning, llms can achieve both high scoring agreement and rationale alignment. ## method we employed counterfactual interventions to establish causality between target concepts and predicted scores. typically, counterfactual intervention involves manipulating a specific feature or concept while controlling for others and observing the subsequent effect on the model's prediction. we firstly extracted target concepts from scoring rubrics for intervention, and then generated counterfactual samples for different concepts using llms and heuristic rules. ## conclusion we generated linguistically-informed counterfactuals with an integrated approach combining llm and rule-based methods, analyzing their impact on essay scoring results of bert-like models and llms. our findings emphasize that a higher scoring agreement with human raters does not necessarily indicate a better alignment with scoring rubrics, suggesting that a more holistic evaluation approach should consider both aspects. moreover, our study highlights llms' considerable potential in aes domain: while zero-shot prompted llms show lower scoring agreement compared to bert-like models, a major reason for this is that them tend to be conservative or strict when evaluating the essay. fsl and sft could significantly increase the agreement level with annotated essays serve as anchors to neutralize the conservatism. in the mean time, llms demonstrate comprehensive rationale alignment with scoring rubrics. this ability is stably maintained in zsl, fsl and sft settings. lastly, llms are not only sensitive to counterfactual interventions when scoring but can also reflect a large part of these differences in their feedback, an advantage beyond the reach of traditional aes methods. this study sheds light on why a neural model assigns specific scores to essays. it unveils how modifying domain-specific concepts in texts to craft counterfactuals enhances transparency in model decisions-a method applicable across multiple fields. with llms, counterfactual generation has been greatly empowered, boosting transparency and accountability in machine learning applications.