Towards Extracting and Understanding the Implicit Rubrics of Transformer Based Automatic Essay Scoring Models (2023)

Abstract: By aligning the functional components derived from the activations of transformer models trained for AES with external knowledge such as human-understandable feature groups, the proposed method improves the interpretability of a Longformer Automatic Essay Scoring (AES) system and provides tools for performing such analyses on further neural AES systems. The analysis focuses on models trained to score essays based on organization, main idea, support, and language. The findings provide insights into the models’ decision-making processes, biases, and limitations, contributing to the development of more transparent and reliable AES systems.

## introduction since its inception over 50 years ago @xcite , automated essay scoring (aes) has been a valuable approach for evaluating large quantities of student essays. recent developments in the field have sought to harness advanced natural language processing techniques to score essays on par with human raters, achieving significant progress toward that goal @xcite @xcite . the inability to understand the learned representations in deep learning based aes models introduces risk and validity concerns to their widespread use in educational settings @xcite @xcite . in response to this concern, we propose a functional component-based approach to scrutinize the activations of transformer models trained for aes. the primary goal of this study is to provide a method and tool that can provide a coherent and interpretable understanding of the functions per-formed by these neural models, comparing their overlaps and differences, and aligning the learned functions with human-understandable groups of features foot_0 . much in the same way that human evaluators use rubrics to guide their scoring of essays, neural models learn a set of features and connections that, when combined and applied to an essay, repeatably determine the score that they will assign.through the comparison and contrast of these components across models, we investigate how the models prioritize different aspects of writing and make stride towards unveiling that their learned rubrics are, alongside any underlying biases or limitations that they entail. ultimately, this in-depth analysis will enhance our understanding of the neural models' decision-making processes, thereby contributing to the development of more transparent and reliable automated essay scoring systems. our proposed methodology involves extending the emerging domain of neural network interpretation by using abstract functional components, enabling a robust comparison between probed functional components of a network and independent feature groups. this approach specifically builds upon recent work on neural probes and derived methods, aligning a neural network's activations with external knowledge such as task metadata and implicit features (e.g., parts-of-speech, capitalization, etc.) @xcite . we focus our interpretation in the domain of aes where each model in our investigation is trained to score essays based on distinct evaluation traits, namely organization, main idea, support, and language. to probe these models, the features are drawn from several sources that correspond to concepts of both high and low validity for essay scoring: statistical features of an essay (e.g. number of sentences, number of paragraphs, etc.) @xcite , tree features generated from rhetorical structure theory (rst) @xcite parses of the essays @xcite , essay prompt and genre @xcite , and a combination of algorithmically derived @xcite and our own human defined style-based word lists. these features provide a lens that while unable to capture all of the capabilities of the models, provide insight into some of the key differences between them. in the following sections, we provide a detailed description of the methodology used for this analysis, discuss the assumptions underpinning the method, and present potential explanations for correlated function/feature pairs through a series of experiments that validate our method's ability to reflect the internal rubric of each of the neural models. ## related work from the interpretability angle, the most closely related work to this is that of neural model probes @xcite @xcite @xcite which have frequently being used to test whether a model has learned a set of properties @xcite . the primary gap we are working to fill in from this body of literature is that current approaches, with few exceptions @xcite , focus on understanding the roles of individual neurons in the greater neural network. we contend that studying the interpretability of a neural network at the individual neuron level can too easily obscure the broader picture. our interest lies in further progress incorporating a more abstract perspective on what is learned by neural networks, complementing the work that has been done at the neuron level. compared to alternative paradigms for interpretability in machine learning models, such as lime @xcite or shap (lundberg and lee, 2017), which evaluate the contribution of a given feature to the prediction of a model, the functional component based methods allow for a more granular identification of important parts of a model, independent from known features for a task. this can enable model analysts to quickly identify unexplained components and begin to propose alternative pallets of features. furthermore, the functional components can represent intermediate steps within the neural network which would be unobservable with these alternative methods. from the educational technologies and automated essay scoring angle, our work primarily applies to the body of deep learning-based aes models such as recurrent neural network models @xcite , convolutional neural network models @xcite , and transformer models @xcite . while our method could be applied to any type of neural model, we focus on transformers as they represent the state-of-the-art. by integrating the interpretability of neural models with the understanding of the functional components they learn, we hope to bridge the gap between human-understandable features and neural network-based essay scoring. the insights gained from our methodology can guide the development of more effective and efficient aes systems, tailored to the specific needs of educators and students. furthermore, the lessons learned from this research may extend beyond the aes domain, providing valuable insights for the broader field of natural language processing and machine learning interpretability. ## methods in this section we present our interpretation approach (figure 1 ), defining the key concepts of functional components, functional group, feature, and feature group. because the approach notably abstracts away from common terms in the neural network literature, throughout this section we draw an analogy to how one can define and describe the common features between mammals by comparing their common and unique characteristics. ## experiments in this section, we delve into the specific methodology used to analyze the activations of the four transformer models for aes, as well as the steps taken to prepare the data and features for this analysis. ## results in this section, we present aggregate statistics for each model comparison when it comes to computing features and independent feature groups ( ## discussion the results presented in the preceding section demonstrate the efficacy of the proposed method in extracting salient feature groups and functional groups from the neural models, particularly when applied to the dataset under consideration. the true potential of this method, however, lies in its capacity to be broadly applied to any neural aes system, thereby facilitating a deeper understanding of the models and the underlying processes they employ. in the following discussion, we will delve further into the results, emphasizing the prominent trends observed in the alignment of functional groups and their correlation with essay features, as well as the implications of these findings for enhancing the interpretability and transparency of neural aes systems. ## conclusion the neural network interpretation technique presented in this paper demonstrates significant promise in learning the implicit rubrics of neural automated essay scoring models. by effectively mapping the intricate relationships between feature groups and the functional groups of the underlying scoring mechanism, the technique provides a step towards an understanding of the factors contributing to a transformer's evaluation of essay quality. this enhanced understanding enables researchers and educators to not only identify potential biases in scoring models, but also to refine their models to ensure a more reliable and fair assessment of student performance. the code for this method will be released and incorporated into an analysis tool for application to neural models not limited to the ones examined in this work with the goal to pave the way for the development of more transparency in neural aes models. these advancements can contribute to the overarching goal of promoting ethical and responsible ai in education by facilitating the examination and comprehension of complex neural models.