Do Deep Neural Nets Display Human-like Attention in Short Answer Scoring? (2022)

Abstract: Deep Learning (DL) techniques have been increasingly adopted for Automatic Text Scoring in education. However, these techniques often suffer from their inabilities to explain and justify how a prediction is made, which, unavoidably, decreases their trustworthiness and hinders educators from embracing them in practice. This study aimed to investigate whether (and to what extent) DL-based graders align with human graders regarding the important words they identify when marking short answer questions. To this end, we first conducted a user study to ask human graders to manually annotate important words in assessing answer quality and then measured the overlap between these human-annotated words and those identified by DL-based graders (i.e., those receiving large attention weights). Furthermore, we ran a randomized controlled experiment to explore the impact of highlighting important words detected by DL-based graders on human grading. The results showed that: (i) DL-based graders, to a certain degree, displayed alignment with human graders no matter whether DL-based graders and human graders agreed on the quality of an answer; and (ii) it is possible to facilitate human grading by highlighting those DL-detected important words, though further investigations are necessary to understand how human graders exploit such highlighted words.

## introduction automatic text scoring refers to the task of applying computational techniques to score written text based on certain grading criteria @xcite . since its inception @xcite , automatic text scoring has been actively investigated and applied to assist educators in scoring student-written text, e.g., short answer questions and essays, which are often referred to as automated short answer scoring (asas) @xcite and essay scoring @xcite . driven by the great success of deep learning (dl) techniques in various nlp tasks, researchers have endeavored to apply them to construct asas systems in recent years @xcite , some of which displayed performance comparable to human graders. for instance, @xcite showed that the average performance of an attention-based bidirectional lstm model could be up to 0.71 (measured by the metric quadratic weighted kappa) in the asas competition organized by the hewlett foundation, which can be deemed as achieving a substantial agreement with human graders. though being effective, dl-based asas systems have been widely plagued by the inability to explain how the quality of an answer is graded. the lack of understanding the underlying working mechanism of these systems, beyond question, may stop educators from adopting them in teaching practice as there remain concerns that the use of such asas systems might unintentionally encourage students to produce formulaic writings, i.e., writing that is often lengthy and involve complex words, but not much quality content @xcite @xcite . inspired by the research efforts in the broader nlp communities, i.e., those focusing on dissecting complex deep neural net architectures and explaining how they work @xcite @xcite @xcite , in the study presented in this paper we aimed to gain a better understanding of how dl-based asas systems work. specifically, we investigated (i) the alignment between dl-based graders and human graders in terms of the words they think are important in the task of short answer scoring and (ii) whether the important words identified by dlbased graders can be of use to human graders in the marking process. formally, this study was guided by the following two research questions: rq1 to what extent do dl-based graders align with human graders regarding the words that are important in assessing answer quality? rq2 can the important words identified by dlbased graders be used to facilitate human graders to perform marking? we conducted two user studies to answer rq1 and rq2. for rq1, with the dataset provided by the hewlett foundation, we constructed relatively simple but effective bert-based graders (i.e., coupling bert with a single classification layer for prediction), whose performances were comparable to those reported in recent studies @xcite . to locate words that were essential in determining answer quality, we extracted weights allocated to different input words in the self-attention layers of bert. then, we ran a user study to ask human graders to not only score an answer but also annotate the words they believed were important in contributing to or hurting the answer quality. we measured the alignment between bert-based graders and human graders with the aid of the jaccard coefficient. building upon the results of rq1, we further implemented a randomized controlled trial to investigate whether displaying the important words identified by bert-based graders can help human graders improve marking accuracy and efficiency to answer rq2. in summary, our work contributes to the research on automated text scoring with the following main findings: (i) text spans contained in an answer that increase the answer quality, compared to those decreasing answer quality, are more likely to be identified by human graders; (ii) there exists a certain level of alignment between dl-based graders and human graders regarding the words they think are important in assessing answer quality no matter whether they agree on the quality score of an answer; and (iii) the important words detected by dl-based graders can be potentially used to facilitate human grading, though more research efforts are required to understand how these words are to be utilized by human graders. ## methods this study was approved by the human research ethics committee at monash university (project id 30074). in the following, we first describe the task and dataset based on which we examined the alignment between dl-based graders and human graders, followed by the construction of the dlbased graders. then, we detailed the setup of the two user studies we implemented to answer rq1 and rq2. ## results recall that all answers used in this study were assessed on a grading scale of [0, 3]. for the following analysis, we aggregated and denoted the answers whose ground-truth score1 is 0 or 1 in the original dataset as low quality, and those of score 2 or 3 as high quality. ## discussions and conclusion given the increasing popularity of dl-based automatic text scoring systems in education, this work investigated the interpretability of these sys-tems regarding their alignment with human graders in the task of asas. through two well-designed user studies, we demonstrated that (i) there exists certain alignment between dl-based graders and human graders in terms of the important words they identified for assessing answer quality; and (ii) it is possible to utilize the important words detected by dl-based graders to facilitate human grading. though several interesting findings have been enabled, it is not the time yet to advocate that we should incorporate such dl-identified important words into asas systems to facilitate human grading due to the following limitations in our study. firstly, as observed in study 1, dl-based graders displayed certain agreement with human graders, even in the circumstances of being unable to correctly assess the quality of an answer. it remains unknown what types of words (or concepts) dl-based graders agree/disagree on with human graders. for example, would it be possible that both types of graders tend to hold similar misconceptions (e.g., identifying the same type of plausibly-correct words)? this calls for future research to characterize and analyze the different types of important answer spans identified by dlbased graders and human graders under different circumstances. by doing this, we can potentially derive more insights on how to further improve the prediction performance of dl-based graders. secondly, and more importantly, it remains unknown the sense-making process the human graders follow to make use of the dl-detected important words. as showed in study 2, the human graders in the experimental group neither agreed nor disagreed that the dl-highlighted words were important, but they considered the dl-highlighted words were of value and delivered better grading accuracy (at the cost of a lower grading efficiency). would it be the case that the highlighted words provoke more in-depth thinking and understanding of an answer among human graders and thus enabling them to better assess the answer quality? this motivates us to conduct further studies to investigate human graders' marking process in using dl-highlighted words in the future.