Answer-state Recurrent Relational Network ( A s RRN ) for Constructed Response Assessment and Feedback Grouping (2023)

Abstract: STEM educators must trade off the ease of assessing selected response (SR) questions, like multiple choice, with constructed response (CR) questions, where students articulate their own reasoning. Our work addresses a CR type new to NLP but common in college STEM, consisting of multiple questions per context. To relate the context, the questions, the reference responses, and students’ answers, we developed an Answer-state Recurrent Relational Network (AsRRN). In recurrent time-steps, relation vectors are learned for specific dependencies in a computational graph, where the nodes encode the distinct types of text input. AsRRN incorporates contrastive loss for better representation learning, which improves performance and supports student feedback. AsRRN was developed on a new dataset of 6,532 student responses to three, two-part CR questions. AsRRN outperforms classifiers based on LLMs, a previous relational network for CR questions, and few-shot learning with GPT-3.5. Ablation studies show the distinct contributions of AsRRN’s dependency structure, the number of time steps in the recurrence, and the contrastive loss.

## introduction stem educators must trade off the ease of assessing selected response (sr) questions, like multiple choice, with constructed response (cr) questions, where students articulate their own reasoning. natural language processing (nlp) research has turned increasingly towards assessment of constructed response (cr) questions in recent years, but models remain relatively simple, in part because datasets are either relatively small for neural methods, or proprietary. in addition, public datasets are limited to standalone questions. timely feedback is important in the context of formative assessment throughout a course. as a result, formative assessments in stem currently rely largely on sr items, where assessment is immediate. this conficts, however, with the current emphasis in stem on critical thinking, reasoning and writing @xcite . manual assessment of cr is time consuming, and more so when feedback comments accompany a score. we present a relational neural network that can be adapted to handle different cr formats, and that outperforms strong baselines on a new dataset of statistics questions developed to measure statistical reasoning at the introductory college level. moreover, it also groups responses for similar feedback, in an unsupervised way. the most well known and reasonably sized nlp datasets for cr questions are the beetle and scientsbank data from semeval 13 task 7 @xcite and the kaggle dataset asap @xcite . the former has 253 prompts across 16 stem domains; the latter has 10 prompts where the stem is mostly from biology, with 22,000 items. assessment scales range from 2-way to 5-way. there are often multiple reference answers for each point level. asap also contains scoring rubrics. in both datasets, the prompts are standalone questions, but there are many other cr formats @xcite . we present istudio, a new dataset with context-based questions, a common format in stem. recent automatic assessment of cr questions using neural models has achieved good performance on the benchmark datasets mentioned above. the standard approach is to encode the question prompt and student answer separately, and send their concatenation to a classifier output layer. @xcite achieved good performance on asap with a bilstm. later work used transformers (e.g., bert @xcite ). using a model that encodes rubric elements, @xcite achieved context 2 some people who have a good ear for music can identify the notes they hear when music is played. one method of note identification consists of a music teacher choosing one of seven notes (a, b, c, d, e, f, g) at random and playing it on the piano. the student is asked to name which note was played while standing in the room facing away from the piano so that he or she cannot see which note the teacher plays on the piano. questions [2a] should statistical inference be used to determine whether carla has a "good ear for music"? explain why you should or should not use statistical inference in this scenario. ## related work deep neural networks (dnns) have been applied to cr assessment at least since @xcite . as noted in the introduction, most dnns for cr encode the (reference answer, student answer) pair and use various output functions. @xcite used long-short term memory (lstm) networks enhanced by word-specific scores, and achieved high score correlations on asap. @xcite combined convolutional neural networks (cnns) with lstms. model performance with these earlier dnns was sometimes shown to improve in combination with hand-crafted features @xcite ) and text mining @xcite . as noted above, pretrained lms like roberta have performed well @xcite . we rely on pretrained lms to encode the input layer, and use a novel relational network. explicit use of rubric information is less common. autop achieves a performance boost on asap through automatic generation of regular expression patterns from top-tier answers and rubrics @xcite . @xcite integrate rubric information via word-level attention, finding that the rubric information compensated for less data, but did not otherwise boost performance. we found that incorporating rubrics into asrrn did not improve performance on is-tudio. this may be due to the compositional logic of rubric statements, that can have multiple conjunctions and disjunctions (see relational networks (rns) were initially developed as an alternative to other graph-based models for reasoning problems in visual, linguistic, or symbolic domains (e.g., physics @xcite ), where distinct types of input elements have correspondingly distinct interrelations. rns have surpassed human performance on the clevr visual question answering dataset @xcite , and have become a general framework for few-shot learning @xcite . @xcite showed that an rn for short answer assessment could efficiently learn relations among encodings of the question, reference answer, and student answer. recurrent relational networks (rrns) @xcite eschew the flat relational structure of an rn for a fully connected relational graph with pairwise message passing. we found this original rrn structure to be disadvantageous for our cr tasks (see discussion accompanying figure 3 ). for istudio, asrrn adopts a dependency structure that reflects how people reason about the text elements shown in fig. 1 . contrastive learning (cl) aims to improve representation learning by making similar data instances closer and separating dissimilar ones @xcite . cl can be applied in unsupervised or supervised learning, and benefits a wide array of visual, language and multi-modal tasks. @xcite pioneered the introduction of contrastive loss in dimensionality reduction. subsequently, @xcite proposed triplet loss, which employs an anchor to concurrently handle positive and negative samples. later, @xcite introduced the highly-regarded selfsupervised normalized temperature-scaled cross entropy loss (nt-xent), with strong results on unsupervised learning in imagenet. @xcite extended nt-xent loss for application to supervised learning; cert @xcite improved pretrained language models (e.g., bert) using contrastive self-supervised learning at the sentence level. @xcite and @xcite developed new text classification methods using contrastive learning loss, and simcse @xcite delivered state-of-the-art sentence embeddings on semantic textual similarity (sts) tasks. we propose a variant of nt-xent with a distinct treatment for different classes: we assume multiple correct reference examples to the same question should be similar, whereas different ways to be partially correct should be dissimilar (see figure 1 ). ## dataset the context-based format is a novel challenge for nlp, but is widely used in stem, e.g., in socalled two-tier assessments that simultaneously address students' content knowledge/skills in tier one and explanatory reasoning in tier two, as in computer science @xcite , mathematics @xcite , physics @xcite and other stem subjects. automatic methods for low-stakes formative assessment can support timely feedback to identify which conceptual issues to address next, thus can lead to greater student success in stem. cr questions that address reasoning skills are preferable to selected response items but burdensome for humans to assess. the istudio data come from an assessment instrument for statistical reasoning called the introductory statistics transfer of understanding and discernment outcomes (istudio) that was found to be educationally valid and reliable (beckman, 2015), meaning there is strong evidence that it measures well what it intends to measure. the istudio dataset consists of responses from 1,935 students to three 2-part questions. the assessment rubric, which uses a 3-way scale of (essentially) correct, partially correct, incorrect, specifies the scoring criteria and provides example student answers for each correctness class. the data is de-identified, and the original irb confirmed that it can be made public for research purposes in its de-identified form. students' schools of origin are identified as large, medium or small. 1 we evaluated inter-rater agreement among two of the co-authors (a, b), and a graduate student in statistics (c). there were 63 items scored by a, b, and c; two additional sets of 63 were scored by a, b, and b, c. these pairs of raters achieved quadratic weighted kappa (qwk) > 0.79, and fleiss kappa showed substantial agreement among a, b and c (@xmath0.70). the remaining unlabeled data was assessed independently by a, b and c. 2 to prepare istudio for nlp research, we performed data cleaning to eliminate non-answers and responses that were unusually long (greater than 125 word tokens). this yielded 7,258 individual student response items, ten percent of which was held in reserve for future research. ninety percent of the remainder (@xmath1,532) was partitioned in the proportion 8:1:1 into training (@xmath2,226), validation (@xmath3, and test (@xmath4. the test set contains all the responses with labels from multiple raters so that at test time, inter-rater agreement of model predictions can be computed with the reliable human labels. ## approach the istudio dataset contains four distinct types of textual input: the context c for a given question prompt, the question prompt q, one or more reference answers r for each correctness class @xmath5, 1, 2}, and the student answers a. 3 among many possible ways to structure the set of encodings of each text type-such as concatenation, stacking, attention layers across sequences of text types, and graph networks-we hypothesized that semantic relations among the five text types could be structured as a dependency graph rooted at c, with the learned representation of each dependent node being directly influenced by its parent, and indirectly by its ancestors. as discussed in detail below, the specific chain of dependencies we hypothesized is shown in figure 2 . our hypothesis is similar in spirit to the motivation for recurrent relational networks, namely to process a chain of interdependent steps of relational inference, as in @xcite . here, recurrence refers to inferential time-steps for the entire graph, rather than recurrence of processing units. temporal recurrence allows the network to iteratively progress towards the full set of relations among all the nodes of the graph @xcite . asrrn also incorporates a super-node that is dependent on all other nodes, as in @xcite , to stabilize the flow of information in the graph network, and to implicitly represent the current answer state at each pass over the whole graph. we test our hypothesized relational structure among c, q, (r rubic , ), r and a through extensive ablations of alternative dependency structures, and by comparison to strong baselines that either have a simpler relational structure, or no relational structure at all. in this section, we present details on the graph structure, the flow of information within the graph (message-passing), the temporal recurrence, the output classifier layer, and our training objective that incorporates contrastive learning. asrrn's relational graph for istudio has the four layers shown in schematic form in figure 2 . the input layer to each node type relies on a pre-trained model to encode the distinct types of text input at time t 0 . note that the c, q, and a layers are shown as consisting of a single node, whereas the r layer has four nodes. this indicates that the pretrained encoding of a single text input is used to initialize the layers for c, q and a whereas for the r layer we initialize with multiple, different reference answers for each correctness class. again through experimentation, we found that four reference answers for each correctness class achieved the best performance. for simplicity, the figure shows only four r nodes instead of the full set of twelve per q (four per class). the arrows showing the dependencies between layers (referred to below as message passing) essentially correspond to relation vectors computed from the hidden states at each node. the super-node s is a global relational vector that incorporates information from the nodes that correspond to the distinct text inputs. the figure also illustrates the temporal recurrence for updating the flow of information within the graph. before describing the information flow, or message passing, we point out here that asrrn can be adapted for cr datasets that have different structures. for example, the c layer can be removed to fit the types of standalone question prompts found in benchmark datasets such as semeval-2013 task 7 @xcite and asap @xcite . this is how we perform asrrn experiments on these two benchmarks, as reported below. asrrn's recurrent message passing over time corresponds to the computation of relational information among nodes within the neural network graph structure. in asrrn, message passing is restricted to pairs of nodes i and j at time t, and each message m t ij (or relational vector) is: classification of a student's answer for correctness utilizes the hidden state at node a at the final time step t. the output distribution o t is: here, where we have n reference examples for each of the three correctness classes, we treat the n reference answers for the predicted class as positive exemplars, and the 2n reference answers for the other two classes as negative exemplars. our cl function depends on the true class label. for our use case, we make three strong assumptions: 1) that there is one way for a student answer to be correct regardless of the exact wording, 2) that there are a few ways, each distinct from the others, for a student's answer to be partially correct that will cover many of the answers, and 3) that there are many diverse ways to be incorrect. for the correct class, the n reference answers are assumed to be near paraphrases, thus for a student answer x i whose label is correct, l cl incorporates a term s j for the average of its similarities to the n correct reference examples: in sum, correct predictions are encouraged to be similar to the average of the correct reference examples and dissimilar to the reference examples for partially correct and incorrect. partially correct (and incorrect) predictions are encouraged to be similar to one of the partially correct (or incorrect) predictions, and dissimilar to the reference examples for the other two classes. the full loss function is a weighted sum of the cross entropy loss and the contrastive loss: @xmath6, where λ is a hyperparameter that governs the proportion of the two types of loss. cross entropy loss is: ## experiments gpt models often perform well using chain-ofthought reasoning for zero-shot and few-shot settings @xcite , so we tested this method on istudio, using gpt-3.5. in all conditions in evidence suggests that timely and specific feedback helps students learn @xcite . for formative assessment feedback, instructor effort is best spent on partially correct (pc) answers, where there is an opportunity to scaffold students towards more complete understanding. incorrect answers are far more diverse, and students may be more confused about the reasoning steps and less likely to integrate feedback into their thinking. to assess the results on whether asrrn groups student responses with the same reference answers human would, we find mixed results when we compare two humans and asrrn on the grouping of partially correct answers. recall that there were four reference answers for all three correctness classes, for all six questions. for the partially correct class, we selected four reference answers that were distinct from one another. here we report on how consistent two experts (a and b) were with each other, and each with asrrn, in pairing a student answer with the same reference answer for questions 2b and 4b; these are two of the three most challenging questions, based on the average student scores. we used qwk to report agreement of the three pairs. for both questions, a and b worked independently to pick one of the four reference answers as equivalent to each students' partially correct answer, or "none of the above." for the subset of accurate partially correct predictions from asrrn (90 out of 107 for 2b; 100 out of 110 for 4b), we show qwk for the expert pair, and each expert with asrrn. cases where asrrn's maximum probability class had @xmath7.85 were treated as "none of the above" based on locations of sharp decreases of output probabilities in scatterplots. the high accuracy of asrrn, and its potentially useful association of student answers with a specific reference answer, argues for the quality of the representation learning. appendix b includes t-sne visualizations of the learned representations of the test set with and without contrastive learning. ## conclusion asrrn is a novel recurrent relational network where the nodes in the neural graph each correspond to a distinct type of textual input, a super-node represents global information across all nodes, and the answer state at the final time step becomes the input to the final classification layer. the graph structure (direction of message passing), leads to efficient learning of relational information that supports multi-step problems. asrrn can have more or fewer layers, depending on the number of distinct types of text input in the dataset. a novel loss function for contrastive learning improves classification accuracy, and provides a step towards feedback for instructors and students on partially correct responses, by learning which of multiple reference answers a response is most similar to. a thorough set of ablations provides evidence for the chosen dependency structure, number of time steps in the temporal recurrence, and the contrastive learning. on istudio, asrrn significantly outperforms strong baselines.