A Semantic Feature-Wise Transformation Relation Network for Automatic Short Answer Grading (2021)

Abstract: Automatic short answer grading (ASAG) is the task of assessing students’ short natural language responses to objective questions. It is a crucial component of new education platforms, and could support more wide-spread use of constructed response questions to replace cognitively less challenging multiple choice questions. We propose a Semantic Feature-wise transformation Relation Network (SFRN) that exploits the multiple components of ASAG datasets more effectively. SFRN captures relational knowledge among the questions (Q), reference answers or rubrics (R), and labeled student answers (A). A relation network learns vector representations for the elements of QRA triples, then combines the learned representations using learned semantic feature-wise transformations. We apply translation-based data augmentation to address the two problems of limited training data, and high data skew for multi-class ASAG tasks. Our model has up to 11% performance improvement over state-of-the-art results on the benchmark SemEval-2013 datasets, and surpasses custom approaches designed for a Kaggle challenge, demonstrating its generality.

## introduction educators at every level rely on classroom assessments to evaluate students' knowledge, often through quizzes. multiple choice questions can be graded automatically, but many studies have shown that short answer, constructed response questions provide greater benefit to students @xcite @xcite . manual assessment of short asnwer questions is time-consuming, and has bias and errors @xcite . automatic short answer grading (asag) applies nlp methods to reduce the assessment burden for short answer questions @xcite , with potential to assist educators in providing more timely feedback to students on a preferred assessment method. with increased reliance on virtual learning environments and educational technology, the potential impact of asag has grown. the most common asag approach classifies students' answers into two or more categories. the key challenge is that the classification problem is inherently relational, involving the relation of the student's answer to the question, as well as to one or more reference answers. another challenge is that existing datasets are relatively small, especially for the kinds of neural network models that perform best on other nlp tasks. much of the asag work is conducted in industry labs with proprietary methods and datasets @xcite . creation of benchmark datasets, however, has fostered broader interest in the problem from the nlp community. the main benchmark dataset, semeval-2013 @xcite , covers multiple stem domains, and has multiple classification tasks regarding both the number of classes, and the level of generalization required. for example, one task addresses unseen answers to known questions (ua), another addresses unseen questions (uq) within the same subject domain, and a third is unseen domains (ud) for student answers to topics and questions not seen in the training data. while this is a rich dataset, it is not large enough to support complex models for all the classification tasks. a somewhat larger dataset, asap-sas foot_0 is less well structured, and contains a range of supporting information other than reference answers, such as rubrics. while several challenge models performed well, they were not described in publications. we use both datasets. besides the potential benefits for educational technology, we believe asag can push nlp research in new directions due to the relational nature of the task, and the graded difficulty of the different classification tasks that asag presents. figure 1 shows a question (q), reference answer (r), and student answer (a) from semeval-2013. the color coding of words simulates a venn diagram of conceptual overlap of all three components (green), versus only q and r (blue), only r and a (yellow), and only q and a (red). we hypothesize that a relation network can better exploit these different similarity spaces. the input to our neural network consists of qra triples, which are then modeled as a 3-way relation. we present a semantic feature-wise transformation relation network (sfrn). it is inspired by vision research using relation networks @xcite and feature mapping @xcite functions, which were both motivated by a desire for greater generalization ability (referred to as reasoning), combined with computational efficiency and low model complexity. sfrn is an end-to-end model with three components. an encoder first encodes each component of a qra triple, producing vectors for the question, one of a set of reference answers, and the student answer. when there are multiple reference answers, a relation network converts each triple of vectors for a given student answer into a single relation vector, and a learned feature-wise transformation function merges all the relation vectors for the student answer by leveraging the attentions calculated by a qra triple. finally, a classifier determines which class the student answer belongs to. to address data insufficiency and class imbalance, we adopt a simple data augmentation method, back-translation, which was studied for augmentation of paraphrase data @xcite , and has been used in other nlp problems. most asag datasets are relatively small, especially com-pared to the typical training sets for large neural networks, and the multiway classification problems have extreme class imbalance. for example, the semeval-2013 beetle subset for the 5-way classification task has only 4,146 training samples, and one of the classes has only 195 examples. here, as in @xcite , we apply back-translation to generate examples from existing ones without changing the label. we find that data augmentation is beneficial for simple models, like logistic regression, and for complex models, including our most complex model, sfrn+bert. sfrn+bert combined with data augmentation achieves up to 11% performance improvement over the state-of-the-art. our contributions are: sfrn+, a novel relation network that outperforms the state-of-the-art, use of data augmentation to address data insufficiency and imbalance, and ability to learn asag relations from either reference answers or rubrics. the next section presents related work on asag, relation networks, and data augmentation. section 3 presents our sfrn and sfrn+ models. section 4 describes the datasets and classification tasks. section 5 presents our data augmentation method. section 6 presents our experiments and results. relation networks (rn) originated as an alternative to other kinds of graph-based neural models to develop relation-based representations, and were designed to overcome the limitations of cnns and mlps for reasoning problems in vision, nlp and symbolic domains such as physics @xcite . rn performance on a visual question answering dataset clevr @xcite surpassed human performance. rns also prove effective at improving object detection models @xcite . moreover, rns have became a general framework for few-shot learning @xcite . we adapt rns for the three-way relation represented by the questions, reference answers, and student answers in asag datasets. rns typically combine learned representations of relational vectors using vector addition. we combine the relation vectors that represent multiple reference answers for the same question and student answer using a learned relation fusion function. for the fusion function, we use feature-wise transformation, based on its success with multi-modal data, e.g., images and text. @xcite developed film, an approach to merge information from language and visual input. a language vector serves as conditioning input, to control the scaling and shifting of the visual feature map in a featurewise fashion. similarly, we train a function with learnable parameters to combine qra triples for a given student answer. we have not seen feature wise transformation used for vector combination in asag research, which typically relies instead on fixed arithmetic operations or concatenation. data augmentation is less common in nlp, compared with computer vision, where image data augmentation is standard. operations on images such as rotating an image a few degrees, or converting it to grayscale, do not change their essential meaning. in general, data augmentation is used both to increase the size of the training data and to add irrelevant noise to examples to improve the robustness of learned models. recently, data augmentation has been found to significantly improve performance on nlp tasks as various as paraphrasing @xcite , natural language generation @xcite , semantic parsing @xcite , or various sentiment and opinion classification tasks @xcite . one method is to substitute a random word with a synonym drawn from a lexical database like word-net @xcite @xcite , or to use word embeddings to find synonyms @xcite . back-translation leverages machine translation to paraphrase a text while retaining the meaning @xcite @xcite . we adopt back-translation for its ease of use, given that machine translation methods have achieved very high performance. ## sfrn a relation network (rn) @xcite is particularly suitable for asag because it is designed to infer higher order generalizations, meaning generalizations that hold across tuples of examples, in a data efficient manner. rns have been used in vision to efficiently learn generalizations across pairs of objects without having to learn individual weights for all possible object pairs in the data. we extend the rn framework to handle relations across vectorized text triples using a data fusion function. in its simplest form, an rn is a composite function: in this section, we first present the sfrn model to convert the three vectors for each q, r and a into a relational vector, then we introduce the semantic feature-wise transformation unit that fuses relation vectors. although encoding the textual q, r and a inputs into vectors is the first step in the model, we postpone discussion of the different encoders we try out until the last subsection. ## datasets the asag datasets we use are semeval-2013, and the asap-sas kaggle competition dataset. the two were created for different purposes, and have distinct structures. with asap, we use only the components that correspond roughly to the se-meval format, as explained further below. semeval-2013 @xcite provides two training datasets, beetle and scientsbank, that have 2-way, 3-way (correct, contradictory, incorrect) and 5-way (correct, partially correct incomplete, contradictory, irrelevant, non domain) class labels. scientsbank has three classification tasks comprising unseen answers (ua), unseen questions (uq) or unseen domains (ud), and beetle has the first two of these. the 5-way labels were chosen to potentially provide tutoring feedback. the beetle dataset consists of 56 questions on basic electricity and electronics, with approximately 3,000 student answers. scientsbank contains approximately 10,000 answers to 197 assessment questions across 15 different science domains. each question has from 1 to 14 reference answers. to test the generality of our method, we also apply it to a dataset that lacks reference answers, automated student assessment prize short answer scoring (asap-sas), used in a kaggle competition @xcite . it has 10 prompts from science, biology and english language arts (ela), with 17,207 training examples and 5,224 test examples. responses are rated by two annotators: four prompts are rated in {0, 1, 2, 3}, and others are in {0, 1, 2}. this dataset has a wide array of other information, including rubrics. therefore we test sfrn on triples that use rubrics in place of reference answers. we exclude all information other than the questions, rubrics, and student answers. many secondary and post-secondary stem courses that use short answer questions rely on rubrics rather than reference answers. we find that sfrn performs as well as the models that use the full resources in asap, which shows the generalization ability of sfrn, and makes it potentially more useful to educators. we use the score assigned by the first annotator as the label and evaluate with quadratic weighted kappa (qwk; a method to measure the agreement between the graders), based on the kaggle competition guidelines. ## data augmentation since the semeval asag dataset has limited training data, and high data skew for the 3-way and 5-way tasks, we utilize back-translation to efficiently generate more examples for any given class @xcite . back-translation refers to translation from a source language to one or more pivot languages, followed by translation from the pivot(s) back to the source. we found good performance using two one-step pivot languages, french and chinese. we interleaved use of a state-ofthe-art neural machine translation (nmt) system, easynmt, with the google translation api. this provides us with greater control over the scale of new examples-given that google limits calls to google translate-as well as more noise injection for robustness. we randomly select sentence-label pairs to generate variant sentences with the same label. if the easynmt back-translation is not different from the source, we call google translate. figure 3 shows two examples of backtranslation, one for each pivot language we use. with chinese as the pivot, the original word gap was converted to distance, and the two prepositional phrase arguments of separated were swapped. with french as the pivot, space replaces gap, and word order is preserved. through trial-and-error, we found that data balancing was not effective unless there was a large enough gap between the original size of the rebalanced class and the largest class. also, there were limits to the maximum augmentation that worked well. if there was a five-fold difference in the size of a class and the largest one, we doubled the size of the small class. otherwise, data augmentation either resulted in little improvement or in degradation of performance. we speculate that increasing the diversity of linguistic form along with the number of examples might allow for greater increases in class size. ## experiments the research questions our experiments address are: 1) how well does sfrn perform compared to the state-of-the art? 2) does data augmentation and rebalancing improve sfrn performance? for both the semeval-2013 and asap-sas datasets, we compare the two sfrn variants-with the lstm (sfrn) or bert encoders (sfrn+)-against multiple baselines. on semeval, we also compare performance after training on the augmented semeval-2013+. performance metrics are accuracy, and macro-averaged f1 (m-f1). sfrn+ performs competitively on most semeval-2013 tasks without data augmentation. with data augmentation, sfrn+ outperforms the state-of-the-art on all beetle tasks, and on most scientsbank tasks. on asap-sas, performance is measured using quadratic weighted kappa (qwk). sfrn+ outperforms all baselines, including three kaggle models that use the full datasets, while sfrn+ uses a subset of the data that is more analogous to the semeval datasets. 2 we calculated 95% confidence intervals for all results for our models in ## conclusion we have presented a new type of relation network, sfrn, that learns relational information from qra triples for automatic short answer grading (asag). it can learn from two types of training data, using reference answers or rubrics. sfrn+, the version with the bert encoder, outperforms previous stateof-the-art by 8-11%, depending on the dataset and classification task, when combined with a simple data augmentation method to compensate for the small and unbalanced training data. as relational meaning is central to nlp, our future work will investigate ways to improve sfrn, to understand its behavior, and to apply it to new problems. another key avenue we aim to explore, however, is how to improve data augmentation and balancing for asag, and for other nlp tasks where data is difficult to come by.