Multi-Relational Graph Transformer for Automatic Short Answer Grading (2022)

Abstract: The recent transition to the online educational domain has increased the need for Automatic Short Answer Grading (ASAG). ASAG automatically evaluates a student’s response against a (given) correct response and thus has been a prevalent semantic matching task. Most existing methods utilize sequential context to compare two sentences and ignore the structural context of the sentence; therefore, these methods may not result in the desired performance. In this paper, we overcome this problem by proposing a Multi-Relational Graph Transformer, MitiGaTe, to prepare token representations considering the structural context. Abstract Meaning Representation (AMR) graph is created by parsing the text response and then segregated into multiple subgraphs, each corresponding to a particular relationship in AMR. A Graph Transformer is used to prepare relation-specific token embeddings within each subgraph, then aggregated to obtain a subgraph representation. Finally, we compare the correct answer and the student response subgraph representations to yield a final score. Experimental results on Mohler’s dataset show that our system outperforms the existing state-of-the-art methods. We have released our implementation https://github.com/kvarun07/asag-gt , as we believe that our model can be useful for many future applications.

## introduction grading student work is critical for assessing their course understanding. however, answer grading can become monotonous and tedious for teachers. automatic short answer grading (asag) task is to assign ordinal scores to student answers, given some 'model' answer by an academician or instructor. figure 1 presents a sample question, model answer, and student answers from an undergraduate computer science course @xcite . one of the early approaches for solving the asag task has been to build models based on humandesigned features @xcite . recent works utilize deep learning methods such as convolutional neural networks (cnns), long short-term memory networks (lstms), transformer @xcite to learn the representation of student responses and to avoid designing features manually @xcite @xcite @xcite . due to semantic heterogeneity, the main problem in assessing student responses given instructor-provided model answers is a complex natural language understanding task (the same answer could be articulated in different ways) @xcite . we hypothesise that a student's answer will be considered correct if the keywords in the answer are in the right relationship with the corresponding words from the model answer. as can be seen from figure 1 , student a is awarded full marks because the words like testing phase and coding are adequately associated with words in the model answer like testing stage and coding phase. however, other students are awarded partial marks because not a lot of words in those correspond to some relation in the model answer, i.e. the decreasing score corresponds to the decreasing number of relations in the model answer being captured in the student response. this motivates us to incorporate structural relationship context information for effective comparison. we discuss the various types of relations captured in detail in the further sections. this paper applies the principle of short text matching to solve the problem of grading short subjective student's response. the key steps for textual matching are efficient textual representation, followed by semantic matching. in literature, we see that short text matching is broadly based on two approaches: sequence-based and structurebased. sequence-based models fully exploit semantic information of sentences without incorporating syntactic information @xcite , @xcite @xcite . recent works by @xcite @xcite have found that the structural information of sentences is beneficial for sentence representation. therefore, structurebased neural networks @xcite @xcite , @xcite exhibit better performance than sequence-based models. graph convolutional network (gcn) @xcite can extract semantic and syntactic information of sentences simultaneously from the graph. gcn first propagates information among nodes and their neighbors and then provides node representation by aggregating the received information. however, gcns are designed for homogeneous graphs and cannot handle different types of nodes and relationships in the graph. recently there have been attempts to explore relationships in the graphs. @xcite introduces rgcn to handle relationships in knowledge graphs by using specific matrices for each relationship. nevertheless, it focuses only on characteristics of the relations and does not study different types of features associated with a node. this paper introduces a multi-relational graph transformer (mitigate) for asag to incorporate the structural context. we first transform a sentence into an abstract meaning representation (amr) graph @xcite . amr parses a sentence into a rooted directed graph. then subgraphs are prepared corresponding to the relationships (types of edges) in the original amr graph. for each subgraph, mitigate prepares relationspecific token representations and aggregates them to obtain a subgraph representation. finally, these relation-enriched subgraph representations of the student and the model answer are matched using multi-perspective matching @xcite and the matching result yields the student score. we evaluate our model on the benchmark mohler's dataset @xcite and it outperforms the current state of the art models. our main contributions can be summarized as follows: application of gcn on nlp: gcn is a simplified graph neural network (gnn) introduced by @xcite to perform semi-supervised classification. in nlp, gcn is mainly explored in tasks such as semantic role labeling @xcite , machine translation @xcite . @xcite first model a whole corpus as a graph where documents and words are regarded as nodes. however, most gnns were designed for homogeneous graphs and could not handle different nodes and relations in heterogeneous graphs. unlike the existing methods, we consider the role of relations to improve the learning of more fine-grained node representation. ## methodology we formally define the asag short text matching problem as follows: given two sentences a @xmath0, w m 2 , . . . } and a @xmath1, w s 2 , . . . }, where a m and a s refers to model and student answer respectively with w as words in the sentence, the goal of a text-matching model f (a m , a s ) is to compute the semantic similarity of a m and a s . in this section, we discuss a graph-based matching model. to create graphs from the input sentences, we first parse each sentence into an amr graph (section 3.1). further, we prepare subgraphs g sub from amr graphs corresponding to each relation (section 3.2). the intuition behind subgraph splitting is to get relation-enriched structural information which can improve matching performance and interestingly can be used to provide a reasonable feedback to students (section 5.5). we then create relation-specific token representation h w,r from each subgraph and aggregate them to a final subgraph representation denoted as g ϕ r,m for model and g ϕ r,s for student answer (section 3.3). lastly, we compare them in section 3.4 to predict the grading score in section 3.5. as shown in figure 3 , our model consists of five layers namely, text to amr conversion, subgraph preparation layer, graph transformer encoder layer, subgraph matching layer and lastly, score prediction layer. we discuss each layer below. ## results and analysis in this section, we attempt to answer following questions: rq1. how does each subgraph influence the final results? (section 5.2) rq2. does incorporating edge representation while computing node representation improve the final results? (section 5.3) rq3. can mitigate provide feedback to students i.e. why did a student lose marks? (section 5.5) @xcite . @xmath2, @xmath3; @xmath4; @xmath5; @xmath6; @xmath7; @xmath8, @xmath9, @xmath10. ## conclusion in this paper, we have proposed mitigate for asag. it prepares token embeddings considering the structural context of a sentence and thus provides a more efficient matching method by considering multiple relations at a granular level. experimental results show that mitigate outperforms the existing asag systems by a significant margin, and can be extended to give an intuitive feedback to explain the provided score. in the future, we would like to investigate how to deal with long and multi-lingual answers. our approach uses an amr graph, and thus such tasks will need a compatible amr parser. we also aim to incorporate the explainability of the final scoring to generate more comprehensive evaluator feedback.