Mistake Captioning: A Machine Learning Approach for Detecting Mistakes and Generating Instructive Feedback (2021)

Abstract: Giving feedback to students is not just about marking their answers as correct or incorrect, but also finding mistakes in their thought process that led them to that incorrect answer. In this paper, we introduce a machine learning technique for mistake captioning, a task that attempts to identify mistakes and provide feedback meant to help learners correct these mistakes. We do this by training a sequence-to-sequence network to generate this feedback based on domain experts. To evaluate this system, we explore how it can be used on a Linguistics assignment studying Grimm’s Law. We show that our approach generates feedback that outperforms a baseline on a set of automated NLP metrics. In addition, we perform a series of case studies in which we examine successful and unsuccessful system outputs.

## introduction giving feedback is one of the most critical parts of education and training. it allows the instructor to correct errors in students' understanding and guide them to the correct solution. in automated learning systems, such as intelligent tutoring systems (itss) the feedback and hints are focused on getting the student to complete the assignment, but this may not take into account their mastery of that skill. while better feedback can be made by considering student intent, we find that current models are lacking in this regard. they are typically focused on learning objective mastery and may not necessarily give the whole picture when determining why students answered something the way they did. but there is a natural solution to this, in that skilled educators are capable of intuitively modeling a student's thought process and determining mistakes in it. this skill is often developed over time. the difficulty of this task for educators depends on the field of study, with some more free form fields like writing and programming being particularly difficult. by indirectly modeling a student's intent by instead modeling how the instructor gives feedback, we can use this model to generate novel feedback sentences. we propose a system that when given a mistake that has been identified in the task of deriving sound changes from proto-indo-european to proto-germanic, will automatically generate an explanation that identifies the mistake and the reasoning to why this sound change should not apply in this case. we call this method mistake captioning as it takes inspiration from image captioning methods and post-hoc ai rationale generation methods such as @xcite automated rationale generation. we apply a suite of automated machine translation and image captioning metrics to test our system. we also take a closer look at 3 cases from a more complete retraining of the network to better understand how the network is performing and how well the metrics perform regarding our task. we find that this method is a good start, but more data and work need to be done to be able to integrate this into a serious learning environment. ## background and related work in the space of itss a problem is how to get a student to the correct solution. one of the ways to achieve this is by having an expert to author the correct solution and the path to take to get there @xcite @xcite . these all require the use of an expert to either structure the assignment such that feedback can be extracted or to label assignments to create a system that can learn the expert knowledge. in some domains this is not necessary as the students can provide the data for the path to the solution themselves. this data-driven approach uses data from multiple students to create all possible paths that lead to a solution, as is shown by @xcite in their work on hint factory. this was later extended to create paths that optimize for productivity metrics by @xcite . in vast solution spaces generating all possible paths to the solution is not feasible, and in @xcite work this is done by abstracting states so that a single state can represent a large space of states. our method does not attempt to establish a correct answer like these previous methods as they do not tackle an underlying problem as to why students make mistakes. we structure our problem not as a method of guiding the student to the correct path, but by trying to correct the thought process that led them to their current answer. this is similar to the methods used in image captioning @xcite where they train a model to recognize what is in an image by relating human authored captions to images. this model is then used to generate captions on similar features in new pictures. this is also like the work done in post-hoc ai rationale generation like that of @xcite . in this work they relate a set of human actions to human rationales of those actions for the use of explaining ai behavior. in our system, we make use of human actions in the form of mistakes and relate them to human explanations of the mistake. ## experiments we conducted two experiments to evaluate our method: a quantitative approach using a set of automated metrics to judge the candidate explanations, and a qualitative case study to demonstrate what types of outputs were generated. for both the qualitative and quantitative evaluation we used 5 different automated metrics: bleu @xcite , meteor (banerjee and lavie, 2005), rouge @xcite , cider @xcite , and bertscore @xcite . for our quantitative evaluation, we used k-fold cross-validation to control for the variance in our data. we compare our approach against a majority baseline. since several of the explanations in the data were repeated, we decided that a better baseline would be the majority baseline and used the most repeated explanation in our data. this explanation was used 15 times in the data, and for the majority baseline we set the generated output to all be this exact sentence. this explanation can be seen in for our qualitative analysis, we opted to look at outputs for a retrained model that was trained on all but 8 random examples of the data. since our dataset is small, we decided that training on all but a small number of examples would give better insights on what the network was learning. we examined how the model performed in generating explanations for each of the 8 examples that were held out during training. from these 8, we used their automated metric performance in conjunction with their correctness to divide them into three categories: correct, partially incorrect, and incorrect. we then explore why this occurred and reason about the implications of these results in practice. during training, we use the following hyperparameter values for our model. we used a teacher forcing ratio of 0.5, and a learning rate of 0.01. for the hidden layers (consisting of the attention and gru layers), a fixed size of 256 was used. all outputs were limited to a maximum length of 30 but this limit was not reached. to test the model, we implemented beam search to select the best candidate explanation instead of the default greedy search. in addition to improving the accuracy of the model this allowed us to look at multiple final outputs, each scored by the model. this is useful in manually judging if the alternatives to the top output were more correct. we limited the beam width to 5. ## results we found that our method consistently outperformed the baseline in all metrics as can be seen in applying a 2-sided t-test we find that the difference between the baseline and the 5-fold is statistically significant (@xmath0.0001) in all cases. when comparing the baseline to 8 case studies this also holds true (@xmath1.001). there is also no statistically significant difference between the case studies and the 5-fold. ## discussion in this section, we discuss the results in more detail, going over the results of the automated metrics and case studies. ## future work due to the way that our method works in identifying a single mistake at a time, we can use the fill-in-the-blank task to generate explanations on multiple mistakes. this is done by first comparing the answer given to the correct answer and finding the difference between the two and isolating each individual mistake. each of these individual mistakes can be then applied to the original correct answer to create separate faulty answers which can subsequently be run through our system. using this method, we can also change the fill-in-the-blank task to a simple response, where the student is tasked to correctly make all the changes to the pie word to produce a pgmc word. either through separate models for separate rules (and enough annotations on the input) or by training a single more complex model, it may be possible to create a system that can caption all mistakes for a given task. this could work in conjunction with automatically grading the assignments to quickly provide feedback on the mistakes. ## conclusion in this paper we show a method of the novel task of creating automated captions of mistakes. these captions serve as explanations for what the thought process behind the mistake was, and why it is not applicable in this case. we apply two methods to test our model, comparing against a baseline with a suite of automated metrics, and manually identifying and analyzing the sentences in a series of case studies. our experiments show that our method has promise in creating these automated captions but that there are significant challenges that need to be overcome. we set out tasks in the domain of linguistics pedagogy, and in this we can make improvements with representation of the input data and with gathering more and more varied data. we also hope to expand this to include a wider set of rules covering a larger set of sound changes, and eventually evaluate the method using a human study. we hope that this method of mistake captioning can be applied to other fields of study.