D i VERT : Distractor Generation with Variational Errors Represented as Text for Math Multiple-choice Questions (2024)

Abstract: High-quality distractors are crucial to both the assessment and pedagogical value of multiple-choice questions (MCQs), where manually crafting ones that anticipate knowledge deficiencies or misconceptions among real students is difficult. Meanwhile, automated distractor generation, even with the help of large language models (LLMs), remains challenging for subjects like math. It is crucial to not only identify plausible distractors but also understand the error behind them. In this paper, we introduce DiVERT (Distractor Generation with Variational Errors Represented as Text), a novel variational approach that learns an interpretable representation of errors behind distractors in math MCQs. Through experiments on a real-world math MCQ dataset with 1,434 questions used by hundreds of thousands of students, we show that DiVERT, despite using a base open-source LLM with 7B parameters, outperforms state-of-the-art approaches using GPT-4o on downstream distractor generation. We also conduct a human evaluation with math educators and find that DiVERT leads to error labels that are of comparable quality to human-authored ones.

## introduction multiple-choice questions (mcqs) are arguably the most common form of questions found in standardized tests. each mcq contains a question stem that states the context of the questions and the task to be completed, and a series of options: a key, i.e., the correct answer, embedded among several other distractors, i.e., incorrect answers. see figure 1 for an example. mcqs are widely used in realworld, large-scale educational/psychological tests and surveys, mainly due to the ease in automated grading @xcite @xcite . however, constructing a good set of distractors can be quite challenging: on the one hand, they should correspond to clear errors in the factual recall or reasoning process required by the question's task. on the other hand, these errors should not be too obvious such that no student would select the distractors. therefore, designing high-quality mcqs that measure specific knowledge components/concepts/skills and more importantly, corresponding distractors good at capturing specific knowledge deficiencies among real students/test takers is very important to the development of high-quality mcqs. the typical approach to mcq distractor development primarily relies on extensive human effort, which can be burdensome for educators, which motivated the development of automated approaches. prior work on automated distractor generation primarily focuses on i) cloze tasks in language learning to assess vocabulary recall or grammatical knowledge and ii) reading comprehension question answering to assess comprehension of a given text or article @xcite . approaches include using knowledge graphs, encoder-decoder models, and with help from large language models (llms). see section 7 for a more detailed review of related work. for other subjects where common tasks require (possibly complex) reasoning ability, such as math, there exist relatively few approaches to automated mcq distractor generation. in these domains, generating high-quality distractors is more challenging than in language learning or reading comprehension: distractors need to reflect abstract mathematical misconceptions and/or procedural errors in the mathematical reasoning process. earlier works either use symbolic, rule-based approaches to generate distractors @xcite , which have limited generalizability beyond template-based questions, or sample incorrect generations during a math problem solving process as distractors @xcite . more recently, @xcite investigate a wide array of llm-based approaches for math mcq generation, particularly focusing on the plausibility of distractors, i.e., how likely is a distractor going to be selected among real students. however, to the best of our knowledge, no existing approach has attempted to generate explanations of errors underlying mcq distractors. in assessment scenarios where one's goal is to measure the overall ability of a student/test taker, being able to generate a good set of distractors may be sufficient. however, in real-world educational scenarios where one's goal is to maximize the learning gain of students, being able to interpret the cause of error behind a distractor is highly important; this information can be used for student knowledge diagnosis, i.e., identifying areas where they lack sufficient knowledge @xcite or even pinpoint specific misconceptions they exhibit @xcite . such diagnosis can be used to provide feedback to both teachers, to help them better understand students' learning progress and directly to students in real-time, through intelligent tutoring systems @xcite , online learning platforms @xcite , or via chatbots powered by llms @xcite . therefore, interpreting the errors behind students selecting specific distractors in math mcqs is important yet challenging, partly due to the mathematical reasoning processes required by these questions. ## problem formulation we denote each math mcq as @xmath0, k, f, t, d, e}, which contains a set of textual components, including the question stem s, the key k, (optionally) an explanation of the key f , (optionally) question topic/concept tags t, and a set of distractors d: we denote each distractor itself as d @xmath1, with e @xmath2, e.g., a misconception. in this paper, we do not consider mcqs with figures or diagrams since the open-source llms we work with cannot process visual input. all of these textual components are sequences of words and math symbols and we denote them as a series of tokens {w 1 , . . . , w l }, with l being the length of the sequence. we study the task of learning an interpretable space of errors behind distractors in math mcqs. since it is difficult to quantitatively evaluate error representations, we also study the downstream task of distractor generation @xcite . specifi-cally, our two major goals are: we formulate the error generation task as learning a function that outputs a plausible error in an mcq, given the question stem and attributes including the key and (optionally) explanation and topic tags, i.e., g err (s, k, f, t) → ê. similarly, we formulate the distractor generation task as learning a function that outputs a distractor that corresponds to a plausible error, i.e., g dis (s, k, f, t, ê) → d. ## methodology we now detail our approach to error representation learning and distractor generation in math mcqs. ## experimental evaluation in this section, we detail our experiments on a realworld math mcq dataset. for quantitative evaluation, we compare divert against state-of-the-art approaches and strong baselines on the task of distractor generation. for qualitative evaluation, we perform a human evaluation of the generated error explanations with math educators. ## results, analysis, and discussion in this section, we quantitatively evaluate the quality of generated distractors, qualitatively evaluate both errors and distractors, perform error analyses on failed cases, and conduct an ablation study. ## 1 implausible error, plausible but inconsistent distractor. divided by the denominator instead of the numerator. implausible error, implausible and inconsistent distractor. subtracts instead of divides. ## related work for automated distractor generation in language learning and reading comprehension, prior works have explored ranking candidate distractors based on semantic similarity to the key and using knowledge graphs @xcite @xcite . more recent works use an end-to-end pipeline for distractor generation, which lead to longer and higher-quality distractors @xcite @xcite @xcite , also leveraging open-source llms such as bert and t5 @xcite @xcite @xcite ; these approaches are similar to the baselines we use in this work. other works prompt state-of-the-art proprietary llms such as chatgpt and gpt-4 to generate distractors, with carefully crafted prompts, for a wider range of subjects including math and computer science @xcite @xcite . evaluating the quality of distractors, however, remains challenging: @xcite proposes a series of features that can be used to evaluate the quality of distractors, although those features are mostly about surface semantics and do not apply to subjects like math that require deeper reasoning. another line of recent work in @xcite proposes to measure the quality of distractors through how likely they are going to be selected by real students, which requires training another model to predict student option selection behavior. however, most existing mcq datasets do not come with such student response information. at a high level, an alternative to our approach of representing errors as text is to learn latent error representations @xcite , i.e., characterizing errors as a latent stochastic error vector, possibly in the text embedding space. however, in our experiments, we found this approach to be uninterpretable and ineffective, significantly reducing distractor generation performance, possibly because errors behind distractors in math mcqs correspond to complex reasoning processes and are harder to capture than different user writing styles. our approach in improving the diversity of generated errors also bears some resemblance to the works in @xcite , although the nature of our task is more complex than dialogue generation and question answering. ## conclusions and future work in this paper, we proposed divert, a novel variational approach to jointly learn an interpretable, textual representation of errors in math mcqs and how to generate distractors that correspond to these errors. on a real-world math mcq dataset, we showed that divert results in better downstream distractor generation performance over state-of-theart approaches. we also conducted a human evaluation with math educators and found that divert leads to errors as interpretable as those provided by humans, and outperforms gpt-4o, despite using a much smaller base llm. there are many avenues for future work. first, we plan to apply the learned error representations to another downstream task, feedback generation, and explore whether our approach leads to a more accurate student profile/model, which will in turn result in higher quality feedback in math tutoring chatbots. second, we plan to test the across-topic generalizability of our approach and investigate whether the error representations can generalize to previously unseen math topics. third, we hope to develop new, llm-based metrics to evaluate the mathematical validity of an error and the equivalence between two error explanations, to alleviate the need for human evaluation of error quality.