Improving Automated Distractor Generation for Math Multiple-choice Questions with Overgenerate-and-rank (2024)

Abstract: Multiple-choice questions (MCQs) are commonly used across all levels of math education since they can be deployed and graded at a large scale. A critical component of MCQs is the distractors, i.e., incorrect answers crafted to reflect student errors or misconceptions. Automatically generating them in math MCQs, e.g., with large language models, has been challenging. In this work, we propose a novel method to enhance the quality of generated distractors through overgenerate-and-rank, training a ranking model to predict how likely distractors are to be selected by real students. Experimental results on a real-world dataset and human evaluation with math teachers show that our ranking model increases alignment with human-authored distractors, although human-authored ones are still preferred over generated ones.

## introduction and related work multiple-choice questions (mcqs) are commonly used to assess student knowledge across all levels of education, including math, since they can accurately assess student knowledge while being easy to administer and grade at scale @xcite @xcite ). an mcq is comprised of a question stem and several answer options. the question stem establishes the context and presents a problem for students to solve. among the options, there exists a key, which is the correct answer, and multiple distractors, which are the incorrect answers specifically designed to reflect student errors or misconceptions. although mcqs offer numerous advantages for assessing student knowledge, crafting high-quality distractors poses a significant challenge for teachers and educators. high-quality distractors should be sufficiently challenging so students do not quickly identify them as incorrect answers. additionally, they should be designed to target specific errors or * these authors contributed equally to this work. misconceptions, enticing students who make these errors or hold these misconceptions to choose them. this delicate balance makes the creation of such high-quality distractors a time and labor-intensive endeavor @xcite . earlier works on automatic distractor generation for math mcqs use constraint logic programming @xcite or manually crafted rules @xcite to generate distractors. however, these methods are restricted to templategenerated mcqs, which have limited applicability in a broader context. more recent work @xcite trains a neural network to solve math problems and sample incorrect answers as distractors. not surprisingly, the generated distractors fail to capture student errors or misconceptions. the most recent works @xcite explore this task using state-of-the-art large language models (llms), such as chatgpt. the authors experiment with several different approaches, including few-shot in-context learning @xcite and zero-shot chain-of-thought (cot) prompting @xcite , showing that llms can often generate distractors that are mathematically relevant to the mcq. however, the overall alignment level with human-authored distractors that are thought to reflect student errors or misconceptions is not high. these works indicate a need to understand what errors or misconceptions are common among students and to use this information to improve the quality of generated distractors. ## methodology this section contains the details of the task definition and our over-generate-and-rank method. ## experiments this section provides a comprehensive overview of our dataset, outlines the evaluation metrics and the experimental setup, and details the findings from experiments and human evaluation. ## conclusions and future work in this paper, we propose an overgenerate-and-rank method for generating distractors for math mcqs. we train a ranking model to predict which distractors students would select more often, and this ranking model can be applied to any existing distractor generation method. we experimentally validate its performance on a real-world dataset and test its limitations through human evaluation. avenues for future work include but are not limited to further improving the ranking model through a student-specific distractor selection prediction objective that considers their knowledge state @xcite , developing a human-in-the-loop approach for distractor selection percentage prediction, and using the same approach for feedback generation @xcite . finally, extending our work from multiple-choice questions to openended questions is important, since open-ended student responses contain much more detailed information on their errors @xcite @xcite .