Exploring Automated Distractor Generation for Math Multiple-choice Questions via Large Language Models (2024)

Abstract: Multiple-choice questions (MCQs) are ubiquitous in almost all levels of education since they are easy to administer, grade, and are a reliable format in assessments and practices. One of the most important aspects of MCQs is the distractors, i.e., incorrect options that are designed to target common errors or misconceptions among real students. To date, the task of crafting high-quality distractors largely remains a labor and time-intensive process for teachers and learning content designers, which has limited scalability. In this work, we study the task of automated distractor generation in the domain of math MCQs and explore a wide variety of large language model (LLM)-based approaches, from in-context learning to fine-tuning. We conduct extensive experiments using a real-world math MCQ dataset and find that although LLMs can generate some mathematically valid distractors, they are less adept at anticipating common errors or misconceptions among real students.

## introduction multiple-choice questions (mcqs) are widely used to evaluate student knowledge because they enable quick and accurate administration and grading. mcqs are reliable because they are designed to measure specific learning objectives consistently @xcite @xcite . mcqs are constructed in a specific format. see figure 1 for an example. the stem refers to the statement on the problem setup and context, followed by a question that needs to be answered. among the options, the correct one can be referred to as the key, while incorrect ones can be referred to as distractors. as the name implies, distractors in mcqs are typically formulated to align with the common errors students would make or misconceptions students would exhibit. these distractors are chosen because students either i) lack the necessary * these authors contributed equally to this work. prior work on automatic distractor generation primarily focuses on language learning and reading comprehension tasks, where distractors are used to assess students' comprehension of a given text or article. early works use a ranking approach based on semantic similarity and word collocation information or a pre-defined ontology to produce distractors @xcite @xcite . more recent works use encoder-decoder models with attention mechanisms for distractor generation, resulting in longer and higher-quality distractors @xcite @xcite . additionally, several recent works use pre-trained large language models (llms) such as bert and t5 for distractor generation in the context of swedish reading and cloze test @xcite @xcite . other works prompt llms such as chatgpt and gpt-4 to generate distractors, either by providing detailed instructions or in-context examples in their prompts, for computer science course quiz questions and questions testing language mastery or factual knowledge @xcite . however, there is limited work on automatic distractor generation for math mcqs. this problem is more challenging than generating distractors for reading comprehension tasks because plausible distractors are not necessarily contained or can be inferred from the passage. a model for math mcq distractor generation should have some math problem-solving capability and more importantly, an understanding of the common errors or misconceptions among real students. existing works either use constraint logic programming (tomás and leal, 2013) or manually constructed rules @xcite to generate distractors. however, these works only applies to math mcqs generated by templates. the work in @xcite explores generating distractors using a neural network. however, their approach is training a math problem solver model and treating the incorrect outputs as distractors, which cannot capture common errors or misconceptions among real students. ## experiments in this section, we detail the specifics of our dataset, the evaluation metrics, the experimental setup, and report results from a series of quantitative, qualitative experiments, and human evaluation. ## conclusions and future work in this paper, we explore automated distractor generation for math multiple-choice questions via large language models. we conduct experiments on a real-world math mcq dataset and find that the incontext learning-based approach knn, achieves the best performance when compared to other approaches such as fine-tuning, chain-of-thought prompting, and various baselines. we also conduct human evaluation and observe that llms are capable of generating mathematically valid distractors but are not fully aware of common errors or misconceptions among real students. our initial question stem solve this problem: 3 ? 216 " ? llm-generated feedback message i think you have multiplied by 3. the question is asking for the cube root. llm-generated distractor 648 llm-generated feedback message i think you have written the digits as a new number. the question is asking for the cube root.