Distractor Generation in Multiple-Choice Tasks: A Survey of Methods, Datasets, and Evaluation (2024)

Abstract: The distractor generation task focuses on generating incorrect but plausible options for objective questions such as fill-in-the-blank and multiple-choice questions. This task is widely utilized in educational settings across various domains and subjects. The effectiveness of these questions in assessments relies on the quality of the distractors, as they challenge examinees to select the correct answer from a set of misleading options. The evolution of artificial intelligence (AI) has transitioned the task from traditional methods to the use of neural networks and pre-trained language models. This shift has established new benchmarks and expanded the use of advanced deep learning methods in generating distractors. This survey explores distractor generation tasks, datasets, methods, and current evaluation metrics for English objective questions, covering both text-based and multi-modal domains. It also evaluates existing AI models and benchmarks and discusses potential future research directions.

## introduction objective questions @xcite such as fillin-the-blank and multiple-choice questions require an examinee to select one valid answer from a set of invalid options @xcite . these types of questions contribute to fair assessment across various domains (e.g., science @xcite , english @xcite , @xcite , and medicine (ha and yaneva, 2018)). they are also beneficial for educators in assessing large capacity of students with unbiased results @xcite . however, creating objective questions manually is a laborious task, as it requires selecting plausible false options, known as distractors, that can effectively confuse the examinee. distractor generation (dg) @xcite ) is the process of generating an erroneous plausible 1 resources are available at https://github.com/ distractor-generation/dg_survey . despite the emerging interest in the dg research, there is no literature review in this field, to the best of our knowledge. existing relevant surveys focus on generating multiple-choice questions @xcite @xcite without discussing dg tasks. a recent work @xcite discussed dg as a subtask of natural language generation (nlg) in the text abbreviation tasks, rather than a subtask in objective questions. we aim to fill the gap and conduct the first survey for dg in objective type of questions. to this end, we collected over 100 highquality papers from top conferences such as acl, aaai, ijcai, iclr, emnlp, naacl, coling, and aied and journals such as acm computing ## methods -distractor generation the methods range from traditional to advanced ai approaches, including deep neural networks and pre-trained language models. ## evaluation methods evaluation methods for dg include automatic and manual approaches that rely on human judgment. ## discussions and findings this section provides analysis of the current ai models utilized for dg, along with an overview of the existing and emerging benchmarks. ## future directions this section outlines directions for future research. ## conclusion distractor generation (dg) is critical in assessment and has received significant attention with advanced ai models. this paper surveys dg tasks, including fill-in-the-blank and multiple-choice question across text and multi-modal domains. it categorizes the tasks within relevant datasets and provides a comprehensive analysis of the components in the available datasets. this paper also provides a detailed discussion of the current methods, summarizes the evaluation metrics, and discusses the main findings, including the analysis of ai models and benchmarks. it also outlines potential future research directions to facilitate further improvements and explorations. to enhance research in distractor generation, this paper also provides a continuously updated reading list available on a github repository at https://github.com/ distractor-generation/dg_survey .