D is G e M : Distractor Generation for Multiple Choice Questions with Span Masking (2024)

Abstract: Recent advancements in Natural Language Processing (NLP) have impacted numerous sub-fields such as natural language generation, natural language inference, question answering, and more. However, in the field of question generation, the creation of distractors for multiple-choice questions (MCQ) remains a challenging task. In this work, we present a simple, generic framework for distractor generation using readily available Pre-trained Language Models (PLMs). Unlike previous methods, our framework relies solely on pre-trained language models and does not require additional training on specific datasets. Building upon previous research, we introduce a two-stage framework consisting of candidate generation and candidate selection. Our proposed distractor generation framework outperforms previous methods without the need for training or fine-tuning. Human evaluations confirm that our approach produces more effective and engaging distractors. The related codebase is publicly available at https://github.com/obss/disgem.

## introduction multiple-choice cloze tests are a prevalent form of assessment that not only evaluates a student's reading comprehension but also challenges their ability to deduce the most fitting option from a set of alternatives. rooted in the concept of the traditional cloze test, where specific words are omitted from a passage and students are required to fill in the blanks with appropriate terms, the multiple-choice variant enhances the testing methodology by presenting a selection of potential options for each blank. this approach adds an element of complexity, requiring students to not only comprehend the context but also discern the most contextually appropriate answer among the * equal contribution. ## related work early nlp research focused on automatic question generation, with particular attention to multiple-choice questions (mcqs), addressing the challenge of distractor generation, i.e. generating plausible distractors alongside correct answers (ch and saha, 2020). traditional methods, including semantic analysis and ontologies, were initially employed for distractor generation @xcite @xcite . recent approaches leverage ontologies to select distractors based on semantic relationships @xcite . advancements in deep learning introduced endto-end frameworks such as bi-lstm and sequenceto-sequence models for distractor generation @xcite . transformer models further improved distractor generation, with approaches like round-trip neural machine translation @xcite and one encoder, three decoders architecture (maurya and desarkar, 2020). transformer finetuning strategies have also been applied @xcite @xcite . two noteworthy frameworks emerged, both featuring a candidate set generator and a distractor selector @xcite . @xcite integrates knowledge bases, ensuring semantic and grammatical relatedness, and uses a learning-torank model for distractor selection. building on this, chiang et al. (2022) proposes a pre-trained language model-based distractor generator that outperforms the former. however, fine-tuning limitations and computational costs are noted. in contrast, our model harnesses pre-trained masked language models without requiring fine-tuning, offering a simpler and more practical solution. ## methodology our approach, building upon the cdgp framework @xcite , is a dual-stage procedure: first, a candidate set generator (csg) creates potential options, followed by a distractor selector (ds) that finalizes the distractor set. see figure 1 for our framework's structure. notably, instead of requiring a training regimen, our methodology leverages a pre-trained language model to produce candidates. in the second stage, these candidates undergo a meticulous two-step elimination process to select the final distractors. for a detailed view of the distractor generation pipeline and the algorithm, we refer readers to appendix b. ## experiments we conducted two primary evaluations to assess our distractor generation method: instructionfollowing large language models (llms) evaluations and human evaluations. for llm evaluations, we used squad context/answer pairs to test different generation hyperparameters and compared the best to the previous work. for human evaluations, we used questions from the cloth dataset, comparing gold distractors, our generated distractors, and those from previous work, and measured human accuracy and quality ratings. additional quantitative results from experiments on the cloth dataset are provided in appendix c. ## discussion the evaluation results from both llm and human evaluations provide significant insights into the effectiveness of our distractor generation method, disgem, compared to previous methods. in the llm evaluations, we observed that the optimal configuration of a matching token count, a dispersion parameter of 1, and the cocktail shaker (ctl) decoding strategy showed a significant improvement over cdgp, as indicated by the final comparison scores. in human evaluations, the results were similarly encouraging. the cloze test scores revealed that evaluators had a slightly lower correctness rate on disgem's questions compared to gold and cdgp. this lower correctness rate indicates that disgem's distractors were more effective at introducing uncertainty and challenging the evaluators. the quality and difficulty ratings from evaluators were also higher for disgem, suggesting that our method produces distractors that are perceived as more engaging and effective at creating uncertainty. ## conclusion we propose a training-free dual-stage distractor generation framework, disgem. our framework possesses two main features extending previous works. disgem in particular, also, due to the training-free nature of our framework can directly be used on any passage without the effort of additional training/fine-tuning, and hence saves a lot of time. additionally, it alleviates the heavy hyper-parameter tuning stage. our evaluations with llm and human participants demonstrate disgem's effectiveness. llm evaluations showed disgem outperforms previous methods. human evaluations confirmed this with lower correctness rates and higher quality and difficulty ratings, indicating more effective and engaging distractors.