Automating True-False Multiple-Choice Question Generation and Evaluation with Retrieval-based Accuracy Differential (2024)

Abstract: Creating high-quality True-False (TF) multiple-choice questions (MCQs), with accurate distractors, is a challenging and time-consuming task in education. This paper introduces True-False Distractor Generation (TFDG), a pipeline that leverages pre-trained language models and sentence retrieval techniques to automate the generation of TF-type MCQ distractors. Furthermore, the evaluation of generated TF questions presents a challenge. Traditional metrics like BLEU and ROUGE are unsuitable for this task. To address this, we propose a new evaluation metric called Retrieval-based Accuracy Differential (RAD). RAD assesses the discriminative power of TF questions by comparing model accuracy with and without access to reference texts. It quantitatively evaluates how well questions differentiate between students with varying knowledge levels. This research benefits educators and assessment developers, facilitating the efficient automatic generation of high-quality TF-type MCQs and their reliable evaluation.

## introduction multiple-choice questions (mcqs) are an essential part of evaluative instruments for education. however, creating mcqs manually can be timeconsuming and laborious. the core challenge part for mcqs' design is to craft distractors (wrong options). as a result, researchers have been working on automatic mcq's distractor generation for different exam settings, such as reading comprehension @xcite , cloze quiz @xcite , knowledge qa @xcite . despite significant progress in the field, the generation of distractors for true-false (tf) mcqs has received limited attention. tf-type mcqs typically present four statement options, one correct and three incorrect, as shown in figure 1 , requiring respondents to identify the correct option. these questions are commonly used in knowledge-based assessments, where participants must judge the accuracy of given statements. however, there is a notable research gap in the automatic generation of tf-type distractors. while distractor generation has advanced in cloze tests @xcite @xcite @xcite and reading comprehension @xcite @xcite , the challenges of crafting true-false distractors remain underexplored. to address this, we introduce tfdg, a pipeline that integrates pre-trained language models and sentence retrieval techniques for true-false distractor generation. furthermore, a challenge in tfdg lies in the evaluation of its effectiveness. traditional tokenbased metrics, like bleu or rouge, do not quite encapsulate the essence of performance. these scores predominantly gauge n-gram overlap between the generated content and a reference. however, the essence of tf generation is not just about matching a reference but ensuring the crafted statements stand accurate and contextually relevant. while human evaluation, as utilized by @xcite , might seem a plausible route, it is not devoid of complications, such as potential subjectivity or varied review standards. as such, developing a robust evaluation metric for tf question generation presents another challenge. to address this issue, we propose evaluating performance by considering the discriminative power of the questions, which refers to their ability to distinguish between respondents with varying levels of knowledge. a well-constructed multiple-choice question (mcq) should exhibit high discriminative power, clearly differentiating between students who are familiar with the material and those who are not. to facilitate this, we introduce the retrievalbased accuracy differential (rad), a metric that gauges the accuracy disparity of the mcq answering model with and without access to reference texts. by comparing the model's accuracy across these conditions using generated mcqs, we can discern how effectively the model can select the correct answer, thereby evaluating the discriminative power of the mcqs. this method highlights our ability to quantitatively measure the quality of mcqs, enhancing the robustness of mcq evaluation. to further illustrate this concept, consider a visual representation shown in figure 2 comparing the behavior difference of mcq answering models when faced with questions of varying discriminative power. in this depiction, the difference in model behavior, with and without access to external reference texts, can shed light on the discriminative power of the mcqs. mcqs with high discriminative power should show a significant divergence in the model's behavior when external resources are either accessible or withheld, signifying that a well-crafted question can be resolved based on the prior knowledge provided. conversely, for mcqs with low discriminative power, the model's behavior is anticipated to remain consistent across both scenarios, suggesting that such questions might be too simplistic, ambiguous, or not thoroughly aligned with the tested content. the contributions of this paper are as follows. in this section, we review the literature related to this work. existing distractor generation (dg) methods can be broadly categorized into two main approaches: cloze distractor generation and reading comprehension (rc) distractor generation. in the cloze dg task, the problem is approached as a word filling challenge. typically, the first step involves extracting distractor candidates from the context or a knowledge base, followed by ranking the extracted distractors to produce the final result. existing models in this field primarily rely on similarity heuristics @xcite or supervised learning @xcite @xcite . on the other hand, the rc-type dg focuses on generating sentence-level distractors for reading comprehension level testing, such as summarizing an article or understanding author's opinion @xcite @xcite . for sentence-level distractor generation, neural models are commonly employed. delving into the available literature, the study by @xcite emerges as closely aligned with our research aims. the authors introduce an unsupervised true/false question generation technique (tf-qg). nevertheless, their methodology is tailored toward reading comprehension assessments intended for english learners. this deviates from our goal of crafting tf questions for knowledge-centric quizzes. as a result, there is a need to develop a new method for generating tf questions that is more aligned with our goal. furthermore, in @xcite , performance evaluation was conducted through human evaluation. however, assessing the quality of a question through human evaluation can lead to issues distractor level model type question type word/phrase sentence extractive generative @xcite y y y r.c. @xcite y y cloze @xcite y y cloze @xcite y y y cloze @xcite y y cloze @xcite y y y r.c. @xcite ## methodology our framework begins with a user-provided keyword, related to a specific topic of interest. as shown in figure 3 , our framework works as follows. ## conclusion in this paper, we address two main issues: how to automatically create incorrect true-false options and how to assess the quality of these generated options. specifically, we propose a pipeline that generates true-false incorrect options based on user-provided keywords. additionally, we introduce the rad metric to evaluate the generated results. preliminary experiments demonstrate that our pipeline effectively generates medium-level questions, as evidenced by the rad metric comparison. however, our current architecture struggles to generate more challenging questions that require reasoning and logical judgment. therefore, our current achievements are primarily applicable to modifying literal distractors. furthermore, we also need to refine the rad metric to account for cases where the initial model's answer accuracy is low due to multiple correct options in the generated results.