Assessing Distractors in Multiple-Choice Tests (2023)

Abstract: Multiple-choice tests are a common approach for assessing candidates’ comprehension skills. Standard multiple-choice reading comprehension exams require candidates to select the correct answer option from a discrete set based on a question in relation to a contextual passage. For appropriate assessment, the distractor answer options must by definition be incorrect but plausible and diverse. However, generating good quality distractors satisfying these criteria is a challenging task for content creators. We propose automated assessment metrics for the quality of distractors in multiple-choice reading comprehension tests. Specifically, we define quality in terms of the incorrectness, plausibility and diversity of the distractor options. We assess incorrectness using the classification ability of a binary multiple-choice reading comprehension system. Plausibility is assessed by considering the distractor confidence - the probability mass associated with the distractor options for a standard multi-class multiple-choice reading comprehension system. Diversity is assessed by pairwise comparison of an embedding-based equivalence metric between the distractors of a question. To further validate the plausibility metric we compare against candidate distributions over multiple-choice questions and agreement with a ChatGPT model’s interpretation of distractor plausibility and diversity.

## introduction multiple-choice tests are an efficient and effective way of assessing candidates' comprehension skills @xcite with key advantages such as being a standardized format, eliminating subjective grading and being easy to grade. these advantages make them a highly popular assessment method widely adopted in a range of settings @xcite , such as university exams, job screening and qualification accreditation. a challenging aspect of generating suitable multiple-choice questions is in selecting the incorrect options, i.e the distractors @xcite . selecting good distractors is a subtle process, which requires the option to possess several properties @xcite ; 1) the distractor should not be a possible correct answer, as this would make marking the question subjective. 2) the distractor option should not be too obviously invalid, as then candidates may easily avoid them. currently, test creators conduct a pre-test phase where questions are internally reviewed and then tested on a subset of real candidates @xcite , an evaluation process that is very manual and can be both subjective and expensive. automating the process to evaluate distractors would lead to improved efficiency in the test creation process, and may aid test designers to create high-quality questions. however, currently, assessing the quality of distractors is a challenging task. to the best of our knowledge, there are no existing datasets targeted towards assisting automated distractor evaluation (beyond sequence overlap measures @xcite ), and therefore any approach has to port information from other resources. further, validating the efficacy of approaches is a challenging task, especially without manual labels of distractor quality, which themselves due to the nature of the task are at risk of being subjective. in this paper, we propose the distractor assessment framework (daf), a collection of systems that can be used to automatically determine the quality of distractors. our framework provides automatic scores for the 3 previously mentioned important properties of the distractors: incorrectness, plausibility and diversity. the incorrectness detector is a binary machine reading comprehension system that predicts whether a given distractor could be the correct answer, the plausibility evaluator leverages system confidence, while the diversity assessor considers the average similarity score between all pairs of distractors. we further propose several methods to probe existing largescale foundation models, specifically chatgpt instruction fine-tuned @xcite from gpt-3 @xcite , to validate the suitability of our quality metrics and demonstrate that our methods do reasonably capture elements of the considered properties. additionally, we validate the plausibility metric against human candidate distributions on multiple-choice questions. our contributions can be summarized as follows: ## related work previous automatic distractor assessment methods proposed to compare the similarity of generated distractors with the ground-truth distractors present in the dataset @xcite or consider rule-based approaches @xcite . following standard reference-based evaluation, n-gram overlap metrics such as bleu @xcite , rouge @xcite and meteor (banerjee and lavie, 2005) have been considered, where these metrics measure the overlap between generated distractors and the distractors from a set of human-annotated ground truth sequences. however, having reference-based distractor evaluation approaches has notable shortcomings @xcite . in particular, for a given multiple-choice question, the set of annotated distractors is unlikely to span the set of all possible good distractors, and some options may get unfairly penalised simply because no similar ones exist in the annotated set. we therefore focus on decomposing the distractors in terms of individual qualities (incorrectness, plausibility and diversity) and consider quality as an amalgamation of the above. these approaches have been considered for the assessment of alternate qualities of questions. @xcite investigate answer-agnostic generated questions in terms of the qualities of relevance, interpretability and acceptability with comparison against human markers. raina and gales (2022b) assess multiple-choice questions in terms of grammatical fluidity, answerability, diversity and complexity. our work specifically explores distractor assessment in multiple-choice questions with a focus on automated assessment. ## multiple-choice comprehension in this section, we describe the multiple-choice reading comprehension task, and the architecture of standard machine reading comprehension systems. note that the machine reading comprehension system will later be leveraged in several components of the daf (see section 4). ## distractor assessment framework as discussed in section 2, n-gram reference-based assessment metrics may not be appropriate for distractor assessment as 1) the number of valid distractors for a given multiple choice reading question is vast, which a limited set of references may fail to capture; 2) they are only valid when a set of reference distractors are available, which requires human intervention and limits the advantages of automatic distractor evaluation. the next section discusses the daf, which uses three different reference-free methods to estimate the quality of distractors' incorrectness, plausibility and diversity independently. incorrectness for multiple-choice reading comprehension questions, distractors must by definition be incorrect, and the answer option must be the only valid answer in the set of options. for automated multiple-choice question generation pipelines, it is particularly important to ensure that any generated distractors satisfy the requirement of being incorrect and do not cause a subjective interpretation of the question. here, we detail the approach used to assess whether distractors satisfy the incorrectness requirement. the binary mrc system from section 3.1 is used to assess incorrectness. the system returns a probability score, p c , which is the probability that the system thinks the given option is correct. an appropriate threshold, τ can then be selected, such if the probability score is less than the threshold, a distractor is deemed to satisfy the incorrectness requirement, as indicated in equation 1 (where @xmath0, correct} denotes the binary output decision). the selected threshold is a design choice of the test creator depending on how stringent the incorrectness criteria should be. for example, the incorrectness detector can be the first stage of a test creation pipeline to filter out the generated questions with multiple options that could be valid. the test creator may then select an operating point with low precision and high recall (in terms of incorrectness) in order to capture a larger pool of questions which should be considered in the subsequent stages of the evaluation pipeline. conversely, for high-stakes educational settings, an operating point which leads to higher precision at the cost of recall may be preferred. plausibility as emphasised by @xcite , good quality distractors should be both incorrect yet also plausible and not obviously invalid. unlike the binary incorrectness metric, plausibility is a continuous property, and distractors can be plausible to different degrees. for assessing the plausibility of a distractor within a distractor set, one can consider the model confidence of the multi-class multiple-choice machine reading comprehension system. the motivation for this approach is that a high confidence score is one that the mrc system finds more plausible, which one can assume would be similar for real candidates. we further define the plausibility score as the sum of confidence scores corresponding to each of the distractors in a question, which can similarly be calculated as the difference between 1 and the confidence score asso-ciated with the correct answer. this is expressed in equation 2, where p θ denotes the probability distribution learnt by the multi-class machine reading comprehension system. diversity as a human candidate, when attempting a multiple-choice question all four options are considered together. if distractors are similar or identical, then one can eliminate multiple options simultaneously using the same information, limiting the amount of comprehension that a question may require. therefore, it becomes increasingly important to ensure the distractors are diverse. particularly, diversity has been demonstrated to be a concern for automated question generation systems (raina and gales, 2022b), where systems are quite susceptible to frequently generating repeated distractors. this demands a need for automated approaches to determine the diversity amongst distractors to select, or at least be aware of, the distractor set with the maximum diversity. in this work, the bert equivalence metric (bem) @xcite is leveraged for assessing the diversity of the distractors. bem is a semantic similarity measure for question answering, where the equivalence score between an answer and the reference is returned. bem takes the text of a predicted answer, the text of the answer option and the question, concatenates them together and a bert system then returns a scalar score, 0 ≤ @xmath1. this score captures the equivalence between the candidate and the reference, where a score of @xmath2. bem is trained explicitly on an answer equivalence dataset and has been shown to out-perform zero-shot equivalence measures such as the bertscore from zhang et al.. in this work, bem is applied pair-wise to all possible pairs of distractors in a given question. the context is not concatenated to the question since initial experiments demonstrated that the long contexts diluted the differences between pairs of distractors. since bem is not order invariant, we average the output from bem with both orderings for the pair of distractors considered. the overall diversity is quoted as the 1 minus the average pairwise bem scores between the distractors, as indicated by equation 3 where the k distractors associated with a given question are denoted as {d 1 , d 2 , . . . , d k }. ## results in this section, we present results for assessing incorrectness, plausibility and diversity as part of the daf for standard multiple-choice reading comprehension datasets. it is observed that the incorrectness rate remains consistent across all the splits for race++. a similar consistency is evident on the cmcqrd dataset. in general, it can be seen that the plausibility scores tend to be higher for more challenging questions for both race++ and cmcqrd. this is potentially explainable by the fact that more challenging questions can expect to have a greater probability mass attributed to the distractors compared to the correct answer option. loosely, the average diversity score follows a similar pattern where more challenging questions can expect to have more diverse distractors. possibly a low diversity in the distractors offers fewer opportunities to distract the candidates. we have presented the incorrectness, plausibility and diversity scores on the race++ and cmc-qrd datasets. the subsequent sections aim to provide a form of verification for each of these metrics to demonstrate they are suitable for the respective qualities that they are assessing. ## conclusions this work proposes the distractor assessment framework, an automatic approach for assessing the quality of distractors on three key properties: incorrectness, plausibility and diversity. by leveraging multi-class and binary machine reading comprehension systems, and semantic similarity metrics, we propose intuitive methods for calculating automatic scores for the 3 properties. we validate the metrics by refining distractors with chatgpt. further there is a positive correlation indicated between candidate and system plausibilities.