None of the Above, Less of the Right Parallel Patterns in Human and LLM Performance on Multi-Choice Questions Answering (2025)

Abstract: Multiple-choice exam questions with “None of the above” (NA) options have been extensively studied in educational testing, in which existing research suggests that they better assess true knowledge. However, their impact on Large Language Models (LLMs) evaluation remains underexplored. Through systematic experiments with 28 LLMs on the MMLU benchmark, we examine how NA options affect model performance and confidence calibration. Our analysis reveals that NA options, when used as the correct answer, lead to a consistent 30-50% performance drop across models regardless of scale–suggesting that LLMs lack the meta-cognitive ability to systematically evaluate and reject all given options when none are correct. This degradation shows strong domain dependence, with minimal impact on mathematical reasoning (14.6% drop) but severe effects on tasks requiring uncertainty handling like business ethics (48.1% drop). Our results highlight important implications for benchmark design and raise questions about LLMs’ ability to handle uncertainty in real-world applications.

## introduction multiple-choice question answering (mcqa) benchmarks-such as mmlu @xcite and mmlu-pro @xcite -have become a cornerstone for evaluating large language models (llms) by measuring their domain-specific knowledge and reasoning capabilities. originally designed for human educational assessments, these benchmarks adhere to well-established guidelines for item construction and distractor design (e.g. @xcite ). yet a critical gap persists: guidelines developed for human test situations, which can enhance both reliability and discrimination @xcite , are rarely scrutinized in the context of llm evaluation. a longstanding debate in educational measurement concerns the use of "none of the above" (na) as an answer option. research by @xcite and @xcite shows that including na as the correct answer tends to increase question difficulty, which is often reflected by lower average student exam scores (0.614 to 0.418 in dibattista et al. (2014) ), by prompting them to rely on elimination strategies when uncertain. conversely, under refined experimental conditions, @xcite found that na options can enhance both difficulty and discrimination. a parallel phenomenon is observed in eyewitness identification research: as demonstrated by @xcite , witnesses are prone to erroneously selecting an option even when the correct response should be to abstain from identification. this bias towards action hints at potential pitfalls when na is used in tests designed to evaluate ability rather than guessing behavior. ## background and education assessment principle educational assessment guidelines by @xcite and @xcite establish best practices for designing multiple-choice question alternatives (mcqas), emphasizing clarity in stems, plausibility of distractors, and alignment with learning objectives. among their recommendations, the inclusion of "none of the above" (na) and "all of the above" as answer choices remains controversial. studies suggest na introduces unique psychometric effects: when na is the correct answer, question difficulty increases (higher p-values) but discriminative power decreases. this occurs because students with knowledge deficiency (i.e., incomplete understanding) may strategically guess na by eliminating other options @xcite , rather than demonstrating positive knowledge. for example, @xcite found that the kr-20 values were .828 for non-na items and .865 for na items (with half serving as answers and half as distractors). they also reported discrimination index scores of 0.584 and 0.581, respectively, and noted that test reliability is generally unaffected by this change. a detailed explanation of kr-20 and discrimination index metrics is introduced in section 4. ## metrics for question quality assessment item quality in educational testing is evaluated using two standard metrics: the discrimination index and the kuder-richardson formula 20 (kr-20) reliability coefficient. in the context of educational assessment, reliability refers to the extent to which a test consistently measures the underlying construct of interest. specifically, a test's reliability is determined by the uniformity and precision of its items in capturing the intended concept, rather than by the characteristics of the llm. we adopt these measures both to assess our modified mmlu questions and to benchmark llm performance. discrimination index: this discriminative metric measures how effectively a question differentiates between high-performing and low-performing testtakers @xcite . it is calculated as: kr-20 reliability coefficient: kr-20 coefficient @xcite quantifies the internal consistency of the test, given binary outcomes (correct/incorrect). the kr-20 is defined as: ## experiments all experiments are conducted using 0-shot chainof-thought prompting @xcite , can be found in appendix e. in the following sections, we describe our evaluation of 28 llms ranging from 1.5b to 671b for 19 open weights models, 9 closed weights models. all models are evaluated under multiple settings along with additional analyses on test quality, confidence, and fine-tuning. ## conclusion in this study, we examined the performance of large language models on mcqa benchmarks when "none of the above" (na) is applied to both answer and distractor choice. our findings reveal a dramatic performance drop-from approximately 63.2% under standard conditions down to 28.5% when the correct answers are replaced by nas-highlighting a fundamental limitation in the models' ability to reject invalid options. while our informed fine-tuning strategy managed to improve na accuracy to 57.7%, a significant gap remains compared to the standard accuracy. these results underscore the need to rethink mcqa benchmarks for llms, recognizing that tasks designed for human evaluation may not directly translate to machine understanding and uncertainty handling.