“World Knowledge” in Multiple Choice Reading Comprehension (2023)

Abstract: Recently it has been shown that without any access to the contextual passage, multiple choice reading comprehension (MCRC) systems are able to answer questions significantly better than random on average. These systems use their accumulated “world knowledge” to directly answer questions, rather than using information from the passage. This paper examines the possibility of exploiting this observation as a tool for test designers to ensure that the form of “world knowledge” is acceptable for a particular set of questions. We propose information-theory based metrics that enable the level of “world knowledge” exploited by systems to be assessed. Two metrics are described: the expected number of options, which measures whether a passage-free system can identify the answer a question using world knowledge; and the contextual mutual information, which measures the importance of context for a given question. We demonstrate that questions with low expected number of options, and hence answerable by the shortcut system, are often similarly answerable by humans without context. This highlights that the general knowledge ‘shortcuts’ could be equally used by exam candidates, and that our proposed metrics may be helpful for future test designers to monitor the quality of questions.

## introduction reading comprehension (rc) exams are used extensively in a wide range of language competency examinations @xcite , and have become a ubiquitous assessment method to probe how well candidates can read a passage and understand the text's core meaning. a fundamental assumption of rc exams is that to answer any of the questions correctly, one has to read the passage, comprehend * equal contribution figure 1 : output probabilities of our model (trained with contexts omitted) on real race++ @xcite examples. 'effective number of options' is a metric that captures the model's confidence. in this paper we analyse this phenomena and for several standard mcmrc datasets @xcite @xcite verify that passage-free baselines are able to achieve performance significantly better than random. we show that a subset of questions can be answered accurately and confidently without access to the contextual passage, where further analysis shows this is partly due to the presence of low-quality distractors, i.e. options that can be eliminated using only the question. as an example, given the question "mina's sister's name is:", one can eliminate any options that use a traditionally male name (see figure 1 ). this highlights a potential 'shortcut' candidates could use to answer questions while bypassing the context. our work raises awareness to this potential flaw, and proposes a simple solution to catch questions that can be answered without comprehension. the proposed metrics might be a useful tool for future multiple-choice rc test designers to ensure that all questions truly assess ## multiple choice reading comprehension multiple-choice reading comprehension is a popular task where given a context passage c and question q, the correct answer must be deduced from a set of answer options {o}. current sota mrc systems are dominated by pre-trained language models (prlms) based on the transformer encoder architecture @xcite @xcite . model architecture our question-answering system follows the standard mcmrc architecture of figure 2 @xcite . each option is individually encoded along with the question and the context into a score, and a softmax layer converts the 4 options' scores into a probability distribution. at inference, the predicted answer is the option with the greatest probability. for well designed questions, one would expect systems with missing information (i.e. the 'shortcut' models) to have no information of what the answer is. this would correspond to a uniform distribution output (the distribution of maximum entropy), with an effective number of options equal to the total number of answer options. however, if the effective number of options is significantly lower than the total number of answer options, then this implies that prior information stored during training can be used to answer the question, without comprehension. ## conclusions for popular mcmrc datasets, systems can achieve reasonably high performance without performing any comprehension. without passage information, 'shortcut' systems can confidently determine some correct answer options, eliminate some unlikely distractors, and use general knowledge to gain information. rather than focusing on average system performance, our work analyses individual question's reliance on world knowledge. we propose a metric based on the shortcut systems to automatically flag questions that are answerable without comprehension. we further provide evidence that the flagged questions are answerable by humans without any context. lastly, using an approximation of the mutual information, we show that the importance of context varies over the questions in the dataset, and reason that high mi questions can be thought of as candidates for high-quality questions that truly measure comprehension abilities.