Answer Uncertainty and Unanswerability in Multiple-Choice Machine Reading Comprehension (2022)

Abstract: Machine reading comprehension (MRC) has drawn a lot of attention as an approach for assessing the ability of systems to understand natural language. Usually systems focus on selecting the correct answer to a question given a contextual paragraph. However, for many applications of multiple-choice MRC systems there are two additional considerations. For multiple-choice exams there is often a negative marking scheme; there is a penalty for an incorrect answer. In terms of an MRC system this means that the system is required to have an idea of the uncertainty in the predicted answer. The second consideration is that many multiple-choice questions have the option of none-of-the-above (NOA) indicating that none of the answers is applicable, rather than there always being the correct answer in the list of choices. This paper investigates both of these issues by making use of predictive uncertainty. Whether the system should propose an answer is a direct application of answer uncertainty. There are two possibilities when considering the NOA option. The simplest is to explicitly build a system on data that includes this option. Alternatively uncertainty can be applied to detect whether the other options include the correct answer. If the system is not sufficiently confident it will select NOA. As there is no standard corpus available to investigate these topics, the ReClor corpus is modified by removing the correct answer from a subset of possible answers. A high-performance MRC system is used to evaluate whether answer uncertainty can be applied in these situations. It is shown that uncertainty does allow questions that the system is not confident about to be detected. Additionally it is shown that uncertainty outperforms a system explicitly built with an NOA option.

## introduction machine reading comprehension (mrc), where the correct answer must be deduced for a question from a context paragraph, plays a crucial role in developing systems for natural language processing and understanding. in recent years, popular mrc datasets @xcite @xcite @xcite @xcite have consistently observed increasingly competitive systems topping public leaderboards @xcite @xcite @xcite and surpassing human performance. however, systems in deployment should not necessarily always aim to answer a posed reading comprehension question. there are two modes of interest in which an mrc system may choose to abstain from giving an answer: answer uncertainty and unanswerability. if a system is uncertain about its prediction, it is likely that the predicted answer will be incorrect. in particular, negative marking schemes, which are shown to improve the reliability of multiple-choice assessment as guessing is deterred @xcite , penalise a system for predicting an incorrect answer while abstaining carries no penalty, and of course the correct answer has a positive reward. in such cases, it would be sensible for a system to abstain from answering if there is answer uncertainty in the prediction. unanswerability is where the answer to a question is not deducible from the associated context. consequently, a system should abstain from answering a question if it believes the answer is not present in the context. answer uncertainty is when the system is unsure about its prediction while unanswerability is where the system (confidently) believes the question cannot be answered. a fair amount of work has investigated the challenge of tackling unanswerability in span-based reading comprehension @xcite with the hope of encouraging systems to truly understand the comprehension task beyond simple word matching with remarkable success @xcite @xcite . however, limited work has been completed with regard to unanswerability for multiple-choice reading comprehension datasets, where most work focuses on developing state-of-the-art systems on the default task such as @xcite ; @xcite . this work investigates both answer uncertainty and unanswerability in multiple-choice mrc. one challenge for this problem is that unanswerable examples are often not available at training time, and the possible range of incorrect answers even to valid questions is vast. uncertainty measures have been demonstrated to be effective at out-of-distribution detection across a wide range of tasks @xcite @xcite . this work studies the potential viability of using uncertainty measures at test time to identify examples for which the system should abstain for both settings of answer uncertainty for optimising performance with a negative marking scheme and handling unanswerability. the transformer models are usually trained with pairs of sentences separated by the [sep] token. the context is used as the first sentence and the question concatenated with an option is used as the second sentence. the construct is repeated for each of the four options. these four pairs of sentences are passed in parallel to the transformer encoder architecture where the weights are shared for each of the inputs. the hidden state embedding associated with the [cls] token is passed to a final linear head (with a non-linear activation) at the end of the transformer encoder that calculates output scores for each answer option which is then converted to a discrete probability distribution over the four answer options using the softmax activation. typically, at test time, the predicted answer option is the one with the greatest probability mass. the work in this paper focuses on reclor (a reading comprehension dataset requiring logical reasoning) introduced by @xcite that encourages the development of mrc systems beyond a superficial understanding of the context as the dataset was designed to focus on more challenging logical reasoning questions compared to previous multiple-choice datasets including dream @xcite , mctest @xcite , arc @xcite and race @xcite . results are presented on race for comparison against reclor. additional numbers are provided on cosmosqa @xcite in the appendix a.3. the architecture of figure 1 based on the baseline systems introduced by @xcite is used for simplicity as the focus here is on answer uncertainty and unanswerability. the selected model in this paper deviates from the baseline systems as electra is specifically selected as the prlm given that it has been proven to achieve state-ofthe-art results in other forms of mrc @xcite whilst also being smaller than equivalently competitive albert @xcite systems. ## uncertainty research in uncertainty estimation is popular in recent years with model averaging @xcite @xcite as the standard approach. in particular, ensemblebased and sampling-based uncertainty estimates have demonstrated effectiveness for both identifying misclassifications and out-of-distribution inputs @xcite . this work focuses on ensemble-based approaches for multiple-choice mrc as ensembles consistently outperform single models @xcite and offer interpretable uncertainty estimates. for multi-class classification, various measures of predictive uncertainty can be calculated using the predicted probability distributions over the classes from each of the ensemble members. measures of knowledge uncertainty include mutual information, expected pair-wise kl divergence, and reverse mutual information; measure of data uncertainty is the average of the entropy of each predicted distribution (expected entropy); while measures of total uncertainty include (negated) confidence and entropy of the average prediction @xcite . we present results using the expected entropy as the uncertainty measure for abstaining to answer for both a measure of answer uncertainty in a negative marking scheme and a measure of unanswerability when a system does not encounter unanswerable examples at training time foot_1 . formally, expected entropy, e[h], for a given input is defined as: ## data and experimental set-up all experiments are based upon the reclor and race datasets @xcite or their variants. this section discusses how the default datasets are modified to perform experiments for answer uncertainty and unanswerability as well as performance criteria. ## results and discussion this section discusses the main findings of how the electra system fares against existing systems on the reclor dataset and the role of uncertainty measures in using answer uncertainty for tackling negative marking schemes or detecting unanswerable examples for reclor and race. expected entropy is the chosen uncertainty measure. see the appendix for other uncertainty measures' results. ## conclusion this paper addresses answer uncertainty and unanswerability in multiple-choice mrc. measures of answer uncertainty are required to identify examples that the system may struggle to get correct and hence should abstain from answering such questions. unanswerability detection is required for when the answer cannot be deduced using the information provided. an electra prlm achieve competitive results on the default reclor dataset, achieving up to 67.1% accuracy on the evaluation split. ensemble-based predictive uncertainty measures are explored for both modes of operation: answer uncertainty for negative marking schemes and the presence of unanswerability. it is shown that uncertainty in the prediction such as expected entropy is correlated with the error rate of the mrc system allowing better than vanilla performance with an aggressive negative marking scheme for reclor and race. interestingly, it is found that expected entropy from the predictions of an implicitly trained system is competitive at unanswerability detection and is able to out-compete map decoding from an explicitly trained system that has been trained with unanswerable examples for reclor.