What do we expect from Multiple-choice QA Systems? (2020)

Abstract: The recent success of machine learning systems on various QA datasets could be interpreted as a significant improvement in models’ language understanding abilities. However, using various perturbations, multiple recent works have shown that good performance on a dataset might not indicate performance that correlates well with human’s expectations from models that “understand” language. In this work we consider a top performing model on several Multiple Choice Question Answering (MCQA) datasets, and evaluate it against a set of expectations one might have from such a model, using a series of zero-information perturbations of the model’s inputs. Our results show that the model clearly falls short of our expectations, and motivates a modified training approach that forces the model to better attend to the inputs. We show that the new training paradigm leads to a model that performs on par with the original model while better satisfying our expectations.

## introduction question answering (qa) has been a prevalent format for gauging advances in language understanding. recent advances in contextual language modelling have led to impressive results on multiple nlp tasks, including on several multiple choice question answering (mcqa, depicted in fig. 1 ) datasets, a particularly interesting qa task that provides a flexible space of candidate answers along with a simple evaluation. however, recent work @xcite @xcite has questioned the interpretation of these qa successes as progress in natural language understanding. indeed, they exhibit, in various task 1 resources for this work are available at: http://cogcomp.org/page/publication_view/913 figure 1 : an example from arc easy dataset @xcite showing the three mcqa task inputs. we formulate the following set of (nonexhaustive) expectation principles that a mcqa model should satisfy. ## multi-choice question answering in this section, we briefly describe the multiplechoice question answering (mcqa) task, and the model and datasets we use in this work. 2 mcqa task in a k-way mcqa task, a model is provided with a question q, a set of candidate options @xmath0, . . . , o k }, and a supporting context for each option @xmath1, . . . , c k }. the model needs to predict the correct answer option that is best supported by the given contexts. figure 1 shows an example of a 4-way mcqa task. ## model vs. our expectations in this section, we define the perturbations we design to evaluate a model against our expectation principles (defined in §1). we then analyze how well the baseline model satisfies these expectations. ## proposed training approach to address the aforementioned limitations, and reduce the tendency of the model to exploit dataset artifacts, we propose the following modifications to the training methodology. ## results the performance of our proposed training approach (+ our training) competitively (within 2.6 points) with the baseline on all three datasets suggesting that our proposed training approach only has minor impact on the overall model performance. in our pio setting, the new model outperforms the baseline on all three datasets by a large margin (55.5 compared to baseline's 25.4 on aristo indicating an improvement over the baseline with regard to our monotonicity expectation. even though the data augmentation did not augment examples with this perturbation, our training approach helps the model better read the inputs and avoid distractor options. when evaluating over unreasonable inputs in the no and nq settings, the resulting model performs poorly compared to the baseline (13.7 vs. 50.2 and 12.3 vs. 34.3 on qasc), showing that our training approach helps the model to not rely on dataset bias and satisfy the sanity expectation. finally, the new model performs poorly when we remove the contexts (e.g 20.6 on race), indicating how it is able to meet our reading expectations. the results also show the resulting model's reliance on the context for information required to correctly answer questions. moreover, it implies that the resulting model is able to achieve performance similar to the baseline by heavily relying on information from the contexts, as opposed to the baseline that exploits dataset artifacts (as previously shown). results showing the performance of the model trained using binary classification loss, without the data augmentation, are attached in the appendix. ## related work our work builds on numerous recent works that challenge the robustness of neural language models @xcite or, more generally, neural models @xcite @xcite . our evaluation settings -hiding one of the three inputs to the mcqa models -are similar to kaushik and lipton 2018's partial input settings which were designed to point out the existence of dataset artifacts in reading comprehension datasets. however, we argue that our results additionally point to a need for more robust training methodologies and propose an improved training approach. our data augmentation approach builds on recent works @xcite @xcite @xcite ) that try to leverage augmenting training data to improve the performance and/or robustness of models. however most of these works are semi-automatic or require human annotation while our augmentation approach requires no additional annotation. ## conclusion we formulated three expectation principles that a mcqa model must satisfy, and devised appropriate settings to evaluate a model against these principles. our evaluations on a roberta-based model showed that the model fails to satisfy any of our expectations, and exposed its brittleness and reliance on dataset artifacts. to improve learning, we proposed a modified training objective to reduce the model's sensitivity to the relative difficulty of candidate options, and an unsupervised data augmentation technique to encourage the model to rely on all the input components of a mcqa problem. the evaluation of our proposed training approach showed that the resulting model performs competitively with the original model while being robust to perturbations; hence, closer to satisfying our expectation principles.