Using Adversarial Attacks to Reveal the Statistical Bias in Machine Reading Comprehension Models (2021)

Abstract: Pre-trained language models have achieved human-level performance on many Machine Reading Comprehension (MRC) tasks, but it remains unclear whether these models truly understand language or answer questions by exploiting statistical biases in datasets. Here, we demonstrate a simple yet effective method to attack MRC models and reveal the statistical biases in these models. We apply the method to the RACE dataset, for which the answer to each MRC question is selected from 4 options. It is found that several pre-trained language models, including BERT, ALBERT, and RoBERTa, show consistent preference to some options, even when these options are irrelevant to the question. When interfered by these irrelevant options, the performance of MRC models can be reduced from human-level performance to the chance-level performance. Human readers, however, are not clearly affected by these irrelevant options. Finally, we propose an augmented training method that can greatly reduce models’ statistical biases.

## introduction reading comprehension tasks are useful to quantify language ability of both humans and machines @xcite @xcite . deep neural network (dnn) models have achieved high performance on many mrc tasks, but these models are not easily explainable @xcite . it is also shown that dnn models are often sensitive to adversarial attacks @xcite @xcite . furthermore, it has been shown dnn models can solve mrc tasks with relatively high accuracy when crucial information is removed so that the tasks are no longer solvable by humans @xcite @xcite . all such evidence suggests that the high accuracy dnn models achieve on mrc tasks does not solely rely on these models' language comprehension ability. at least to some extent, the high accuracy reflects exploitation of statistical biases in the datasets @xcite @xcite . here, we propose a new model-independent method to evaluate to what extent models solve mrc tasks by exploiting statistical biases in the dataset. as a case study, we only focus on the classic race dataset @xcite , which requires mrc models to answer multiple-choice reading comprehension questions based on a passage. the advantage of multiple-choice questions is that its performance can be objectively evaluated. at the same time, it does not require the answer to be within the passage, allowing to test, e.g., the summarization or inference ability of models. nevertheless, since models are trained to select the right option from 4 options, which are designed by humans and may contain statistical biases, models may learn statistical properties of the right option. consequently, models may tend to select options with these statistical properties similar to the properties of the right option without referring to the passage and question. our method is designed to reveal this kind of statistical bias. the logic of our method is straightforward: for each multiple-choice question, we gather a large number of options that are irrelevant to the question and passage. we ask the model to score how likely each irrelevant option is the right option. if a model is biased, it may always assign higher scores to some irrelevant options than others, even if all the options are irrelevant. if a model is so severely biased, which turns out to be true for all models tested here, it may assign higher scores to some irrelevant options than the true answer and select the irrelevant option as the answer. here, the irrelevant options that are often selected as the answer are referred to as magnet options. ## dataset and pre-trained models we used race dataset in our experiment @xcite , which is a large-scale reading comprehension data set covering more than 28,000 passages and nearly 100,000 questions. the task was to answer multi-choice questions based on a passage. specifically, each question contained a triplet (p i , q i , o i ), where p i denoted a passage, q i denoted a question, and o i denoted a candidate set of 4 options, i.e., o @xmath0,1 , o i,2 , o i,3 , o i,4 }. only one option was the correct answer, and the accuracy was evaluated by the percent of questions being correctly answered. we tested 3 pre-trained language models, i.e., bert @xcite , roberta @xcite , and albert @xcite . for each model, we separately tested the base version and large version. we built our models based on pretrained transformer models in the huggingface @xcite . we fine-tuned pre-trained models based on the race dataset and the parameters we used for fine-tuning were shown in appendix a.1. the passage, question, and an option were concatenated as the input to models, i.e., [cls, p i , sep , q i , o i,j , sep ]. the 4 options were separately encoded. the concatenated sequence was encoded through the models and the output embedding of cls was denoted as c i,j . we used a linear transformation to convert vector c i,j into a scalar s(o i,j ), i.e., s(o i,j ) = w c i,j . the scalar s(o i,j ) was referred to as the score of the option o i,j . a score was calculated for each option, and the answer to a question was determined as the option with the highest score, i.e., argmax j s(o i,j ). ## results and analyses 4.1 experiments setup to screen potential magnet options, we constructed a large set of irrelevant options, i.e., o a , by randomly selecting 300 passages from the race test set, which were associated with 1064 questions. furthermore, to test whether options in the training set can cause stronger interference, we also randomly selected 300 passages from the race training set, which had 1029 questions. the options from the test and training set were pooled to create o a , which had 8372 options in total. for such a large number of irrelevant options, it was computationally challenging to evaluate the interference score of each option based on each question in the race test set. therefore, as a screening procedure, we first randomly selected 100 passages from the race test set, which have a total of 346 questions. the interference score for each of the 8372 irrelevant options was evaluated based on the 346 questions. after potential magnet options were determined by the screening procedure, the interference score of magnet options were further evaluated using all questions in race test set. for race test set, the accuracy of the models ranged between about 0.6 and 0.85, with roberta-large achieving the highest performance ( ## related work our attack strategy distinguishes from previous work in two ways. first, unlike, e.g., gradient-based methods @xcite , our method does not require any knowledge about the structure of dnn models. second, some methods manipulate the passage in a passagedependent way @xcite @xcite , while our method manipulate the options in a passage-independent way. furthermore, we proposed a strategy to train more robust models that are insensitive to our attack. here, we restricted our discussion to race, but our method is applicable to other tasks in which the answer is selected from a limited set of options. for example, for span extraction tasks, such as squad, the method will insert a large number of irrelevant phrases into the passage and analyze which phrases are often selected as the answer. in this way, our method is similar to the trigger-based attack methods @xcite , but the difference is that our method test whether the inserted irrelevant phrase is selected as the answer while the trigger-based methods test whether the content following the trigger phrase is selected. ## conclusion in summary, we propose a new method to evaluate the statistical biases in mrc models. it is found that current mrc models have strong statistical biases, and are therefore sensitive to adversarial attack. when attacked using the method proposed here, model performance can drop from humanlevel performance to chance-level performance. to alleviate sensitivity to such attacks, we provided an augmented training procedure that effectively enhances the robustness of models. 2 were separately shown in figure 6 . it denoted that irrelevant options from the training and test sets had similar interference score. only in bert-large and albert-large models, the interference scores of the irrelevant options from the training set were higher than those