Evaluating Neural Model Robustness for Machine Comprehension (2021)

Abstract: We evaluate neural model robustness to adversarial attacks using different types of linguistic unit perturbations – character and word, and propose a new method for strategic sentence-level perturbations. We experiment with different amounts of perturbations to examine model confidence and misclassification rate, and contrast model performance with different embeddings BERT and ELMo on two benchmark datasets SQuAD and TriviaQA. We demonstrate how to improve model performance during an adversarial attack by using ensembles. Finally, we analyze factors that effect model behavior under adversarial attack, and develop a new model to predict errors during attacks. Our novel findings reveal that (a) unlike BERT, models that use ELMo embeddings are more susceptible to adversarial attacks, (b) unlike word and paraphrase, character perturbations affect the model the most but are most easily compensated for by adversarial training, (c) word perturbations lead to more high-confidence misclassifications compared to sentence- and character-level perturbations, (d) the type of question and model answer length (the longer the answer the more likely it is to be incorrect) is the most predictive of model errors in adversarial setting, and (e) conclusions about model behavior are dataset-specific.

## introduction deep neural models have recently gained popularity, leading to significant improvements in many natural language understanding (nlu) tasks @xcite . however, the research community still lacks in-depth understanding of how these models work and what kind of linguistic information is actually captured by neural networks @xcite . evaluating model robustness to manipulated inputs and analyzing model behavior during adversarial attacks can provide deeper context: one of the most famous people born in warsaw was maria skłodowska-curie, who achieved international recognition for her research on radioactivity and was the first female recipient of the nobel prize. question: what was maria curie the first female recipient of? answer: nobel prize in this work, we evaluate neural model robustness on machine comprehension (mc), a task designed to measure a system's understanding of text. in this task, given a context paragraph and a question, the machine is tasked to provide an answer. we focus on span-based mc, where the model selects a single contiguous span of tokens in the context as the answer (tab. 1). we (1) quantitatively measure when and how the model is robust to manipulated inputs, when it generalizes well, and when it is less susceptible to adversarial attacks, (2) demonstrate that relying on ensemble models increases robustness, and (3) develop a new model to predict model errors during attacks. our novel contributions shed light on the following questions: broader implications we would like to stress the importance of this type of work to ensure diversity and progress for the computational linguistics community. we as a community know how to build new models for language understanding, but we do not fully understand how these models work. when we deploy these models in production, they fail to perform well in real-world conditions, and we fail to explain why they fail; the reason being we have not performed through evaluation of model performance under different experimental conditions. neural model evaluation and thorough error analysis, especially for tasks like machine comprehension, are critical to make progress in the field. we have to ensure our research community goes beyond f1 scores and incremental improvements and gains deeper understanding of models decision making processes to drive revolutionary research rather than evolutionary. ## background there is much recent work on adversarial nlp, surveyed in @xcite and @xcite . to situate our work, we review relevant research on the black-box adversarial setting, in which one does not have access or information about the model's internals, only the model's output and its confidence about the answer. 1 in an adversarial setting, the adversary seeks to mislead the model into producing an incorrect output by slightly tweaking the input. recent work has explored input perturbations at different linguistic levels: character, word, and sentence-level. for character-level perturbations, nlp systems generally do not take into account the visual characteristics of characters. researchers have explored the effects of adding noise by randomizing or swapping characters and examining its effect on machine translation (mt) @xcite , sentiment analysis and spam detection @xcite , and toxic content detection @xcite . @xcite replaced with similar looking symbols, and developed a system to replace characters with nearest neighbors in visual embedding space. for word-level perturbations, @xcite used a genetic algorithm to replace words with contextually similar words, evaluating on sentiment analysis and textual entailment. for sentence-level perturbations, @xcite generated adversarial paraphrases by controlling the syntax of sentences and evaluating on sentiment analysis and textual entailment tasks. @xcite found that augmenting the training data with paraphrases can improve performance on natural language inference, question answering, and mt. @xcite use adversarial paraphrases for dialog models. other related work includes @xcite ; @xcite , who generated natural looking adversarial examples for image classification, textual entailment, and mt. specifically for mc, @xcite added a distractor sentence to the end of the context, @xcite extracted sentence perturbation rules from paraphrases created by translating to and then from a foreign language and then manually judged for semantic equivalence, and @xcite focused on evaluating model robustness for mc. unlike earlier work, we empirically show how neural model performance degrades under multiple types of adversarial attacks by varying the amount of perturbation, the type of perturbation, model architecture and embedding type, and the dataset used for evaluation. moreover, our deep analysis examines factors that can explain neural model behavior under these different types of attacks. concurrent with the development of our paper, there has also been a slew of relevant work tackling robustness in neural nlp models, including adversarial robustness toolbox @xcite , advertorch @xcite , foolbox @xcite , advbox @xcite , openattack @xcite , teapot @xcite , textattack @xcite , textfooler @xcite , and robustness gym @xcite . ## methods we perform comprehensive model evaluation for machine comprehension over several dimensions: the amount of perturbation, perturbation type, model and embedding variation, and datasets. ## evaluation results fig. 1 summarizes our findings on how model behavior changes under noisy perturbations and adversarial attacks. here, we briefly discuss how perturbation type, perturbation amount, model, and embeddings affect model misclassification rate. in addition, we contrast model performance across datasets and report how to mitigate model error rate using ensembling. detailed analyses are presented in sec. 5. key findings are italicized. the effect of perturbation type to assess whether perturbations changed the meaning, we ran a human study on a random sample of 100 perturbed contexts from squad. we found (as expected) that the two annotators we employed could not distinguish char-perturbed text from the original. for word perturbations, the meaning of the context remained in 65% of cases, but annotators noted that sentences were often ungrammatical. for sentence-level perturbations, the meaning remained in 83% of cases. for a model trained on clean data, character perturbations affect the model the most, followed by word perturbations, then paraphrases. to a machine, a single character perturbation results in a completely different word; handling this type of noise is important for a machine seeking to beat human performance. word perturbations are context independent and can make the sentence ungrammatical. 6 nevertheless, the context's meaning generally remains coherent. paraphrase perturbations are most ideal because they retain meaning while allowing more drastic phrase and sentence structure modifications. in sec. 4.2, we present a more successful adversarially targeted paraphrasing approach. the effect of perturbation amount perturbed training data improves the model's performance for character perturbations (1 st column of fig. 1a ), likely due to the models' ability to handle unseen words: bidaf with elmo utilizes character embeddings, while bert uses word pieces. our results corroborate @xcite 's findings (though on a different task) that without adversarial training, models perform poorly on perturbed test data, but when models are trained on perturbed data, the amount of perturbed training data does not make much difference. we do not see statistically significant results for word and paraphrase perturbations (2 nd and 3 rd columns in each heatmap in fig. 1 ). we conclude that perturbing 25% of the words and the non-strategic paraphrasing approach were not aggressive enough. the effect of model and embedding as shown in fig. 1a and b, the bert model had less errors than the elmo-based model regardless of the perturbation type and amount on squad data. while train test model answer none char here" half char orientalism full char orientalism none word orientalism half word behaviourism identities full word the discourse of orientalism none char orientalism half char . . . the east as a negative full char orientalism the two models are not directly comparable, our results indicate that the bert model is more robust to adversarial attacks compared to elmo. the effect of the data holding the model constant (fig. 1b and c ), experiments on triviaqa resulted in more errors than squad regardless of perturbation amount and type, indicating that triv-iaqa may be a harder dataset for mc and may contain data bias, discussed below. ## robustness analysis here, we do a deeper dive into why models make errors with noisy input. we investigate data charac-teristics and their association with model errors by utilizing crosscheck @xcite , a novel interactive tool designed for neural model evaluation. unlike several recently developed tools for analyzing nlp model errors @xcite and understanding ml model outputs @xcite @xcite , crosscheck is designed to allow rapid prototyping and cross-model comparison to support experimentation. foot_8 ## conclusion and future work our in-depth analysis of neural model robustness sheds light on how and why mc models make errors in adversarial training, and through our error prediction model, we discovered features of the data e.g., question types that are strongly predictive of when a model makes errors during adversarial attacks with noisy inputs. our results on evaluating the effect of the data e.g., questions and context length will not only explain model performance in context of the data, but will also allow to build future neural models more resilient to adversarial attacks and advance understanding of neural model behavior across a variety of nlu tasks and datasets and its strengths and weaknesses. for future work, we see many avenues for extension. we plan to experiment with more aggressive and more natural perturbations, and deeper counterfactual evaluation @xcite . while recent research has made great strides in increasing model performance on various nlp tasks, it is still not clear what linguistic patterns these neural models are learning, or whether they are learning language at all @xcite . more broadly, as ai becomes more entrenched in our lives, ai models need to be held to higher standards including but not limited to accountability (e.g. wang et al., 2018, 2019, genie 10 ), fairness (e.g. saleiro et al., 2018; bellamy et al., 2018; bird et al., 2020; ahn and lin, 2020, 11 ; 12 ; 13 ), and transparency (e.g. @xcite @xcite @xcite .