longhorns at DADC 2022: How many linguists does it take to fool a Question Answering model? A systematic approach to adversarial attacks. (2022)

Abstract: Developing methods to adversarially challenge NLP systems is a promising avenue for improving both model performance and interpretability. Here, we describe the approach of the team “longhorns” on Task 1 of the First Workshop on Dynamic Adversarial Data Collection (DADC), which asked teams to manually fool a model on an Extractive Question Answering task. Our team finished first (pending validation), with a model error rate of 62%. We advocate for a systematic, linguistically informed approach to formulating adversarial questions, and we describe the results of our pilot experiments, as well as our official submission.

## introduction rapid progress in nlp has resulted in systems obtaining apparently super-human performance on popular benchmarks such as glue @xcite , squad @xcite , and snli @xcite . dynabench @xcite proposes an alternative approach to benchmarking: a dynamic benchmark wherein a human adversary creates examples that can "fool" a stateof-the-art model but not a human language user. the idea is that, by generating and compiling examples that fool a particular system, the community can gain a better idea of that system's actual strengths and weaknesses, as well as ideas and data for iteratively improving it. there is no straightforward recipe, however, for generating successful adversarial examples. to contribute to that knowledge base, this paper describes the strategy used by team "longhorns" in task 1 of the first workshop on dynamic adversarial data collection (dadc), which was on extractive question answering (answering a question about a passage by pointing to a particular span of text within that passage). 2 we focus not only on describing the details of our strategy, but also on our process for approaching the task. at the time of this paper submission, pending expert validation of the results, our team ranked first in the competition, obtaining 62% model error rate (mer). our approach towards creating adversarial examples was designed to be systematic, analytical, and draw on linguistically informed ideas. we first compiled a list of linguistically inspired "attack strategies" and used it to create adversarial examples in a systematic manner. we then analyzed some existing biases of the model-in-the-loop and its performance on a variety of different attacks. we used this piloting phase to select the best performing attacks for the official submission. based on the approaches that were most successful both in our pilot studies and in our official submission, we posit that the following broad areas should be of particular interest for theoretically motivated adversarial attacks on contemporary nlp systems, as evidenced by their strong performance on our target task: the model was proficient at identifying the correct kind of named entity being asked for (e.g., a person for a "who" question, a place for a "where" question), but was biased to give answers which were salient (either topically or because they appeared first @xcite )) or which had high lexical overlap with the question. thus, picking a distractor with the same entity type as the target answer (e.g., another person mentioned in the text when the question was a "who" question) was often effective. this result is broadly consistent with observations that modern nlp systems can perform well in the general case but can be biased towards frequency-based priors (e.g., @xcite ) that mean they are sometimes "right for the wrong reasons" @xcite . psycholinguists who study human language processing often study constructions that are grammatical but difficult for humans to process in real time, such as garden path sentences @xcite and complex coreference resolution @xcite . we found that the model was indeed often fooled by questions that included these types of constructions. while we did not collect any human data, the sentences that fooled the model are likely to be hard for humans as measured by tests of real-time processing difficulty (e.g., eye tracking, selfpaced reading), even though humans would be able to successfully process these sentences given enough time. because these strategies and this general approach are broad and theoretically motivated, we believe that our methods could be used to generate adversarial examples on other natural language understanding tasks besides question answering. in what follows, we characterize our approach in both the pilot phase and official submission, provide our list of attack strategies, and discuss the limitations of the task and model. ## task definition in task 1 of dadc, titled "better annotators", each participating team submits 100 "official" extractive question answering (qa) examples through the dynabench platform. the organizers of the shared task provide short passages as context and the participants have to create questions that "can be correctly answered from a span in the passage and do not require a yes or no answer". the objective is to find as many model-fooling examples as possible -the winning team is the one with the highest validated model error rate (vmer) 3 . the competition uses dynabench @xcite : "an open-source platform for dynamic dataset creation and model benchmarking". dynabench aims to facilitate human-and-model-in-theloop dataset creation. the annotators' aim to generate examples that will be misclassified by an automated model, but can be answered correctly by competent human speakers. dynabench has been used to create data for question answering @xcite , natural language inference @xcite , online hate detection @xcite , and sentiment analysis @xcite , among others. ## approach our team consisted of faculty, postdocs, and students from the ut austin linguistics department, computer science department, information school, and electrical and computer engineering department. we approached the problem of creating adversarial attacks in a systematic manner, informed by ideas from computational linguistics, psycholinguistics, and theoretical linguistics. we composed a list of linguistic phenomena and reasoning capabilities that we hypothesized would be difficult for a state-of-the-art qa model. we then used some of those phenomena to create our official submission of adversarial examples. while the list is not exhaustive, it covers a wide range of potential attack strategies and can be used to guide the creation of adversarial examples for other tasks and systems. separate from our official submission for the competition, we ran a series of pilot experiments in which we used the list as a guide for experimenting with a variety of strategies for creating adversarial example. to ensure a fair and competitive official submission, all pilot experiments were carried out either before the official start of the shared task or after the official submission was made. our objective when evaluating the different adversarial strategies was to explore the space of potential attack strategies to determine the most successful ones for fooling the model. for each strategy, we measured the model error rate (mer) on a small sample of example, and we also analyzed how frequently the attack can be used. based on the results of these pilot experiments, we targeted the best strategies for our official submission. official question submissions were made by subsets of the team, in group sizes ranging from 1 to around 10. since not all strategies can be used for all example passages, we used the specific passages we were presented with in order to guide our decision about what strategy to focus on for a particular question. when more than one participant was present, question submissions were made by consensus agreement among those present. anecdotally, we found that the attacks were often more successful when multiple team members are present, with each member hypothesizing model behaviors from diverse angles. overall, we found the adversarial question generation process nontrivial, taking 5-10 minutes per passage, although we became faster over time. we also chose to skip passages occasionally when the passage covers very well-known entity, is too simple, or is not promising to most of our strategies (not having distractor entities, etc). we generated multiple questions for promising passages. ## pilot experiments: evaluating adversarial strategies many of our "adversarial strategies" are inspired by prior work in adversarial data generation and unit testing for question answering, natural language inference, and paraphrase identification @xcite @xcite @xcite @xcite @xcite @xcite @xcite . we propose the following linguistic and reasoning phenomena as a source for potential adversarial attacks: lexical knowledge examples that require understanding lexical properties and in partiuclar lexical entailments that require knowledge of hypernyms/hyponyms (e.g., knowing that @xmath0, but @xmath1, named entities and their properties (e.g., knowing shakira is a singer), nominalization (e.g., knowing "a submission" implies something has been submitted), (a)symmetrical relations (e.g., knowing that john marrying mary implies mary marrying john, but john loving mary does not imply mary loving john), polarity substitutions (e.g., knowing that a movie is good implies that it is not bad), converse substitution (e.g., knowing that if something has been provided, it has been received), comparisons with antonyms (e.g., knowing that if clara is the tallest, she is not shorter than mary), reasoning about modal verbs (e.g., understanding that if something could happen, that does not mean it did happen), and reasoning about quantifers (e.g., knowing that if some swans are white, that does not imply all swans are white). ## exploring model biases during the pilot experiment step in section 4 we found that the model-in-the-loop performs surprisingly well on a variety of different attacks. we hypothesized that at least in some situations, the strong performance is due to spurious correlations, the nature of the underlying language model, and the nature of the task. we further carried out a set of experiments to determine some specifics of the model behavior. we briefly discuss two "shortcuts" used by the model. semantic similarity figure 1 illustrates the model bias towards "semantic similarity" on a nonsensical question. when the model is unsure what to do, or like in this example, when the question is not a valid english sentence, it identifies parts of the context that are similar to the question and predicts neighboring words. due to the relatively short length of most of the passages, this strategy often gets the correct answer without actually understanding the question. ## official submission after analyzing and discussing the results of our preliminary experiments, for our official submission we focus on the following strategies: using distractors, numerical reasoning, temporal reasoning, garden path questions, complex coreference, list manipulations, and common-sense reasoning. we also used the model biases to either confuse the model or reduce it's ability to rely on heuristics. in the rest of this section we briefly describe each of our strategies and provide examples. ## discussion a fundamental feature of language is that it is a cooperative enterprise @xcite that enables efficient communication between parties @xcite . therefore, in ordinary language, people typically talk about discourse-relevant entities @xcite , avoid difficult syntactic constructions @xcite , and structure information in a way that is easy to produce and understand @xcite . if anything unifies all of our most successful attack strategies, it is that they eschew these principles in the context of the given task and passages. instead, successful attacks ask about surprising aspects of the text (e.g., by including distractors), often using complex language (e.g., garden path sentences and complex coreference resolution) and reasoning (e.g., temporal and numeric reasoning). so, in some ways, the successful attack questions are less likely to be encountered in ordinary language use (leading to claims that adversarial examples are brittle, e.g., @xcite . but another key property of human language is that it is flexible and generative, such that people can produce and understand surprising and unexpected utterances. to that end, we think these adversarial questions are a fair target for improving systems precisely because they are linguistically unusual: human language is not just for the "average case" but can be used to express meanings that are subtle, interesting, and complicated. perhaps because these questions also require humans to think creatively outside their ordinary linguistic experience, we also found that we achieved better performance when we had larger groups of people working on generating questions at once, so that there was a wider diversity of ideas. indeed, while some questions may be less likely to appear in a "extractive question answering" dataset, they are understandable by humans and are likely to be useful for efficient communication in real-world settings. the objective behind "extractive qa" is that a machine should answer any question that a human would, given the passage. a variety of real-world tasks can be reduced to extractive qa and in many cases the "correct" passage corresponding to the question is not known a priori. asking questions such as "where was x at a time y" and "what is the difference between 737-200 and 737-200c" may be less natural for a human that has access to the passage, but are questions that someone would, for example, ask their automated assistant. therefore, a well functioning model needs to embrace the creativity and be able to correctly answer adversarial questions. finally, the adversarial attacks that we present are not just interesting from the scientific point of view, but also have clear practical implications. most of the attacks correspond to specific capacities of the model-in-the-loop such as coreference resolution, numerical and temporal reasoning. the consistently high mer indicates that the model underperforms in tasks that require those capacities. our approach towards creating adversarial examples allows us to implicitly evaluate the performance of the model and the quality of the data with respect to a wide variety of linguistic and reasoning categories. overall, we found that the model-in-theloop performs impressively good on the majority of question types. only a small subset of the strategies could consistently obtain above 50% mer and these strategies did not necessarily work for all questions. for instance, questions with relatively few possible entities matching the question type meant fewer possibilities for distractors. the performance of the model is also a function of the varying difficulty of the passages. we found the majority of the passages to be short declarative texts with a simple syntactic structure, few named entities, and low amount of information. generating and answering questions from those passages is a rather trivial task. the selection of more complex paragraphs will likely result in a lower performance of the model and a lot more possibilities for creative and successful adversarial attacks. ## conclusions in this paper we presented the strategies used by team "longhorns" for task 1 of dadc: generating high-quality adversarial examples. we obtain the best results in the competition by taking a systematic approach, using linguistic knowledge, and working in a collaborative environment. our approach outperforms prior work in terms of model error rate and also provides a variety of insights. for instance, our pilot analysis covers a large number of linguistic and reasoning phenomena and explores different model biases. this facilitates a more in-depth analysis of the performance of the model. the systematic approach also gives us insight into the quality and difficulty of the data. our strategies for generating adversarial examples are not limited to extractive question answering. they can be adopted at larger scale to improve the quality of models and data on a variety of different tasks. we believe that our work opens new research directions with both scientific and practical implications.