On the Efficacy of Adversarial Data Collection for Question Answering: Results from a Large-Scale Randomized Study (2021)

Abstract: In adversarial data collection (ADC), a human workforce interacts with a model in real time, attempting to produce examples that elicit incorrect predictions. Researchers hope that models trained on these more challenging datasets will rely less on superficial patterns, and thus be less brittle. However, despite ADC’s intuitive appeal, it remains unclear when training on adversarial datasets produces more robust models. In this paper, we conduct a large-scale controlled study focused on question answering, assigning workers at random to compose questions either (i) adversarially (with a model in the loop); or (ii) in the standard fashion (without a model). Across a variety of models and datasets, we find that models trained on adversarial data usually perform better on other adversarial datasets but worse on a diverse collection of out-of-domain evaluation sets. Finally, we provide a qualitative analysis of adversarial (vs standard) data, identifying key differences and offering guidance for future research.

## introduction across such diverse natural language processing (nlp) tasks as natural language inference (nli; @xcite , question answering (qa; @xcite , and sentiment analysis @xcite , researchers have discovered that models can succeed on popular benchmarks by exploiting spurious associations that characterize a particular dataset but do not hold more widely. despite performing well on independent and identically distributed (i.i.d.) data, these models are liable under plausible domain shifts. with the goal of providing more challenging benchmarks that require this stronger form of generalization, an emerging line of research has investigated adversarial data collection (adc), a scheme in which a worker interacts with a model (in real time), attempting to produce examples that elicit incorrect predictions (e.g., @xcite . the hope is that by identifying parts of the input domain where the model fails one might make the model more robust. researchers have shown that models trained on adc perform better on such adversarially collected data and that with successive rounds of adc, crowdworkers are less able to fool the models @xcite . while adversarial data may indeed provide more challenging benchmarks, the process and its actual benefits vis-a-vis tasks of interest remain poorly understood, raising several key questions: (i) do the resulting models typically generalize better out of distribution compared to standard data collection (sdc)?; (ii) how much can differences between adc and sdc be attributed to the way workers behave when attempting to fool models, regardless of whether they are successful? and (iii) what is the impact of training models on adversarial data only, versus using it as a data augmentation strategy? in this paper, we conduct a large-scale randomized controlled study to address these questions. focusing our study on span-based question answering and a variant of the natural questions dataset (nq; @xcite , we work with two popular pretrained transformer architectures-bert large @xcite and electra large @xcite )each fine-tuned on 23.1k examples. to eliminate confounding factors when assessing the impact of adc, we randomly assign the crowdworkers tasked with generating questions to one of three groups: (i) with an incentive to fool the bert model; (ii) with an incentive to fool the electra model; and (iii) a standard, non-adversarial setting (no model in the loop). the pool of contexts is the same for each group and each worker is asked to generate five questions for each context that they see. workers are shown similar instructions (with minimal changes), and paid the same base amount. we fine-tune three models (bert, roberta, and electra) on resulting datasets and evaluate them on held-out test sets, adversarial test sets from prior work @xcite , and 12 mrqa @xcite datasets. for all models, we find that while fine-tuning on adversarial data usually leads to better performance on (previously collected) adversarial data, it typically leads to worse performance on a large, diverse collection of out-of-domain datasets (compared to fine-tuning on standard data). we observe a similar pattern when augmenting the existing dataset with the adversarial data. results on an extensive collection of out-of-domain evaluation sets suggest that adc training data does not offer clear benefits vis-à-vis robustness under distribution shift. to study the differences between adversarial and standard data, we perform a qualitative analysis, categorizing questions based on a taxonomy @xcite . we notice that more questions in the adc dataset require numerical reasoning compared to the sdc sample. these qualitative insights may offer additional guidance to future researchers. absent further assumptions, learning classifiers robust to distribution shift is impossible @xcite . while few nlp papers on the matter make their assumptions explicit, they typically proceed under the implicit assumptions that the labeling function is deterministic (there is one right answer), and that covariate shift @xcite applies (the labeling function p(y|x) is invariant across domains). note that neither condition is generally true of prediction problems. for example, faced with label shift @xcite ) p(y|x) can change across distributions, requiring one to adapt the predictor to each environment. ## study design in our study of adc for qa, each crowdworker is shown a short passage and asked to create 5 questions and highlight answers (spans in the passage, see fig. 1 ). we provide all workers with the same base pay and for those assigned to adc, pay out an additional bonus for each question that fools the qa model. finally, we field a different set of workers to validate the generated examples. ## experiments and results our study allows us to answer three questions: (i) how well do models fine-tuned on adc data generalize to unseen distributions compared to finetuning on sdc? (ii) among the differences between adc and sdc, how many are due to workers trying to fool the model regardless of whether they are successful? and (iii) what is the impact of training on adversarial data only versus using it as a data augmentation strategy? datasets for both bert and electra, we first identify contexts for which at least one question elicited an incorrect model prediction. note that this set of contexts is different for bert and electra. for each such context c, we identify the number of questions k c (out of 5) that successfully fooled the model. we then create 3 datasets per model by, for each context, (i) choosing precisely those k c questions that fooled the model (bert fooled and electra fooled ); (ii) randomly choosing k c questions (out of 5) from adc data without replacement (bert random and electra random )-regardless of whether they fooled the model; and (iii) randomly choosing k c questions (out of 5) from the sdc data without replacement. thus, we create 6 datasets, where all 3 bert datasets have the same number of questions per context (and 11.3k total training examples), while all 3 electra datasets likewise share the same number of questions per context (and 14.7k total training examples). see ## qualitative analysis finally, we perform a qualitative analysis over the collected data, revealing profound differences with models in (versus out of) the loop. recall that be- to begin, we analyze 100 questions from each dataset and categorize them using the taxonomy introduced by @xcite . 6 we also look at (f) sdc-electra figure 3: frequency of question types based on the taxonomy introduced by hovy et al. (2000). the first word of the wh-type questions in each dev set (fig. 3 ) and observe key qualitative differences between data via adc and sdc for both models. in case of adc with bert (and associated sdc), while we observe that most questions in the dev sets start with what, adc has a higher proportion compared to sdc (587 in bert fooled and 492 in bert random versus 416 in sdc). furthermore, we notice that compared to bert fooled dev set, sdc has more when-(148) and who-type (220) questions, the answers to which typically refer to dates, places and people (or organizations), respectively. this is also reflected in the taxonomy categorization. interestingly, the bert random dev set has more whenand who-type questions than bert fooled (103 and 182 versus 50 and 159, respectively). this indicates that the bert model could have been better at answering questions related to dates and people (or organizations), which could have further incentivized workers not to generate toplevel.html such questions upon observing these patterns. similarly, in the 100-question samples, we find that a larger proportion of questions in adc are categorized as requiring numerical reasoning (11 and 18 in bert fooled and bert random , respectively) compared to sdc (7). it is possible that the model's performance on numerical reasoning (as also demonstrated by its lower performance on drop compared to fine-tuning on adc or sdc) would have incentivized workers to generate more questions requiring numerical reasoning and as a result, skewed the distribution towards such questions. similarly, with electra, we observe that what-type questions constitute most of the questions in the development sets for both adc and sdc, although data collected via adc has a higher proportion of these (641 in electra fooled and 619 in electra random versus 542 in sdc). we also notice more how-type questions in adc (126 in electra random ) vs 101 in sdc, and that the sdc sample has more questions that relate to dates (223) but the number is lower in the adc samples (157 and 86 in electra random and electra fooled , respectively). as with bert, the electra model was likely better at identifying answers about dates or years which could have further incentivized workers to generate less questions of such types. however, unlike with bert, we observe that the electra adc and sdc 100-question samples contain similar numbers of questions involving numerical answers (8, 9 and 10 in electra fooled , electra random and sdc respectively). lastly, despite explicit instructions not to generate questions about passage structure (fig. 1 ), a small number of workers nevertheless created such questions. for instance, one worker wrote, "what is the number in the passage that is one digit less than the largest number in the passage?" while most such questions were discarded during validation, some of these are present in the final data. overall, we notice considerable differences between adc and sdc data, particularly vis-avis what kind of questions workers generate. our qualitative analysis offers additional insights that suggest that adc would skew the distribution of questions workers create, as the incentives align with quickly creating more questions that can fool the model. this is reflected in all our adc datasets. one remedy could be to provide workers with initial questions, asking them to edit those questions to elicit incorrect model predictions. similar strategies were employed in @xcite , where breakers minimally edited original data to elicit incorrect predictions from the models built by builders, as well as in recently introduced adversarial benchmarks for sentiment analysis @xcite . ## conclusion in this paper, we demonstrated that across a variety of models and datasets, training on adversarial data leads to better performance on evaluation sets created in a similar fashion, but tends to yield worse performance on out-of-domain evaluation sets not created adversarially. additionally, our results suggest that the adc process (regardless of the outcome) might matter more than successfully fooling a model. we also identify key qualitative differences between data generated via adc and sdc, particularly the kinds of questions created. overall, our work investigates adc in a con-trolled setting, offering insights that can guide future research in this direction. these findings are particularly important given that adc is more timeconsuming and expensive than sdc, with workers requiring additional financial incentives. we believe that a remedy to these issues could be to ask workers to edit questions rather than to generate them. in the future, we would like to extend this study and investigate the efficacy of various constraints on question creation, and the role of other factors such as domain complexity, passage length, and incentive structure, among others.