Beat the AI : Investigating Adversarial Human Annotation for Reading Comprehension (2020)

Abstract: Innovations in annotation methodology have been a catalyst for Reading Comprehension (RC) datasets and models. One recent trend to challenge current RC models is to involve a model in the annotation process: Humans create questions adversarially, such that the model fails to answer them correctly. In this work we investigate this annotation methodology and apply it in three different settings, collecting a total of 36,000 samples with progressively stronger models in the annotation loop. This allows us to explore questions such as the reproducibility of the adversarial effect, transfer from data collected with varying model-in-the-loop strengths, and generalization to data collected without a model. We find that training on adversarially collected samples leads to strong generalization to non-adversarially collected datasets, yet with progressive performance deterioration with increasingly stronger models-in-the-loop. Furthermore, we find that stronger models can still learn from datasets collected with substantially weaker models-in-the-loop. When trained on data collected with a BiDAF model in the loop, RoBERTa achieves 39.9F1 on questions that it cannot answer when trained on SQuAD—only marginally lower than when trained on data collected using RoBERTa itself (41.0F1).

## introduction data collection is a fundamental prerequisite for machine learning-based approaches to natural language processing @xcite . innovations in data acquisition methodology, such as crowdsourcing, have led to major breakthroughs in scalability and preceded the ''deep learning revolution'', for which they can arguably be seen as co-responsible @xcite @xcite . annotation approaches include expert annotation, for example, relying on trained linguists @xcite , crowd-sourcing by non-experts @xcite , distant supervision @xcite , and leveraging document structure @xcite . the concrete data collection paradigm chosen dictates the degree of scalability, annotation cost, precise task structure (often arising as a compromise of the above) and difficulty, domain coverage, as well as resulting dataset biases and model blind spots @xcite @xcite . a recently emerging trend in nlp dataset creation is the use of a model-in-the-loop when composing samples: a contemporary model is used either as a filter or directly during annotation, to identify samples wrongly predicted by the model. examples of this method are realized in build it break it, the language edition @xcite , hotpotqa @xcite , swag @xcite , mechanical turker descent @xcite , drop @xcite , codah @xcite , quoref @xcite , and adversarialnli @xcite . 1 this approach probes model robustness and ensures that the resulting datasets pose a challenge to current models, which drives research to tackle new sets of problems. we study this approach in the context of reading comprehension (rc), and investigate its robustness in the face of continuously progressing models-do adversarially constructed datasets quickly become outdated in their usefulness as models grow stronger? figure 1 : human annotation with a model in the loop, showing: i) the ''beat the ai'' annotation setting where only questions that the model does not answer correctly are accepted, and ii) questions generated this way, with a progressively stronger model in the annotation loop. based on models trained on the widely used squad dataset, and following the same annotation protocol, we investigate the annotation setup where an annotator has to compose questions for which the model predicts the wrong answer. as a result, only samples that the model fails to predict correctly are retained in the dataset-see figure 1 for an example. we apply this annotation strategy with three distinct models in the loop, resulting in datasets with 12,000 samples each. we then study the reproducibility of the adversarial effect when retraining the models with the same data, as well as the generalization ability of models trained using datasets produced with and without a model adversary. models can, to a considerable degree, learn to generalize to more challenging questions, based on training sets collected with both stronger and also weaker models in the loop. compared to training on squad, training on adversarially composed questions leads to a similar degree of generalization to non-adversarially written questions, both for squad and naturalquestions @xcite . it furthermore leads to general improvements across the model-in-theloop datasets we collect, as well as improvements of more than 20.0f 1 for both bert and roberta on an extractive subset of drop @xcite , another adversarially composed dataset. when conducting a systematic analysis of the concrete questions different models fail to answer correctly, as well as non-adversarially composed questions, we see that the nature of the resulting questions changes: questions composed with a model in the loop are overall more diverse, use more paraphrasing, multihop inference, comparisons, and background knowledge, and are generally less easily answered by matching an explicit statement that states the required information literally. given our observations, we believe a model-in-the-loop approach to annotation shows promise and should be considered when creating future rc datasets. to summarize, our contributions are as follows: first, an investigation into the model-in-theloop approach to rc data collection based on three progressively stronger models, together with an empirical performance comparison when trained on datasets constructed with adversaries of different strength. second, a comparative investigation into the nature of questions composed to be unsolvable by a sequence of progressively stronger models. third, a study of the reproducibility of the adversarial effect and the generalization ability of models trained in various settings. ## qualitative analysis having applied the general model-in-the-loop methodology on models of varying strength, we next perform a qualitative comparison of the nature of the resulting questions. as reference points we also include the original squad questions, as well as drop and naturalquestions, in this comparison: these datasets are both constructed to overcome limitations in squad and have subsets sufficiently similar to squad to make an analysis possible. specifically, we seek to understand the qualitative differences in terms of reading comprehension challenges posed by the questions in each of these datasets. ## discussion and conclusions we have investigated an rc annotation paradigm that requires a model in the loop to be ''beaten'' by an annotator. applying this approach with progressively stronger models in the loop (bidaf, bert, and roberta), we produced three separate datasets. using these datasets, we investigated several questions regarding the annotation paradigm, in particular, whether such datasets grow outdated as stronger models emerge, and their generalization to standard (non-adversarially collected) questions. we found that stronger models can still learn from data collected with a weak adversary in the loop, and their generalization improves even on datasets collected with a stronger adversary. models trained on data collected with a model in the loop further generalize well to non-adversarially collected data, both on squad and on natu-ralquestions, yet we observe a gradual deterioration in performance with progressively stronger adversaries. we see our work as a contribution towards the emerging paradigm of model-in-the-loop annotation. although this paper has focused on rc, with squad as the original dataset used to train model adversaries, we see no reason in principle why findings would not be similar for other tasks using the same annotation paradigm, when crowdsourcing challenging samples with a model in the loop. we would expect the insights and benefits conveyed by model-in-the-loop annotation to be the greatest on mature datasets where models exceed human performance: here the resulting data provides a magnifying glass on model performance, focused in particular on samples which models struggle on. on the other hand, applying the method to datasets where performance has not yet plateaued would likely result in a more similar distribution to the original data, which is challenging to models a priori. we hope that the series of experiments on replicability, observations on transfer between datasets collected using models of different strength, as well as our findings regarding generalization to non-adversarially collected data, can support and inform future research and annotation efforts using this paradigm.