Adversarial Augmentation Policy Search for Domain and Cross-Lingual Generalization in Reading Comprehension (2020)

Abstract: Reading comprehension models often overfit to nuances of training datasets and fail at adversarial evaluation. Training with adversarially augmented dataset improves robustness against those adversarial attacks but hurts generalization of the models. In this work, we present several effective adversaries and automated data augmentation policy search methods with the goal of making reading comprehension models more robust to adversarial evaluation, but also improving generalization to the source domain as well as new domains and languages. We first propose three new methods for generating QA adversaries, that introduce multiple points of confusion within the context, show dependence on insertion location of the distractor, and reveal the compounding effect of mixing adversarial strategies with syntactic and semantic paraphrasing methods. Next, we find that augmenting the training datasets with uniformly sampled adversaries improves robustness to the adversarial attacks but leads to decline in performance on the original unaugmented dataset. We address this issue via RL and more efficient Bayesian policy search methods for automatically learning the best augmentation policy combinations of the transformation probability for each adversary in a large search space. Using these learned policies, we show that adversarial training can lead to significant improvements in in-domain, out-of-domain, and cross-lingual (German, Russian, Turkish) generalization.

## introduction there has been growing interest in understanding nlp systems and exposing their vulnerabilities through maliciously designed inputs @xcite @xcite . adversarial examples are generated using search @xcite , heuristics @xcite or gradient @xcite based techniques to fool the model into giving the wrong outputs. often, the model is further trained on those adversarial examples to make it robust to similar attacks. in the domain of reading comprehension (rc), adversaries are qa samples with distractor sentences that have significant overlap with the question and are randomly inserted into the context. by having a fixed template for creating the distractors and training on them, the model identifies learnable biases and overfits to the template instead of being robust to the attack itself @xcite . hence, we first build on @xcite 's work of adding randomness to the template and significantly expand the pool of distractor candidates by introducing multiple points of confusion within the context, adding dependence on insertion location of the distractor, and further combining distractors with syntactic and semantic paraphrases to create combinatorially adversarial examples that stress-test the model's language understanding capabilities. these adversaries inflict up to 45% drop in performance of rc models built on top of large pretrained models like roberta @xcite . next, to improve robustness to the aforementioned adversaries, we finetune the rc model with a combined augmented dataset containing an equal number of samples from all of the adversarial transformations. while it improves robustness by a significant margin, it leads to decline in performance on the original unaugmented dataset. hence, instead of uniformly sampling from the various adversarial transformations, we propose to perform a search for the best adversarial policy combinations that improve robustness against the adversarial attacks and also preserve/improve accuracy on the original dataset via data augmentation. however, it is slow, expensive and inductive-biased to manually tune the transformation probability for each adversary and repeat the process for each target dataset, and so we present rl and bayesian search methods to learn this policy combination automatically. for this, we create a large augmentation search space of up to 10 6 , with four adversarial methods, two paraphrasing methods and a discrete binning of probability space for each method (see figure 1 ). @xcite showed via autoaugment that a rnn controller can be trained using reinforcement learning to find the best policy in a large search space. however, autoaugment is computationally expensive and relies on the assumption that the policy searched using rewards from a smaller model and reduced dataset will generalize to bigger models. alternatively, the augmentation methods can be modelled with a surrogate function, such as gaussian processes @xcite , and subjected to bayesian optimization @xcite , drastically reducing the number of training iterations required for achieving similar results (available as a software package for computer vision). 2 hence, we extend these ideas to nlp and perform a systematic comparison between au-toaugment and our more efficient bayesaugment. finally, there has been limited previous work exploring the role of adversarial data augmentation to improve generalization of rc models to out-of-domain and cross-lingual data. hence, we also perform automated policy search of adversarial transformation combinations for enhancing generalization from english wikipedia to datasets in other domains (news, web) and languages (russian, german, turkish). policy search methods like bayesaugment can be readily adapted for lowresource scenarios where one only has access to a small development set that the model can use as a black-box evaluation function (for rewards, but full training or gradient access on that data is unavailable). we show that augmentation policies for the source domain learned using target domain performance as reward, improves the model's generalization to the target domain with only the use of a small development set from that domain. similarly, we use adversarial examples in a pivot language (in our case, english) to improve performance on other languages' rc datasets using rewards from a small development set from that language. our contributions can be summarized as follows: our experiments show that efficient bayesian optimization achieves similar results as autoaugment with a fraction of the resources. overall, the goal of our paper is to make reading comprehension models robust to adversarial attacks as well as out-of-distribution data in cross-domain and cross-lingual scenarios. zhao et al. (2018) employ gans to generate semantically meaningful adversaries. ren et al. (2019) and alzantot et al. (2018) use a synonymsubstitution strategy while ebrahimi et al. (2018) create gradient-based perturbations. iyyer et al. (2018) construct a syntactic paraphrasing network to introduce syntactic variance in adversaries. augmentation and generalization: goodfellow et al. (2015) and miyato et al. (2018) use adversarial training to demonstrate improvement in image recognition. xie et al. (2020) improve the adversarial training scheme with auxiliary batch normalization modules. back-translation @xcite , pre-training with other qa datasets @xcite @xcite and virtual adversarial training @xcite @xcite and significantly reduce the compute time required by autoaugment. we are the first to adapt randaugment style techniques for nlp via our bayesaugment method. randaugment enforces uniform transformation probability on all augmentation methods and collapses the augmentation policy search space to two global parameters. bayesaugment eliminates the need to choose between adversarial methods and optimizes only for their transformation probabilities (see sec. 3.2). ## adversary policy design as shown by @xcite , qa models are susceptible to random, semantically meaningless and minor changes in the data distribution. we extend this work and propose adversaries that exploit the model's sensitivity to insert location of distractor, number of distractors, combinatorial adversaries etc. after exposing the model's weaknesses, we strengthen them by training on these adversaries and show that the model's robustness to adversarial attacks significantly increases due to it. finally, in sec. 4, we automatically learn the right combination of transformation probability for each adversary in response to a target improvement using policy search methods. ## automatic policy search next, we need to perform search over the large space of augmentation policies in order to find the best policy for a desired outcome. performing naive search (random or grid) or manually tuning the transformation probabilities is slow, expensive and largely impractical due to resource constraints. hence, we compare two different approaches for learning the best augmentation policy in fewer searches: autoaugment and bayesaugment. we follow the optimization procedure as demonstrated in figure 1 . for @xmath0, 2, ..., do: ## analysis and discussion having established the efficacy of automated policy search for adversarial training, we further probe the robustness of adversarially trained models to unseen adversaries. we also analyze the convergence of bayesaugment for augmentation policy search and contrast its requirement of computational resources with that of autoaugment. see appendix for more analysis on domain independence of adversarial robustness and augmentation data size. ## conclusion we show that adversarial training can be leveraged to improve robustness of reading comprehension models to adversarial attacks and also to improve performance on source domain and generalization to out-of-domain and cross-lingual data. we present bayesaugment for policy search, which achieves results similar to the computationallyintensive autoaugment method but with a fraction of computational resources. by combining policy search with rewards from the corresponding target development sets' performance, we show that models trained on squad can be generalized to newsqa and german, russian, turkish crosslingual datasets without any training data from the target domain or language. perturbanswer (semantic paraphrasing): following alzantot et al. (2018), we perform semantic paraphrasing of the sentence containing the answer span. instead of using genetic algorithm, we adapt their perturb subroutine to generate paraphrases in the following steps: 1. select word locations for perturbations, which includes locations within any content phrase that does not appear within the answer span. here, content phrases are verbs, adverbs and adjectives. 2. for location k i in the set of word locations {k}, compute 20 nearest neighbors of the word at given location using glove embeddings, create a candidate sentence by perturbing the word location with each of the substitute words and rank perturbed sentences using a language model. 3. select the perturbed sentence with highest rank and perform step 2 for the next location k i+1 using the perturbed sentence. we use the openai-gpt model @xcite to evaluate paraphrases. perturbquestion (syntactic paraphrasing): we use the syntactic paraphrase network introduced by @xcite to generate syntactic adversaries. sentences from the context of qa samples tend to be long and have complicated syntax. the corresponding syntactic paraphrases generated by the paraphrasing network usually miss out on half of the source sentence. therefore, we choose to perform paraphrasing on the questions. we generate 10 paraphrases for each question and rank them based on cosine similarity, computed between the mean of word embeddings @xcite of source sentence and generated paraphrases @xcite . finally, we combine negative perturbations with positive perturbations to create adversaries which double-down on the model's language understanding capabilities. it always leads to a larger drop in performance when tested on the reading comprehension models trained on original unaugmented datasets. semantic difference check: to make sure that the distractor sentences are sufficiently different from the original sentence, we perform a semantic difference check in two steps: 1. extract content phrases from original sentence. content phrase is any common ner phrase or one of the four: noun, verb, adverb, adjective. 2. there should be at least 2 content phrases in the original text that aren't found in the distractor. we examined 100 randomly sampled originaldistractor sentence pairs and found that our semantic difference check works for 96% of the cases.