Leveraging Extracted Model Adversaries for Improved Black Box Attacks (2020)

Abstract: We present a method for adversarial input generation against black box models for reading comprehension based question answering. Our approach is composed of two steps. First, we approximate a victim black box model via model extraction. Second, we use our own white box method to generate input perturbations that cause the approximate model to fail. These perturbed inputs are used against the victim. In experiments we find that our method improves on the efficacy of the ADDANY—a white box attack—performed on the approximate model by 25% F1, and the ADDSENT attack—a black box attack—by 11% F1.

## introduction machine learning models are ubiquitous in technologies that are used by billions of people every day. in part, this is due to the recent success of deep learning. indeed, research in the last decade has demonstrated that the most effective deep models can match or even outperform humans on a variety of tasks @xcite . despite their effectiveness, deep models are also known to make embarrassing errors. this is especially troublesome when those errors can be categorized as unsafe, e.g., racist, sexist, etc. @xcite . this leads to the desire for methods to audit models for correctness, robustness andabove all else-safety, before deployment. unfortunately, it is difficult to precisely determine the set of inputs on which a deep model fails because deep models are complex, have a large number of parameters-usually in the billionsand are non-linear @xcite . in an initial attempt to automate the discovery of inputs on which these embarrassing failures occur, researchers developed a technique for making calculated perturbations to an image that are imper-ceptible to the human eye, but cause deep models to misclassify the image @xcite . in addition to developing more effective techniques for creating adversarial inputs for vision models @xcite , subsequent research extends these ideas to new domains, such as natural language processing (nlp). nlp poses unique challenges for adversarial input generation because: 1. natural language is discrete rather than continuous (as in the image domain); and 2. in nlp, an "imperceptible perturbation" of a sentence is typically construed to mean a semantically similar sentence, which can be difficult to generate. nevertheless, the study of adversarial input generation for nlp models has recently flourished, with techniques being developed for a wide variety of tasks such as: text classification, textual entailment and question answering @xcite @xcite . these new techniques can be coarsely categorized into two groups: white box attacks, where the attacker has full knowledge of the victim modelincluding its parameters-and black box attacks, where the attacker only has access to the victim's predictions on specified inputs. unsurprisingly, white box attacks tend to exhibit much greater efficacy than black box attacks. in this work, we develop a technique for black box adversarial input generation for the task of reading comprehension that employs a white box attack on an approximation of the victim. more specifically, our approach begins with model extraction, where we learn an approximation of the victim model @xcite ; afterward, we run a modification of the addany @xcite attack on the model approximation. our approach is inspired by the work of @xcite for images and can also be referred to as a black box evasion attack on the original model. since the addany attack is run on an extracted (i.e., approximate) model of the victim, our modification encourages the attack method to find inputs for which the extracted model's top-k responses are all incorrect, rather than only its top response-as in the original addany attack. the result of our addany attack is a set of adversarial perturbations, which are then applied to induce failures in the victim model. empirically, we demonstrate that our approach is more effective than addsent, i.e., a black box method for adversarial input generation for reading comprehension @xcite . crucially, we observe that our modification of ad-dany makes the attacks produced more robust to the difference between the extracted and victim model. in particular, our black box approach causes the victim to fail 11% more than addsent. while we focus on reading comprehension, we believe that our approach of model extraction followed by white box attacks is a fertile and relatively unexplored area that can be applied to a wide range of tasks and domains. ## background in this section we briefly describe the task of reading comprehension based question answering, which we study in this work. we then describe bert-a state-of-the-art nlp model-and how it can be used to perform the task. ## method our goal is to develop an effective black box attack for rc based qa models. our approach proceeds in two steps: first, we build an approximation of the victim model, and second, we attack the approximate model with a powerful white box method. the result of the attack is a collection of adversarial inputs that can be applied to the victim. in this section we describe these steps in detail. ## experiments in this section we present results of our proposed approach. we begin by describing the dataset used, and then report on model extraction. finally, we compare the effectiveness of addany-kbest to 2 other black box approaches. ## conclusion in this work, we propose a method for generating adversarial input perturbations for black box reading comprehension based question answering models. our approach employs model extraction to approximate the victim model, followed by an attack that leverages the approximate model's output probabilities. in experiments, we show that our method reduces the f1 score on the victim by 11 points in comparison to addsent-a previously proposed method for generating adversarial input perturbations. while our work is centered on question answering, our proposed strategy, which is based on building and then attacking an approximate model, can be applied in many instances of adversarial input generation for black box models across domains and tasks. future extension of our work could explore such attacks as a potential proxy for similarity estimation of victim and extracted models in not only accuracy, but also fidelity @xcite .