Imitation Attacks and Defenses for Black-box Machine Translation Systems (2020)

Abstract: Adversaries may look to steal or attack black-box NLP systems, either for financial gain or to exploit model errors. One setting of particular interest is machine translation (MT), where models have high commercial value and errors can be costly. We investigate possible exploitations of black-box MT systems and explore a preliminary defense against such threats. We first show that MT systems can be stolen by querying them with monolingual sentences and training models to imitate their outputs. Using simulated experiments, we demonstrate that MT model stealing is possible even when imitation models have different input data or architectures than their target models. Applying these ideas, we train imitation models that reach within 0.6 BLEU of three production MT systems on both high-resource and low-resource language pairs. We then leverage the similarity of our imitation models to transfer adversarial examples to the production systems. We use gradient-based attacks that expose inputs which lead to semantically-incorrect translations, dropped content, and vulgar model outputs. To mitigate these vulnerabilities, we propose a defense that modifies translation outputs in order to misdirect the optimization of imitation models. This defense degrades the adversary’s BLEU score and attack success rate at some cost in the defender’s BLEU and inference speed.

## introduction nlp models deployed through apis (e.g., google translate) can be lucrative assets for an organization. these models are typically the result of a considerable investment-up to millions of dollarsinto private data annotation and algorithmic improvements. consequently, such models are kept hidden behind black-box apis to protect system integrity and intellectual property. we consider an adversary looking to steal or attack a black-box nlp system. stealing a produc-tion model allows an adversary to avoid long-term api costs or launch a competitor service. moreover, attacking a system using adversarial examples @xcite allows an adversary to cause targeted errors for a model, e.g., bypassing fake news filters or causing systems to output malicious content that may offend users and reflect poorly on system providers. in this work, we investigate these two exploits for black-box machine translation (mt) systems: we first steal (we use "steal" following @xcite ) production mt systems by training imitation models, and we then use these imitation models to generate adversarial examples for production mt systems. we create imitation models by borrowing ideas from knowledge distillation @xcite : we query production mt systems with monolingual sentences and train imitation (i.e., student) models to mimic the system outputs (top of figure 1 ). we first experiment with simulated studies which demonstrate that mt models are easy to imitate (section 3). for example, imitation models closely replicate the target model outputs even when they are trained using different architectures or on outof-domain queries. applying these ideas, we imitate production systems from google, bing, and systran with high fidelity on @xmath0. for example, bing achieves 32.9 bleu on wmt14 @xmath1.4 bleu. we then demonstrate that our imitation models aid adversarial attacks against production mt systems (section 4). in particular, the similarity of our imitation models to the production systems allows for direct transfer of adversarial examples obtained via gradient-based attacks. we find small perturbations that cause targeted mistranslations (e.g., bottom of figure 1 ), nonsense inputs that produce malicious outputs, and universal phrases that cause mistranslations or dropped content. the reason we identify vulnerabilities in nlp figure 1 : imitating and attacking an @xmath2. in phase one (model imitation), we first select sentences from english corpora (e.g., wikipedia), label them using the black-box api, and then train an imitation model on the resulting data. in phase two (adversarial attacks), we generate adversarial examples against our imitation model and transfer them to the production systems. for example, we find an input perturbation that causes google to produce a factually incorrect translation, see the link here (all attacks work as of april 2020). ## how we imitate mt models we have query access to the predictions (but no probabilities or logits) from a victim mt model. this victim is a black box: we are unaware of its internals, e.g., the model architecture, hyperparameters, or training data. our goal is to train an imitation model @xcite that achieves comparable accuracy to this victim on held-out data. moreover, to enhance the transferability of adversarial examples, the imitation model should be functionally similar to the victim, i.e., similar inputs translate to similar outputs. ## imitating black-box mt systems we first study imitation models through simulated experiments: we train victim models, query them as if they are black boxes, and then train imitation mismatch data test ood inter transformer victim 1x 34.6 19.8 -all same 1x 34.4 19.9 69.7 data different 3x 33.9 19.3 67.7 convolutional imitator 1x 34.2 19.2 66.2 data different + conv 3x 33.8 18.9 63.2 convolutional victim 1x 34.3 19.2 -transformer imitator 1x 34.2 19.3 69.7 ## attacking production systems thus far, we have shown that imitation models allow adversaries to steal black-box mt models. here, we show that imitation models can also be used to create adversarial examples for black-box mt systems. our attack code is available at https: //github.com/eric-wallace/adversarial-mt. ## defending against imitation models our goal is not to provide a recipe for adversaries to perform real-world attack. instead, we follow the spirit of threat modeling-we identify vulnerabilities in nlp systems in order to robustly patch them. to take first steps towards this, we design a new defense that slightly degrades victim model bleu while more noticeably degrading imitation model bleu. to accomplish this, we repurpose ## conclusion we demonstrate that model stealing and adversarial examples are practical concerns for production nlp systems. model stealing is not merely hy-6 a naïve defense would equally degrade the bleu score of the victim and imitation models. for example, the victim could simply deploy a worse mt system. pothetical: companies have been caught stealing models in nlp settings, e.g., bing copied google's search outputs using browser toolbars @xcite . moving forward, we hope to improve and deploy defenses against adversarial attacks in nlp, and more broadly, we hope to make security and privacy a more prominent focus of nlp research.