The Thieves on Sesame Street are Polyglots - Extracting Multilingual Models from Monolingual API s (2020)

Abstract: Pre-training in natural language processing makes it easier for an adversary with only query access to a victim model to reconstruct a local copy of the victim by training with gibberish input data paired with the victim’s labels for that data. We discover that this extraction process extends to local copies initialized from a pre-trained, multilingual model while the victim remains monolingual. The extracted model learns the task from the monolingual victim, but it generalizes far better than the victim to several other languages. This is done without ever showing the multilingual, extracted model a well-formed input in any of the languages for the target task. We also demonstrate that a few real examples can greatly improve performance, and we analyze how these results shed light on how such extraction methods succeed.

## introduction deploying machine learning models typically involves significant cost, including the expense of data acquisition, data cleaning, and model training and tuning. recent work by @xcite has demonstrated that deployed nlp models can be stolen by adversaries by querying victim models with gibberish input data that consists of random sequences of words. in particular, they showed that the following approach is sufficient for stealing text classification and question answering models. first, unlabeled data is created by randomly sampling words from a vocabulary. second, a deployed api is queried with each random input sequence to obtain a label for each. third, a pre-trained language model such as bert @xcite is fine-tuned on the victim-labeled gibberish data. the resulting model retains a significant fraction of the victim model's performance without ever seeing a single well-formed input sentence. this victim monolingual nli model pre-trained multilingual language model (english) hypothesis input data labels extraction phase static iconic changing building, 1933, (for policy mistake writes successor contradiction entailment grand jesse often visit device change… televisions, initial due leave towers justin… (english) premise (multilingual) premise inference phase contradiction मु झे तै यारी शु रू करनी पड़ी contradiction extracted multilingual nli model मु झे पाठ्यक्रम में प्र श ि क्षण शु रू करना पड़ा। । (multilingual) hypothesis process of "stealing" from an api, or "extracting" a local copy of a victim model, is not specific to nlp tasks but rather is a more general phenomenon @xcite @xcite . notably, it does not succeed when the extractor model is trained from scratch; a pre-trained model, such as bert @xcite or roberta @xcite , appears to be critical @xcite . the costs for creating and hosting multilingual nlp models can be even greater than for monolingual models. therefore, extracting a multilingual model is potentially more valuable for an adversary. we demonstrate that it is possible to create multilingual models by stealing the task-specific knowledge from a monolingual victim model and extracting it into a new model pre-trained for multilingual language modeling, such as mbert @xcite and xlmr @xcite . these models are similar to bert and roberta discussed above but extend pre-training and finetuning to multiple languages. even when finetuned in one language, say english, these models achieve good zero-shot performance in other pretraining languages. this phenomenon, known as zero-shot cross-lingual transfer, forms the basis of our approach. combining it with model stealing, or extracting, we demonstrate cross-lingual transfer of task-specific knowledge stolen from a monolingual victim model without collecting a single grammatically correct sentence in any language. our investigation has ramifications for the discussion of model apis as intellectual property and motivates the need to build defenses against such attacks. since models could be deployed by adversaries in multiple languages without collecting real examples in any, defenses such as watermarking @xcite would be rendered useless. while the reason for the surprising phenomenon is unknown, it is hypothesized @xcite that the dynamics of extraction is similar to that of model distillation @xcite . during model distillation, a (student) model is trained with labels as the outputs of another (teacher) model rather than the ground truth to achieve similar or better performance than the teacher @xcite . the success of distillation could thus help explain that of model extraction. for this short paper, we consider the problem of natural language inference on the multilingual xnli dataset @xcite and show that: ## methodology we study the problem of natural language inference (nli): classifying the relationship between a pair of sentences (premise and hypothesis) as either entailment, contradiction or neutral. we focus on this problem given the availability of data in several languages and a history of results on the benchmark @xcite . we consider the setting where an nli classification model is available as a black-box. it can be queried with any input data and returns hard labels. consistent with earlier work, we call this model the victim model. we consider a separate model, the extractor model, that is trained by extracting task-specific knowledge from the victim model. we aim to study how multilingual pre-training affects the extractor and show that it allows transfer of task-specific knowledge from the victim model to other pre-trained languages. we consider two instances in our experiments: (i) one where the extractor has access to no real data and only queries gibberish, and (ii) extractor has access to some real data in english. here, we refer to real data as data which was also used to train the victim model. in our experiments, the victim model is trained on the mnli dataset @xcite . we perform all cross-lingual experiments on the xnli benchmark @xcite . this benchmark contains nli instances in several languages whose test sets were translated by humans using the mnli dataset. in order to generate gibberish input data, we follow the approach of @xcite . for the hypothesis, we generate sentences of random length by sampling words uniformly from the wordlevel vocabulary of wikitext-103. the length of the sentence is sampled based on the distribution of lengths in wikitext-103. for the premise, we randomly swap three words of the hypothesis for random words leaving the rest identical. this is to mimic common nli inputs which have several overlapping words in the hypothesis and premise. for all input sentences, we then perform inference using the victim model and use the hard labels as ground truth labels for the gibberish input data. the gibberish dataset is generated to be the same size as the mnli dataset (∼392k examples). when training the extractor, we tune the learning rate and maximum iterations, and we use the huggingface library @xcite . for both the victim and extractor, we consider variants of the roberta model @xcite . for the victim model, we use the roberta-large variant and, for the extractor we use the xlmr architecture @xcite . the former is a language model pre-trained on a large amount of english data whereas the latter is similar but trained on data from over 100 languages. xlmr demonstrates zero-shot cross-lingual transfer: when finetuned on one language, say english, it is able perform well on other languages without seeing a single training example in those languages. ## experimental results pre-trained multilingual models also succeed at model extraction with gibberish inputs. in models extracted with multilingual pretrained language models perform well on zeroshot cross-lingual transfer. in performance of extracted models greatly improves with a fraction of real data. we now consider the case when the adversary has access to some real labeled data. here, we sample 1, 5, or 10% of the mnli (english) data and investigate two ways of using it: during extraction by adding it to the gibberish data, or after extraction as another fine-tuning stage similar to supplementary training @xcite . the results show that even a small fraction of real data can significantly improve zero-shot performance. in particular, 5% of the mnli english training dataset is enough to lift the performance of extraction to that of xlmr-base for all languages. further, the results show that presenting data after extraction is better than mixing it during extraction. this is in line with results from @xcite . ## analysis the dynamics of model extraction are hypothesized to be similar to that of model distillation @xcite . nlp models ascribe high confidence to gibberish data @xcite . by distilling a model from such queries, the stolen model's decision boundary approximates that of the victim's. we further validate this hypothesis by demonstrating that the embeddings of pre-trained language models show similar behavior whether the input data is real or gibberish. we compute representations for each example of the mnli dataset and the gibberish dataset by max-pooling the outputs of the last layer of pre-trained roberta. for 1000 examples from the mnli development and 1000 gibberish samples, we compute the minimum inner-product distance between each example and the mnli training data. we plot this nearest distance in figure 2 . overlap in the distribution suggests the distinction between real and fake is difficult to make by embeddings alone. though gibberish samples appear random, they sufficiently mimic the input distribution to allow distillation from the victimteacher model into the local-extracted model. finally, we demonstrate that extraction depends heavily on the vocabulary used for random sequence generation and not only on the properties of the models. instead of using the vocabulary from wikitext-103, we use vocabulary of a dataset derived from papers on covid-19 1 . the extraction performance drops from 78.6% to random chance. this suggests that model extraction is unlikely to succeed if the domain of the victim model and the input sampling distribution are different. the most common words of the covid-19 dataset included influenza, rna, infection, respiratory, patients, viral which substantially differ from the more common terms in wikitext-103 such as television, fam-1 https://www.kaggle.com/allen-institute-for-ai/cord19-research-challenge ily, government, military, system. whereas our earlier experiments demonstrated domain extension is possible by extracting into a multilingual model, this transfer requires input queries to reasonably mimic the domain of the victim model. ## conclusion we study the problem of extracting multilingual models by stealing from a monolingual model. we query the monolingual victim model with gibberish data. we then use the victim's labels as groundtruth to fine-tune a separate multilingual. this extracts the task-specific knowledge from the victim and transfers it to languages seen by the multilingual model during its own self-supervised pretraining. we show that high accuracy can be obtained on several languages using this approach, and that this performance improves when the extractor has access to a small fraction of real data. we also show that post-hoc fine-tuning on real data is better than mixing real and gibberish data during extraction. we present results underscoring the importance of vocabulary on the extraction performance, and we provide preliminary evidence to support the hypothesis that the dynamics of model extraction are similar to that of model distillation. our work prompts a deeper investigation into associated topics such as theoretical similarities to distillation, defenses against such multilingual extractions, and improving performance on out-ofdomain vocabulary.