Membership Inference Attacks on Sequence-to-Sequence Models: I s My Data In Your Machine Translation System? (2020)

Abstract: Data privacy is an important issue for “machine learning as a service” providers. We focus on the problem of membership inference attacks: Given a data sample and black-box access to a model’s API, determine whether the sample existed in the model’s training data. Our contribution is an investigation of this problem in the context of sequence-to-sequence models, which are important in applications such as machine translation and video captioning. We define the membership inference problem for sequence generation, provide an open dataset based on state-of-the-art machine translation models, and report initial results on whether these models leak private information against several kinds of membership inference attacks.

## motivation there are many situations in which private entities are worried about the privacy of their data. for example, many companies provide black-box training services where users are able to upload their data and have customized models built for them, without requiring machine learning expertise. a common concern in these ''machine learning as a service'' offerings is that the uploaded data be visible only to the client that owns it. currently, these entities are in the position of having to trust that service providers abide by the terms of their agreements. although trust is an important component in relationships of all kinds, it has its limitations. in particular, it falls short of a well-known security maxim, originating in a russian proverb that translates as, trust, but verify. 1 ideally, customers would be able to verify that their private data was not being slurped up by the serving company, whether by design or accident. this problem has been formalized as the membership inference problem, first introduced by @xcite and defined as: ''given a machine learning model and a record, determine whether this record was used as part of the model's training dataset or not.'' the problem can be tackled in an adversarial framework: the attacker is interested in answering this question with high accuracy, whereas the defender would like this question to be unanswerable (see figure 1 ). since then, researchers have proposed many ways to attack and defend the privacy of various types of models. however, the work so far has only focused on standard classification problems, where the output space of the model is a fixed set of labels. in this paper, we propose to investigate membership inference for sequence generation problems, where the output space can be viewed as a chained sequence of classifications. prime examples of sequence generation includes machine translation and text summarization: in these problems, the output is a sequence of words whose length is undetermined a priori. other examples include speech synthesis and video caption generation. sequence generation problems are more complex than classification problems, and it is unclear whether the methods and results developed for membership inference in classification problems will transfer. for example, one might imagine that whereas a flat classification model might leak private information when the output is a single label, a recurrent sequence generation model might obfuscate this leakage when labels are generated successively with complex dependencies. we focus on machine translation (mt) as the example sequence generation problem. recent advances in neural sequence-to-sequence models have improved the quality of mt systems significantly, and many commercial service providers are deploying these models via public api's. we pose the main question in the following form: given black-box access to an mt model, is it possible to determine whether a particular sentence pair was in the training set for that model? in the following, we define membership inference for sequence generation problems ( §2) and contrast with prior work on classification ( §3). next we present a novel dataset ( §4) based on state-of-the-art mt models. 2 finally, we propose several attack methods ( §5) and present a series of experiments evaluating their ability to answer the membership inference question ( §6). our conclusion is that simple one-off attacks based on shadow models, which proved successful in classification problems, are not successful on sequence generation problems; this is a result that favors the defender. nevertheless, we describe the specific conditions where sequence-to-sequence models still leak private information, and discuss the possibility of more powerful attacks ( §7). ## problem definition we now define the membership inference attack problem for sequence-to-sequence models in detail. following tradition in the security research literature, we introduce three characters: 2 we release the data to encourage further research in this new problem: https://github.com/sorami/ tacl-membership alice (the service provider) builds a sequenceto-sequence model based on an undisclosed dataset a train and provides a public api. for mt, this api takes a foreign sentence f as input and returns an english translation ê. bob (the attacker) is interested in discerning whether a data sample was included in alice's training data a train by exploiting alice's api. this sample is called a ''probe'' and consists of a foreign sentence f and its reference english translation, e. together with the api's output ê, bob has to make a binary decision using a membership inference classifier g(•), whose goal is to predict: we term in-probes to be those probes where the true class is in, and out-probes to be those whose true class is out. importantly, note that bob has access not only to f but also to e in the probe. intuitively, if ê is equivalent to e, then bob may believe that the probe was contained in a train ; however, it may also be possible that alice's model generalizes well to new samples and translates this probe correctly. the challenge for bob is to make this distinction; the challenge for alice is to prevent bob from doing so. carol (the neutral third-party) is in charge of setting up the experiment between alice and bob. she decides which data samples should be used as in-probes and out-probes and evaluates bob's classification accuracy. carol is introduced only to clarify the exposition and to set up a fair experiment for research purposes. in practical scenarios, carol does not exist: bob decides his own probes, and alice decides her own a train . ## attack results we now present a series of results based on the shadow model attack method described in section 5. in section 6.1 we will observe that bob has difficulty attacking alice under our definition of membership inference. in sections 6.2 and 6.3 we will see that alice nevertheless does leak some private information under more nuanced conditions. section 6.4 describes the possibility of attacks beyond sentence-level membership. section 6.5 explores the attacks using external resources. the results of the attacks on the alice model show that it is around 50%, meaning that the attack is not successful and the binary classification is almost the same as a random choice. 9 the accuracy is around 50% for bob:valid, meaning that bob also has difficulty attacking his own simulated probes, therefore the poor performance on a in probe and a out probe is not due to mismatches between alice's model and bob's model. ## discussions and conclusions we formalized the problem of membership inference attacks on sequence generation tasks, and used machine translation as an example to investigate the feasibility of a privacy attack. our results in section 6.1 and section 6.5 show that alice is generally safe and it is difficult for bob to infer the sentence-level membership. in contrast to attacks on standard classification problems @xcite , sequence generation problems maybe be harder to attack because the input and output spaces are far larger and complex, making it difficult to determine the quality of the model output or how confident the model is. also, the output distribution of class labels is an effective feature for the attacker for standard classification problems, but is difficult to exploit in the sequence case. however, this does not mean that alice has no risk of leaking private information. our analyses in sections 6.2 and 6.3 show that bob's accuracy on out-of-domain and out-of-vocabulary data is above chance, suggesting that attacks may be feasible in conditions where unseen words and domains cause the model to behave differently. further, section 6.4 shows that for a looser definition of membership attack on groups of sentences, the attacker can win at a level above chance. our attack approach was a simple one, using shadow models to mimic the target model. bob can attempt more complex strategies, for example, by using the translation api multiple times per sentence. bob can manipulate a sentence, for example, by dropping or adding words, and observe how the translation changes. we may also use the metrics proposed by @xcite as features for bob; they show how recurrent models might unintentionally memorize rare sequences in the training data, and propose a method to detect it. bob can also add ''watermark sentences'' that have some distinguishable characteristics to influence the alice model, making attack easier. to guard against these attacks, alice's protection strategy may include random subsampling of training data or additional regularization terms. finally, we note some important caveats when interpreting our conclusions. the translation quality of the alice and bob mt models turned out to be similar in terms of bleu. this situation favors bob, but in practice bob is not guaranteed to be able to create shadow models of the same standard, nor verify how well it performs compared with the alice model. we stress that when one is to interpret the results, one must evaluate both on bob's test set and alice probes side-by-side, like those shown in tables 2 , 3 , and 7, to account for the fact that bob's attack on his own shadow model translations is likely an optimistic upper-bound on the real attack accuracy on alice's model. we believe our dataset and analysis is a good starting point for research in these privacy questions. although we focused on mt, the formulation is applicable to other kinds of sequence generation models such as text summarization and video captioning; these will be interesting as future work.