O PTIMAL S TRATEGIES A GAINST G ENERATIVE A TTACKS
Anonymous authors
Paper under double-blind review
A BSTRACT
Generative neural models have improved dramatically recently.
With this progress comes the risk that such models will be used to attack systems that rely on sensor data for authentication and anomaly detection.
Many such learning systems are installed worldwide, protecting critical infrastructure or private data against malfunction and cyber attacks.
We formulate the scenario of such an authentication system facing generative impersonation attacks, characterize it from a theoretical perspective and explore its practical implications.
In particular, we ask fundamental theoretical questions in learning, statistics and information theory: How hard is it to detect a “fake reality”?
How much data does the attacker need to collect before it can reliably generate nominally-looking artiﬁcial data?
Are there optimal strategies for the attacker or the authenticator?
We cast the problem as a maximin game, characterize the optimal strategy for both attacker and authenticator in the general case, and provide the optimal strategies in closed form for the case of
Gaussian source distributions.
Our analysis reveals the structure of the optimal attack and the relative importance of data collection for both authenticator and attacker.
Based on these insights we design practical learning approaches, and show that they result in models that are more robust to various attacks on real-world data.
Code will be made publicly available upon publication.
I NTRODUCTION
Generative models have attracted considerable attention since the introduction of Generative Adversarial Networks (Goodfellow et al., 2014a).
Empirically, GANs have been shown to generate novel data instances that resemble those in the true distribution of the data.
The success of GANs also comes with the risk that generative models will be used for attacking sensor-based security systems.
One example is identity authentication systems, where an individual is identiﬁed via her images, and
GANs might be able to generate such images to gain access (Thies et al., 2016).
Another is anomaly detection systems protecting critical infrastructure.
As demonstrated by recent cyber-attacks (notably the Stuxnet attack) sensors of these systems can be hijacked, so that GANs can be used to generate “normal” looking activity while the actual system is being tampered with.
The latter is in fact a new form of a man-in-the-middle attack.
Our goal here is to construct a theoretical framework for studying the security risk arising from generative models, and explore its practical implications.
We begin with a simple key insight.
If the attacker (i.e. the generative model) has unlimited observations of the source it is trying to imitate, it will be able to fool any authenticator.
On the other hand, if the attacker has access to fewer sensor observations than the number of fake observations it needs to generate, it seems intuitively clear that it cannot always succeed (as we indeed prove in Sec. 4).
Therefore, the optimal defense and attack strategies depend crucially on the amount of information available to the attacker and authenticator.
Motivated by the above insight, we cast the authentication setting as a two player maximin game (authenticator vs. attacker) where all observations are ﬁnite.
Speciﬁcally, there are three key observation sets to consider: those available to the attacker, those that the attacker needs to generate, and those available to the authenticator when designing the system.
Our goal is to understand how these three information sources determine the optimal strategies for both players.
Under the realistic assumption that cyber attackers are sophisticated enough to play optimal or close to optimal strategies, a characterization of the maximin authentication strategy can be of signiﬁcant value.
We prove several theoretical results characterizing the optimal strategy for both players.
These results highlight the role played by the available observations as well as the functional form for an optimal attacker and authenticator.
We refer to the setting above as “GAN in The Middle” (GIM) due to its similarity to “man in the middle” attacks.
After describing our theoretical results, we show how to learn both authenticator and attacker policies in practice, where both are based on neural architectures.
Our GIM method can be applied to multiple practical problems.
The ﬁrst is building an authentication system that is robust to impersonation attacks.
The second is building a data generating mechanism that can generate novel data instances.
Finally, we evaluate the method empirically, showing that it outperforms existing methods in terms of resilience to generative attacks, and that it can be used effectively for data-augmentation in the few-shot learning setting.
P ROBLEM STATEMENT
We begin by motivating our problem, and formulating it as a two player zero sum game.
As a simple illustrative example, consider a face authentication security system whose goal is to maximize authentication accuracy.
The system is initialized by registering k images of an individual θ (the ”source”), whose identity is to be authenticated.
At test-time, each entity claiming to be θ is required to present to the system n of its images, and the authentication system decides whether the entity is θ or an impersonator.
We let m denote the maximum number of “leaked” images any attacker obtained.
We observe that if an attacker obtained m ě n images of θ , it can present n of those images.
Thus the observations generated by the attacker are indistinguishable from ones generated by θ , leading to failure of the authentication system (see Sec.
4.2 below).
Namely, the number of images of θ that the attacker obtains and the size of the authentication sample are of key importance.
We now turn to formally stating the problem.
Notation.
The set of possible observations is denoted by X .
Let H denote the known set of possible sources θ , where each source θ P H is deﬁned by a probability density f θ , and an observation of a source θ P H is an X -valued random variable with density f θ .
We assume that subsequent observations of the source are IID, so that n sequential observations have density f p n q θ p x 1 , . . .
, x n q : “ ś ni “ 1 f θ p x i q .
We allow θ to be sampled from a known distribution Q on
H and denote the corresponding H -valued random variable by Θ .
In what follows we will denote the number of observations leaked to the attacker by m , the number of “registration” observations available to the authenticator by k , and the number of observations required at authentication by n (these may be generated by either the attacker or the true source).
The Authentication Game.
The game begins with a random source Θ being drawn from H according to Q .
The authenticator ﬁrst receives information about the drawn source, and then chooses a decision rule for deciding whether a given test sequence x P X n is an authentic sequence of observations sampled from f p n q θ , or a fake sequence generated by the attacker.
Formally the authenticator learns about the source by seeing k IID “registration” observations
A “ A 1 , . . . ,
A k „ f p k q θ .
The set of all possible decisions rules is then D :
X k ˆ X n Ñ t 0 , 1 u (where a decision of 1 corresponds to the true source, and 0 to an attacker).
After the authenticator ﬁxes its strategy, the attacker can seek the best attack strategy.
We assume that the attacker has access to m “leaked” IID observations
Y “ Y 1 , . . .
, Y m „ f p m q θ as information about the source θ .
Then it generates an attack sequence
X P X n and presents it to the authenticator which uses its decision rule to decide whether X is an authentic sequence of observations sampled from f p n q θ , or a fake sequence generated by the attacker.
Formally, the strategy set of the attacker is all functions G : X m Ñ ∆ p X n q , where ∆ p X n q is the set of probability distributions over X n , and g X | Y is the associated conditional probability density.
We note that the set H , the parametric family f θ , and the prior probability Q are known to both players.
Also, note that the leaked sample Y revealed to the attacker is not available to the authenticator, and the “registration” sample A is not available to the attacker.
The goal of the authenticator is to maximize its expected accuracy, and the goal of the attacker is to minimize it (or equivalently maximize its success probability).
We deﬁne the utility (payoff) of the game as the expected prediction accuracy of the authenticator.
To deﬁne expected accuracy we consider the case of equal priors for and real samples.
Formally, for a pair of strategies p D , G q and a speciﬁc source θ , the expected accuracy of the authenticator is then:
[Equation]
Since this utility only depends on G in the second term, minimizing it is equivalent to G maximizing its success probability.
To obtain the overall utility for the authenticator, we take the expected value w.r.t θ and deﬁne V p D , G q “ E Θ „ Q V p Θ , D , G q .
Finally, we arrive at the following maximin game:
[Equation]
where D , G are the set of all possible authenticator and attacker strategies, respectively.
In Sec.
4 we show that this game has a Nash equilibrium, we characterize the optimal strategies and game value in general, and ﬁnd them in closed form for the case of Multivariate Gaussian sources.
R ELATED WORK
Adversarial hypothesis testing (AHT): Hypothesis testing (HT) is a rich ﬁeld in statistics which studies how one can detect whether a sample was generated by one of two sets of distributions.
A variant of HT that is related to our work but distinct from it is AHT (e.g., see Brand ˜ ao
et al., 2014;
Barni & Tondi, 2013b;a; 2014; Bao et al., 2011; Zhou et al., 2019;
Br ¨ uckner & Scheffer, 2011;
Br ¨ uckner et al., 2012).
These works describe an HT setting where the sample is generated by one of two hypotheses classes, but is then modiﬁed by an adversary in some restricted way.
e.g., in Barni & Tondi (2013b;a; 2014) the adversary can change the sample of one class up to a ﬁxed distance (e.g., Hamming).
Given the quality of current generative models and the rapid pace of progress, when considering an impersonation attack, it seems that the only relevant restriction one can assume on an attacker is on the information it has.
This is not captured by prior work since it assumes that the adversary has a restricted strategy set.
In contrast, our work considers a novel problem setting where both players are not limited in their strategy set in any way.
This leads to a novel analysis that focuses on the dependence on the ﬁnite information available to each player ( m, n, k ).
Adversarial Examples: It has been observed (Goodfellow et al., 2014b) that deep learning models can be very sensitive to small changes in their input.
Such “misleading” inputs are known as adversarial examples and much recent work has analysed the phenomenon (Ilyas et al., 2019; Shamir et al., 2019; Zhang et al., 2019), addressed the problem of robustness to these (Moosavi-Dezfooli et al., 2016;
Papernot et al., 2017; Yuan et al., 2017), and certifying robustness (Raghunathan et al., 2018; Wong et al., 2018).
The setting of robust classiﬁcation in the presence of adversarial examples can also be thought of as a speciﬁc case of AHT (see above), where a classiﬁer is required to predict the class of an observation that could have been perturbed by a restricted adversary (Wong et al., 2018; 2019) or generated by an adversary limited to generating examples that will be classiﬁed correctly by humans (Song et al., 2018).
In contrast, in our setting the attacker is not limited in any way, nor does it have another utility in addition to impersonating the source.
Furthermore, in adversarial examples there is no notion of limited information for the adversary, whereas our work focuses on the dependence of the game on the information available to the players (sample sizes n, m, k ).
GAN: The GAN model is a game between a generator and discriminator.
While our concept of generative attacks is inspired by a GAN it is very different from it: a successful GAN generator is not necessarily a successful attacker in our setting, and vice-versa (e.g., given sufﬁciently expressive generators and discriminators, GANs can “memorize” the training data, and thus the discriminator will perform at chance level.
2 Such a discriminator will not be useful as a defence against generative attacks).
Unlike GANs, in our setting, sample sizes are of key importance.
Thus, our attacker will not memorize the data it sees, as this will be detected when generating n ą m examples.
Conditional GANs:
In conditional GANs (Mirza & Osindero, 2014) the generator uses side information for generating new samples.
The attacker in our approach (analogous to GAN generator) has input, but this input is not available to the authenticator (analogous to GAN discriminator).
Thus, the objective of the learning process is fundamentally different.
Few shot learning and generation:
Our work relates to few shot learning (Snell et al., 2017;
Vinyals et al., 2016; Finn et al., 2017; Lake et al., 2011; Koch et al., 2015) and few shot generative models (Rezende et al., 2016; Zakharov et al., 2019;
Lake et al., 2015; Edwards & Storkey, 2017;
Hewitt et al., 2018) in the sense that both authenticator and attacker need to learn from a limited set of observations.
However, in our setting the authenticator is required to predict whether a sample came from the true source or an attacker impersonating the source, while taking into consideration the amount of information both players have.
These are notions that are not part of the general few shot learning setup.
Also, in prior work on few-shot generation, the generator is either measured through human evaluation (Lake et al., 2015) or trained to maximize the likelihood of its generated sample (Rezende et al., 2016; Edwards & Storkey, 2017; Hewitt et al., 2018).
In contrast, in our setting the attacker’s objective is to maximize the probability that its generated sample will be labeled as real by an authenticator.
To this end, we show that the attacker must consider the sample sizes m, n, k , which the generative models in other works do not account for.
Furthermore, we show in
Sec.
F.4, and in Figures 1c,6, that the ML solution is indeed sub-optimal in our setting.
Image to image translation: Several GAN models have been introduced for mapping between two domains (Zhu et al., 2017; Huang et al., 2018; Isola et al., 2017; Wang et al., 2017; Park et al., 2019).
This relates to our work since the attacker also needs to learn to map the leaked sample to an attack sample.
However, in our setting the mapping is not to a different domain but rather to other images from the same distribution, which results in a different objective.
Data Augmentation: Generative models have also been used for augmenting data in supervised learning, and in particular few-shot learning (Koch et al., 2015;
Snell et al., 2017; Vinyals et al., 2016;
Lake et al., 2011).
One such approach is Data Augmentation GAN (DAGAN)
(Antoniou et al., 2018), which takes as input an image and generates a new image.
It relates to our framework in the limited case of m “ 1 , n “ 2 , k “ 1 , in the sense that the generator objective is to map one image to two.
However, in DAGAN the only goal of the discriminator is to improve the generator, and the generator is limited to the functional form of adding a new image to the existing one, which is a sub-optimal attack strategy, as can be seen from the Gaussian case of our problem.
T HEORETICAL RESULTS
In this section we study the game deﬁned in Eq.
First, in Sec.
4.1 we show the existence of a
Nash equilibrium and characterize the optimal strategies for both players.
Speciﬁcally we show that the optimal attacker strategy minimizes a certain divergence between its marginal distribution and f p n q Θ .
Next, Sec.
4.2 shows that when there are more leaked samples m than generated samples n , the authenticator will fail.
Finally, in Sec.
4.3 we provide a closed form solution for both attacker and authenticator for the case of multivariate Gaussian distributions, and analyze the signiﬁcance of the observations’ dimension, and the sample sizes m, n and k .
Proofs are provided in the appendix.
C HARACTERIZING THE OPTIMAL STRATEGIES
We begin by showing that the game deﬁned in Eq.
2.2 admits a Nash equilibrium.
Namely, Theorem 4.1 below shows that there exists a pair of strategies p D ˚ , G ˚ q that satisfy:
[Equation]
Theorem 4.1.
Consider the attacker G ˚ deﬁned by:
[Equation]
Where f A p a q “ ş θ P H Q p θ q f p k q θ p a q dθ is the marginal density of A , Q Θ | A is the posterior over H given A , f Y | A p y | a q “ ş θ P H f p m q θ p
y q Q Θ | A p θ | a q dθ , and f X
| A p x | a q “ ş θ P H f p n q θ p x q Q Θ | A p θ | a q dθ , g X | A p x | a q “ ş y P X m g X | Y p x | y q f Y |
A p y | a q dy are the conditional densities of X given A generated by the source and the attacker respectively.
Consider the authenticator deﬁned by:
D ˚ p a, x q “ I ” f X | A p x | a q ą g ˚ X |
A p x | a q ı , where I is the indicator function.
Then p D ˚ , G ˚ q is a solution of Eq.
2.2 that satisﬁes Eq. 4.1.
The proof (see Sec. D) follows by ﬁrst showing that since D p a, x q P t 0 , 1 u , then for any G its optimal strategy is a MAP test between the two hypotheses (true source or attacker).
We then show that given
D ˚ , the game objective for G becomes the (cid:96) 1 distance over the space
X k ˆ X n between the true source distribution and the marginal distribution of the attacker, and thus any optimal G minimizes it.
Therefore it holds that min G V p D ˚ , G q “ V p D ˚ , G ˚ q “ max D V p D , G ˚ q
and it follows that
D ˚ , G ˚ satisfy Eq. 4.1.
4. R EPLAY A TTACKS : A UTHENTICATION FAILURE FOR n ď m
When n ď m , the attacker generates a number of observations that is at most the number of observations it has seen.
Intuitively, an optimal attack in this case is to simply “replay” a subset of size n from the m observations.
This is known as a replay-attack (Syverson, 1994).
This subset constitutes an IID sample of length n of the observed source, and is therefore a legitimate “fresh” sample.
In this case, it seems like the attack cannot be detected by the authenticator.
Indeed it is easy to show using Theorem 4.1 that this attack is optimal and therefore for n ď m we have: max
[Equation]
G P G
[Equation] p D , G q [Equation] 5 (see Sec.
E)
4. T HE G AUSSIAN CASE
We now turn to the case of multivariate Gaussian distributions where we can ﬁnd the exact form of the attacker and authenticator, providing insight into the general problem.
Speciﬁcally, we consider the setting where the source distributions are d -dimensional multivariate Gaussians with an unknown mean and known covariance, and the prior Q over H is the improper uniform prior.
3 We assume n ą m to keep the problem non-trivial.
Let the observations be d -dimensional Gaussian vectors with a known covariance matrix Σ P R d ˆ d and an unknown mean vector θ P R d .
The set of possible sources H becomes R d , the Gaussian mean vectors.
For any sample of n examples z P R n ˆ d we let z i denote the i ’th example, and ¯ z “ 1 n ř ni “ 1 z i denote the sample mean.
Finally, for any v P R d , B P R d ˆ d , we deﬁne } v } 2 B “ v T Bv .
The following theorem gives a closed-form solution for both attacker and authenticator for the game deﬁned in Eq. 2.2.
Theorem 4.2.
Deﬁne δ “ m { n ď 1 and let ρ “ m { k .
Consider the attacker G ˚ deﬁned by the following generative process:
Given a leaked sample Y P R m ˆ d , G ˚ generates a sample X P R n ˆ d as follows: it ﬁrst samples n vectors W 1 , . .
, W n iid „
N p 0 , Σ q and then sets: X i “ W i ´ ¯ W ` ¯ Y .
Deﬁne the authenticator D ˚ by:
[Equation]
Then p D ˚ , G ˚ q is a solution of Eq.
2.2 that satisﬁes Eq. 4.1.
The proof (see Sec. F) starts by showing that @
α ą 0 , given D p a, x q “ I r} ¯ x ´ ¯ a } 2Σ ´
ă α s , the optimal strategy for G is to set ¯ x “ ¯ y with probability 1 (as done in G ˚ ).
To prove this, we ﬁrst use the Prekopa-Leindler inequality (Prkopa, 1973) to show that in this case G ’s maximization objective is log-concave.
We then show that any G that satisﬁes ¯ x “ ¯ y with probability 1 is a local extremum, and since the objective is log-concave it follows that it is the global maximum.
We continue by showing that given G ˚ , D ˚ is optimal.
To do so, we ﬁrst ﬁnd the distribution of G ˚ ’s attack sample using the Woodbury matrix identity, and then show that D ˚ is indeed the optimal decision rule.
Finally, using the max-min inequality, this implies that p D ˚ , G ˚ q satisfy Eq.
There are several interesting observations about the above optimal strategies.
Perhaps the most intuitive strategy for the attacker would have been to sample n elements from a Gaussian with mean ¯
Y and the known covariance Σ .
In expectation this sample would have the correct mean.
However, this turns out to be sub-optimal, as can be seen in Figures 1c, 6 (we refer to this as an ML attack, see Sec.
F.4 in the appendix for the derivation and visualizations).
Instead, the optimal attacker begins by drawing an IID sample, W , from a Gaussian distribution with mean 0 , and then “forces” the sample mean to be exactly ¯ Y by shifting the sample points by ¯ Y ´ ¯ W .
This optimal attacker strategy can be viewed as matching the sufﬁcient statistics of the leaked sample Y in the generated sample X .
The optimal authenticator is a MAP test for the optimal attacker, as in Theorem 4.1.
As a corollary to Theorem 4.2 we obtain the value of the game (i.e., the accuracy of D ˚ ).
Corollary 4.3.
Deﬁne δ and ρ as in Theorem 4.2.
Then the game value for the Gaussian case is:
[Equation]
Where γ is the lower incomplete gamma function, and Γ is the Gamma function.
The proof (see Sec. F) follows by showing that the test statistic used by D ˚ is Gamma distributed.
Fig.
1 demonstrates several interesting aspects of the above results.
First, Fig.
1a shows that the authenticator accuracy improves as n the size of the test sample grows.
Furthermore, accuracy also improves as the dimension d grows, meaning that for a speciﬁed authentication accuracy, the required ratio n { m becomes smaller with the dimension d .
This is a very encouraging result, since although this dimensional dependency is proved only for Gaussian sources, it suggests that for real world high-dimensional sources (e.g. faces, video, voice, etc.)
the authenticator can achieve high accuracy even when requiring a small (and practical) authentication sample.
Intuitively it may seem like authentication is impossible when the authenticator has less data than the attacker (i.e., m ą k ).
Surprisingly, this is not the case.
As can be seen in Fig.
1b, even when m ą k , the authenticator can achieve non trivial accuracy.
An intuitive explanation to this phenomenon is that the test statistic used by the authenticator is ››
¯ X
´ ¯ A ›› , which, due to the variance in the attacker estimation, has higher variance when X is generated by an attacker than when X is generated by the true source.
This in turn allows the authenticator to discriminate between the hypotheses.
A closed-form solution for the general case remains an open problem.
We believe that solving for Gaussians is an important step forward, since it exposes interesting structural properties of the solution, which we use in practice.
Furthermore, if G has an encoder-decoder structure, it is not unreasonable that the source distribution in latent space can be approximately Gaussian (as in VAE).
GAN IN THE M IDDLE N ETWORKS
So far we explored the general formalism of authentication games.
Here we consider speciﬁc architectures for D and G .
As in GAN based models (Mirza & Osindero, 2014; Mescheder et al., 2018;
Karras et al., 2018a;b), we use neural nets to model these, while using insights from our theoretical analysis.
Below we provide implementation details for the GIM model (see Sec.
H and code for more details).
In our analysis we considered the non-differentiable zero-one loss since it is the real accuracy measure.
In practice we will use cross-entropy as used in most GAN approaches.
Authenticator Architecture:
The authenticator is implemented as a neural network D p a, x q that maps from a source information sample a P X k and a test sample x P X n to a probability that the test sample came from the true source.
Our framework does not restrict the authenticator to any speciﬁc function type, but in practice one must of course implement it using some model.
We recall that our theoretical results do suggest a certain functional form.
The Gaussian results in Sec. 4.
show that the optimal authenticator is a test on the sufﬁcient statistic of the source parameters.
Motivated by this result, and in the spirit of Siamese networks (Koch et al., 2015; Chopra et al., 2005), we consider the following form for the authenticator.
We deﬁne a function T D that maps a sample to a ﬁxed sized vector, analogous to the sufﬁcient statistic in the theorem.
We apply T D to both a and x .
Then, these two outputs are used as input to a comparison function σ which outputs a scalar reﬂecting their similarity.
Thus the authenticator can be expressed as: D p a, x q “ σ p T D p a q , T D p x qq .
Attacker Architecture:
The attacker is implemented as a stochastic neural network G p y q that maps a leaked sample y P
X m to an attack sample x P X n .
Our theoretical results suggest a certain functional form for this network.
The Gaussian analysis in Sec.
4.3 shows that the optimal attacker generates a sample whose sufﬁcient statistic matches that of the leaked sample.
Motivated by this result, we consider the following functional form for the attacker.
First it applies a function T G that maps the leaked sample Y to a ﬁxed sized vector T G p Y q , analogous to the sufﬁcient statistic in the theorem.
It then draws n random latent vectors W 1 , . .
, W n and matches their mean to the leaked sufﬁcient statistic to obtain the latent vectors W 1 i .
I.e it sets W 1 i “ W
i
´
¯ W ` T G p Y q as done in
Theorem 4.2.
Finally, it uses a decoder function ϕ that maps each latent vector W 1 i to the domain
X .
Thus, the attacker can be expressed as: G p Y q i “ ϕ p
W i ´ ¯ [Equation] G p Y qq @
i P r n s .
Optimization Details: Each iteration begins when a source θ is chosen randomly from the set of sources in the training dataset (e.g., a person to be authenticated).
Samples A, Y, X θ are drawn from the set of examples available for θ , where X θ represents a test sample from θ .
Then, given a leaked sample Y , G generates a fake sample X G , passes it to D and suffers the appropriate loss.
Finally, D receives as input the source information sample A , outputs a prediction for each of the test samples
X θ , X G , and suffers the appropriate loss.
Optimization is done via gradient ascent on authenticator parameters and descent on attacker parameters, as is typical for GAN problems.
E XPERIMENTS
We next evaluate our method empirically.
In all experiments we use the model described in Sec. 5.
We optimize the model with adversarial training, using the loss suggested by Mescheder et al.
and Adam (Kingma & Ba, 2015).
See Sec.
H for further Implementation details.
Gaussian sources: For the case of Gaussian sources we arrived at a complete characterization of the solution in Sec. 4.3.
Thus, we can learn the models using our GIM algorithm and check whether it ﬁnds the correct solution.
This is important, as the GIM objective is clearly non-convex and GANs are generally hard to train in practice and lack convergence guarantees (Mescheder et al., 2018).
We ran all experiments using a multivariate Gaussian with Σ “ I d , and in each game the source mean was drawn from the prior distribution Q “ N p 0 ,
10 I d q .
This approximates the improper uniform prior since the prior has much larger variance than the sources.
Fig.
1a shows the empirical game value compared to the theoretical one as a function of the test sample size n , for ﬁxed m “ 1 , k “ 10 , and different values of d .
It can be seen that there is an excellent ﬁt between theory and experiment.
[Figure]
Authentication on Faces and Characters: We next evaluate GIM in an authentication setting on two datasets: the VoxCeleb2 faces dataset (Nagrani et al., 2017; Chung & Zisserman, 2018), and the Omniglot handwritten character dataset (Lake et al., 2015).
Additional information about the datasets, splits, and modeling details is provided in Sec.
G, H. Our goal is to check whether the GIM authenticator is more robust to generative attacks than a state of the art authentication system.
To evaluate this, we consider several attackers: 1) A “random source” attacker (RS): a naive attacker that ignores the leaked sample Y . It simply draws a random source from the dataset, and samples n real images of that source.
From the authenticator’s perspective it’s equivalent to a sample version of the veriﬁcation task (Koch et al., 2015; Schroff et al., 2015; Deng et al., 2018), in which an agent is presented with a pair of real images and needs to decide whether they are from the same source or not.
2) Replay attacker (Replay): an attacker which upon seeing a leaked sample Y , draws n random images of the leaked sample (with replacement).
A GIM attack, which is the “worst case” attacker G , learned by our GIM model.
For VoxCeleb we compare the GIM authenticator to the ArcFace method (Deng et al., 2018), which is currently state of the art in face veriﬁcation.
As a baseline for Omniglot we use the Siamese network suggested by Koch et al.
(2015), which achieves state of the art in the veriﬁcation task on Omniglot.
Results are shown in Table 1.
It can be seen that on average across attacks, GIM outperforms the baselines.
The only attack for which GIM is inferior is RS.
This is not surprising as this is the objective that both baselines are trained for.
Qualitative evaluation of attacker:
In Fig.
2 we provide images generated by the GIM attacker for the test set of both Omniglot and Voxceleb.
The images demonstrate qualitatively the strategy learned by the attacker.
In the Voxceleb2 dataset, face images are drawn from a video of the person talking.
Note that as in real samples from the data, the attack sample varies in pose and expression and not in the background or clothing.
[Table]
[Figure]
Data augmentation: Finally, we use GIM for data augmentation in one-shot classiﬁcation on Omniglot.
This is done by using the GIM attacker to generate more data for a given class.
We ﬁrst train
GIM on the training set with parameters m “ 1 , n “ 5 , k “ 5 .
Then, during both training and testing of one shot classiﬁcation, given an example, we use the attacker to augment the single example available for each of the classes, by adding to it the n “ 5 examples our attacker generated from it.
We use Prototypical Nets (Snell et al., 2017) as the baseline model.
We ﬁnd that without using our augmentation method, Prototypical Nets achieve 95 .
9% accuracy on the test split, and with our method they achieve 96 .
5% , which is similar to the improvement achieved by Antoniou et al.
(2018) with Matching networks (Vinyals et al., 2016)
as the few shot classiﬁcation algorithm.
C ONCLUSIONS
We deﬁned the notion of authentication in the face of generative attacks, in which a generative model attempts to produce a fake reality based on observations of reality.
These attacks raise numerous interesting theoretical questions and are very important and timely from a practical standpoint.
We proposed to study generative attacks as a two-person zero-sum game between attacker and authenticator.
In our most general setup both attacker and authenticator have access to a ﬁnite set of observations of the source.
We show that this game has a Nash equilibrium, and we characterize the optimal strategies.
In the Gaussian version of the game a closed form of the optimal strategies is available.
A nice outcome of the analysis is that the game value depends on m, n, k only through their ratios δ “ m { n (i.e., the ”expansion ratio” between attack and leaked sample sizes) and ρ “ m { k (i.e., the “information ratio” between the number of source observations available to attacker and authenticator).
As we show in Fig.
1b, there is a large range of values for which high accuracy authentication is possible, and as d grows we observe that the high authentication accuracy region in the p δ, ρ q plane grows sharply.
We introduce the GIM model, which is a practical approach to learning both authenticator and attacker, and whose structure is inspired by our analysis.
GIM achieves accuracy that is very close to the theoretical rates in the Gaussian case, and is also more robust to attacks, when compared to state of the art authenticators on real data.
Many theoretical and practical questions remain.
For example ﬁnding closed form optimal strategies for other distributions, and going beyond IID generation.
The non IID setting is of particular importance for the problem of fake video (Thies et al., 2016) and audio (Arik et al., 2018) generation, which we intend to study in the future.
A I NTRODUCTION TO THE APPENDIX
Here we provide additional information and proofs for the main paper.
In Sec.
B we provide a visualization of the game setting.
In Sec.
C we plot the game value in the Gaussian case for different parameter values.
In Sec.
D we provide a proof for Theorem 4.1, in Sec.
E we formally state the theorem discussed in Sec.
and provide proof, and in Sec.
F we prove Theorem 4.2 and also derive the game value for the sub-optimal ”ML attacker”.
Finally in Sec.
G,H we provide additional details about the experiments and the implementation of GIM.
B G AME FORMULATION VISUALIZATION
Our problem setup is illustrated in Figure 3 for a face authentication scenario.
A source θ (in this case an individual) generates IID observations (images).
k images are used by the authenticator to study the source, which it aims to authenticate.
m images are “leaked” and obtained by an attacker who wishes to impersonate the source and pass the authentication.
At test time the authenticator is presented with n images that were generated either by the true source θ , or by the attacker, and decides whether the entity that generated the images was the source θ or an attacker.
[Figure]
( (cid:3036) (cid:3036)(cid:2880)(cid:2869)(cid:3041) (cid:3037) (cid:3037)(cid:2880)(cid:2869)(cid:3038) )
[Figure]
source.
C G AME VALUE VISUALIZATIONS
In corollary 4.3 we provide the game value (the expected authenticator accuracy) for multivariate
Gaussian sources.
In this section we present additional visualizations of the game value as a function of the different parameters of the game.
Figure 4 visualizes the game value for different values of d (the observation dimension), as a function of δ “ mn , the “expansion ratio” between the source information available to the attacker through the leaked sample and the size of the attacker’s attack sample, and ρ “ mk , the information ratio between number of source observations available to attacker and authenticator.
One can clearly see that as d , the observation dimension, grows large, so does the accuracy of the authenticator.
Even for δ values higher then 0 .
5 , and ρ values which intuitively would give the attacker an advantage (e.g., m k “ 10 ).
Figure 5 visualizes the game value for different values of d (the observation dimension), as a function of δ “ mn and (cid:15) “ nk , the“information expansion” of the attacker with respect to the authenticator source information.
Again, one can clearly see that as d , the observation dimension, grows large, so does the accuracy of the authenticator.
D A DDITIONAL THEOREMS AND PROOFS FOR S EC . 4.
We begin with some additional notation.
Let
[Equation]
denote the marginal density of A , let Q Θ |
A denote the posterior probability over H given A .
That is:
[Equation]
Also let f Y | A , f X | A , g X | A , denote the conditional densities deﬁned by:
[Equation]
Lemma D.1.
Let G be an attacker deﬁned by the conditional probability distribution g X | Y .
Then @ a P X k , x P X n a best response strategy for D is:
[Equation]
Proof.
Given an attacker strategy g X | Y , the objective for D is given by:
[Equation]
Note that D can be optimized independently for each pair a, x P X k ˆ X n .
Hence, @ a, x P X k ˆ X n the objective is:
[Equation]
And thus the optimal decision rule for D is:
[Equation]
As required.
Lemma D.2.
@ a, x P X k ˆ X n let the strategy for D be deﬁned by:
[Equation]
Then G ˚ is a best response strategy for G , iff it minimizes the (cid:96) 1 distance between the distributions f
X | A , g X |
A over the space X k ˆ X n .
Namely:
[Equation]
Proof.
Let D be deﬁned as in Eq.
D.1, the objective for G is:
[Equation]
As required.
Where in Eq.
D.3 we used the deﬁnition of D .
Theorem 4.1.
Consider the attacker deﬁned by:
[Equation]
and let G ˚ be the corresponding map from X m to the set of probability distributions over X n .
Consider the authenticator deﬁned by:
[Equation]
where I is the indicator function.
Then p D ˚ , G ˚ q is a solution of Eq.
2.2 that satisﬁes Eq. 4.1.
Proof.
From Lemmas D.1, D.2, we have that max D V p D , G ˚ q “ V p D ˚ , G ˚ q “ min G V p D ˚ , G q form which it follows that Eq.
4.1 is satisﬁed and thus p D ˚ , G ˚ q is a solution of Eq.
E T HEOREM AND PROOF FOR S EC . 4.
Theorem E.1.
For all n ď m it holds that:
[Equation]
Proof.
Consider the attacker G replay deﬁned by the following generative process:
Given a leaked sample Y P R m ˆ d , G replay generates a sample X P R n ˆ d such that X i “ Y i @
i P r n s .
(This is possible since we assumed n ď m ).
Namely, we have:
[Equation]
Where δ is the Dirac delta.
Thus @ a, x P X k ˆ X n :
[Equation] [Equation]
ż
[Equation]
H
[Equation]
θ p y
[Equation]
Q
[Equation]
| A p θ | a q “ ż
[Equation]
dθ ż
[Equation]
dy n ź i “δ p x i ´ y
[Table]
q f p m q
[Table]
p y
[Table]
|
[Table]
p
[Table]
| a
[Table]
“ ż θ P H dθQ
[Table]
|
[Table]
p
[Table]
| a q ż
[Table]
ź
[Table] [Table]
p
[Table]
ż
[Table]
[Equation]
ż θ P H dθQ Θ | A p
[Table]
| a q ż
[Table]
n ź
[Table] [Table]
x
[Table]
´ y
[Table]
p y q
[Equation]
Deﬁne:
[Equation]
Then according to Theorem 4.1 p D 0 , G replay q is a solution of Eq.
2.2 that satisﬁes Eq. 4.1, and therefore: max
[Equation]
As required.
F A DDITIONAL THEOREMS AND PROOFS FOR S EC . 4. F. N OTATION AND DEFINITIONS
In this section we consider the case where the sources are d -dimensional Gaussian vectors with a known covariance matrix Σ “ CC T P R d ˆ d and unknown mean vector θ P R d .
That is the set of possible sources is H “ R d , and given θ P H the associated probability density over the domain
X
[Equation]
R d is
[Equation]
A sample of n examples x P X n , is considered a matrix x P R n ˆ d , where the ﬁrst dimension is the sample and the second is the dimension in the observation space, R d . i.e. x ij P R is the j ’th element of the i ’th example in the sample x P R n ˆ d .
We continue with a few more notations that simplify the proofs.
Given a matrix x P R n ˆ d , we let x c “ “ x T 1 , . . .
, x Tn ‰ T P R nd be the concatenation vector representing x .
Given a vector θ P R d we let θ c,n “ “ θ T , . . .
, θ T ‰ T P R nd be the concatenation of n copies of θ .
Given a matrix x P R n ˆ d we let ¯ x ” 1 n ř ni “ 1 x i P R d denote its mean along the sample dimension.
For any matrix
B P R d ˆ d , we denote:
[Equation]
and
[Equation]
Finally we deﬁne strategies for both attacker and authenticator which we prove in what follows to be the optimal strategies for the game.
Deﬁnition F.1.
Let G ˚ denote an attacker deﬁned by its associated conditional probability g ˚ X | Y as follows:
Given a leaked sample Y P R m ˆ d , G ˚ generates an attack sample X P R n ˆ d as follows.
It ﬁrst samples n vectors W 1 , . .
, W n iid „
N p 0 , Σ q
and then sets:
[Equation]
Deﬁnition F.2.
For any α P R ` , let D α denote an authenticator deﬁned as:
[Equation]
Where I is the indicator function
F. T ECHNICAL LEMMAS
Lemma F.3.
Let X 1 , . . .
, X k iid „ N p µ, Σ q
Where µ P R d , Σ P R d ˆ d .
Then
[Equation]
Proof.
We begin by observing that: X c „ N p µ c,k , diag p Σ , k qq .
Let B “ 1 k r I d ¨ ¨ ¨
I d s P R d ˆ kd and observe that ¯ X “ BX c .
Therefore, since this is an afﬁne transformation of a Gaussian vector we have:
[Equation]
As required.
Lemma F.4.
Let X P R d be a Gaussian vector s.t X „ N p µ, Σ q , and let X c,n P R nd be the concatenation of n copies of X .
Then:
[Equation]
Proof.
Let
[Equation]
And observe that X c,n “ BX .
Therefore, since this is an afﬁne transformation of a gaussuian vector we have:
[Equation]
As required.
Lemma F.5.
Let θ P R d , Σ P R d ˆ d represent the mean and covariance of a Gaussian distribution.
Let X P R n ˆ d be a random sample generated by the attacker deﬁned in Def. F.1.
Then:
[Equation]
Proof.
Observe that W c „ N p 0 , diag p Σ , n qq .
Using Lemma F.3 we get ¯ Y „ N p θ, 1 m Σ q and observe that: ¯ W c,n “ rep p 1 n I d , n q W c .
and using Lemma F.4 we get: ¯ Y c,n „ N p θ c,n , 1 m rep p Σ , n qq .
We deﬁne the following block matrices
[Equation]
and observe that:
[Equation]
Note that X c “ W c ´ ¯ W c,n ` ¯ Y c,n “ BZ , and therefore we get:
[Equation]
as required.
Lemma F.6.
Let Σ “ CC T P R d ˆ d represent the covariance of a Gaussian distribution, and consider the following covariance matrix:
[Equation]
Then its inverse is:
[Equation]
And the determinant is:
[Equation]
Proof.
We begin by deﬁning the following block matrices:
[Equation]
And note that rep pp n ´ mmn q Σ , n q “ UV T .
Then:
[Equation]
As required.
Where in p˚q we used the Woodbury matrix identity
[Equation]
and in p˚˚q we used the inverse of a diagonal block matrix.
Next
[Equation]
we turn to ﬁnding the determinant of Ψ :
[Equation]
Where in p˚q we used the matrix determinant lemma, and in p˚˚q we used the determinant of a diagonal block matrix.
Lemma F.7.
Let
[Equation]
And deﬁne the function:
[Equation]
Then ψ p µ q is log-concave over the space R d .
Proof.
We begin by noting that the function:
[Equation]
is convex w.r.t µ , hence its negative is concave, and by deﬁnition the function:
[Equation]
is log-concave w.r.t µ .
We now show that h p x, µ q is log concave w.r.t µ .
Let β P r 0 , 1 s , then we have that:
[Equation]
Therefore h p µ q is log-concave.
Where in p˚q we used Eq.
F.4, and in p˚˚q we used the fact that
I rt x
P R d : x T Σ ´ 1 x ă α us P t 0 , 1 u .
Finally, by using Prekopa-Leindler inequality (Prkopa, 1973)
we have that ψ p µ q is log concave, as required.
[Equation]
Lemma F.8.
Consider the attacker G ˚ , deﬁned in Def.
F.
The best response strategy for the authenticator against this attacker is:
[Equation]
Where
[Equation]
Proof.
The best response authenticator satisﬁes:
[Equation]
D can be chosen independently for each pair p a, x q P X k ˆ X n therefore for any p a, x q P X
k ˆ X n the decision rule for D p a, x q “ 1 is:
[Equation]
[Table]
Observing the LHS and using the improper uniform prior assumption we have:
[Equation]
[Table]
[Equation]
Observing the RHS and using the improper uniform prior assumption we have:
[Equation]
[Table] [Table]
[Equation]
Therefore the decision rule is
[Equation] [Equation]
As required.
Lemma F.9.
Consider the authenticator D α as deﬁned in Def.
F.2.
Then any attacker G , represented by a conditional probability g X | Y
[Equation]
that satisﬁes the condition: ¯ x “ ¯ y for any leaked sample y P
R m ˆ d
[Equation]
and attacker generated sample x P t R n ˆ d : g X | Y p x | y q ą 0 u satisﬁes:
[Equation]
Proof.
The best response attacker satisﬁes:
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Note that since g X | Y p x | y q can be chosen independently for each y P R m ˆ d
[Equation]
we can optimize it independently for each y P R m ˆ d
[Equation]
and we have:
[Equation]
j ´ θ qu
Note that for any PDF f over R n ˆ d and a function ϕ : R n ˆ d Ñ R
[Equation]
it holds that ş
[Equation]
dxf p x q ϕ p x q ď sup x ϕ p x q .
Therefore there exists a deterministic distribution g 1 X | Y p x | y q “ δ p x ´ x 1 q that achieves the maximum.
And thus its sufﬁcient to ﬁnd a vector x G that achieves the maximum:
[Equation]
[Table]
[Equation]
Where in p˚q we used the fact that Q p θ q is the improper uniform prior.
Note that the expression depends only on the mean ¯ x , therefore its sufﬁcient to ﬁnd an ¯ x that maximizes the expression.
We substitute the integration variable to ϕ “ ¯ a ´ ¯ x and obtain:
[Equation]
Where ψ is deﬁned as in Lemma F.7 (with σ “ m ` k mk )
[Equation]
from which it follows that ψ p µ q is logconcave
[Equation]
and therefore has at most one local extremum which can be only a maximum.
Therefore it is sufﬁcient to show that µ “ 0 (i.e ¯ x “ ¯ y ) is a local extremum by equating the gradient at the point to 0.
[Equation]
Therefore:
[Equation]
And since the domain of integration is symmetric about the origin and the integrand is odd with respect to the integration variable, the integral is equal to zero.
i.e BB µ ψ p µ q| µ “ 0 “ 0 .
Therefore ¯ x “ ¯ y ( µ “ 0 ) achieves the global maximum, and any attacker that satisﬁes the condition: ¯ x “ ¯ y for any leaked sample y P R m ˆ d and attacker generated sample x P t R n ˆ d : g X | Y p x | y q ą 0 u satisﬁes:
[Equation]
As required.
Corollary F.10.
Consider an authenticator D α as deﬁned in Def.
F.2.
Then the attacker G ˚ deﬁned in Def.
F.1 is a best response.
i.e.:
[Equation]
Proof.
Directly from Lemma F.
Theorem F.11.
The game value is:
[Equation]
Where γ is the lower incomplete gamma function.
Proof.
From the max-min inequality we have:
[Equation]
On the other hand using Lemma F.8 and Corollary F.10 we have:
[Equation]
And therefore:
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
The game value is given by:
[Equation]
Observing the ﬁrst term we have:
[Equation]
“p˚q
Observe that
[Equation]
And since:
[Equation]
q
Then:
[Equation]
We denote ˜ Z “ b nkn ` k Z „ N p 0 d , I d q , and thus ˜ Z 1 , . . .
, ˜ Z d are independent standard normal random variables and ˜ Z T ˜
Z „ χ 2 p d q .
Therefore Z T Z “ n ` k nk ˜ Z T ˜ Z „ Γ p k “ d 2 , θ “ 2 n ` k nk q
and:
[Equation]
Where in p i q We used the CDF of the Gamma distribution in which γ is the lower incomplete gamma function.
Similarly, observing the second term we have:
[Equation]
Where:
[Equation]
Using the deﬁnition of G ˚ (Deﬁnition F.1) and Lemma F.3 we have: „
[Equation]
Therefore:
[Equation]
And similarly to the ﬁrst term we get:
[Equation]
And thus:
[Equation]
Therefore the game value is given by:
[Equation]
As required.
Finally we prove Theorem 4.2 and Corollary 4.3.
Theorem 4.2.
Deﬁne δ “ m { n ď 1 and let ρ “ m { k .
Consider the attacker G ˚ deﬁned by the following generative process:
Given a leaked sample Y P R m ˆ d , G ˚ generates a sample X P R n ˆ d as follows: it ﬁrst samples n vectors W 1 , . .
, W n iid „
N p 0 , Σ q
and then sets: X i “ W i ´ ¯ W ` ¯ Y .
Deﬁne the authenticator D ˚ by:
[Equation]
Then p D ˚ , G ˚ q is a solution of Eq.
2.2 that satisﬁes Eq. 4.1.
Proof.
Directly from Lemma F.8, Corollary F.10, and Theorem F.11 by assigning δ “ mn , ρ “ mk .
Corollary 4.3.
Deﬁne δ and ρ as in Theorem 4.2.
Then the game value for the Gaussian case is:
[Equation]
Where γ is the lower incomplete gamma function, and Γ is the Gamma function.
Proof.
Directly from Theorem F.11 by assigning δ “ m n , ρ “ m k .
F. G AME VALUE FOR A MAXIMUM LIKELIHOOD ATTACKER
In this section we show that the intuitive “optimal attacker” is sub optimal as can be seen in Figure 1c in the main paper.
We consider an attacker that draws the attack sample from a Gaussian distribution with the maximum likelihood estimate of the mean and the known covariance.
We denote this attacker by the name ML attacker.
We ﬁnd the best response autenticator for this attacker and associated the game value.
Figure 6 visualizes the difference in theoretical game value between the
ML attacker (see Deﬁnition F.12) and the optimal attacker (see Deﬁnition F.1) for different values of d (the observations dimension), and demonstrates that the ML attacker is indeed sub-optimal.
[Figure]
Deﬁnition F.12.
Let G ML denote an attacker deﬁned by the following generative process: Given a leaked sample Y P R m ˆ d , G ML generates an attack sample X iid „ N p ¯ Y , Σ q
Lemma F.13.
Let θ P R d , Σ P R d ˆ d represent the mean and covariance of a Gaussian distribution.
Let X P R n ˆ d be a random sample generated by the attacker deﬁned in Def.
F.12.
Then:
[Equation]
Proof.
Let W 1 , . .
, W n iid „
N p 0 , Σ q , observe that X i “ ¯ Y ` W i @
i P r n s , and thus:
[Equation]
c “ ¯ Y c,n ` W c
Where W c „ N p 0 ¨ 1 dn , diag p Σ , n q .
Using Lemma F.3 we have ¯ Y „ N p θ, 1 m Σ q , and using Lemma
F.4 we have ¯ Y c,n „ N p θ c,n , rep p 1 m Σ , n qq .
Let Z “ „
W c ¯ Y c,n  and B “ r I nd ˆ nd , I nd ˆ nd s , then X c “ W c ` ¯ Y c,n “ BZ .
Note that:
[Equation]
q and therefore we have:
[Equation]
as required.
Lemma F.14.
Let Σ “ CC T P R d ˆ d represent the covariance of a Gaussian distribution, and consider the following covariance matrix: Ψ ML “ diag p Σ , n q ` rep p 1 m Σ , n q .
Then:
[Equation]
And the determinant is:
[Equation]
Proof.
To ﬁnd the inverse of Ψ ML we ﬁrst deﬁne
[Equation]
then we have:
[Equation]
As required.
Where in p i q we used the Woodbury matrix identity, and in p ii q we used the inverse of a diagonal block matrix.
Next, we turn to ﬁnding the determinant of Ψ ML :
[Equation] [Equation]
q
[Equation]
Where in p iii q we used the matrix determinant lemma
[Equation]
and in p iv q we used the determinant of a diagonal block matrix.
Lemma F.15.
Consider the attacker G ML , deﬁned in F [Equation]
The best response strategy for the authenticator against this attacker is:
[Equation]
Where:
[Equation]
Proof.
The best response authenticator satisﬁes:
[Equation]
ı
[Equation]
[Equation]
Lemma F.
[Equation]
D can be chosen independently for each pair p a
[Equation]
x q P X k ˆ X n therefore for any p a, x q P
X
k
ˆ X n the decision rule for D p a [Equation] x q “ 1 is:
[Equation]
[Table] [Table]
the LHS and using the improper uniform prior assumption we obtain:
[Table]
[Equation]
[Table]
the RHS and
[Table]
the improper uniform prior assumption we have:
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Where in p i q we denoted v “ nm ¯ x ` k p n ` m q ¯ a nm ` nk ` mk .
Therefore the decision rule is
[Equation] [Equation]
As required.
Theorem F.16.
Fix the attacker to be G ML as deﬁned in F.12, then the game value is:
[Equation]
Where ρ “ mk , δ “ mn
[Equation]
and γ is the lower incomplete gamma function
Proof.
The game value is given by:
[Equation]
ı
Observing the ﬁrst term
[Equation]
we can see that by replacing α ˚ with α ML in the analogue part of the proof for Theorem F.11 we get:
[Equation]
Again, similarly to analogue part of the proof for Theorem F.
[Equation]
observing the second term we have:
[Equation]
Where:
[Equation]

Using the deﬁnition of G ML (Deﬁnition F.12) we have ¯ X „ N p θ, n ` m nm Σ q , using Lemma F.3 we have ¯ A „ N p θ, 1 k Σ q , and thus: „
[Equation] [Equation]
q
Therefore:
[Equation]
We denote ˜ V “ b nmk nm ` nk ` mk V and thus ˜ V 1 , . . .
, ˜ V d are independent standard normal random variables and ˜ V T ˜ V „ χ 2 p d q .
Therefore
[Equation]
q
And thus:
[Equation]
Therefore the game value is given by:
[Equation]
required.
G E XPERIMENTS - DATASETS
Below we provide details on the datasets used in the authentication on faces and characters experiments.
The VoxCeleb2 (Nagrani et al., 2017; Chung & Zisserman, 2018) dataset contains cropped face videos of 6112 identities.
We used the original split of 5994 identities for training and 118 for test.
For each identity we saved every ﬁfth frame and resized each frame to 64 ˆ 64 and augmented it using horizontal ﬂip.
The Omniglot dataset (Lake et al., 2015) contains handwritten character images from 50 alphabets.
There are 1623 different characters, and 20 examples for each character.
We use the splits and augmentations suggested by Vinyals et al.
and used by Snell et al.
H E XPERIMENTS - IMPLEMENTATION DETAILS
In this section we describe our implementation of the GIM model in detail for the different experiments.
Recall from Sec.
5, that in general the authenticator is a neural network D p a, x q that can be expressed as D p a, x q “ σ p T D p a q , T D p x qq , and the generator is a neural network G p y q that can be expressed as G p Y q i “ ϕ p
W i ´ ¯ W ` T G p Y qq @
i P r n s .
In what follows we describe our implementation of these models for each of the experiments.
H. G AUSSIAN SOURCES
Authenticator Architecture: For the statistic function T D , we use a concatenation of the mean and standard deviation of the sample.
For the comparison function σ , we use the element-wise absolute difference between the statistics T D p a q , T D p x q , followed by a linear layer.
Attacker Architecture:
For the statistic function T G we use the sample mean, i.e T G p y q “ ¯ y .
The noise vectors W i are generated as follows:
First, n Gaussian noise vectors Z 1 , . . .
, Z n „ N p 0 , I d q are drawn, then each vector Z i is passed through a linear layer to obtain W i .
Finally, the decoder ϕ is the identity function.
Optimization details
The model is trained in an authentication setup as in our theoretical setup, using alternating gradient descent as is common in GAN optimization (Mescheder et al., 2018).
Each iteration begins when a source θ P R d is drawn from the prior distribution Q “ N p 0 ,
10 I d q .
Samples A P R k ˆ d , Y P R m ˆ d , X θ P R n ˆ d are drawn IID from f θ “ N p θ, I d q , where X θ represents a real sample from θ .
The attacker, given the leaked sample Y , generates a fake sample X G “ G p Y q P R n ˆ d , passes it to D , and suffers the loss ´ log p sigmoid p D p A , X G qqq .
The authenticator, D receives as input the source information sample
A , outputs a prediction for each of the test samples X θ , X G , and suffers the binary cross entropy loss.
´ 0 .
5 p log p sigmoid p D p A ,
X θ qqq ` log p sigmoid p 1 ´ D p A , X G qqqq .
Each experiment is trained for 200 K iterations with a batch size of 4000 using the Adam optimizer (Kingma & Ba, 2015) with learning rate 10 ´ 4 .
H. E XPERIMENTS ON V OXCELEB 2 AND O MNIGLOT
To describe the models we begin with some notation.
We let c denote the number of image channels, h denote the image size (we only consider square images of size c ˆ h ˆ h ), and l denote the latent dimension of the model.
Authenticator Architecture: The authenticator model consists of the following components: Two image encoders E D ,i : r 0 , 1 s c ˆ h
ˆ h Ñ R l that map an image to a latent vector, a statistic module ζ that maps a set of latent vectors to a single ﬁxed sized vector of size 5 l , and the comparator module σ that takes in two latent vectors and outputs a scalar.
The exact speciﬁcation of these components can be seen in the published code.
The statistic function T D maps a sample of n images x P r 0 , 1 s n ˆ c
ˆ h
ˆ h to a statistic vector T D p x q P R 5 l as follows:
Every image X i in the sample is mapped using E D , 1 and E D , 2 to latent vectors v i , u i P R l .
The latent sample u is passed through ζ to obtain a single vector ˜ u P R 5 l which gets concatenated to ¯ v , the sample mean of v .
Namely,
T D
[Equation]
The comparison function σ : R 5 l Ñ R receives two latent vectors s A , s X representing the statistics of the samples A and X respectively.
The vectors are concatenated and then passed through a
Multi-Layered Perceptron which outputs a scalar reﬂecting their similarity.
Namely:
[Equation]
The full architecture of the authenticator is depicted in Fig.
Attacker Architecture: Our implementation of the attacker is inspired by the architecture suggested in (Zakharov et al., 2019), which relies on an implicit assumption that an image could be modeled as a mapping of two latent vectors to image space.
The ﬁrst vector represents the source θ and is the same for any image of θ , the second vector represents the environment (e.g pose, lighting, expression) and is different for each image of the source.
The attacker model consists of the following components:
An image encoder E G ,src : r 0 , 1 s c ˆ h ˆ h Ñ R l that maps an image to a latent vector representing the source θ .
an image encoder
E G ,env : r 0 , 1 s c ˆ h
ˆ h Ñ R l that maps an image to a latent vector representing the environment, a Multi-layered Perceptron MLP G : R l Ñ R l that maps gaussian noise to the environment latent space, an environment decoder ϕ env : R l Ñ R c
ˆ h
ˆ h that maps a latent vector to an environment image, which could represent aspects of the environment such as facial landmarks 5 , and ﬁnally a generator φ : R 2 c
ˆ h
ˆ h
ˆ R l Ñ r 0 , 1 s c ˆ h
ˆ h that maps an environment image concatenated to the real image to a new image.
The generator is based on the image to image model used in (Zakharov et al., 2019; Johnson et al., 2016) and uses the source latent vector as input for Adaptive instance normalization (Huang & Belongie, 2017).
The exact speciﬁcation of these components can be seen in the published code.
The attacker generates a fake sample
X P r 0 , 1 s n ˆ c
ˆ h ˆ h based on a leaked sample Y P r 0 , 1 s m ˆ c
ˆ h
ˆ h in the following way: Each image Y j in the leaked sample is mapped using E G ,src and E G ,env to latent vectors src j , env j P R l .
A latent environment vector, ˜ env i , for each fake image
X i is constructed in the following way: First, n IID Gaussian noise vectors Z 1 , . . .
, Z n „ N p 0 , I l q are drawn, then each vector Z i is passed through MLP G to obtain W i , and ﬁnally ˜ env i is obtained by matching the mean of the latent environment vectors to ¯ env .
Namely:
[Equation]
useful
Each fake image X i is then generated deterministically from the latent vectors src, ˜ env i .
˜ env i is used as input to the decoder ϕ env which outputs an environment image.
This image is concatenated along the channel dimension to a random image from the leaked sample Y , and then passed as input to the generator φ , which also receives src as input to its Adaptive instance norm layers.
The output of the generator is the fake image
X i for all i P r n s .
The full architecture of the attacker is depicted in Fig. 8.
Optimization details:
The model is trained in an authentication setup as in our theoretical setup, using alternating gradient descent with the regularization parameter as suggested by
Mescheder et al.
Each iteration begins when a source θ P R d is drawn uniformly from the dataset.
Samples A P r´ 1 , 1 s k ˆ c
ˆ h
ˆ h , Y P r´ 1 , 1 s m ˆ c
ˆ h
ˆ h ,
X θ P r´ 1 , 1 s n ˆ c ˆ
h ˆ h are sampled uniformly from the images available for the source θ .
The attacker, given the leaked sample Y , generates a fake sample X G “ G p Y q , passes it to D , and suffers the loss ´ log p sigmoid p D p A , X G qqq .
The authenticator, D receives as input the source information sample A , outputs a prediction for each of the test samples
X θ , X G , and suffers the binary cross entropy loss ´ 0 .
5 p log p sigmoid p D p A ,
X θ qqq ` log p sigmoid p 1 ´ D p A , X G qqqq .
The experiments on Omniglot were trained for 520 k iterations with batch size 128 using the Adam optimizer (Kingma & Ba, 2015) with learning rate 10 ´ 6 for D , 10 5 for G and 10 7 for MLP G (as done in (Karras et al., 2018b)).
The regularization parameter was set to 0 .
The experiments on Voxceleb2 were trained for 250 k iterations with batch size 64 using the Adam optimizer (Kingma & Ba, 2015) with learning rate 10 ´ 4 for both D and G and 10 6 for MLP G .
The regularization parameter was set to 10 (as done in (Karras et al., 2018b)) since we noticed that it stabilized and sped up the training, and in contrast to Omniglot and the Gaussian experiments did not seem to hurt the results.