B AYESIAN F RAMEWORK FOR G RADIENT L EAKAGE
A BSTRACT
Federated learning is an established method for training machine learning models without sharing training data.
However, recent work has shown that it cannot guarantee data privacy as shared gradients can still leak sensitive information.
To formalize the problem of gradient leakage , we propose a theoretical framework that enables, for the ﬁrst time, analysis of the Bayes optimal adversary phrased as an optimization problem.
We demonstrate that existing leakage attacks can be seen as approximations of this optimal adversary with different assumptions on the probability distributions of the input data and gradients.
Our experiments conﬁrm the effectiveness of the Bayes optimal adversary when it has knowledge of the underlying distribution.
Further, our experimental evaluation shows that several existing heuristic defenses are not effective against stronger attacks, especially early in the training process.
Thus, our ﬁndings indicate that the construction of more effective defenses and their evaluation remains an open problem.
I NTRODUCTION
Federated learning (McMahan et al., 2017) has become a standard paradigm for enabling users to collaboratively train machine learning models.
In this setting, clients compute updates on their own devices, send the updates to a central server which aggregates them and updates the global model.
Because user data is not shared with the server or other users, this framework should, in principle, offer more privacy than simply uploading the data to a server.
However, this privacy beneﬁt has been increasingly questioned by recent works (Melis et al., 2019; Zhu et al., 2019;
Geiping et al., 2020;
Yin et al., 2021) which demonstrate the possibility of reconstructing the original input from shared gradient updates.
The reconstruction works by optimizing a candidate image with respect to a loss function that measures the distance between the shared and candidate gradients.
The attacks typically differ in their loss function, their regularization, and how they solve the optimization problem.
Importantly, the success of these attacks raises the following key questions: (i) what is the theoretically worst-case attack ?, and (ii) how do we evaluate defenses against gradient leakage ?
This work In this work, we study and address these two questions from both theoretical and practical perspective.
Speciﬁcally, we ﬁrst introduce a theoretical framework which allows us to measure the expected risk an adversary has in reconstructing an input, given the joint probability distribution of inputs and their gradients.
We then analyze the Bayes optimal adversary, which minimizes this risk and show that it solves a speciﬁc optimization problem involving the joint distribution.
Further, we phrase existing attacks (Zhu et al., 2019;
Geiping et al., 2020;
Yin et al., 2021) as approximations of this optimal adversary, where each attack can be interpreted as implicitly making different assumptions on the distribution of gradients and inputs, in turn yielding different loss functions for the optimization.
In our experimental evaluation, we compare the Bayes optimal adversary with other attacks, those which do not leverage the probability distribution of gradients, and we ﬁnd that the Bayes optimal adversary performs better, as explained by the theory.
We then turn to practical evaluation and experiment with several recently proposed defenses (Sun et al., 2021; Gao et al., 2021; Scheliga et al., 2021) based on different heuristics and demonstrate that they do not protect from gradient leakage against stronger attacks that we design speciﬁcally for each defense.
Interestingly, we ﬁnd that models are especially vulnerable to attacks early in training, and thus we advocate that defense evaluation should take place during and not only at the end of training.
Overall, our ﬁndings suggest that creation of effective defenses and their evaluation is a challenging problem, and that our insights and contributions can substantially advance future research in the area.
Main contributions Our main contributions are:
• Formulation of the gradient leakage problem in a Bayesian framework which enables phrasing Bayes optimal adversary as an optimization problem.
• Interpretation of several prior attacks as approximations of the Bayes optimal adversary, each using different assumptions for the distributions of inputs and their gradients.
• Practical implementation of the Bayes optimal adversary for several defenses, showing higher reconstruction success than previous attacks, conﬁrming our theoretical results.
We make our code publicly available at https://github.com/eth-sri/ bayes-framework-leakage .
• Evaluation of several existing heuristic defenses demonstrating that they do not effectively protect from strong attacks, especially early in training, thus suggesting better ways for evaluating defenses in this space.
R ELATED WORK
We now brieﬂy survey some of the work most related to ours.
Federated Learning Federated learning (McMahan et al., 2017) was introduced as a way to train machine learning models in decentralized settings with data coming from different user devices.
This new form of learning has caused much interest in its theoretical properties (Koneˇcn´y et al., 2016a) and ways to improve training efﬁciency (Koneˇcn´y et al., 2016b).
More speciﬁcally, besides decentralizing the computation on many devices, the fundamental promise of this approach is privacy, as the user data never leaves their devices.
Gradient Leakage Attacks Recent work (Zhu et al., 2019;
Geiping et al., 2020) has shown that such privacy assumptions, in fact, do not hold in practice, as an adversarial server can reliably recover an input image from gradient updates.
Given the gradient produced from an image x and a corresponding label y , they phrase the attack as a minimization problem over the (cid:96) 2 -distance between the original gradient and the gradient of a randomly initialized input image x (cid:48) and label y (cid:48) : ( x ∗ , y ∗ ) = arg min ( x
[Equation]
Here h θ is a neural network and l denotes a loss used to train the network, usually cross-entropy.
Follow-up works improve on these results by using different distance metrics such as cosine similarity and input-regularization (Geiping et al., 2020), smarter initialization (Wei et al., 2020), normalization (Yin et al., 2021), and others (Mo et al., 2021; Jeon et al., 2021).
A signiﬁcant improvement proposed by Zhao et al.
(2020) showed how to recover the target label y from the gradients alone, reducing Eq.
(1) to an optimization over x (cid:48) only.
In Section 4 we show how these existing attacks can be interpreted as different approximations of the Bayes optimal adversary.
In response to the rise of privacy-violating attacks on federated learning, many defenses have been proposed (Abadi et al., 2016; Sun et al., 2021; Gao et al., 2021).
Except for DP-SGD (Abadi et al., 2016), a version of SGD with clipping and adding Gaussian noise, which is differentially private, they all provide none or little theoretical privacy guarantees.
This is partly due to the fact that no mathematically rigorous attacker model exists, and defenses are empirically evaluated against known attacks.
This also leads to a wide variety of proposed defenses: Soteria (Sun et al., 2021) prunes the gradient for a single layer, ATS (Gao et al., 2021) generates highly augmented input images that train the network to produce non-invertible gradients, and PRECODE (Scheliga et al., 2021) uses a VAE to hide the original input.
Here we do not consider defenses that change the communication and training protocol (Lee et al., 2021; Wei et al., 2021).
B ACKGROUND
Let X ⊆
R
d be an input space, and let h θ : X → Y be a neural network with parameters θ classifying an input x to a label y in the label space Y .
We assume that inputs ( x, y ) are coming from a distribution D with a marginal distribution p ( x ) .
In standard federated learning, there are n clients with loss functions l 1 , ..., l n , who are trying to jointly solve the optimization problem:
[Equation]
To ease notation, we will assume a single client throughout the paper, but the same reasoning can be applied to the general n -client case.
Additionally, each client could have a different distribution
D , but the approach is again easy to generalize to this case.
In a single training step, each client i ﬁrst computes ∇ θ l i ( h θ ( x i ) , y i ) on a batch of data ( x i , y i ) , then sends these to the central server that performs a gradient descent step to obtain the new parameters θ (cid:48)
= θ − αn (cid:80)
ni =1 ∇ θ l i ( h θ ( x i ) , y i ) , where α is a learning rate.
We will consider a scenario where each client reports, instead of the true gradient ∇ θ l i ( h θ ( x i ) , y i ) , a noisy gradient g sampled from a distribution p ( g | x ) , which we call a defense mechanism.
The purpose of the defense mechanism is to add enough noise to hide the sensitive user information from the gradients while retaining high enough informativeness so that it can be used for training.
Thus, given some noisy gradient g , the central server would update the parameters as θ (cid:48)
=
θ − αg (assuming n = 1 as mentioned above).
Typical examples of defenses used in our experiments in Section 6, each inducing p ( g | x ) , include adding Gaussian or Laplacian noise to the original gradients, as well as randomly masking some components of the gradient.
This setup also captures the common DP-SGD defense (Abadi et al., 2016) where p ( g | x ) is a Gaussian centered around the clipped true gradient.
Naturally, p ( x ) and p ( g | x ) together induce a joint distribution p ( x, g ) .
Note that a network that has no defense corresponds to the distribution p ( g | x ) , which is concentrated only on the true gradient at x .
B AYESIAN ADVERSARIAL FRAMEWORK
Here we describe our theoretical framework for gradient leakage in federated learning.
As we show, our problem setting captures many commonly used defenses such as DP-SGD (Abadi et al., 2016) and many attacks from prior work (Zhu et al., 2019;
Geiping et al., 2020).
Adversarial risk We ﬁrst deﬁne the adversarial risk for gradient leakage and then derive the Bayes optimal adversary that minimizes this risk.
The adversary can only observe the gradient g and tries to reconstruct the input x that produced g .
Formally, the adversary is a function f : R k
→ X mapping gradients to inputs.
Given some ( x, g ) sampled from the joint distribution p ( x, g ) , the adversary outputs the reconstruction f ( g ) and incurs loss L ( x, f ( g )) , which is a function L :
X × X → R .
Typically, we will consider a binary loss that evaluates to 0 if the adversary’s output is close to the original input, and 1 otherwise.
If the adversary wants to reconstruct the exact input x , we can deﬁne the loss to be L ( x, x (cid:48) ) :
= 1 x (cid:54) = x (cid:48) , denoting with 1 the indicator function.
If the adversary only wants to get to some δ -neighbourhood of the input x in the input space, a more appropriate deﬁnition of the loss is L ( x, x (cid:48) ) :
= 1 d ( x,x (cid:48) ) >δ .
In this section, we will assume that the distance d is the (cid:96) 2 -distance, but the approach can be generalized to other notions as well.
This deﬁnition is well suited for image data, where (cid:96) 2 distance captures our perception of visual closeness, and for which the adversary can often obtain a reconstruction that is very close to the original image, both visually and in the (cid:96)space.
Note that if we let δ approach 0 in our second deﬁnition of the loss, we essentially recover the ﬁrst loss.
We can now deﬁne the risk R ( f ) of the adversary f as
[Equation]
Bayes optimal adversary
We consider white-box adversary who knows the joint distribution p ( x, g ) , as opposed to the weaker alternative based on security through obscurity, where adversary does not know what defense is used, and therefore does not have a good estimate of p ( x, g ) .
We also do not consider adversaries that can exploit other vulnerabilities in the system to obtain extra information.
Let us consider the second deﬁnition of the loss for which L ( x, f ( g )) :
= 1 || x − f ( g )
|| 2 >δ .
[Figure]
We can then rewrite the deﬁnition of the risk as follows:
[Equation]
Here B ( f ( g ) , δ ) denotes the (cid:96) 2 -ball of radius δ around f ( g ) .
Thus, an adversary which wants to minimize their risk has to maximize E g (cid:82) B ( f ( g ) ,δ ) p ( x | g ) dx , meaning that the adversarial function f can be deﬁned as f ( g ) :
= arg max x 0 (cid:82) B ( x 0 ,δ ) p ( x | g ) dx .
Intuitively, the adversary predicts xwhich has the highest likelihood that one of the inputs in its δ -neighborhood was the original input that produced gradient g .
Note that if we would let δ → 0 , then f ( g ) → arg max x p ( x | g ) , which would be the solution for the loss that requires the recovered input to exactly match the original input.
As we do not have the closed form for p ( x | g ) , it can be rewritten using Bayes’ rule:
[Equation]
Computing the optimal reconstruction now requires evaluating both the input prior p ( x ) and the conditional probability p ( g | x ) , which is determined by the used defense mechanism.
Given these two ingredients, Eq.
(2) then provides us with a way to compute the output of the Bayes optimal adversary by solving an optimization problem involving distributions p ( x ) and p ( g | x ) .
Approximate Bayes optimal adversary
While Eq.
(2) provides a formula for the optimal adversary in the form of an optimization problem, using this adversary for practical reconstruction is difﬁcult due to three main challenges: (i) we need to know the exact prior distribution p ( x ) , (ii) in general computing the integral over the δ -ball around x 0 is intractable, and (iii) we need to solve the optimization problem over X .
However, we can address each of these challenges by introducing [Table] appropriate approximations.
We ﬁrst apply Jensen’s inequality to the logarithm function:
[Equation]
Here C =
1 /δ d is a normalization constant.
For image data, we approximate log p ( x ) using the total variation image prior, which has already worked well for Geiping et al.
Alternatively, we could estimate it from data using density estimation models such as PixelCNN (van den Oord et al., 2016) or Glow (Kingma & Dhariwal, 2018).
We then approximate the integral over the δ -ball via Monte Carlo integration by sampling k points x 1 , ..., x k uniformly in the ball to obtain the objective:
[Equation]
Finally, as the objective is differentiable, we can use gradient-based optimizer such as
Adam (Kingma & Ba, 2015), and obtain the attack in Algorithm 1.
Fig.
1 shows a single run of this adversary which is initialized randomly, and gets closer to the original image at every step.
x (1) ← attack init() for i = 1 to m − 1 do
Sample x 1 , ..., x k uniformly from B ( x ( i ) , δ ) x
[Equation]
+
[Equation]
← [Equation] i ) Existing attacks as approximations
of the Bayes optimal adversary
We now describe how existing attacks can be viewed as different approximations of the Bayes optimal adversary.
Recall that the optimal adversary f searches for the δ -ball with the maximum value for the integral of log p ( g | x ) + log p ( x ) .
Next we show previously proposed attacks can in fact be recovered by plugging in different approximations for log p ( g | x ) and log p ( x ) into Algorithm 1, estimating the integral using k = 1 samples located at the center of the ball.
For example, suppose that the defense mechanism adds Gaussian noise to the gradient, which corresponds to the conditional probability p ( g | x ) =
N ( ∇ θ l ( h θ ( x ) , y ) , σ 2 I ) .
This implies log p ( g | x ) =
C − 12 σ 2 || g − ∇ θ l ( h θ ( x ) , y ) || 22 , where C is a constant.
Assuming a uniform prior (where log p ( x ) is constant), the problem in Eq.
(2) simpliﬁes to minimizing the (cid:96) 2 distance between g and ∇ θ l ( h θ ( x ) , y ) , recovering exactly the optimization problem solved by Deep Leakage from Gradients (DLG) (Zhu et al., 2019).
Inverting Gradients (Geiping et al., 2020) uses a total variation (TV) image prior for log p ( x ) and cosine similarity instead of (cid:96) 2 to measure similarity between gradients.
Cosine similarity corresponds to the distribution log p ( g | x ) =
C − [Table]
|which also requires the support of the distribution to be bounded. This happens, e.g., when gradients are clipped.
GradInversion (Yin et al., 2021) introduces a more complex prior based on a combination of the total variation, (cid:96) 2 norm of the image, and a DeepInversion prior while using l 2 norm to measure distance between gradients, which corresponds to Gaussian noise, as observed before.
Note that we can work with unnormalized densities, as multiplying by the normalization constant does not change the argmax in Eq.
We summarize these observations in Table 1, showing the respective conditional and prior probabilities for each of the discussed attacks.
B REAKING EXISTING DEFENSES
In this section we provide practical attacks against three recent defenses, showing they are not able to withstand stronger adversaries early in training.
While in Section 4 we have shown that Bayes optimal adversary is the optimal attack for any defense (each with different p ( g | x ) ), we now show how practical approximations of this attack can be used to attack existing defenses (more details in Appendix A.5).
In Section 6 we show experimental results obtained using our attacks, demonstrating the need to evaluate defenses against closest possible approximation to the optimal attack.
Soteria The Soteria defense (Sun et al., 2021) perturbs the intermediate representation of the input at a chosen defended layer l of the attacked neural network H : R n 0 → R n L with L layers of size n 1 , . . .
, n L and input size n 0 .
Let
X and X (cid:48) ∈
R n 0 denote the original and reconstructed images on
H and h i,j : R n i
→ R n j denote the function between the input of the i th layer of H and the output of the j th .
For the chosen layer l , Sun et al.
(2021) denotes the inputs to that layer for the images X and X (cid:48) with r = h 0 ,l − 1 ( X ) and r (cid:48)
=
h 0 ,l − 1 ( X (cid:48) ) , respectively, and aims to solve
[Equation]
Intuitively, Eq.
(4) is searching for a minimal perturbation of the input to layer l that results in maximal perturbation of the respective reconstructed input X (cid:48) .
Despite the optimization being over the intermediate representation r (cid:48) , for an attacker who observes neither r nor r (cid:48) the defense amounts to a perturbed gradient at layer l .
In particular let ∇ W = {∇ W 1 , ∇ W 2 , . . .
∇ W L } be the set of gradients for the variables in the different layers of H .
Sun et al.
ﬁrst solves Eq.
(4) to obtain r (cid:48) .
It afterwards uses this r (cid:48) to generate a perturbed gradient at layer l denoted with ∇ W (cid:48) l .
Notably r (cid:48) is not propagated further through the network and hence the gradient perturbation stays local to the defended layer l .
Hence, to defend the data X , Soteria’s clients send the perturbed set of gradients ∇ W (cid:48)
= {∇ W 1 , . . .
, ∇ W l − 1 , ∇ W (cid:48) l , ∇ W l +1 , . . .
∇ W L } in place of ∇ W .
Sun et al.
(2021) show that Soteria is safe against several attacks from prior work (Zhu et al., 2019;
Geiping et al., 2020).
In this work, we propose to circumvent this limitation by dropping the perturbed gradients ∇ W (cid:48)
l from ∇ W (cid:48) to obtain ∇ W ∗ = {∇ W 1 , . .
. , ∇ W l − 1 , ∇ W l +1 , . . .
∇ W L } .
As long as ∇ W ∗ contains enough gradients, this allows an attacker to compute an almost perfect reconstruction of X .
Note that the attacker does not know which layer is defended, but can simply run the attack for all.
Automated Transformation Search
The Automatic Transformation Search (ATS) (Gao et al., 2021) attempts to hide sensitive information from input images by augmenting the images during training.
The key idea is to score sequences of roughly 3 to 6 augmentations from AutoAugment library (Cubuk et al., 2019) based on the ability of the trained network to withstand gradient-based reconstruction attacks and the overall network accuracy.
Similarly to Soteria, Gao et al.
(2021) also demonstrate that ATS is safe against attacks proposed by Zhu et al.
(2019) and Geiping et al.
In this work we show that, even though the ATS defense works well in later stages of training, in initial communication rounds we can easily reconstruct the input images using Geiping et al.
(2020)’s attack.
Our experiments in Section 6 on the network architecture and augmentations introduced in
Gao et al.
(2021) indicate that an attacker can successfully extract large parts of the input despite heavy image augmentation.
The main observation is that the gradients of a randomly initialized network allow easier reconstruction independent of the input.
Only after training, which changes gradient distribution, can defenses such as ATS empirically defend against gradient leakage.
PRECODE PRECODE (Scheliga et al., 2021) is a proposed defense which inserts a variational bottleneck between two layers in the network.
Given an input x , PRECODE ﬁrst encodes the input into a representation z
= E ( x ) , then samples bottleneck features b ∼ q ( b | z ) from a latent Gaussian distribution.
Then, they compute new latent representation ˆ z = D ( b ) , and ﬁnally obtain the output
ˆ y
=
O
(ˆ z ) .
For this defense we focus on an MLP network, which Scheliga et al.
(2021) evaluates against several attacks and shows that images cannot be recovered.
Our attack is based on the attack presented in Phong et al.
(2017), later extended by Geiping et al.
(2020) – it shows that in most cases the inputs to an MLP can be perfectly reconstructed from the gradients.
Next we detail this attack as presented in Phong et al.
; Geiping et al.
Let the output of ﬁrst layer of the MLP (before the activation) be y
=
A 0 x T + b 0 , with x ∈ R n 0 , [Figure] y 0 , b 0 ∈ R n 1 and A 0 ∈ R n 0 × n 1 .
Assume a client sends the unperturbed gradients dldA 0 and dldb 0 for that input layer.
Let ( y 0 )
i and ( b 0 )
i denote the i th entry in these vectors.
By the chain rule we have:
[Equation]
Combining this result with the chain rule for
[Equation]
: we further obtain:
[Equation]
where ( A 0 ) i, : denotes the i -th line of the matrix A 0 .
Assuming dl d
[Equation]
b 0 ) i (cid:54) = 0 , we can precisely calculate x
[Equation]
dl d
[Equation]
0 ) i, : as there is typically at least one non-zero entry in b 0 .
E XPERIMENTAL EVALUATION
We now evaluate existing defenses against strong approximations of the Bayes optimal adversary.
Evaluating existing defenses In this experiment, we evaluate the three recently proposed defenses described in Section 5, Soteria, ATS and PRECODE, on the CIFAR-10 dataset (Krizhevsky, 2009).
For ATS and Soteria, we use the code from their respective papers.
In particular, we evaluated ATS on their ConvNet implementation, a network with 7 convolutional layers, batch-norm, and ReLU activations followed by a single linear layer.
We consider 2 different augmentation strategies for ATS, as well as a hybrid strategy.
For Soteria, we evaluate our own network architecture withconvolutional and 3 linear layers.
The defense is applied on the largest linear layer, directly after the convolutions.
Both attacks are implemented using the code from Geiping et al.
(2020), and using cosine similarity between gradients, the Adam optimizer (Kingma & Ba, 2015) with a learning rate of 0 . 1 , a total variation regularization of 10 − 5 for ATS and 4 × 10 − 4 for Soteria, as well as 2000 and 4000 attack iterations respectively.
We perform the attack on both networks using batch size 1.
For PRECODE, which has no public code, we use our own implementation.
We use MLP architecture consisting of 5 layers with 500 neurons each and ReLU activations.
Following Scheliga et al.
we apply the variational bottleneck before the last fully connected layer of the network.
The results of this experiment are shown in Fig.
We attack ATS for the ﬁrst 1000 steps of training and Soteria for the ﬁrst 10 epochs, measuring peak signal-to-noise ratio (PSNR) of the reconstruction obtained using our attack at every step.
In all of our PRECODE experiments we obtain perfect reconstruction with PSNR values > 150 so we do not show PRECODE on the plot.
We can observe that, generally, each network becomes less susceptible to the attack with the increased number of training steps.
However, early in training, networks are very vulnerable, and images can be reconstructed almost perfectly.
Fig.
3 visualizes the ﬁrst 40 reconstructed images obtained using our attacks on Soteria, ATS and PRECODE.
We can see that for all defenses, our reconstructions are very close to their respective inputs.
This indicates that proposed defenses do not reliably protect privacy under gradient leakage, especially in the earlier stages of training.
Our ﬁndings suggest that creating effective defenses and properly evaluating them remains a key challenge. [Figure] [Figure] [Table]
Bayes optimal adversary in practice
In the following experiment, we compare the approximation of Bayes optimal adversary described in Section 4 with three variants of Inverting Gradients attack (Geiping et al., 2020), based on (cid:96) 2 , (cid:96) 1 , and cosine distance, on the MNIST (Lecun et al., 1998) and CIFAR-10 datasets.
Recall from Table 1 that the (cid:96) 2 , (cid:96) 1 and cosine attacks make different assumptions on the probability distribution of the gradients, which are not optimal for every defense.
To this end, we evaluate the performance of these attacks on a diverse set of defenses.
For both MNIST and CIFAR-10, we train a CNN with ReLU activations, and attack gradients created on image batches of size 1 both at the initialization and step 500 of the training.
For all attacks, we use anisotropic total variation image prior, and we initialize the images with random Gaussian noise.
Further experiments using prior based on pixel value range are presented in Appendix A.4.
We optimize the loss using Adam (Kingma & Ba, 2015) with exponential learning rate decay.
For this experiment, we modiﬁed Eq.
(3) to add an additional weighting parameter β for the prior ask (cid:80) ki =1 log p ( g | x i ) + β log p ( x i ) which compensates for the imperfect image prior selected.
Experimentally, we observed that the success of the different attacks is very sensitive to the choice of their parameters.
Therefore, to accurately compare the attacks, we use grid search that selects the optimal parameters for each of them individually.
In particular, for all attacks we tune their initial learning rates and learning rate decay factors as well as the weighting parameter β .
Further, for the (cid:96) 2 , (cid:96) 1 and cosine attacks the grid search also selects whether exponential layer weighting, introduced by Geiping et al.
(2020), is applied on the gradient reconstruction loss.
In this experiment, the approximation of the Bayes optimal adversary uses k = 1 Monte Carlo samples.
Further ablation study on the effects of k is provided in Appendix A.6.
Our grid search explores 480 combinations of parameters for the (cid:96) 2 , (cid:96) attacks 540 combinations
Bayesian attacks respectively provide more details on this parameter search Appendix A.
For this experiment, we consider defenses for which we can exactly compute conditional probability distribution of the gradients for a given input, denoted earlier as p ( g | x ) .
First we consider two defenses which add Gaussian noise with standard deviation σ = 0 .
1 and Laplacian noise of scale b = 0 .
1 to the original gradients, thus corresponding to the probability distributions p ( g | x ) =
N ( g − ∇ θ l ( h θ ( x ) , y ) , σ 2 I ) and p ( g | x ) =
Lap ( g − ∇ θ l ( h θ ( x ) , y ) , bI ) , respectively.
Note that the Gaussian defense with additional gradient clipping corresponds to the well known DP-SGD algorithm (Abadi et al., 2016).
We also consider defenses based on soft pruning that set random 50% entries of the gradient to 0, and then add Gaussian or Laplacian noise as in previous defenses.
In Table 2 we report the average PSNR values on the ﬁrst 100 images of the training set for each combination of attack and defense.
First, we observe, as in Fig.
2, that network is signiﬁcantly more vulnerable early in training.
We can observe that Bayes optimal adversary generally performs best, showing that the optimal attack needs to leverage structure of the probability distribution of the gradients induced by the defense.
Note that, in the case of Gaussian defense, (cid:96) 2 and Bayes attacks are equivalent up to a constant factor (as explained in Section 4), and it is expected that they achieve a similar result.
In all other cases, Bayes optimal adversary outperforms the other two attacks.
Overall, this experiment provides empirical evidence for our theoretical results from Section 4.
[Figure]
Approximations of Bayes optimal adversary
In this experiment, we compare the Bayes optimal adversary with attacks that have suboptimal approximations of p ( g | x ) and p ( x ) .
As vision datasets have complex priors p ( x ) which we cannot compute exactly, we now consider a synthetic dataset where p ( x ) is 20-dimensional unit Gaussian.
We deﬁne the true label y := arg max
( Wx ) where W is a ﬁxed random matrix, and perform an attack on a 2-layer MLP defended by adding Laplacian noise with a 0. scale.
Thus, p ( x ) is a Gaussian, and p ( g | x ) is Laplacian.
In this study, we consider 4 different variants of the attack, obtained by choosing Laplacian or Gaussian for prior and conditional.
Fig.
4 shows, for each attack, the distance from the original input for 200 steps.
We can observe that Bayes optimal attack (with Gaussian prior and Laplacian conditional) converges signiﬁcantly closer to the original input than the other attacks, providing empirical evidence for our theoretical results in Section 4.
Discussion and future work Our experimental evaluation suggests the evaluation of the existing heuristic defenses against gradient leakage is inadequate.
As these defenses are signiﬁcantly more vulnerable at the beginning of the training, we advocate for their evaluation throughout the entire training, and not only at the end.
Ideally, defenses should be evaluated against the Bayes optimal adversary, and if such adversary cannot be computed, against properly tuned approximations such as the ones outlined in Section 4.
Some interesting future work includes designing better approximations of the Bayes optimal adversary, e.g. by using better priors than total variation, and designing an effective defense for which it is tractable to compute the distribution p ( g | x ) so that it can be evaluated using Bayes optimal adversary similarly to what we did in Table 2.
We believe our ﬁndings can facilitate future progress in this area, both in terms of attacks and defenses.
C ONCLUSION
We proposed a theoretical framework to formally analyze the problem of gradient leakage, which has recently emerged as an important privacy issue for federated learning.
Our framework enables us to analyze the Bayes optimal adversary for this setting and phrase it as an optimization problem.
We interpreted several previously proposed attacks as approximations of the Bayes optimal adversary, each approximation implicitly using different assumptions on the distribution over inputs and gradients.
Our experimental evaluation shows that the Bayes optimal adversary is effective in practical scenarios in which it knows the underlying distributions.
We additionally experimented with several proposed defenses based on heuristics and found that they do not offer effective protection against stronger attacks.
Given our ﬁndings, we believe that formulating an effective defense that balances accuracy and protection against gradient leakage during all stages of training remains an exciting open challenge.
A A PPENDIX
Here we provide additional details for our work.
A. S OTERIA
For Soteria, we built directly on the Sun et al.
(2021) repository using their implementation of the defense and the included Inverting Gradient (Geiping et al., 2020) library.
Testing was conducted primarily on the ConvBig architecture presented below:
The architecture consists of a convolutional ”feature extractor” followed by three linear layers, shown in Table 3.
We chose this architecture because (i) it is simple in structure while providing reasonable accuracies on datasets such as CIFAR10 and (ii) because (unlike many small convolutional networks) it has more than one layer with a signiﬁcant fraction of the overall network parameters.
In particular, the ﬁrst linear layer roughly contains around 80% of the network weights and the second one 20% .
This is particularly relevant for our attack, as cutting out a large layer of weights will negatively affect the reconstruction quality.
As it is in practice uncommon to have the majority of weights in a single layer, we believe our architecture provides a reasonable abstraction.
To justify this, we show in Fig.
5 that no matter which of the larger two layers is defended by Soteria, we can attack the network successfully.
For all Soteria defenses we set the pruning rate (refer to (Sun et al., 2021) for details) to 80% .
Note however that our attack works independent of the pruning rate, as we always remove the entire layer.
Conv2d(3, 1 * width, kernel size=3, padding=1), BatchNorm2d(), ReLU() Conv2d(1 * width, 2 * width, kernel size=3, padding=1), BatchNorm2d(), ReLU()
Conv2d(2 * width, 2 * width, kernel size=3, padding=1), BatchNorm2d(), ReLU() Conv2d(2 * width, 4 * width, kernel size=3, padding=1), BatchNorm2d(), ReLU()
Conv2d(4 * width, 4 * width, kernel size=3, padding=1), BatchNorm2d(), ReLU() Conv2d(4 * width, 4 * width, kernel size=3, padding=1), BatchNorm2d(), ReLU()
MaxPool2d(3),
Conv2d(4 * width, 4 * width, kernel size=3, padding=1), BatchNorm2d(), ReLU() Conv2d(4 * width, 4 * width, kernel size=3, padding=1), BatchNorm2d(), ReLU()
Conv2d(4 * width, 4 * width, kernel size=3, padding=1), BatchNorm2d(), ReLU()
[Table] [Table]
We use the Adam optimizer with a learning rate of 0.1 with decay.
As similarity measure, we use cosine similarity.
Besides cutting one layer of weights, we weigh all gradients equally.
The total variation regularization constant is 4 × 10 − 4 .
We initialize the initial guess randomly and only try reconstruction once per image, attacking one image at a time.
For training, we used a batch size of 32.
We trained the network without defense and applied the defense at inference time to speed up the training.
A. A UTOMATED T RANSFORMATION S EARCH
For ATS, we built upon the repository released alongside (Gao et al., 2021).
We use the ConvNet architecture with a width of 64 also proposed in (Gao et al., 2021) and train with the augmentations ”7-4-15”, ”21-13-3”, ”21-13-3+7-4-15” which perform the best on ConvNet with CIFAR100.
We present the ConvNet architecture in Table 4.
For reconstruction, we use the Adam optimizer with a learning rate of 0.1 with decay.
As similarity measure, we use cosine similarity weighing all gradients equally.
The total variation regularization constant is 1 × 10 − 5 .
We initialize the initial guess randomly and only try reconstruction once per image, attacking one image at a time.
For training, we used a batch size of 32 and trained individually for every set of augmentations.
In Figure 6 we show how the quality of the reconstructed inputs degrades during training.
Nevertheless we can recover high-quality inputs during the ﬁrst 10 to 20 training steps.
A. P ARAMETER SEARCH FOR THE ATTACKS
Since the range of β for which the different attacks perform well is wide, prior to the grid search we need to calculate a range of reasonable values for β for each of the attacks.
We do this by searching for values of β for which the respective attack is optimal.
The rest of the parameters of these attacks are set to the same set of initial values.
The values of β considered are in the range [1 × 10 − 7 , 1 × 10 5 ] and are tested on logarithmic scale.
The ﬁnal range for β used in the grid search is given by the range [0 .
5 β ∗ , 2 β ∗ ] , where β ∗ is the values that produced the highest PSNR.
A. E XPERIMENT WITH STRONGER PRIORS
In this section, we compare the effects of different image priors on the gradient leakage attack.
In particular, we compared a simple anisotropic total variation image prior and a prior that uses combination of the same anisotropic total variation and an error term ﬁrst introduced in Geng et al.
(2021) that encourages the reconstructed image’s pixels to be in the [0 , 1] range:
[Equation]
where φ ∈ [0 , 1] balances the total variation error term L TV and the pixel range error term L [0 , 1] and clip ( x, 0 , 1) clips the values of x in the [0 , 1] range.
We compared both priors on the MNIST
[Figure]
[Table]
network trained for 500 steps from Table 2 using (cid:96) 2 conditional distribution.
We compared on all defenses considered in Table 2.
We used the same grid search procedure, but we extended it to search for optimal value of φ as well.
The results are presented in Table 5.
From the results, we see that the addition of the pixel range prior term improved the attack for all defenses, showing that despite its simplicity the pixel range prior is very effective.
A. O UR PRACTICAL ATTACKS AS APPROXIMATIONS OF THE B AYES OPTIMAL ADVERSARY
In this subsection we interpret our attacks on Soteria, ATS and PRECODE as different approximations of Bayes optimal adversary.
We summarize this in Table 6, and also describe details of each approximation in separate paragraphs.
Attack on Soteria
In this section, we interpret the attack on Soteria that we presented in Section 5, as an approximation to the Bayes optimal attack in this setting.
As described in Section 5, Soteria takes the original network gradient ∇ W = {∇ W 1 , ∇ W 2 , . .
. , ∇ W L } and defends it by substituting the gradient at layer l with the modiﬁed gradient ∇ W (cid:48) l , producing g k = (cid:26)
[Equation]
where g m is the subvector of the client gradient update vector g , corresponding to the m th layer of the network.
In Section 4, we derived the Bayes optimal attacker objective as
[Equation]
[Table]
Conditional
[Table]
TV
[Table]
Gaussian
[Table]
Next, we show how we approximate this objective for the Soteria defense.
Since Soteria modiﬁes the network gradients per-layer, we model the gradient probability distribution p ( g | x ) of our Bayes optimal attack as per-layer separable:
[Equation]
To model the design choice of Soteria to not change gradients of layers different than l , we set p ( g m | x ) =
N ( ∇ W m , σ m ) with σ m → 0 , for all layers m (cid:54)
=
l .
With that choice, the Bayes objective becomes
[Equation]
As σ m → 0 , the contribution of log p ( g l | x ) to the overall objective becomes exceedingly smaller.
Therefore, one can choose σ m = σ for some σ > 0 , such that the optimal solution of the Bayes objective in Eq.
(5) is very close to the optimal solution of the surrogate objective:
[Equation]
The objective in Eq.
(6) is the exact objective that we optimize to produce the Soteria attacks in Section 5.
Attack on ATS As described in Section 5, we attack ATS using the attack from Geiping et al.
This attack corresponds to the prior with total variation, and conditional distribution is based on cosine similarity.
Attack on PRECODE Our attack on PRECODE is based on the observation that the distribution p ( x | g ) for PRECODE corresponds to a Dirac delta distribution.
Namely, given gradient g , there is always unique input x 0 that produced this gradient, and the density p ( x | g ) is then a Dirac delta centered at x 0 (uniqueness of the solution follows from the derivation in Phong et al.
(2017) for linear layers).
As shown in Eq.
(2), this x 0 corresponds to maximizing the value of p ( x | g ) .
In this case, we can in fact solve for the maximum of p ( x | g ) exactly using attack described in Section 5, and we do not have to rewrite the objective using Bayes rule and apply optimization.
In terms of p ( x ) and p ( g | x ) , we can have arbitrary p ( x ) , while p ( g | x ) is mixture of Dirac delta distributions where the mixture comes from the random samples of the VAE used by PRECODE.
A. A BLATION STUDY OF THE M ONTE C ARLO ESTIMATION
In this experiment we perform an ablation study for the Monte Carlo estimation of the adversarial objective. More speciﬁcally, we vary the number of samples k used for the Monte Carlo estimate. [Figure] We show the PSNR value averaged over 100 different sampled images for each step of the optimization process in Figure 7.
We used a convolutional network on the MNIST dataset, at the initial training step, defended by Gaussian noise of standard deviation 0.1 and using δ = 9 . 0 in the deﬁnition of the adversarial risk.
The results indicate that higher number of samples in the Monte Carlo estimate results in faster convergence of the attacker towards the reconstruction image closer to the original. This is expected as estimate using larger number of samples results in an Monte Carlo estimator that has lower variance. A. R ELATIONSHIP BETWEEN B AYES ATTACK AND L ANGEVIN DYNAMICS In this section, we discuss the relationship between our proposed Bayes optimal attack and Langevin updates proposed by Yin et al.
We focus on the case where the number of samples in the
Monte Carlo estimate is k = 1 .
As shown in Algorithm 1, our update is
[Equation]
where x 1 is a point sampled from the ball B ( x, δ ) uniformly at random.
At the same time, the update with Langevin dynamics proposed by Yin et al.
(2021) is given by
[Equation]
where η ∼ N (0 , I ) is noise sampled from unit Gaussian, and α n is noise scaling coefﬁcient.
As Yin et al.
(2021) explain, we can view Langevin updates as a method to encourage exploration and diversity.
The main differences between the updates are: (i) we sample a point x 1 from the ball
B ( x, δ ) uniformly at random, while Langevin update in Yin et al.
samples noise η from unit Gaussian, (ii) we compute gradient at the sampled point x 1 ∼ B ( x, δ ) , while Langevin update evaluates gradient directly at x , and adds noise η to x afterwards.
Note that Yin et al.
(2021) do not provide ablation study to investigate the effect of Langevin update, so we do not know important it is in the overall optimization.
Overall, both Monte Carlo estimation and Langevin updates can be seen as different ways to add noise to the optimization process, though currently we do not see a way to formally view Langevin updates as a Bayesian prior.