Learning to Attack Federated Learning: A Model-based Reinforcement Learning Attack Framework
Abstract
We propose a model-based reinforcement learning framework to derive untargeted poisoning attacks against federated learning (FL) systems.
Our framework first approximates the distribution of the clients’ aggregated data using model updates from the server.
The learned distribution is then used to build a simulator of the FL environment, which is utilized to learn an adaptive attack policy through reinforcement learning.
Our framework is capable of learning strong attacks automatically even when the server adopts a robust aggregation rule.
We further derive an upper bound on the attacker’s performance loss due to inaccurate distribution estimation.
Experimental results on real-world datasets demonstrate that the proposed attack framework significantly outperforms state-of-the-art poisoning attacks.
This indicates the importance of developing adaptive defenses for FL systems.
Introduction
Federated learning (FL) is a promising machine learning framework that allows multiple devices with private data to jointly train a learning model (coordinated by a server) without sharing their local data.
It has recently been applied to consumer digital products [41], credit risk prediction [1], drug discovery [42], and digital health [49].
However, federated learning systems are vulnerable to adversarial attacks [39] such as model poisoning attacks [11, 65, 20, 8], data poisoning attacks [9, 23, 26], and inference attacks [43, 29, 74].
To this end, various robust aggregation rules such as coordinate-wise median [67], trimmed mean [67], Krum [12], norm clipping [59], geometric median [46] and FLTrust [15] have been proposed.
However, these defenses are mainly evaluated against manually crafted myopic attack policies [53].
Their robustness in the face of advanced attacks remains unknown.
Due to the distributed nature of FL systems, a malicious device typically has limited knowledge about benign devices and system dynamics.
To fully reveal the vulnerabilities of FL systems, it is therefore crucial to develop strong attacks that can best utilize the limited global knowledge.
In this work, we take a first step in this direction by considering the white-box attack setting where the attacker has some global knowledge about the FL system and the server’s algorithm, but has no access to the private data of benign devices, a reasonable assumption for real-world FL systems.
To derive strong adaptive attacks, we propose to leverage the power of model-based reinforcement learning (RL) by integrating distribution learning and policy learning .
A key observation of our approach is that although accurate information about individual devices can be hard to obtain in FL, it is often possible to infer their aggregated data distribution from publicly available model updates, which ˚ Equal contribution.
is sufficient to derive strong attacks.
In particular, the set of malicious devices first cooperatively estimate the aggregated data distribution through gradient inversion [29, 74].
The learned distribution is then used to build a simulator of the FL environment, which is utilized to derive an adaptive attack policy through reinforcement learning.
We focus on untargeted model poisoning in this work, where the malicious devices aim to reduce the accuracy of the global model as much as possible by sending crafted gradient information to the server.
However, our proposed framework can potentially be applied to other types of attacks in both the white-box and the more challenging black-box settings.
Our model-based approach distinguishes from existing work on reinforcement learning based adversarial attacks against machine learning systems [58, 71, 72].
In particular, we consider a more realistic threat model where the attacker might not always be selected due to subsampling nor does it have prior information about the distribution of the aggregated data.
The attackers need to efficiently learn the distribution along the federated learning process in real time.
In contrast, previous works typically assume more powerful attackers that can attack at any time and have full knowledge about the environment.
Thus, they typically adopt a purely model-free approach, which is infeasible in attacking FL systems due to the large number of samples needed to be effective.
Our contributions.
We advance the state-of-the-art in the following aspects.
First, we propose a novel reinforcement learning attack framework against federated learning systems by integrating distribution learning and policy learning.
Second, we theoretically quantify the effect of inaccurate distribution learning and heterogeneous local data distributions on the optimal attack performance.
Third, our experiments on real-world datasets demonstrate that our RL-based attack method consistently outperforms existing model poisoning attacks [11, 20, 65] and significantly reduces the global model accuracy even when robust aggregation rules are applied.
These findings indicate the importance of developing adaptive defenses for FL systems.
Related Work
Poisoning attacks and defenses.
To compromise the integrity of the target model in federated learning, both targeted poisoning attacks [11, 8, 9] that aim to misclassify a specific set of inputs, and untargeted attacks [20, 65, 52] aiming to reduce the global model accuracy have been proposed.
Existing approaches typically adopt a heuristics-based method [65]) or optimize a myopic goal [20, 52]), and are usually sub-optimal, especially when a robust aggregation rule is adopted.
Further, they often require access to benign agents’ local updates or the accurate global model parameters of the next round [65, 20]) to make significant attack impact.
In contrast, our reinforcement learning based attack requires less global knowledge and targets a long-term attack goal.
Various defenses have been proposed for model poisoning attacks including robust-aggregationbased approaches and detection-based approaches.
The former includes dimension-wise filtering that considers each dimension of local updates separately [10, 67], client-wise filtering that aims to restrict or even remove the impact of potentially malicious clients [12, 12, 46, 59], and approaches that require the server to have access to a small amount of root data [15].
Our RL-based attack is effective against all these defenses.
In addition, time-coupled robust aggregation methods [4, 5, 32] that target adaptive attacks and anomaly detection-based defenses [35] have been proposed recently.
Our approach can potentially be extended to compromise them by explicitly encoding the history information into states or utilizing a recurrent structure.
RL-based attacks.
Reinforcement learning has recently been utilized for developing strong attacks in various settings, including corrupting training data of online supervised and unsupervised learning [72], manipulating the combinatorial structure of graph data [18], and injecting malicious nodes into a graph [58].
RL-based attacks have also been developed to damage the performance of reinforcement learning itself, by perturbing the reward signals during the training stage [71] and corrupting the state signals received by an agent during the testing stage [70, 56].
However, these methods typically assume that the attacker has access to an accurate MDP model or simulator and has unlimited time for training the attack policy.
In contrast, our method first builds a world model by learning a data distribution from the FL model updates and then constructs an approximate simulator for training our attacks.
Both distribution learning and policy learning happen when FL training is ongoing.
Further, previous work has mainly focused on attacking a single RL agent by an external agent rather than an insider attack in a distributed learning environment as we consider in this work.
Approach Overview
In this section, we describe the federated learning setting considered in this work, the threat model, and the proposed RL-based attack framework.
Federated learning.
We consider an FL setting that is similar to federated averaging (FedAvg) [40].
The FL system consists of a server and K workers (also known as devices or clients) in which each worker has some private data.
Let r K s “ t 1 , 2 , ..., K u denote the set of workers.
Coordinated by the server, the set of workers cooperate to train a machine learning model within T epochs by solving the following problem: min θ f p θ q where f p θ q : “ ř Kk “ 1 p k F k p θ q .
Here F k p¨q is the local objective of worker k and p k is the weight assigned to worker k and satisfies p k ě 0 and ř k p k “ 1 .
The local objective F k p θ q is usually defined as the empirical risk over worker k ’s local data with model parameter θ P Θ .
That is, F k p θ q “ 1 N k ř N k j
“ 1 ℓ p θ ; z jk q , where N k is the number of data samples available locally on worker k , ℓ p¨ ; ¨q is the loss function, and z jk : “ p x jk , y jk q is the j th data sample that is drawn i.i.d.
from some distribution P k .
It is typical to set p k “ N k N , where N “ ř k N k is the total number of data samples across workers.
If all the workers’ local data distributions are the same (i.e., P k “ P k 1 for all k, k 1 P r K s ), we call the workers’ data are i.i.d.
; otherwise, the data are non-i.i.d. .
We write p P k as the empirical distribution of the N k data samples drawn from P k , and let p P : “ ř Kk “ 1 N k N p P k denote the mixture empirical distribution across workers.
The FL algorithm (see Algorithm 1 in Appendix B) works as follows: at each time step t ě 0 , a random subset S t of size w is uniformly sampled without replacement from the workers set r K s by the server for synchronous aggregation [37].
The process of selecting workers for aggregation is called subsampling .
Let κ “ w { N denote the subsampling rate , i.e., the ratio of the selected workers number w to the total number of workers K .
Each selected worker k P r w s then samples a minibatch b k of size B from its local data distribution p P k .
The worker then calculates the average local gradient g t ` 1 k Ð 1 B ř z P b k ∇ θ ℓ p θ t ; z q and sends the gradient to the server.
The server then uses an aggregation rule to compute the aggregated gradient g t ` 1 Ð Aggr p g t ` 1 k 1 , ..., g t ` 1 k w q where k i P S t , and updates the global model parameters θ t ` 1 Ð θ t ´ ηg t ` 1 where η is the learning rate.
The newly updated model parameters θ t ` 1 is then sent to the selected workers to perform another FL iteration.
Threat Model.
We assume that among the K workers, M p 1 ď M ă K q of them are malicious.
Let A denote the set of malicious workers.
They are coordinated either by one leading attacker or an external agent.
We refer such agent as a leader agent .
These attackers are assumed to be fully cooperative and share the same goal of compromising the FL system.
We consider untargeted model poisoning attacks in this work where the M cooperative attackers send crafted local updates t r g tk u k P A to the server in order to maximize the empirical loss f p θ q .
the batch size B ), and their local data distributions t p P k u k P A (but not the benign workers’ local data distributions).
We further assume that the attackers obtain information about the server’s training algorithm (i.e., the white-box attack setting).
This information includes the server’s learning rate η , the subsampling rate κ , the total number of workers K , the aggregation rule Aggr , and the total number of training epochs T .
RL-based online attack framework.
Our attack framework consists of the following three phases.
• Distribution learning : The malicious workers first jointly learn an approximation of r P from the model updates t θ t u received from the server, using a gradient inversion based inference attack [24].
• Policy learning : The leader agent then builds a simulator of the FL environment using the attackers’ local data and the learned distribution.
An optimal attack policy is then derived through reinforcement learning using data sampled from the simulator.
Note that policy learning can start together with distribution learning and continue after a reasonable distribution is learned.
• Attack execution :
The learned policy is distributed to all the malicious workers to generate attack actions.
Note that attack execution can start once an initial policy is learned, which can be updated during attack execution.
We note that all the three phases happen while the federated learning process is ongoing, thus the lengths of these phases are important hyperparameters to be determined.
For example, with more observations, an attacker can learn a more accurate distribution, which will help obtain a better attack policy.
However, when the total time window available to attacks is limited, a longer distribution learning phase reduces the attack opportunities in Phase 3.
Compared with a purely model-free approach, our model-based approach is more sample efficient, which is especially important for federated learning as a malicious worker can only attack when it is sampled by the server.
Model-based Reinforcement Learning Attack Framework
In this section, we first formulate the model poisoning attack problem as a Markov decision process (MDP).
We then discuss our model-based reinforcement learning attack framework in more details.
4.1 Attackers’ problem as a Markov decision process
The attackers’ problem is formulated as an undiscounted MDP, denoted by M “ p S, A , T, r, H q , where
• S is the state space.
Let τ P t 0 , 1 , ..., H ´ 1 u denote the index of the attack step and t p τ q P t 0 , 1 , ..., T ´ 1 u the corresponding FL epoch when at least one attacker is selected by the server.
The state at step τ is defined as s τ : “ p θ t p τ q , A t p τ q q where A t p τ q q is the set of attackers selected at time t p τ q , which is shared between all malicious workers.
• A “ A M is the space of the attackers’ joint actions where each attacker shares the same action space A .
If attacker i is selected at t p τ q , its action a τi : “ r g t p τ q` 1 i P R d is the local update that attacker i sends to the server at time step t p τ q , where d is the dimension of the model parameters.
The only action available to an attacker not selected at t p τ q is K , indicating that the attacker does not send any information in that step.
• T : S
ˆ A Ñ P p S q is the state transition function that represents the probability of reaching a state s 1 P S from the current state s P S when attackers choose actions a τ 1 , ..., a τM , respectively.
• r : S ˆ A ˆ S Ñ R ě 0 is the reward function.
We define the reward at step τ as r τ : “ f p θ t p τ ` 1 q q ´ f p θ t p τ q q , which is determined by the current state, the next state, and the joint attack actions and is shared by all the attackers.
• H is the number of attack steps in each episode and we have t p H ´ 1 q ă T .
The attackers’ goal is to find a joint attack policy
π “ p π 1 , ..., π M q that maximizes the expected total rewards over H attack steps, i.e., E r ř H ´ 1 τ “ 0 r τ s , where π i : S Ñ P p
A q denotes a stationary policy of attacker i that maps the state to a probability measure over A .
Using the definition of r τ , this objective is equivalent to finding a joint policy π that maximizes E θ t p H q r f p θ t p H q qs .
A key obstacle to solving the MDP is that both the transition probabilities T and the reward function r depend on the joint empirical distribution across workers t p P k u k Pr K s , which is fixed but unknown to the attackers.
Although model-free reinforcement learning can bypass this difficulty, it requires a large number of samples to be effective, which is infeasible in the online attacking scenario we consider.
We therefore adopt model-based reinforcement learning as a principled approach for designing adaptive attacks in the online setting.
An important observation is that although the joint empirical distribution t p P k u k Pr K s is unknown, the attackers can learn an approximation of the mixture distribution p P “ ř Kk “ 1 N k N p P k , denoted by r P , from model updates shared by the server, which is often sufficient to simulate the behavior of benign agents and the server by assuming that each benign agent samples data from r P .
This gives rise to a new MDP Ă M “ p S, A , T 1 , r 1 , H q where T 1 and r 1 are derived from r P .
Thus, our proposed model-based reinforcement learning attack framework naturally consists of the distribution learning, policy learning, and attack execution phases.
4.2 Distribution learning
Initially, the attackers do not perform model-poisoning attacks.
Instead, they jointly learn a mixture distribution r P from the model updates t θ t u using a gradient inversion based inference attack [24, 74].
Various gradient inversion attacks have been proposed in the literature.
In this work, we adapt the inverting gradients (IG) method [24] to distribution learning.
The
IG method reconstructs data samples by optimizing a loss function based on the angle (i.e., cosine similarity) between the gradient generated from true data and that from the reconstructed data.
The primary goal of IG is to reconstruct the original data samples, which is more ambitious than what the attackers need in our setting.
On the other hand, recent works on gradient inversion including IG have focused on the server side, where the true gradients of each individual worker can be easily obtained from model updates.
In contrast, the attackers only obtain approximated and aggregated gradient information from consecutive model updates received from the server, due to model aggregation and subsampling.
Despite these differences, our experiment results show that the r P learned using IG can help derive an effective attacker policy (see Figure 4(c)).
As shown in Algorithm 2 in Appendix B, for each epoch t p τ q that at least one attacker is selected, the leader agent obtains the model update from one of the attackers and calculates the batch-level gradient as ¯ g τ : “ p θ t p τ ´ 1 q ´ θ t p τ q q{p η p t p τ q ´ t p τ ´ 1 qqq .
The leader agent then starts with a batch of (randomly generated) dummy data and dummy labels D dummy , which is updated iteratively by solving the following optimization problem: arg min D dummy 1 ´ cos p ∇ θ F dummy p θ t p τ q q ,
¯ g τ q ` βBř p x,y qP
D dummy TV p x q , where cos p A, B q : “ x A,B y } A }} B } is the cosine similarity between two vectors
A and B , F dummy p θ q “ 1 Bř p x,y qP D dummy ℓ p θ ; p x, y qq , B 1 is the number of reconstructed data per epoch (the size of the dummy data batch), TV p x q is the total variation [50] of x , and β is a fixed parameter.
The process terminates after max iter iterations, then outputs the updated data as the reconstructed data samples.
We observed that although the data samples generated by IG resemble true samples, they contain a certain amount of noise as shown in Figure 2, making the learned distribution less representative of the true distribution.
To reduce noise, we adapt the method of denoising auto-encoder [62].
We utilize the clean data owned by attackers and add Gaussian noise to them to simulate the noise in the reconstructed data samples.
The clean data and the synthetic noisy data are then paired to train an autoencoder for denoising, which is then used to remove the noise in reconstructed data samples as shown in the figure.
The approximated mixture distribution r
P p τ q consists of the reconstructed data up to t p τ q and the M attackers’ local data.
The distribution learning phase starts at the first FL epoch and continues for τ E steps.
The learned distribution is shared with all the attackers.
Although we adopt the IG method in this work due to its simplicity, other more recent approaches such as the GradInversion method [68] and gradient inversion with a trained generative model [30] can be easily incorporated into our framework, which can potentially learn r P in more challenging settings for complex datasets like ImageNet [19], deep networks, and large batch sizes.
On the other hand, we show in the experiments that our RL-based attack trained using attackers’ local data only is still effective and surpasses all the baselines, while distribution learning further boosts the attack performance.
A detailed discussion on gradient inversion attacks and defenses is provided in
Appendix C.
4.3 Policy learning
Once the leader agent obtains the approximated distribution r P , it can simulate the behavior of the server and that of normal workers.
In particular, to simulate the behavior of a normal worker in each
FL training step, a minibatch of size B is i.i.d.
sampled from r P .
With experiences sampled from the simulated environment, the leader agent can learn a joint attack policy that maximizes the empirical loss using a state-of-the-art (deep) reinforcement learning algorithm, e.g., TD3 [22] or PPO [51].
Note that the leader agent does not need to wait until a good distribution has been learned to start training the policy.
Instead, policy learning can start together with distribution learning.
Initially, the leader agent uses the attackers’ local data to train the policy, which will be continuously updated while more data samples are being generated.
To reduce the training overhead, we assume that all the malicious workers share the same attack policy (i.e., π 1 “ π 2 “ ¨ ¨ ¨ “ π M ) in this work, which achieves good attack performance in our experiments.
Extension to general joint policies will be considered in our future work.
When we train a small neural network with federated learning, it is natural to use p θ t p τ q , A t p τ q q as the state, and the crafted gradient r g t p τ q as the action.
When we use the federated learning system to train a large neural network, however, this approach does not scale as it results in an extremely large search space that requires both large runtime memory and long training time, which is usually prohibitive.
To solve this problem, we propose to compress the state and action spaces for high-dimensional models as follows.
To compress the state space, we first observe that as the set of attackers are fully cooperative and share the same policy, there is no need to distinguish them in the state when the server does not track the behavior of individual workers.
Thus, we replace A t p τ q by the number of attackers sampled in t p τ q , denoted by m t p τ q .
Further, instead of using the entire set of model parameters θ t p τ q in the state, the parameters of the last hidden layer of the current neural network model is used.
This is because the last hidden layer passes on values to the output layer and typically carries information about important features of the model [57].
Note that the approximated state is used to define the policy only.
The true state is still the full FL model that determines transition probabilities and rewards.
It is more challenging to compress the action space.
We observe that a policy that manipulates the model parameters of the last hidden layer only works well for certain aggregation rules such as
Krum and coordinate-wise median.
However, it becomes less effective for stronger defenses such as coordinate-wise median with clipping.
Given that it is challenging to identify a small subset of model parameters that can lead to most damage to model accuracy when manipulated, we adopt a different approach in this work.
The main idea is to search for a model update direction that can lead to a large empirical loss using gradient ascent, with its parameters identified by reinforcement learning.
To this end, we define the local search objective as L p θ q : “ p 1 ´ λ q F p θ q ` λ cos p θ t p τ q
´ θ, g p θ t p τ q qq where F p θ q “
E z „ r P r ℓ p θ ; z qs models the empirical loss and g p θ t p τ q q “ E z „ r P r ∇ θ ℓ p θ t p τ q ; z qs captures the average update direction from normal devices, both are estimated from r P .
The second term in L p θ q is used to control the deviation of the model update from the normal direction (measured by the cosine similarity) so that the adversarial input cannot be easily identified by the server.
The parameter λ P r 0 , 1 s is used to balance the loss and the deviation.
To solve the problem, we start with θ “ θ t p τ q and generate G trajectories, where each trajectory involves E model updates.
Each model update involves a single gradient ascent step using a minibatch of size r B sampled from r P .
Let θ k denote the new model parameters found by the k -th trajectory after E updates.
The update direction is then set as 1 G ř Gk “ 1 θ k and each attacker’s action in t p τ q is computed as r g t p
τ q` 1 “ γ p θ t p τ q ´ 1 G ř
[Equation]
The scaling factor γ ě 0 is used to control the magnitude of the crafted gradient, which is needed as most robust aggregation rules apply a certain type of filtering rule to mitigate the effect of malicious attacks.
We assume that G is fixed while γ , E , λ are parameters to be learned by reinforcement learning.
Thus, the action space for the attackers’ MDP becomes a 3-dimensional real space.
4.4 Attack execution
Both distribution learning and policy learning can start from the first epoch of federating learning and continue while federating learning is ongoing.
The simulated environment is updated when a new estimated distribution r P is learned.
Although the attackers may choose to start attacking during distribution learning, we observe that this can blur the gradient information and make distribution learning less accurate.
Thus, we assume that each attacker starts attacking once the distribution learning phase is finished, and applies the latest learned attack policy during the remaining epochs of the federated learning process.
During attack execution, each selected attacker first notifies other attackers so that every one knows the number of attackers that are sampled in that epoch.
Each selected attacker then generates a crafted gradient according to the process described above with the parameters obtained from the latest learned policy.
The attackers’ total training time (including distribution learning and policy learning) should be significantly less than the total FL training time so that the attackers have time to execute the attacks.
In real-world FL training, the server usually must wait for some time (typically ranging from 1 minute to 10 minutes) before it receives responses from the clients [66, 13, 31].
In contrast, the leader agent does not incur such time cost in training attackers’ policies using a simulated FL environment.
Therefore, an epoch in policy learning is typically much shorter than an FL epoch, making it possible to train the attack policy with a large number of episodes.
In addition, the leader agent is usually equipped with GPUs, or other parallel computing facilities and can run multiple training episodes in parallel [16].
We compare the actual running time of our RL-based attack against different defenses in our experiment setting in Appendix E.2.
Impact of Inaccurate Distribution Learning and Data Heterogeneity
Our model-based RL attack employs the estimated data distribution r P to simulate the behavior of benign workers, which can suffer from two types of errors.
First, r P can be far away from the true mixture distribution p P due to inaccurate distribution learning.
Second, benign workers may vary in their local data distributions p P k , which cannot be fully captured by a single mixture distribution.
In this section, we study how the attack performance is affected by these two factors, which provides insights into properly distributing resources between the three phases of our attack method.
Our analysis is adapted from recent works that study the impact of model inaccuracy on the performance of model-based reinforcement learning [69, 38, 71] by addressing two new challenges.
First, we need to establish the connection between the inaccuracy in data distribution r P and the inaccuracy in the corresponding MDP as both the reward function and the transition dynamics depend on r P .
Second, although there are different ways to measure the distance between two models [69], it makes more sense to use the 1-Wasserstein distance [61] to measure the distance between two data distributions.
This, however, requires bounding the Lipschitz constant of the optimal value function [69].
Although this is a challenging task for general RL tasks, we are able to show that this is indeed the case in our setting under the following assumptions.
The first assumption models the inaccuracy of distribution learning as well as the heterogeneity of benign workers’ local data.
Assumption 1.
W 1 p r P, p P k q ď δ for any benign worker k , where W 1 p¨ , ¨q is the 1-Wasserstein distance [61].
We further need the following standard assumptions on the loss function.
Assumption 2.
Let Z denote the domain of data samples across all the workers.
For any s 1 , s 2 P S and z 1 , z 2 P Z , the loss function ℓ : S
ˆ Z Ñ R satisfies:
| ℓ p s 1 , z 1 q ´ ℓ p s 2 , z 2 q| ď L }p s 1 , z 1 q ´ p s 2 , z 2 q}(Lipschitz continuity w.r.t.
s and z ); 2. } ∇ s ℓ p s 1 , z 1 q ´ ∇ s ℓ p s 1 , z 2 q} 2 ď L z } z 1 ´ z 2 }(Lipschitz smoothness w.r.t.
z ); 3.
ℓ p s 2 , z 1 q ě ℓ p s 1 , z 1 q ` x ∇ s ℓ p s 1 , z 1 q , s 2 ´ s 1 y ` α 2 } s 2 ´ s 1 } 22 (strong convexity w.r.t.
s ); 4.
ℓ p s 2 , z 1 q ď ℓ p s 1 , z 1 q ` x ∇ s ℓ p s 1 , z 1 q , s 2 ´ s 1 y ` β 2 } s 2 ´ s 1 }
(strong smoothness w.r.t.
s ); 5.
ℓ p¨ , ¨q is twice continuously differentiable with respect to s .
where }p s 1 , z 1 q´p s 2 , z 2 q} 2 2 : “ } s 1 ´ s 2 } 2 2 `} z 1 ´ z 2 } 2 2 .
For simplicity, we further make the following assumption on the FL environment, although our analysis can be readily applied to more general settings.
Assumption 3.
The server adopts FedAvg without subsampling ( w “ K ).
All workers have same amount of data ( p k “ 1 K ) and the local minibatch size B “ 1 .
In each epoch of federated learning, each normal worker’s local minibatch is sampled independently from the local empirical data distribution p P k .
Since no subsampling is considered in this section, with a slight abuse of notation, we let index t denote both an attack step and the corresponding FL epoch.
Let M “ p S, A , T, r,
H q denote the true
MDP for the attackers, and Ă M “ p S, A , T 1 , r 1 , H q the simulated MDP when the local distribution of any benign worker is estimated as r P .
The following theorem captures the attack performance loss due to inaccurate distribution learning (see Appendix D for the proof).
Theorem 1.
Let J M p
π q : “ E π
[Equation]
T,
[Equation]
0 r ř H ´ 1 t “ 0 r p s t , a t , s t ` 1 qs denote the expected return over H attack steps under M , policy π and initial state distribution µ 0 .
Let π ˚ and r π ˚ be the optimal policies for
M and Ă M respectively, with the same initial state distribution µ 0 .
Then,
[Equation]
where ϵ “ K ´ MK is the fraction of benign nodes, L v ď ř H ´ 1 t “ 0 p K F q t p
L ` LK F q , and K F ď ϵ max t| 1 ´ ηα | , | 1 ´ ηβ |u .
In practice, the learning rate η is typically small enough so that max t| 1 ´ ηα | , | 1 ´ ηβ |u ď 1 .
In this case, L v is bounded by L p 1 ` K F q 1 ´ K F ď L p 1 ` ϵ q 1 ´ ϵ .
Therefore, we have | J M p π ˚ q ´ J M p
r
π ˚ q| “
O p H ϵ 1 ´ ϵ ηδ q .
To ensure convergence, we typically have η “ O p 1 ?
H q [47], thus | J M p π ˚ q ´
J M p r π ˚ q| “ O p ϵ 1 ´ ϵ δ ?
H q .
This result clearly shows how the loss of attack performance depends on the fraction of benign nodes, the inaccuracy of distribution learning, and the time horizon.
Experiments
In this section, we compare our RL-based attack with state-of-the-art model poisoning attacks on real-world datasets.
Our code is available at https://github.com/SliencerX/Learning-toAttack-Federated-Learning .
6.1 Experiment setup
Datasets .
We conduct extensive experiments on four real-world datasets: MNIST [34], FashionMNIST [64], EMNIST [17], and CIFAR-10 [33].
Due to space limitation, experiment results for
Fashion-MNIST, EMNIST, and CIFAR-10 are provided in Appendix E.2.
For the i.i.d.
setting, we randomly split the dataset into K groups, each of which consists of the same number of training samples.
For the non-i.i.d.
setting, we follow the method of [20] to quantify the heterogeneity of local data distribution across clients.
Suppose there are C classes in the dataset, e.g., C “ 10 for the
MNIST and Fashion-MNIST datasets.
We evenly split the worker devices into C groups (with the
M attackers evenly distributed across the C groups), where each group is assigned 1 { C of training samples as follows.
A training instance with label c is assigned to the c -th group with probability q ě 1 { C and to every other group with probability p 1 ´ q q{p C ´ 1 q .
Within each group, instances are evenly distributed.
A higher q indicates a higher non-i.i.d. degree.
We set q “ 0 .
as the default non-i.i.d.
degree.
To demonstrate the power of distribution learning, we assume that the set of attackers share m true data points sampled from the training instances assigned to them.
We set m “ 200 as the default value for MNIST.
Baselines .
We compare our RL-based attack (RL) with no attack (NA), and the state-of-the-art model poisoning FL attack methods: explicit boosting (EB) [11], inner product manipulation (IPM) [65], and local model poisoning attack (LMP) [20].
IPM manipulates the attackers’ gradients so that the inner product between the aggregation result and the true gradient is negative.
This requires access to the average of normal workers’ gradients in each FL epoch, which is usually unavailable in practice.
LMP generates myopic attacks by solving an optimization problem in each FL epoch.
In addition to the server’s aggregation rule, it also requires access to normal workers’ local models.
Although
LMP with partial knowledge is also presented in [20], it performs substantially worse than the full knowledge case when the server uses the coordinate-wise median defense.
We compare the RL-based attack with the more powerful full knowledge LMP below.
[Figure]
We consider four representative robust aggregation rules of different types [53]: Krum [12], geometric median [46], both of which apply client-wise filtering to model updates, coordinate-wise median [67], which adopts a dimension-wise filtering, and FLTrust [15], which requires the server to have access to a small amount of root data.
In the experiments, we actually consider an extension of the vanilla coordinate-wise median where a norm clipping [59] step is first applied.
This gives a more powerful defense as we observed in experiments.
We set the default norm threshold to 2.
Default FL and RL settings .
We adopt the following default parameters for the FL models: number of total workers “ 100 , number of attackers “ 20 , learning rate η “ 0 . 01 , subsampling rate “ 10% , the number of total FL epochs “ 1 , 000 .
For our RL-based attack, both the distribution learning and policy learning phase start at the first FL epoch.
The former ends at the 100 th FL epoch when
RL-based attack starts (all other attacks start at epoch 0).
The policy learning phase ends at the 400 th epoch.
Since both the action space and state space are continuous in our setting, we choose the state-of-the-art Twin Delayed DDPG (TD3) [22] and Proximal Policy Optimization (PPO)
[51] algorithms for training the attack policy in our experiments and find that TD3 gives better results in most cases.
Below we report the results for TD3.
We fix the initial model and the random seeds for subsampling and local data sampling for fair comparisons.
See Appendix E for details of the datasets, experimental setups, and additional results.
6.2 Attack performance
Figure 3 shows how the test accuracy of the global model varies over FL epochs under different attacks when the server uses Krum and clipping median as the aggregation rule, respectively.
Results for geometric median and FLTrust are provided in Appendix E.2.
We observe that our RL-based attack performs significantly better in all the settings, despite the fact that IPM and LMP use the model updates of normal clients while RL does not.
Note that for Krum, RL-based attack quickly drives the global model to a poor state ( „ 10% accuracy) once the attack starts at epoch 100 under both i.i.d.
and non-i.i.d.
local data distributions.
Attacks become harder under clipping median due to the norm clipping but our RL-based attack still reduces global model accuracy to around 50% on average.
This is mainly because it targets long-term return while all other baselines are myopic.
For example, in Figure 3(c), the global model accuracy drops significantly under all the attacks when five malicious devices are sampled around epoch 200.
After that, the RL method keeps the accuracy at a low level, while other baselines’ accuracy rebounds rapidly.
6.3 Ablation studies
Impact of the number of attackers.
Previous studies on untargeted model poisoning in federated learning typically assume a relatively large fraction of attackers.
For example, the default setting is 20% in [20] and 40% in [65].
Figure 4(a) shows that our RL-based attack obtains superb performance even when the number of attackers (among 100 total clients) is as low as 8.
In contrast, neither IPM nor LMP obtains meaningful attack performance even with 10 attackers.
For 7 attackers, none of the baselines including the RL-based attack can cause significant damage to the FL system.
Impact of non-i.i.d. degree .
Figure 4(b) shows the impact of data heterogeneity on attack performace.
We use 5 different random seeds for all attacks and show the error bars.
We observe that all the baselines obtain similar performance under different non-i.i.d.
degrees and the impact of randomness in the testing environment on their performance is limited.
On the other hand, we observe that the RL policies for q “ 0 . 1 and q “ 0 .
5 exhibit large variances, but even the worst-case performance of our attack outperforms the best cases of all the baselines.
For q “ 0 . 3 , the RL-based attack can always lead to a model with a very high loss so that the model accuracy stays at a low level and is close to a constant, which explains the observed low variance in model accuracy.
We expect that the variation across different RL policies is in part because the attackers always use the latest trained policy for attack execution, which does not necessarily give the best performance among all the intermediate policies trained (see also Figure 4(d)).
Importance of distribution learning .
Figure 4(c) compares the global model accuracy of RL-based attack with distribution learning (RL) and that without distribution learning (RL w/o DL).
We observe that in both cases, the model accuracy decreases dramatically after the attack starts at FL epoch 100.
Further, the accuracy of RL w/o DL slightly increases up to 20% , while the accuracy of RL stays below 10% , which is consistent with our expectation that distribution learning allows the attackers to learn a better attack policy.
Figure 4(c) also shows the attack performance of RL 0 , a variant of the
RL-based attack where the attackers only have 200 unlabeled true images used to train the denoising autoencoders, thus completely relying on distribution learning to generate labeled samples needed for policy learning.
Compared with the baseline results in Figure 4(b) ( q “ 0 . 3 ), we observe that RLstill outperforms other baseline methods, further indicating the power of distribution learning.
On the other hand, the fact that RL w/o DL surpasses all the baselines indicates that our approach is still applicable even when distribution learning becomes less effective due to a strong defense against gradient inversion applied at the server side.
Impact of training length on policy learning .
Figure 4(d) shows how the global model loss at the end of an FL training episode (in the simulated environment) varies over the RL policy training steps.
We observe that longer training usually provides a better attack policy, although the training process is not stable.
To fix this, one approach is to set up a separate testing environment to identify best trained policies.
As mentioned above, our RL-based attack achieves promising performance even when the attackers always use the latest policy obtained during policy learning.
Conclusion
We propose a new approach for developing non-myopic attacks that can effectively compromise FL systems even with advanced defense mechanisms applied, by utilizing model-based reinforcement learning as a principled approach.
While we focus on untargeted model poisoning against FL systems in this paper, our attack framework can be extended to targeted attacks (e.g., backdoor attacks) and to objectives beyond global model accuracy (e.g., fairness across clients [44, 36]).
Further, our attack framework can be integrated with meta-learning [21, 27] to generalize the learned policy to different training tasks and develop black-box attacks.
Another direction is to investigate novel methods to defend our adaptive attack methods.
One possible solution would be to dynamically adjust FL parameters such as the subsampling rate or the aggregation rule.
Acknowledgments
This work has been funded in part by NSF grants CNS-1816495 and CNS-2146548 and Tulane
University Jurist Center for Artificial Intelligence.
We thank the anonymous reviewers for their valuable and insightful feedback.
Appendix A Broader Impact
To study the vulnerabilities of federated learning, we propose a model-based reinforcement learning attack framework.
Our work shows that non-myopic attacks can break federated learning systems even when they are equipped with sophisticated defense rules.
This reveals the urgent need of developing more advanced defense mechanisms for federated learning systems.
While we have focused on adversarial attacks against federated learning in our work, we note that one possible solution to defending RL-based attacks would be to dynamically adjust FL parameters such as the subsampling rate or the aggregation rule.
Future work is needed to identify how best to do so.
B Algorithms
Algorithm 1 gives the framework of a standard federated learning algorithm where the aggregation function, Aggr p¨q , can be either a simple average or a robust aggregation rule.
Algorithm 2 gives the details of our distribution learning procedure.
The algorithm first initializes D reconstructed with attackers’ local data.
A synthetic noisy dataset is built by adding Gaussian noise to D reconstructed .
A denoising autoencoder is then learned using paired clean data and noisy data.
In each FL epoch, a batch of dummay data samples are first generated randomly, which are then updated iteratively by matching their average gradient with the aggregated gradient estimated from received model parameters.
When no attacker is sampled in an FL epoch, the same process is applied by reusing the most recent model parameters received from the server.
Due to the randomness of the algorithm, new data samples are generated and added (after denoising) to D reconstructed in each FL epoch during distribution learning.
Algorithm
Learning
Input: Initial weight θ 0 , K workers indexed by k , size of subsampling w , local minibatch size B , step size η , number of global training steps T
Output: θ T
Server executes: for t “ 0 to T ´ 1 do
S t Ð randomly select w workers from K workers for each worker j P S t in parallel do g t ` 1 j Ð WorkerUpdate ( j, θ t ) end for g t ` 1 Ð Aggr p g t ` 1 k 1 , ..., g t ` 1 k w q , k i P S t θ t ` 1 Ð θ t ´ ηg t `end for
WorkerUpdate p j
, θ
q :
Sample a minibatch b of size B g Ð 1 B ř z P b ∇ θ ℓ p θ, z q return g to server
C Discussion on Gradient Inversion Attacks and Defenses
Although federated learning is expected to protect clients’ local data, it has been recently observed that sensitive information can still be inferred from the gradients or model updates shared by clients [29, 74].
In particular, it is shown in [74, 73] that using an optimization based approach, a curious server can extract both the training inputs and labels from the gradients shared by a client for a small batch size ( ď 8 ).
This approach is further improved in [24], where it is shown that by exploiting a magnitude-invariant loss, the proposed inverting gradients (IG) method can reconstruct images in deep non-smooth architectures even in batches of 100 images.
More recently, the GradInversion method [68] and gradient inversion with a trained generative model [30] are capable of reconstructing individual images with high fidelity from averaging gradients even for complex datasets like ImageNet [19], deep networks, and large batch sizes.
Several approaches have been
Input: number of steps for distribution learning τ E , number of iterations for each step max iter , learning rate for FL η learning rate for inverting gradients η 1 , number of reconstructed data per epoch B 1 , and model parameters t θ t p τ q u
Output: D reconstructed
D Reconstructed Ð M attackers’ local data
D Noisy Ð Add Gaussian noise to D reconstructed and clip data to the valid range
Train a denoising autoencoder A denoise using D reconstructed and D noisy for τ “ 0 to τ E do
Generate D dummy with B 1 random data and label pairs
Compute aggregated gradient ¯ g τ Ð p θ t p τ ´ 1 q ´ θ t p τ q q{p η p t p τ q ´ t p τ ´ 1 qqq for i “ 0 to max iter ´ 1 do
F dummy p θ q Ð 1 Bř p x j ,y j qP D dummy ℓ p θ ; p x j , y j qq
L Ð 1 ´ x ∇ θ F dummy p θ t p τ q q , ¯ g τ y || ∇ θ F dummy p θ t p τ
q q||¨|| ¯ g τ || ` βBř p x j ,y j qP D dummy TV p x j q x j Ð x j ´ η 1 ∇ x j L , y j Ð y j ´ η 1 ∇ y j L , @p x j , y j q P D dummy end for
Denoise the dummy batch D dummy using A denoise and add it to D reconstructed end for proposed to counter inference attacks.
This includes methods that inject a limited amount of statistical noise into model updates [2, 25], and approaches that learn to perturb data representation [55] such that the reconstructed data from the representation is dissimilar to the raw data, while FL performance is maintained.
However, it is unclear if these defenses can provide sufficient protection in the face of more advanced attacks.
Further, they introduce extra overhead at the server side.
On the other hand, our attack framework only requires a rough estimate of the joint distribution of clients’ local data and can tolerate a certain level of inaccuracy in the learned dataset, which provides the attacker with extra flexibility.
D Proof of Theorem
D.1 Preliminaries
Our theoretic analysis relies on the following definitions and results.
First, we formally define the
Wasserstein distance [61], which will be used to measure the distance between the estimated and true data distributions as well as the distance between the corresponding transition dynamics introduced by different data distributions.
Definition 1.
(Wasserstein distance)
Let p M , d q be a metric space and
P p p M q the set of all probability measures on M with finite p th moment, then the p th Wasserstein distance between two probability distributions µ 1 and µ 2 in P p p M q is defined as:
[Equation]
where J is the collection of all joint distributions j on M ˆ M with marginals µ 1 and µ 2 .
In the following, we focus on 1-Wasserstein distance and denote W p µ 1 , µ 2 q : “ W 1 p µ 1 , µ 2 q .
Wasserstein distance is also known as “Earth Mover’s distance” and measures the minimum expected distance between two sets of points where the joint distribution is constrained to match their corresponding marginals.
Compared with Kullback-Leibler (KL) divergence and Total Variation (TV) distance,
Wasserstein distance is more sensitive to how far the points are from each other [7].
We will also need the following special form of Lipschitz continuity from [7].
Definition 2.
(Lipschitz Continuity)
Given two metric spaces p M 1 , d 1 q and p M 2 , d 2 q , a function f : M 1 Ñ M 2 is Lipschiz continuous if the Lipschiz constant, defined as
[Equation]
is finite.
Similarly, a function f : M 1 ˆ A Ñ M 2 is uniformly Lipschitz continuous in A if:
[Equation]
is finite.
Let M “ p S, A, T, r q be a generic MDP, where S and A denote the state space and the action space respectively, T p s 1 | s, a q denotes the probability of reaching a state s 1 from the current state s and action a , and r p s, a, s 1 q denotes the reward given the current state s , action a , and the next state s 1 .
We then introduce the concept of Lipschiz model class from [7], which allows us to represent the stochastic transition dynamics of an MDP as a distribution over a set of deterministic transitions.
Definition 3.
(Lipschitz model class)
Given a metric state space p S, d S q and an action space A , let
F g be a collection of functions: F g “ t f : S Ñ S u distributed according to g p f |
a q where a P A .
We say that F g is a Lipschitz model class if
[Equation]
is finite.
We say that a transition function T is induced by a Lipschitz model class F g if T p s 1 | s, a q “ ř f 1 p f p s q “ s 1 q g p f | a q for any s, s 1 P S and a P A .
We will later show that the transition dynamics of our MDP model for attackers is induced by a
Lipschitz model class.
Finally we give a formal definition of finite-horizon value functions [60].
Definition 4.
Given an MDP M and a stationary policy π , the value function of π at time l is defined as V π M ,l p s q : “ E π,T r ř H ´ 1 t “ l r p s t , a t q| s l “ s s , where r p s, a q : “ E s 1 „ T p¨| s,a q r r p s,
[Equation]
, s 1 qs
[Equation]
π M ,l p¨q satisfies the following backward recursion form:
[Equation]
with V π M ,H ´ 1 p s q “ E a „ π p s q r r p s, a qs .
The optimal value function is defined as V ˚ M ,l p s q : “ max π V π M ,l p s q for any s .
To analyze the impact of inaccurate transition probabilities on the value function, we also make use of the following lemmas [7].
Lemma 1.
Given two distributions µ 1 and µ 2 over states S , a transition function T induced by a
Lipschitz model class F g is uniformly Lipschitz continuous in action space
A with a constant:
[Equation]
Lemma 2.
Given a Lipschiz function f : S Ñ R with constant K d S ,d R p f q :
[Equation]
Below we state the assumptions needed for establishing Theorem 1.
The first assumption models the inaccuracy of distribution learning as well as the heterogeneity of benign workers’ local data.
Assumption 1.
W p r P, p P k q ď δ for any benign worker k .
We further need the following standard assumptions on the loss function.
Assumption 2.
Let Z denote the domain of data samples across all the workers.
For any s 1 , s 2 P S and z 1 , z 2 P Z , the loss function ℓ : S
ˆ Z Ñ R satisfies:
| ℓ p s 1 , z 1 q ´ ℓ p s 2 , z 2 q| ď L }p s 1 , z 1 q ´ p s 2 , z 2 q}(Lipschitz continuity w.r.t.
s and z ); 2. } ∇ s ℓ p s 1 , z 1 q ´ ∇ s ℓ p s 1 , z 2 q} 2 ď L z } z 1 ´ z 2 }(Lipschitz smoothness w.r.t.
z ); 3.
ℓ p s 2 , z 1 q ě ℓ p s 1 , z 1 q ` x ∇ s ℓ p s 1 , z 1 q , s 2 ´ s 1 y ` α 2 } s 2 ´ s 1 } 22 (strong convexity w.r.t.
s ); 4.
ℓ p s 2 , z 1 q ď ℓ p s 1 , z 1 q ` x ∇ s ℓ p s 1 , z 1 q , s 2 ´ s 1 y ` β 2 } s 2 ´ s 1 }
(strong smoothness w.r.t.
s );
ℓ p¨ , ¨q is twice continuously differentiable with respect to s .
where }p s 1 , z 1 q ´ p s 2 , z 2 q} 22 : “ } s
´ s 2 } 22 ` } z 1 ´ z 2 } 22 .
For simplicity, we further make the following assumption on the FL environment, although our analysis can be readily applied to more general settings.
Assumption 3.
The server adopts FedAvg without subsampling ( w “ K ).
All workers have same amount of data ( p k “ 1 K ) and the local minibatch size B “ 1 .
In each epoch of federated learning, each normal worker’s local minibatch is sampled independently from the local empirical data distribution p P k .
D.2 Measuring the uncertainty: from data distributions to total returns
Let M “ p S, A , T, r,
H q denote the true MDP for attacking the federated learning system, and Ă
M “ p S, A , T 1 , r 1 , H q the estimated MDP used in the policy learning stage, where T 1 and r 1 are derived from the estimated joint data distribution t r P k u where r P k “ p P k when k is an attacker and r
P k “ r P otherwise.
Our main goal is to compare the optimal attack performance that can be obtained from the true MDP M and that derived from the simulated MDP Ă M .
We will focus on understanding the impact of inaccurate data distributions (obtained from distribution learning) and assume that other system parameters are known to the attackers.
Without loss of generality, we assume that the M attackers’ indexes are from K ´ M ` 1 to K .
Let r M s “ t K ´ M ` 1 , ..., K u denote the set of attackers and ϵ “ K ´ MM the fraction of benign nodes.
We consider the idealized setting where the M attackers are perfectly coordinated by a single leading attacker.
Because of these simplifications, the state s t in each epoch t is completely defined by the current model parameters θ t .
With a slight abuse of notation, we assume S “ Θ in the following.
Let J M p
π q : “ E π,T,µ
r ř H ´ 1 t “ 0 r p s t , a t , s t ` 1 qs denote the expected return over H attack steps under the MDP M , policy π and initial state distribution µ 0 .
Let π ˚ be an optimal policy of M that maximizes J M p π q .
Define J Ă M p π q similarly and let r π ˚ be an optimal policy for Ă M , with the same initial state distribution µ 0 .
Our analysis is built upon the following lemma that compares the performance of π ˚ and that of r π ˚ with respect to the true MDP M .
It extends a similar result in [69] to a finite-horizon MDP where the reward in each step depends on not only the current state and action but also the next state.
Note that the lemma relies on the key assumption that both V ˚ M ,l p¨q and V ˚ Ă M ,l p¨q are L v -Lipschitz continuous (with respect to the l 2 norm of states) for all l .
That is, | V ˚ M ,l p s 1 q ´ V ˚ M ,l p s 2 q| ď L v } s 1 ´ s 2 } 2 for any s 1 , s 2 P S where L v is a constant independent of l .
A similar requirement holds for V ˚ Ă M ,l p¨q .
Let
W p T,
[Equation]
Lemma 3.
Assume Assumption 1 and Assumption 2.1 holds and both V ˚ M ,l p¨q and V ˚ Ă M ,l p¨q are
L v -Lipschitz continuous for all l .
Then,
[Equation]
Proof.
Let F l be the expected return when π ˚ is applied to Ă M for the first l steps, then switching to
M for l to H ´ 1 .
That is,
[Equation]
r t p s t , a t , s t
[Equation]
1 q ff
By the definition of F l , we have J M p
π ˚ q “ F 0 and J Ă M p π ˚ q “ F H , which implies that J M p π ˚ q ´
J Ă M p π ˚ q “ ř H ´ 1 l “ 0 p F l ´ F l ` 1 q .
Note that
[Equation]
where R l ´ 1 is the expected return of the first l ´ 1 steps, which are taken with respect to Ă M .
Thus,
[Equation]
Define G ˚ Ă M ,l p s l , a l q : “ E s l ` 1 „ T p s l
[Equation]
a
[Equation]
r V
[Equation]
p s l `
[Equation]
qs ´ E s l ` 1 „ T 1 p s l ,a l q r V ˚ M
[Equation]
l p s l ` 1 qs .
We have
[Equation]
a l q r 1 K
K ÿ k “ℓ 1 k p s l ` 1 q ´ ℓ 1 k p s l qqs ˙ `
H ´ 1 ÿ l “
E s l ,a l „ T 1 ,π ˚ r G ˚ Ă M ,l p s l , a l qs “
H ´ 1 ÿ l “
E s l ,a l „ T 1 ,π ˚ ˜
E s l ` 1 „ T p s l ,a l q r 1 K
K ÿ k “ℓ k p s l ` 1 qs ´ E s l ` 1 „ T 1 p s l ,a l q r 1 K
K ÿ k “ℓ 1 k p s l ` 1 qs ¸ `
H ´ 1 ÿ l “
E s l ,a l „ T 1 ,π ˚ ˜
K
K ÿ k “ℓ 1 k p s l q ´ 1 K
K ÿ k “ℓ k p s l q ¸ `
H ´ 1 ÿ l “
E s l ,a l „ T 1 ,π ˚ r G ˚ Ă M ,l p s l , a l qs where ℓ k p s q : “ E z k „ p P k r ℓ p s, z k qs , ℓ 1 k p s q : “ E z k „ r P k r ℓ p s, z k qs and the third equality follows from the definition of reward function r p s, a, s 1 q “ 1 K ř Kk
“ 1 ℓ k p s 1 q ´ 1 K ř Kk “ 1 ℓ k p s q , and r 1 p s, a, s 1 q “ 1 K
[Equation]
Since V ˚ M ,l is L v -Lipschitz, we have | G ˚ Ă M ,l p s, a q| ď L v W p T p s, a q , T 1 p s, a qq from the definition of 1-Wasserstein distance.
We further have
[Equation]
where the second inequality follows from the definition of 1-Wasserstein distance and Assumption 2.1, and the last inequality follows from Assumption 1 and the fact that r P k “ p P k for any attacker k
[Equation]
Similarly
[Equation]
we have
[Equation] [Equation]
where the last inequality follows Assumption 1, Assumption 2.1, and the property of 1-Wasserstein distance with respect to product measures.
Combining the above results, we have
[Equation]
A similar argument shows that
[Equation]
Let U : “ H p L v ` L q W p T, T 1 q ` 2 HLϵδ .
We have
[Equation]
As indicated in [69], an important obstacle to applying Lemma 3 to real reinforcement learning problems is to bound the Lipschitz constant L v for optimal value functions.
Further, we need to bound W p T, T 1 q , the 1-Wasserstein distance between two transition functions.
We study these two problems in the following two subsections, respectively.
D.3 Lipschitz constants of value functions
In this section, we show that the Lipschitz constant L v can be upper bounded for any optimal value function in our setting.
We first rewrite the update of model parameters in each epoch of FedAvg as follows:
[Equation]
where s denotes the parameters of the current global model, z “ t z k u denotes the set of data points sampled by each worker.
Note that the above equation gives the one-
step deterministic transition
when the data samples are fixed.
An important observation is that the transition function T is induced by a Lipschitz model class F g “ t f z : z P Z K u with g p
f
z | a q equal to the probability that z is sampled according to the joint distribution ś k Pr K s p P k .
Similarly, T 1 is induced by F g 1 “ t f z : z P Z K u with g 1 p f
z | a q equal to the probability that z is sampled according to the joint distribution r
P K ´ M ś Kk “ K ´ M ` 1 p P k .
This observation allows us to apply the techniques in [7] to bound the
Lipschitz constant L v of an optimal value function once we bound the Lipschitz continuity of individual f z .
We first show that for any joint action a “ t ˜ g i u i Pr M s , the deterministic transition f z p¨ , a q is Lipschitz continuous with a Lipschitz constant K d S ,d S p f z p¨ , a qq that can be upper bounded independent of z .
Lemma 4.
Assume Assumptions 2.3, 2.4, and 2.5 hold.
For any Lipschitz model class F g “ t f z : z P Z K u , we have K F ď max t ϵ | 1 ´ ηα | , ϵ
| 1 ´ ηβ |u .
Proof.
It suffices to show that for any action a , K d S ,d S p f z p¨ , a qq ď max t ϵ | 1 ´ ηα | , ϵ
| 1 ´ ηβ |u .
By (1), we have for any s 1 , s 2 P S ,
[Equation]
[Equation]
where (a) follows from the triangle inequality, (b) follows from the fact that ℓ p s, z q is twice continuously differentiable with respect to s and the mean value theorem, where ¯ s is a point on the line segment connecting s 1 and s 2 , and I is the identity matrix with its dimension equal to the dimension of the model parameters, and (c) is due to the Cauchy–Schwarz inequality.
By the strong convexity and smoothness of ℓ p s, z q with respect to s , the eigenvalues of B 2 ℓ p ¯ s,z k q B sare between α and β [47].
It follows that
[Equation]
Therefore, for any s 1 , s 2 ,
[Equation]
By Definition 2, we then have
[Equation]
Note that by using a small enough learning rate η , K F can be made less than 1 so that the one-step deterministic transition becomes a contraction.
We next show that the optimal value function V ˚ M ,l p¨q has a bounded Lipschitz constant.
Note that the bound is independent of M ; hence it also applies to
V ˚ Ă M ,l p¨q
Lemma 5.
Assume Assumptions 2.1, 2.3, 2.4, and 2.5 hold.
The optimal value function V ˚ M ,l p¨q is
Lipschitz continuous with a Lipschitz constant bounded by ř H ´ l ´ 1 t “
[Equation]
Proof.
The proof is adapted from the proof of Theorem 3 in [7].
Let Q π M ,l p s, a q : “ r p s, a q ` ř s 1 P S T p s 1 | s, a q V π M ,l ` 1 p s 1 q denote the state-action value function, where r p s, a q “
E s 1 „ T p s 1 | s,a q r r p s, a, s 1 qs .
We have for the optimal state-action value function
[Equation]
with Q ˚ M ,H ´ 1 p s, a q “ r p s, a q .
The Lipschitz constant of Q ˚ M ,l is bounded by:
[Equation]
where (a) follows Lemma 2 and (b) is due to the fact that the max operator is 1-Lipschitz, that is,
K }} 8 ,d R p max p x qq “ 1 [6].
From the definition of r p s, a q , we further have
[Equation]
where ℓ k p s q : “ E z k „ p P k r ℓ p s, z k qs .
The first term of the second inequality comes from the
Lipschitz continuity of the loss function ℓ , which gives | ℓ k p s 1 q ´ ℓ k p s 2 q| ď L } s 1 ´ s 2 } 2 for any k , and the second term follows from Lemma 2 by letting f p s q “ ℓ k p s q , which gives
K Ad S ,d R p E s 1 „ T r ℓ k p s
[Equation]
qsq ď LK AW
[Equation]
W p
[Equation]
q for all k .
Since the above inequality holds for any a P A , r p s, a q is uniformly Lipschitz continuous in action space
A with a Lipschitz constant
K Ad S ,d R p r q ď L ` LK AW,W p T q .
Thus, K Ad S ,d R p Q ˚ M
[Equation]
l
[Equation]
Since the optimal value function V ˚ M ,l p s q “ max a P A Q ˚ M ,l p s, a q and the max operator is 1Lipschitz [6], we have K d S ,d R p V ˚ M ,l q ď K Ad S ,d R p Q ˚ M ,l q ď
[Equation]
We obtain the desired result by applying Lemma 1.
The lemma immediately implies that V ˚ M ,l p¨q is L v -Lipschitz for any l where L v ď ř H ´ 1 t “ 0 p K F q t p
L
[Equation]
LK F q .
D.4 Wasserstein distance between transitions
In this section, we bound the 1-Wasserstein distance of transition functions.
Recall that the true transition dynamics T p¨| s, a q depends on the joint distribution ś Kk “ 1 p P k , while T 1 p¨| s, a q depends on r
P K ´ M ś Kk “ K ´ M ` 1 p P k .
We have the following lemma.
Lemma 6.
Assume Assumptions 1-3 hold.
For any state-action pair p s, a q , the 1-Wasserstein distance between transition dynamics T p¨| s, a q and T 1 p¨| s, a q generated from the real FL environment and the estimated environment, respectively, is bounded by ηL z ϵδ , that is,
[Equation]
Proof.
Let z 1 “ t z 1 k u k “ 1 ,...,K ´ M and z 2 “ t z 2 k u k “ 1 ,...,K
´ M denote two data sets of normal workers sampled from ś K ´ M k “p P k and r P K ´ M respectively.
Let j “ ś K ´ M k “ 1 j k denote an arbitrary coupling between the two joint distributions that is independent across workers where j k denotes a coupling between p P k and r P .
Let J denote the set of all such couplings.
Let J s denote the collection of couplings between T p¨| s, a q and T 1 p¨| s, a q generated from the couplings of joint distributions in J .
To simplify the notation, let s p z q : “ f z p s, a q denote the successive state given the current state-action pair p s, a q and the sampled data z of normal workers.
From the definition of 1-Wasserstein distance, we have
[Equation]
“ inf j P J ÿ p z 1 ,z 2 q ›››››
K K
[Table]
M ÿ k “
[Table]
∇ s ℓ p s, z
[Table]
q
[Table]
1 K
K
[Table]
ÿ
[Table] [Table]
∇
[Table]
z
[Table]
q ›››››
[Table]
ź
[Table]
j
[Table]
p z
[Table]
p c q ď ηL
[Table]
K inf
[Table]
ÿ p
[Table]
ÿ
[Table] [Table]
ź
[Table]
ÿ
[Table]
K
[Table]
ÿ
[Table]
ÿ
[Table] [Table]
K
[Table]
ÿ
[Table]
W
[Table]
P
[Table]
p
[Table]
ď ηL
[Table]
K p K
[Table]
M q δ where (a) is due to the fact that we consider a restrictive collection of couplings, (b) is due to the fact that J s is generated from J , (c) follows from the smoothness of ℓ p s, z q with respect to z , (d) is due to j k p z 1 k , z 2 k q ď 1 , @ k , and (e) follows from Assumption 1.
D.5 Difference between expected returns
Combining the results from the previous three sections, we have the following main result.
Theorem 1.
Assume Assumptions 1-3 hold.
Let J M p
π q : “ E π,T,µ
r ř H ´ 1 t “ 0 r p s t , a t , s t ` 1 qs denote the expected return over H attack steps under MDP M , policy π and initial state distribution µ 0 .
Let π ˚ and r π ˚ be optimal policies for M and Ă M respectively, with the same initial state distribution µ 0 .
Then, | J M p π ˚ q ´ J M p r π ˚ q| ď 2 Hϵδ rp L ` L v q ηL z ` 2 L s where L v ď ř H ´ 1 t “ 0 p K F q t p
L ` LK F q and K F ď ϵ max t| 1 ´ ηα | , | 1 ´ ηβ |u .
Proof.
By Lemma 3, | J M p
π ˚ q ´ J M p
r π ˚ q| ď 2 H rp L ` L v q W p T, T 1 q ` 2 Lϵδ s .
From Lemma 6, we have W p T, T 1 q ď ηL z ϵδ .
Thus, | J M p π ˚ q ´ J M p r π ˚ q| ď 2 H rp L ` L v q ηL z ϵδ ` 2 Lϵδ s .
By
Lemma 5 and the comment below it, L v ď ř H ´ 1 t “ 0 p K F q t p
L ` LK F q where K F ď ϵ max t| 1 ´ ηα | , | 1 ´ ηβ |u .
E Experiments E.1 Experiment setup
Datasets.
We consider four real world datasets: MNIST [34], Fashion-MNIST [64], Balanced
EMNIST [17], and CIFAR-10 [33], and a synthetic dataset.
Both MNIST and Fashion-MNIST include 60 , 000 training examples and 10 , 000 testing examples, where each example is a 28 ˆgrayscale image, associated with a label from 10 classes.
Balanced EMNIST includes 112 ,training examples and 18 , 800 testing examples, where each example is a 28 ˆ 28 grayscale image, associated with a label from 47 classes.
CIFAR-10 consists of 60,000 color images in 10 classes of which there are 50 , 000 training examples and 10,000 testing examples.
Details about the synthetic data are given in Appendix E.2.
For the i.i.d.
setting, we randomly split the dataset into K groups, each of which consists of the same number of training samples.
For the non-i.i.d.
setting, we follow the method of [20] to quantify the heterogeneity of local data distribution across clients.
Suppose there are C classes in the dataset, e.g., C “ 10 for the MNIST, Fashion-MNIST, and CIFARdatasets.
We evenly split the worker devices into C groups, where each group is assigned 1 { C of training samples as follows.
A training instance with label c is assigned to the c -th group with probability q ě 1 { C and to every other group with probability p 1 ´ q q{p C ´ 1 q .
Within each group, instances are evenly distributed.
A higher q indicates a higher non-i.i.d. degree.
We set q “ 0 .as
the default non-i.i.d.
degree.
To demonstrate the power of distribution learning, we assume that the set of attackers share m true data points sampled from the training instances assigned to them.
We set m “ 200 for MNIST and Fashion-MNIST, m “ 500 for EMNIST, and m P t 500 , 5000 u for
CIFAR-10.
Federated learning setting.
We adopt the following parameters for the federated learning models: learning rate η “ 0 . 01 ( 0 . 05 for EMNIST and the synthetic data), total number of workers “ 100 , number of attackers “ 20 , subsampling rate “ 10% , and number of total epochs “ 1000 .
For the
MNIST, Fashin-MNIST, and EMNIST datasets, we train a neural network classifier consisting of 8×8, 6×6, and 5×5 convolutional filter layers with ReLU activations followed by a fully connected layer and softmax output.
The cross-entropy loss is used to optimize the model.
For CIFAR-10, we use the ResNet-18 model [28].
We set the local batch size B “ 128 .
We implement the FL model with
PyTorch [45] and run all the experiments on the same 2.30GHz Linux machine with 16GB NVIDIA
Tesla P100 GPU.
We simulate subsampling and local data sampling with different random seeds in each test run.
Error bars are reported in Figure 4(c) in the main paper.
We set cross-entropy as our default loss function, and stochastic gradient descent (SGD) as our default optimizer.
Baselines.
We compare our RL-based attack (RL) with no attack (NA), and the state-of-theart model poisoning FL attack methods: explicit boosting (EB) [11], inner product manipulation (IPM) [65], and local model poisoning attack (LMP) [20].
The EB attack [11] is originally proposed for the targeted setting.
We adapt it to the untargeted setting by using empirical loss as the objective, which is optimized through multi-step gradient ascent using attackers’ local data, where the number of steps is 5 and the step size equals to the FL learning rate η .
The model update is then boosted by a factor of KM .
We compare our RL-based attack with the full knowledge LMP [20], where the attackers have access to not only the aggregation rule but also all normal workers’ updates.
We use the LMP attack tailored to Krum when the Krum defense is used, and the LMP attack tailored to coordinate-wise median when the clipping median defense or the geometric median defense is used.
Further, we implement the adaptive version of LMP introduced in [15], which requires the attackers to know the server’s updates derived from its root data, as a baseline against the FLTrust defense [15].
In our implementation of IPM [65], we set the default boosting factor (i.e., ϵ in [65]) as 5 .
We consider four representative robust aggregation rules of different types [53]: Krum [12] and geometric median [46], both of which apply client-wise filterings to model updates, coordinate-wise median [67], which adopts a dimension-wise filtering, and FLTrust [15], which requires the server to collect a small training dataset D 0 (called root dataset).
In the experiments, we actually consider an extension of the vanilla coordinate-wise median where a norm clipping step [59] is first applied.
This gives a more powerful defense as we observed in experiments.
We set the default clipping threshold to 2.
In geometric median [46], we set the iteration number of the smoothed Weiszfeld algorithm for computing the geometric median [46] to 10 to balance effectiveness and efficiency.
In FLTrust, the root data is used to calculate a server model update g 0 “ 1 | D 0 | ř z P D 0 r ∇ θ ℓ p θ ; z qs in each epoch.
The aggregation weight of each received client’ update is then determined through its ReLU-clipped cosine similarity with g 0 .
Given that the server has no access to the true training data distribution, the root dataset is often biased in practice.
We adopt the approach in [15] to model such bias.
Among the | D 0 | root data samples, a fraction q 0 of them are sampled from a certain class c in the training data, and the rest are sampled from other classes with equal probabilities.
For a dataset with C classes, Dis unbiased only when q 0 “ 1 { C .
We set the size of root dataset | D 0 | “ 100 following [15].
Distribution learning setting.
In distribution learning, we set the step size for inverting gradients η 1 “ 0 . 05 , the total variation parameter β “ 0 . 02 , optimizer as Adam, the number of iterations for inverting gradients max iter “ 10 , 000 , and learn the data distribution from scratch.
The number of steps for distribution learning is set to τ E “ 100 .
32 images are reconstructed (i.e., B 1 “ 32 ) and denoised in each FL epoch.
If no attacker is selected in the current epoch, the aggregate gradient estimated from previous model updates is reused for reconstructing data.
To build the denoising autoencoder, a Gaussian noise sampled from 0 .
3 N p 0 , 1 q is added to each dimension of images in
D reconstructed , which are then clipped to the range of [0,1] in each dimension.
Policy learning setting.
In policy learning, we implement our simulated environment with OpenAI
Gym [14] and adopt OpenAI Stable Baseline3 [48] to implement Twin Delayed DDPG (TD3) [22] and Proximal Policy Optimization (PPO) [51] algorithms.
We find that TD3 gives better results in most cases and report the results for TD3 below.
The default parameters are described as follows: the length of simulating environment = 1 , 000 epochs, policy learning rate = 1 e ´ 7 , the policy model is
MultiInputPolicy , batch size = 256 and gamma = 1 for updating the target networks.
As described in Section 4.3, we compress the MDP state to include the parameters of the last hidden layer of θ t p τ q and the number of attackers sampled, m t p τ q , where each last hidden layer parameter is in r´8 , `8s and m t p τ q is in t 0 , . .
. , 10 u .
In our experiment, we restrict all attackers to take the same action in each epoch.
In solving the local search problem, we fix the number of trajectories
G “ 1 and the size of minibatch r B “ 200 (except for FLTrust where r B “ 500 ).
For the Krum, clipping median, and geometric median defenses, the local search objective is F p θ q “ E z „ r P r ℓ p θ ; z qs (i.e., λ “ 0 ).
In this case, the action space becomes p γ, E q , where γ P r 0 , 10 s and
E P t 0 , . .
, 20 u for the Krum defense, and γ P r 0 , 10 s and E P t 0 , . . .
, 50 u for the clipping median and geometric median defenses.
Since TD3 can only be applied to a continuous action space, we consider a continuous interval for E (e.g., E P r 0 , 20 s for Krum) when updating the policy and round its value to an integer in the feasible range before the action is applied.
For FLTrust, we consider two cases, when the attackers have access to the server’s root data D 0 or equivalently, the model update g 0 in each epoch, and when they only know how D 0 is sampled from the true training data distribution.
Note that even the former setting is more realistic than the adaptive
LMP setting in [15], which also requires access to normal workers’ updates.
In the former case, we slightly modify the local search method described in Section 4.3 by fixing γ p θ t p τ q q “ } g 0 p θ t p τ
q
q}and considering the same local search objective L p θ q : “ p 1 ´ λ q F p θ q ` λ cos p θ t p τ q
´ θ, g 0 p θ t p τ q qq with the extra constraint that } θ t p τ q ´ θ } 2 ď } g 0 p θ t p τ q q} 2 .
This is because FLTrust normalizes all the local model updates using the magnitude of the root update.
In the latter case, we use the same objective but approximate g 0 p θ t p τ q q with E z q 0 „ r P r ∇ θ ℓ p θ t p τ q ; z qs , where q 0 models the bias of root data, which is assumed to be known to the attackers.
In both cases, the action space is then p E, λ q with E P t 0 , . . .
, 20 u and λ P r 0 , 1 s .
We further find that when the root data D 0 is known (or can be well approximated), the RL-based attack can be made more efficient by considering an alternate local search objective L p θ q : “ p 1 ´ λ q F p θ q ´ λF
p θ q , where F 0 p θ q “ 1 | D 0 | ř z P D 0 r ℓ p θ ; z qs is the empirical loss associated with the root data.
Intuitively, the attackers aim to push the model parameters towards the region that can overfit the root data.
In our experiments, the initial model for all training episodes is set as the first model the attackers received from the actual FL environment.
We assume that the server waits for 72 seconds to receive the updates from the workers before performing a model aggregation, which allows 80 , 000 total time steps (i.e., 80 episodes) of policy learning for Krum, 40 , 000 total time steps (i.e., 40 episodes) of policy learning for clipping median, and 40 , 000 total time steps (i.e., 40 episodes) of policy learning for FLTrust within 400 FL epochs in our experiment setting.
It is more time consuming to train an RL policy for clipping median and FLTrust because large attack bounds need to be considered.
See E.2 for a detailed comparison of the running time of different stages of the RL-based attack under different defense scenarios.
Attack execution setting.
Both the distribution learning and policy learning phases in the RL-based attack start at the first FL epoch.
The former ends at the 100 th FL epoch when RL-based attack starts
All other attacks start at epoch 0.
For fair comparisons, we fix all the random seeds for generating the initial model and the root data (for FLTrust), subsampling, and local data sampling when evaluating different attacks.
We observe that both EB and RL can occasionally produce NaNs in model updates, which when incorporated by the server, can lead to bad models in all future steps.
This produces unrealistic attack scenarios as NaNs can be easily detected by the server.
To have a fair comparison with other attacks, we use the built-in VecCheckNan Wrapper in OpenAI Stable Baseline3 [48] to detect abnormal values.
We assume that attackers take less ambitious actions (i.e., p 0 .
5 γ, E ´ 1 q ) in that epoch once they detect a NaN value.
If E “ 0 or γ “ 0 , the attackers send r g t p τ q “ 0 to the server.
E.2 More experiment results
Attack performance on other datasets.
Figures 5 and 6 compare the test accuracy of the global model under different attacks when the server uses Krum or clipping median as the defense for the
Fashion-MNIST and EMNIST datasets.
We consider both i.i.d.
and non-i.i.d.
( q “ 0 . 5 ) settings.
Our RL-based attack constantly outperforms other baselines by a large margin in all the settings.
We observe that in most cases, all attacks are more effective in the non-i.i.d.
setting.
This is mainly because a higher degree of local data heterogeneity increases the variance across normal workers’ updates, making it more difficult to filter out adversarial updates.
Further, clipping median, which adopts both dimension-wise filtering and client-wise norm clipping to model updates, provides a stronger level of defense than Krum, which only applies client-wise filtering to model updates.
In particular, our attack can reduce the model accuracy to an extremely low level under the Krum defense, depending on the number of classes of the dataset used ( „ 10% for Fashion-MNIST and „ 2% for EMNIST).
Figure 7 compares the test accuracy of the global model under different attacks for the CIFARdataset in the i .
i .
d . setting.
Here we assume that our RL-based attack does not perform distribution learning, and the attackers use their local data to train the attack policy and start to execute attack at epoch 100.
This is mainly due to the fact that image reconstruction for CIFAR-10 takes prohibitive amount of time in our experiment environment.
Further, state-of-the-art gradient inversion attacks either cannot reconstruct a large batch of images for CIFAR-10 accurately or have not made their code available yet.
We consider two cases where 500 and 5,000 local samples are used to train the attack policy, respectively.
We observe that in both cases, our approach surpasses all the baselines.
In particular, the RL policy trained using only 500 local samples quickly drives the model accuracy to a very low level ( „ %9 . 52 ) under the Krum defense.
Attack performance under geometric median.
We compare the attack performance of RL-based attack and other baselines (i.e., NA, EB, IPM, and LMP) against geometric median [46] on MNIST dataset in the i.i.d.
setting.
As shown in Figure 8(a), RL-based attack and LMP significantly outperform other baselines.
Further, although our RL-based attack starts attacking at the 100th epoch, it quickly drives the model accuracy to a very low level, while LMP takes much longer time to achieve similar attack performance.
Attack performance under noisy gradients.
We also compare the attack performance of our
RL-based attack and other baselines against clipping median aggregation (with the clipping threshold set to 2) under noisy gradients [63].
In particular, the server injects noise into the global model parameters shared with clients, where the noise is sampled from a Laplace distribution [3] (i.e., double exponential distribution) with 0 mean and 1 e ´ 4 exponential decay.
We observe that although adding noise indeed decreases the quality of reconstructed images, distribution learning is still effective for the MNIST dataset.
Further, our RL-based method still outperforms other baselines in this setting as shown in Figure 8(b). [Figure]
Attack performance under FLTrust.
We compare the attack performance of our RL-based attack with and without access to server’s root data (details are given in E.1 policy learning setting) and other baselines (i.e., NA, IPM, and adaptive LMP) against the FLTrust defense on the EMNIST dataset.
For
RL-based attacks
, the attackers use 5,000 local data samples to simulate the environment and skip the distribution learning phase, and start attacking at FL epoch 100.
All the baselines start from the beginning of FL.
We consider both the cases when the root data are unbiased ( q 0 “ 1 { 47 ) and when they are biased against a single class ( q 0 “ 0 . 3 ).
In the former case, our attack with access to root data leads to a significantly low test accuracy ( „ 50% ) as shown in Figure 8(c), while other attacks, including RL-based attack without access to root data, have limited effect against FLTrust.
This is due to the fact that when the root data are unbiased and representative of the true training dataset, the root update g 0 in each epoch provides a good estimate of the right direction for model updates, making it difficult to reverse the trend.
On the other hand, when the root data is biased, which is likely to happen in practice, the root updates are less representative or even misleading.
As shown in Figure 8(d), our RL-based attack with root data access becomes more effective as expected.
Our
RL-based attack without root data also achieves significant although unstable attack performance.
Here we ignore the second term in the local search objective L p θ q by fixing λ “ 0 to minimize the impact of inaccurate estimate of g 0 .
Actual runtime comparison.
The actual runtime varies across the FL environment, the training method used, and most importantly, the amount of computational resource available.
The tables below report the numbers from our current experiment settings (see Appendix E.1) and the way the simulator is implemented (clients are simulated sequentially in each FL epoch).
For MNIST, Fashion-MNIST, and EMNIST, distribution learning takes around 100 seconds to reconstruct a batch of 32 images and we construct 50 batches within 2 hours.
Note that multiple batches can be generated from a single gradient.
We start policy training from the beginning of FL training, and we set 8 hours limit for policy training.
It takes around 0.05 seconds to simulate a single
FL epoch with 10 sampled clients without parallelization.
Total training steps vary across defense policies as stated in Appendix E.1.
With the above numbers, if we assume that each FL epoch takes 72 seconds to finish and there are in total of 1000 FL epochs during FL training, then distribution learning will end before the 100th FL epoch and policy training ends by the 400th FL epochs, and the total FL training time is aroundhours. [Table]
[Table] For CIFAR-10, we do not perform distribution learning in this work and policy learning alone takes about 20 hours in our experiment environment as we use a much bigger network (i e., Resnet18).
However, we expect that once equipped with more powerful devices, the training time can be significantly reduced by parallelly simulating multiple clients using multiprocessing and multiple episodes using vectorized environments, which will make it possible to simulate large FL systems.
In terms of attack executing time, for MNIST with clipping median defense, IPM takes around 0.
seconds to execute an attack in each FL epoch, LMP takes around 7.7 seconds, EB takes around 0.
seconds.
For CIFAR-10 with clipping median defense, IPM takes around 2.55 seconds to execute an attack in each FL epoch, LMP takes around 30 seconds, EB takes around 5.5 seconds.
The execution time of our RL-based method varies over the action space used and it takes around 5.8 seconds andseconds for MNIST and CIFAR-10 respectively with the default action space described in Section 4.3.
Given that each FL epoch typically lasts a minute or longer (72 seconds in our experiment), a few seconds of search time is completely acceptable.
We observe that for defenses such as Krum, it suffices to use the gradients of the last two layers of model parameters as the action.
This approach does not require any online searching and reduces the attack execution time to 0.5s. [Table]
Results for the synthetic data.
In addition to the four real datasets discussed above, we also consider a two-dimensional synthetic dataset and a small network with 28 model parameters to demonstrate the full potential of our RL-based attack framework (i.e., without state and action compression).
We generate the synthetic data based on the method described in [54].
In particular, we generate 55 , 000 data instances (including 50 , 000 training instances and 5 , 000 testing instances), where for each instance z “ p x, y q , the data x P R 2 „ N p 0 , I q and its label y “ sign p} x } 2 q ´ 2 .
Each worker has 500 data instances.
We train a multilayer perceptron (MLP) with two hidden layers of size four and two, respectively, and use ReLU as the activation function.
For our RL-based attack, we consider both the 2-dimensional action space p γ, E q discussed above as well as the generaldimensional action space where the attackers directly decide r g t p τ q to be sent to the server in each epoch.
In both cases, the state space includes the full 28 model parameters and the number of attackers in each epoch.
Policy learning takes 8 , 000 total time steps (i.e., 8 episodes) to learn the policy, within 10 FL epochs.
The attackers use their local data ( 10 , 000 samples) to build a simulated environment without using distribution learning, and start attacking at epoch 0.
We fix all random seeds for a fair comparison across different attacks.
Figure 9 and Figure 10 illustrate the classification boundaries at the end of a federated learning episode for all the attacks when the clipping median defense and the FLTrust defense are applied respectively.
The root dataset D 0 for FLTrust is assumed to be known for RL-based attacks.
We observe that all the baseline methods and our RL-based attack with 2d actions have limited effect under clipping median and completely fail under FLTrust.
On the other hand, the RL-based attack with the full 28-dimensional action space reduces the classification accuracy to 68 .
90% (worst-case accuracy for the given environment) under both defenses.
These results indicate the potential of considering large state and action spaces in our RL-based attack when equipped with more computational power and longer training time.
Table 3 shows how the global model accuracy under different attacks and the Krum defense varies over the number of attackers.
The results show that our approach is effective even when the fraction of malicious clients is as low as 5% .