Original: R OBUST F EDERATED L EARNING WITH M AJORITY A DVERSARIES VIA P ROJECTION - BASED R E - WEIGHTING
Revised: I NVARIANT A GGREGATOR FOR D EFENDING AGAINST F EDERATED B ACKDOOR A TTACKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Most robust aggregators for distributed or federated learning assume that adversarial clients are the minority in the system. In contrast, this paper considers the majority adversary setting. We first show that a filtering method using a few trusted clients can defend against many standard attacks.
Revised: Federated learning is gaining popularity as it enables training of high-utility models across several clients without directly sharing their private data. As a downside, the federated setting makes the model vulnerable to various adversarial attacks in the presence of malicious clients.

Original: However, a new attack called Mimic-Shift can circumvent simple filtering.
Revised: Specifically, an adversary can perform backdoor attacks to control model predictions via poisoning the training dataset with a trigger. In this work, we propose a mitigation for backdoor attacks in a federated learning setup. Our solution forces the model optimization trajectory to focus on the invariant directions that are generally useful for utility and avoid selecting directions that favor few and possibly malicious clients. Concretely, we consider the sign consistency of the pseudo-gradient (the client update) as an estimation of the invariance.

Original: To this end, we develop a re-weighting strategy that identifies and down-weights the potential adversaries under the majority adversary regime. We show that our aggregator converges to  a neighborhood around the optimum under the Mimic-Shift attack.
Revised: Following this, our approach performs dimension-wise filtering to remove pseudo-gradient elements with low sign consistency. Then, a robust mean estimator eliminates outliers among the remaining dimensions. Our theoretical analysis further shows the necessity of the defense combination and illustrates how our proposed solution defends the federated learning model.

Original: Empirical results further show that our aggregator achieves negligible accuracy loss with a majority of adversarial clients, outperforming strong baselines
Revised: Empirical results on three datasets with different modalities and varying number of clients show that our approach mitigates backdoor attacks with a negligible cost on the model utility.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Federated learning (FL) is a leading framework for collaboratively training a machine learning(ML) model over local datasets.
Revised: Federated learning enables multiple distrusting clients to jointly train a machine learning model without sharing their private data directly.

Original: The decentralized nature of FL systems has raised concerns about vulnerability – as adversaries can connect to an FL system like other benign users and corrupt theML model while evading detection by standard means (Kairouz et al., 2021).
Revised: However, a rising concern in this setting is the ability of potentially malicious clients to perpetrate backdoor attacks. To this end, it has been argued that conducting backdoor attacks in a federated learning setup is practical (Shejwalkar et al., 2022) and can be effective (Wang et al., 2020). For instance, the adversary can connect to a federated learning system as a legitimate user and conduct a backdoor attack that forces the model to mispredict.

Original: To this end, there is growing literature on the adversarial robustness of FL (Blanchard et al., 2017; Chen et al., 2018;
Revised: The impact of such attacks is quite severe in many mission-critical federated learning applications. For example, anomaly detection is a common federated learning task where multiple parties (e.g., banks or email users) collaboratively train a model that detects frauds or phishing emails.

Original: Xie et al., 2019b; Rajput et al., 2019b; Xie et al., 2020; Karimireddy et al., 2021a; 2022; He et al., 2022b), particularly where adversaries can upload malicious updates. Most existing defenses assume that the adversarial clients are the minority in the system (Blanchard et al., 2017;
Revised: Backdoor attacks allow the adversary to successfully circumvent these detection methods. The most common backdoor attack embeds triggers in the data samples and forces the model to make an adversary-specified prediction when the trigger is observed (Liu et al., 2018; Bagdasaryan et al., 2020).

Original: Chen et al., 2018; Rajput et al., 2019b; Karimireddy et al., 2021a; He et al., 2022b). However, in a federated scenario, the decentralized nature means that it is relatively straightforward for the adversary to be the majority and thus break existing defenses. We call such an adversary the “majority adversary”.
Revised: Thus, an adversary can conduct a backdoor attack by generating a trigger that statistically correlates with a particular label. Once the adversary injects these trigger-embedded backdoor data samples into the training data, the model can entangle the trigger-label correlation and predict as the adversary specifies.

Original: Our work joins a growing literature on robustness with majority adversaries, e.g., Xie et al. (2019b; 2020), motivated by noted practical vulnerabilities.
Revised: Meanwhile, the backdoor attack often does not degrade the predictive accuracy on the benign samples, making backdoor detection difficult in practice (Wang et al., 2020).

Original: Although Shejwalkar et al. (2021) argue that the number of registered clients in a production system (e.g., GBoard) may be too large for the adversary to compromise a majority of them, they neglect the client availability issue in FL. In particular, Kairouz et al. (2021) suggests that, at any given time, only a subset ( < 1%) of clients are available for  the server.
Revised: In federated learning, the server aggregates only the client-level updates (a.k.a. pseudo-gradient or gradient for short) without control over the training procedure or any data samples.

Original: Such a low client availability allows the adversary to become the majority and overwhelm the server utilizing compromised networked devices (e.g., IoT devices) in a similar way as the common distributed denial-of-service (DDoS) attack (Specht & Lee, 2003; Bonguet &
Revised: Such limited visibility of the federated learning server on the client-side training makes defending against backdoor attacks challenging. Common defenses against backdoor attacks aim at identifying the backdoor data samples or poisoned model parameters and usually require access to at least a subset of the training data (Tran et al., 2018;

Original: Bella¨ıche, 2017). Some other settings such as crowd-sourced training (Ryabinin & Gusev, 2020)
Revised: Li et al., 2021a), which is prohibitive for a federated learning server.

Original: (a.k.a. volunteer computing) are perhaps even more vulnerable to majority adversaries because the crowd-sourcing systems do not implement access control – allowing the adversary to connect an arbitrary number of clients as volunteers. We consider the adversarial robustness of federated learning against a class of attacks where an adversary aims to decrease the accuracy of thetrained ML model by uploading malicious updates.
Revised: Other defense methods against untargeted poisoning attacks that degrade the model utility

Original: In particular, we are interested in Mimic-type attacks (Karimireddy et al., 2022), asis discussedlater in this section.
Revised: (Shejwalkar et al., 2022) are applicable but lack robustness against backdoor attacks, as discussed in Section 6.2. Our approach.

Original: A key assumption in our setup is the existence of a few trusted clients, e.g., with secure hardware support. We call these trusted clients “reference clients”.
Revised: Our defense leverages the observation that learning from the poisonous data does not benefit the model on benign data and vice versa.

Original: In practice, the number of reference clients could be as small as two in  each round. Similar approaches have been considered inexisting works (Xie et al., 2019b; 2020). One option for secure hardware is the trusted execution environment (TEE) (Pinto & Santos, 2019), which guarantees that the program is not Byzantine.
Revised: Therefore, focusing on the invariant directions that are generally beneficial in the model optimization trajectory helps defending against the aforementioned backdoor attack (which often lead to non-invariant directions). To this end, we develop a defense by examining each dimension of the gradients on the server-side and checking whether the dimension-wise gradients point in the same direction across the clients.

Original: TEE is so far commercialized (e.g., on Google Pixel (GoogleBlog), Apple iPhone (AppleSupport), Sam- sung phones (SamsungDeveloper)).
Revised: Here, a dimension-wise gradient can point to a positive or negative direction, or have a zero value.

Original: Recent works (Mo et al., 2021) bring TEE support to federated learning systems. We propose a combination of defenses – filtering and projection-based re-weighting. Our first defense is a filtering method that constructs a spherical accept region and excludes the updates outside it. The center of  the accept region is an average update from a few trusted clients with secure hardware support.
Revised: In the case of small learning rates and for a specific dimension, two gradients pointing in the same direction means that taking the direction of one gradient can benefit the other.

Original: The radius of the accept region is the sample variance ofreference updates times a scaling factor.
Revised: As such, the invariance of a direction depends on how many dimension-wise gradients align with that direction.

Original: Although thefiltering method is effective against several standard attacks (e.g., sign flipping attack, Gaussian attack), this filtering is easy to circumvent. Building on the recently proposed Mimic attack (Karimireddy et al., 2022), shown to break many existing defenses, we develop an improved attack method called Mimic-Shift.
Revised: Following this intuition, we define the sign consistency of a dimension by the average gradient sign. The higher the sign consistency is, the more invariant direction the gradient dimension may have. Designing such a method carefully selecting only the invariant gradient directions is non-trivial, especially given the non-i.i.d.

Original: Mimic-Shift clients send malicious updates which slightly shift away from benign updates andmislead the aggregated updates away from the expected update.
Revised: gradient distributions across benign clients and the presence of malicious clients.

Original: The slight shift makes Mimic-Shift hard to detect, and the calibrated shifting direction can corrupt the aggregated model.
Revised: Hence, our approach enforces two separate treatments for each gradient dimension.

Original: To perform the Mimic-Shift attack, we consider a man-in-the-middle (MITM) adversary capable of intercepting the message between the clients and the server.
Revised: First, we employ an AND-mask (Parascandolo et al., 2021), a dimension-wise filter setting the gradient dimension with sign consistency below a given threshold to zero.

Original: Computer security researchers have extensively studied the MITM adversary, but existing encryption solutions can be too expensive for resource-constrained client devices.
Revised: However, this alone is not enough: the malicious clients can still use outliers to mislead the aggregation result in the remaining highly consistent dimensions.

Original: For example, the AES (advanced encryption standard) encryption only has a throughput of around 50 MB/s even with a powerful desktop CPU (Gleeson et al., 2014).
Revised: To address this issue, we propose using the trimmed-mean estimator (Xie et al., 2020b; Lugosi & Mendelson, 2021), as a means to remove the outliers.

Original: With a 1 GB moderate size neural network, the encryption takes more than 20 seconds on the client-side, draining the computational resources and increasing the client dropout rate. Our second defense is a projection-based re-weighting method to deal with the Mimic-Shift attack under a majority adversary regime. The main idea is to measure the influence ofeach update on the aggregated update, then down-weight the updates with high influence.
Revised: Our analysis suggests that the AND-mask complements the trimmed-mean estimator well, motivating their composition. We support the proposed approach with a theoretical analysis under a conventional linear regime (Rosenfeld et al., 2021; Wang et al., 2022; Zhou et al., 2022; Manoj & Blum, 2021), showing that the composition of the AND-mask and the trimmed-mean estimator is necessary for defending against backdoor attacks.

Original: Specifically, we compute the scalar projection of the aggregated update on each client’s update.
Revised: Our analysis starts with feature invariance and discusses the connection between feature invariance and gradient sign consistency.

Original: The intuition is that the majority adversarial clients can significantly mislead the aggregated update, resulting in a large scalar projection. Note that the filtering and re-weighting methods complement each other because the re-weighting defense does not deal with the aforementioned standard attacks, as is discussed in
Revised: Then, we outline conditions under which triggerbased backdoor attacks can lead to non-invariant directions and decrease the sign consistency of a dimension. Further analysis results demonstrate the necessity for the combination of both the AND-mask and the trimmed-mean estimator.

Original: Section 4. We further provide some theoretical analysis of our methods under the Mimic-Shift attack. First, false-positives in filtering can eliminate a benign update and perturb the aggregated update. Regarding this concern, we show that the false positive rate decreases quickly w.r.t.
Revised: Simulation results in Appendix D.1 further verify our theoretical results.

Original:  the number of reference clients and the scaling factor of the accept radius, suggesting that our filtering method can work with a few reference clients and a conservative scaling factor. For the re-weighting phase, the probability of malicious updates having larger scalar projections than benign updates increases as the adversary takes more shares in a system, complementing existing results (He et al., 2022b) that guarantee more robustness as the share of the adversary decreases.
Revised: Our empirical evaluation employs the strong edge-case backdoor attack (Wang et al., 2020), as detailed in Section 6.1, to test our defense. Empirical results on tabular (phishing emails), visual (CIFAR-10) (Krizhevsky, 2009; McMahan et al., 2017), and text (Twitter) (Caldas et al., 2018)

Original: Additionally, we discuss the performance of our method under a conventional minority adversary setting.
Revised: datasets demonstrate that our method is effective in defending against backdoor attacks without degrading utility as compared to prior works.

Original: Finally, we show that, in a convex setting, our method converges with a rate of O ( 1 √ T ) toa neighborhood of the optimum.
Revised: On average, our approach decreases the model accuracy on backdoor samples by 61.6% and only loses 1.2% accuracy on benign samples compared to the standard FedAvg aggregator (McMahan et al., 2017).

Original:  Our contributions aresummarized as follows:
Revised: Contributions. Our contributions are as follows:

Original: • We developthe Mimic-Shift attack and show that Mimic-Shift circumvents many defense methods in federated learning. • We develop a two-stage defense using filtering and re-weighting to defend against a broad class of attacks.
Revised: • We develop a combination of defenses using an AND-mask and the trimmed-mean estimator against the backdoor attack by focusing on the dimension-wise invariant directions in the model optimization trajectory.

Original: • We theoretically analyze our strategy and outline conditions under which it helps.
Revised: • We theoretically analyze our strategy and demonstrate that a combination of an AND-mask and the trimmed-mean estimator is necessary in some conditions.

Original: Empirical results on FEMNIST (Caldas et al., 2018), CelebA (Liu et al., 2015) and Shakespeare (McMahan et al., 2017) datasetsshow that our aggregator recovers a near-optimal model under a majority adversary setting with Mimic-Shift attack, outperforming existing methods by a large margin. Also, our method only loses up to 2.4% accuracy under conventional minority adversary settings. Additional empirical results demonstrate that our method is robust to a broad class of attacks, including standard Gaussian and sign-flipping attacks as well as an improved Mimic-ShiftVar attack.
Revised: • We empirically evaluate our method on three datasets with varying modality, model architecture, and client numbers, as well as comparing the performance to existing defenses.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Many existing works on Byzantine robust machine learning assume the adversary is the minority in the system, based on clustering (Blanchard et al., 2017), median (Yin et al., 2018), voting (Bernstein et al., 2019), bucketing (Karimireddy et al., 2022), and robust estimation (Data & Diggavi, 2021).
Revised: Backdoor Attack. Common backdoor attacks aim at misleading the model predictions using a trigger (Liu et al., 2018). The trigger can be digital (Bagdasaryan et al., 2020), physical (Wenger et al., 2021), semantic (Wang et al., 2020), or invisible (Li et al., 2021b).

Original: However, these methods can fail once the adversary becomes the majority. A few papers (Xie et al.,2019b; 2020) leverage a validation dataset to improve Byzantine tolerance to arbitrary numbers of adversarial clients but constructing a global validation dataset in a federated scenario with local private datasets is infeasible.
Revised: Recent works extended backdoor attacks to the federated learning setting and proposed effective improvements such as gradient scaling (Bagdasaryan et al., 2020) or generating edge-case backdoor samples (Wang et al., 2020). The state-of-the-art edge-case backdoor attack shows that using backdoor samples with low probability density on benign clients (i.e., unlikely samples w.r.t.

Original: Other redundancy-based methods with more robustness guarantees do not apply to  federated learning because they assume a  centralized setting with control over the data allocation on each node (Chen et al., 2018; Rajput et al., 2019a). Recently, a clipping-based method (Karimireddy et al., 2021b; He et al., 2022b) showed success against minority adversary in a federated learning setting.
Revised: the training distribution) are hard to defend in the federated learning setting. Centralized Defense. There is a line of work proposing centralized defenses against backdoor attacks where the main aim is either detecting the backdoor samples (Tran et al., 2018) or purifying the model parameters that are poisoned (Li et al., 2021a).

Original: Although it is relatively straightforward tocombine the clipping with our reference clients, we find that the training is unstable, as is discussed in Section 6.
Revised: However, applying such centralized defense to federated learning systems is in practice infeasible due to limited access to the client data in many implementations. Federated Defenses.

Original: Our paper focuses on model poisoning attacks, which aim to decrease the accuracy of the trained ML model by uploading malicious updates (He et al., 2022b).
Revised: Several recent works have attempted to defend against backdoor attacks in federated learning systems. Sun et al. (2019) shows that weak differential-private (weak-dp) federated averaging can mitigate the backdoor attack.

Original: Other attacks, including data poisoning (Steinhardt et al., 2017; Wang et al., 2021) and backdoor  (Wang et al., 2020; Xie et al., 2021), are beyond the scope of this work.
Revised: However, the weak-dp defense is circumvented by the improved edge-case federated backdoor attack (Wang et al., 2020). Nguyen et al. (2021) suggest that the vector-wise cosine similarity can help detect malicious clients performing backdoor attacks.

Revised: The vector-wise cosine similarity is insufficient when the backdoor attacks can succeed with few poisoned parameters, incurring little vector-wise difference (Wu & Wang, 2021).

Original: We focus only ontraining the centralized model.
Revised: Other defenses against untargeted poisoning attacks (Blanchard et al., 2017; Xie et al., 2020b) lack robustness against the backdoor attack. Sign-SGD with majority vote (Bernstein et al., 2018; 2019) is similar to our approach, but it always takes the majority direction instead of focusing on the invariant directions. Section 6.2 discusses the limitation of previous defenses in more detail, along with the empirical evaluation. Unlike existing works, our defense encourages the model to pursue invariant directions in the optimization procedure.

Original: Additional considerations such as personalization are known to be handled effectively using robust centralized training, followed by local fine-tuning Li et al. (2021), thus are beyond the scope of this paper.
Revised: Certification. Unlike the above discussed defenses, certification (Xie et al., 2021) aims at extinguishing backdoor samples within a neighborhood of a benign sample. A direct comparison between certification and our defense is not meaningful due to the different evaluation metrics.

Revised: Certification considers the certification rate of benign samples as the metric, while our defense aims at reducing the accuracy of the backdoor samples.

Revised: However, it would be interesting to investigate whether the proposed defense can ease the certification of a model.

Original:  P ROBLEM S ETUP
Revised:  P ROBLEM S ETUP

Original:  We assume a  federated learning system where Nbenign clients collaboratively train a ML model f : X −→ Y withd -dimensional parameter ζ coordinated by a server. The N benign clients include
Revised: Notation. We assume a synchronous federated learning system, where N clients collaboratively train an ML model f : X −→ Y with parameter w coordinated by a server. An input to the model are the data samples x ∈ X =

Original: N R reference (trusted) clients.
Revised: R d with d features indexed by k and a label y .

Original: There are N ′ adversarial clients who aim to corrupt the ML model during training.
Revised: There are N ′ < Nadversarial clients aiming at corrupting the ML model during training (Shejwalkar et al., 2022).

Original: The i th , i ∈ [1 , ..., N+ N ′ ] , client has n i data samples, being benign for i ∈ [1 , ..., N  ] or being adversarial for i ∈
Revised: The i th , i ∈ [1 , ..., N ] , client has n i data samples, being benign for i ∈ [1 , ..., N − N ′ ] or being adversarial for i ∈

Original: [ N  + 1 , ..., N+ N ′ ] .
Revised: [ N − N ′ + 1 , ..., N ] .

Original: The  federated learning is conducted in T rounds.
Revised: The synchronous federated learning is conducted in T rounds.

Original: In  round t ∈ [1 , ..., T ] , the server broadcasts a model parameterized by ζ t − 1 to each client.
Revised: In each round t ∈ [1 , ..., T ] , the server broadcasts a model parameterized by w t − 1 to all the participating clients.

Original: We omit the subscript t while focusing on one round.
Revised: We omit the subscript t while focusing on a single round.

Original: Then, the i th client optimizes ζ t −with their local data samples  and report ζ t,i to the server.
Revised: Then, the i th client optimizes w t − 1 on their local data samples indexed by j and report the locally optimized w t,i to the server.

Original: We define pseudo-gradient g t,i = ζ t − 1 − ζ t,i being the difference between the locally optimized model and the broadcasted model from the previous round.
Revised: We define pseudo-gradient g t,i = w t − 1 − w t,i being the difference between the locally optimized model and the broadcasted model from the previous round.

Original: Note, for simplicity, that wewill often use the term ”gradient” to refer to the updates.
Revised: Note, for simplicity, that we often use the term “gradient” to refer to the pseudo-gradient.

Original: Once allthe gradients are reported in, the server aggregates the gradients and produce a new model with parameters ζ t using the following rule: ζ t = ζ t − 1 − (cid:80) N + N ′ i = ni (cid:80) N + N ′ i
Revised: Once all gradients are uploaded, the server aggregates them and produces a new model with parameters w t using the following rule: [Equation] n

Original: = n i g t,i .
Revised: [Equation] g t,i .

Original: Our goal  is to minimize a  risk function over the benign clients: F ( ζ )=
Revised: The goal of federated learning is to minimize a weighted risk function over the N clients: F ( w )

Original: (cid:80) Ni =n i (cid:80) Ni = 1 n i F [Equation]
Revised: =

Original: ( ζ ) = (cid:80) Ni =
Revised: (cid:80) Ni =

Original: [Equation]
Revised: [Equation]

Original: [ ℓ ( f ( x ; ζ ) , y )] , where ℓ : R × Y −→ R is a loss function. 
Revised: where ℓ : R × Y −→ R is a loss function. ⊙ denotes the Hadamard product operator.

Original: 3. T HREAT M ODEL
Revised: Threat Model.

Original: Following the standard practice (Blanchard et al., 2017), we consider an omniscient man-in-themiddle (MITM) adversary that knows g i , ∀ i ∈ { 1 , ..., N } and can tell which clients are reference users, e.g., by observing the device identifier in  the message. An omniscient adversary helps explore the limits of the defense. However, an omniscient adversary is not mandatory for our Mimic-Shift attack, which can work with partial information as Section 6 will show.
Revised: The adversary generates a backdoor data sample x ′ by embedding a trigger in a benign data sample x and correlating the trigger with a label y ′ , which is different from the label y of the benign data sample.

Original: The MITM adversary also owns a majority of clients in a system and adopts an attack from the Mimic family (Karimireddy et al., 2022). Specifically, we assume the following Mimic-Shift attack. Mimic-Shift At each round, the MITM adversary first intercepts the gradients from benign clients. Then the adversary computes the average reference gradient ¯ g R = (cid:80) N R i = n Ri (cid:80) NRi =1 n Ri · g R i from the reference users indexed by R i and the average benign gradient ¯ g = (cid:80) N i = n i (cid:80) N i =1 n i · g i from all benign users, including the reference users.
Revised: We use D ′ to denote the distribution of backdoor data samples.

Original: Then,all the adversarial clients report g ′ = ¯ g R + (¯ g R − ¯ g ) to the server.
Revised: Then, the malicious clients connect to the federated learning system and insert backdoor data samples into the training set.

Original: The Mimic-Shift attack iseffective because it tries to push the aggregated gradient away from the expected gradient. The reason is that ¯ g is estimated with more clients and data samples.
Revised: Since the goal of federated learning is to minimize the risk over all clients’ datasets, the model can entangle the backdoor while trying to minimize the risk over backdoor samples on the malicious clients.

Original:  A simple concentration argument suffices to show that ¯ g will be closer to E [¯ g ] than ¯ g R with high probability, resulting in a ¯ g R − ¯ g pointing away from E [¯ g ] .
Revised: Appendix A visualizes some backdoor samples.

Original: Empirical results in Section 6 show that Mimic-Shift decreases the accuracy twice as much as Mimic (Karimireddy et al., 2022), which sets g ′ =
Revised: [Figure]

Original: g i for an i∈ { 1 , ..., N } . The Mimic-Shift attack is also difficult to detect because the malicious g ′ has the same distance to the reference ¯ g R as the benign ¯ g .
Revised: To simplify tedious notation, we assume all users have the same number of samples i.e., n i = n i ′ , ∀ i ̸

Original: We will see how this property lets Mimic-Shift bypass distance-based defenses in Section 6.
Revised: = i ′ .

Original:  M ETHOD
Revised:  M ETHOD

Original: Our robust aggregator has two phases, filtering and re-weighting. The filtering phase, applied first, excludes the gradients outside the accept region defined by the reference gradients, defending against standard attacks.
Revised: This section presents the proposed server-side defense operating on the gradients. We start with an overview of the idea, then introduce the two complementary components of our defense and outline the invariant aggregator steps.

Original: Then, the re-weighting phase further down-weights the potential malicious gradients within the accepting region of the filtering phase, targeting the Mimic-Shift attack.

Original: 4. P HASE 1: F ILTERING
Revised: 4. O VERVIEW

Original: In the filtering phase (Algorithm 1), we first compute the sample mean of reference gradient m R = (cid:80) N R i =
Revised: In the proposed defense, we aim at finding invariant directions to optimize the federated learning model, such that the model is generally utilitarian for most of clients and can exclude the directions that benefit a subset of potentially malicious clients. Since our defense operates with the gradients, we consider the invariant direction from a first-order perspective (Parascandolo et al., 2021). Expanding the loss function around the current weight w on client i , for a parameter update g , we have:

Original: [Equation]
Revised: [Equation]

Original: 1 N R · g R i , making a guess of where the good gradients might be. Next, we set m R as the center of a spherical accept region.
Revised: where R 2 ( w + g ) is a second-order Taylor remainder. With a reasonably small ∥ g ∥ (achievable with small learning rates), the remainder term R 2 ( w + g ) is negligible and the change of the loss function mainly depends on the first-order gradient and the parameter update g . Since learning the trigger benefits the malicious clients exclusively, there exists at least one gradient dimension k and one benign client i ∈ { 1 , ..., N − N ′ } where E D ′ [ ∇ w k ℓ ( x , y ; w )]

Original: Then, we compute the sample variance of the reference gradients,
Revised: ̸ = 0 and the dimension-wise gradient have inconsistent signs, i.e.,

Original: [Equation]
Revised: [Equation]

Original: The sample variance s R is further scaled up by a tunable hyper-parameter c . The product c · s R is  the radius of the spherical accept region.
Revised: If the condition in Equation 2 is not true, then learning the trigger would always benefit the model on benign data, thereby contradicting common empirical observation. A more detailed analysis of the condition in Equation 2 is in Section 5.2.

Original: Finally, we remove all the updates outside the spherical accept region.
Revised: The proposed methods show how to treat the inconsistent signs and defend against the backdoor attack by only allowing invariant directions.

Original: This filtering is effective against attacks that do not know where the expected gradient is (e.g., Gaussian attack) or do not carefully calibrate the adversarial gradient (e.g., sign flipping attack), as is shown in Section 6.
Revised: A dimensionwise analysis is necessary because backdoor attacks can succeed by poisoning few inconsistent dimensions (Wu & Wang, 2021) without incurring much vector-wise difference, as Section 6.2 will show. We consider two treatments for each gradient dimension to help the model avoid the direction specified by E D ′ [ ∇ w ℓ ( x ′ , y ′ ; w )] : (Treatment 1) setting the dimension with inconsistent signs to zero using an AND-mask (Parascandolo et al., 2021) such that no client benefits or (Treatment 2) employing a robust mean estimator (e.g., trimmed-mean (Xie et al., 2020b)) to remove the malicious values that cause the inconsistent sign. To achieve a better result, we combine these two treatments to avoid their failure modes. The following examples illustrate the failure mode of each treatment and motivate the combination of defense.

Original: Algorithm 1 Filtering.
Revised: Failure Mode 1.

Original: Input: A set of reference gradients, { g R i | i ∈ { 1 , ..., N R }} ; A set of reported gradients, { g i | i ∈ { 1 , ..., N
Revised: Figure 1a shows an example where the adversary may exploit a dimension by inserting outliers where the benign values have a consistent sign. The outliers can mislead the average toward the non-invariant direction.

Original: 4. P HASE 2: R E - WEIGHTING
Revised: The robust mean estimator (treatment 2) can trim the outliers and accurately estimate the mean.

Original: Suppose a majority adversary misleads the aggregated gradient. In that case, the malicious gradients likely have a stronger influence on the aggregated gradient compared to benign gradients. Here, we measure the influence via scalar projections.
Revised: In contrast, treatment 1 can fail with a high sign consistency. Because it either lets the highly consistent dimension pass or has to be over-aggressive in zeroing out the dimensions, hurting the model’s accuracy on benign data.

Original: The re-weighting defense is outlined in Algorithm 2.The scaling weights are designed to augment benign gradients and down-weight adversarial updates. This is implemented using a monotonic re-scaling 1 of the scalar projection between the aggregate and the gradients (step 4). The monotonic re-scaling amplifies the difference between the scalar projections, making large projections larger.
Revised: Failure Mode 2.

Original: Therefore, the potential malicious gradients would be down-weighted more. However,  the monotonic re-scaling may also lead to an over-up-weighting on certain benign gradients, which are far from ¯ g ∗ .
Revised: Figure 1b shows some values with inconsistent signs, which treatment 1 can handle. However, this example can fail the robust mean estimator (treatment 2), whose result has the same sign as the malicious values.

Original: such up-weighting leads to instability in the training process.
Revised: The following sections shall detail the two treatments and discuss their complementary relationship.

Original: Thus, we also include a clipping operator (step 6). The clipping addresses the over-up-weighting issue by specifying a bound. An additional benefit of using the power function and clipping is that they prompt a set of uniform weights on benign gradients, stabilizing the training.
Revised: The AND-mask (Parascandolo et al., 2021) computes a dimension-wise mask by inspecting the sign consistency of each dimension across clients.

Original: If we set k to be sufficiently large such that the maximum s i is greater than NF · ( (cid:80) N F i =
Revised: For dimension k , the sign consistency is: | 1 N (cid:80) Ni =1 sign( g i,k ) | .

Original: s i − max i ∈{ 1 ,...,N F } s i ) , we have s i = τ, ∀ i ̸ = arg max i ∈{ 1 ,...,N F } s i . In practice, we set k  to 10 and τ to 0.6.
Revised: If the sign consistency is below a given threshold τ , the mask element m k is set to 0, otherwise, m k is set to 1.

Original: Both hyper-parameters generalize well to the three datasets in our experiments. Note that the filtering phase complements the re-weighting phase because standard attacks can corrupt the scalar projections via uploading 0 or flipping the sign. For a class of inner product-based attacks (Xie et al., 2019a), an additional clipping operator that prevent the scalar projection s i from being negative can help.
Revised: The mask along dimension k is defined as:

Original: Algorithm 2 Re-weighting.
Revised: Definition 1.

Original: [Equation]
Revised: (AND-Mask)

Revised: For the k th dimension in the gradient vector, the corresponding mask m k is defined as:

Original: [Equation]
Revised: [Equation]

Original: We first show that the probability of not filtering out a benign gradient increases at a rate of O (1 − 1 N 2 R ) w.r.t. the number of reference clients N R and O (cid:0) (1 − 1 c 2 ) 2 (cid:1) w.r.t.
Revised: Our defense then multiplies the mask m with the aggregated gradient ¯ g element-wise, setting the inconsistent dimension to zero.

Original: the scaling factor c in the accept radius.

Original: Then, we discuss conditions for the probability of down-weighting the malicious gradients to increase as the adversary owns more clients in the system. Additional discussion considers how our strategy may still help even if the aforementioned conditions do not hold, providing insights to certain of empirical results. Finally, we study the convergence of our method under a convex setting. Before proceeding, we outline some additional assumptions.
Revised: 4. T RIMMED -M EAN

Original: To simplify tedious notation, we assume all users have the same number of samples because the difference in sample size can be merged into the difference between gradients. Assumption 1. Uniform sample size:
Revised: To complement the AND-mask, our defense broadcasts the trimmed-mean estimator to each gradient dimension. The trimmed-mean estimator alleviates the outlier issue by removing the subset of largest and smallest elements before computing the mean.

Original: [Equation]
Revised: The largest and smallest elements appear on the two tails of a sorted sequence. Next, we define order statistics and the trimmed mean estimator.

Original: Assumption 2.
Revised: Definition 2.

Original: The gradients follow a hierarchical distribution in all rounds: [Equation]
Revised: (Order Statistics) (Xie et al., 2020b)

Original: where Stage 1 is a distribution where  the random variable µ has finite expected value E [ µ ] and finite non-zero variance Var[ µ ] . Stage 2 is truncated isotropic Gaussian distribution with a truncation threshold γ + on the L 2 norm.
Revised: By sorting the scalar sequence { x i : i ∈ { 1 , ..., N } , x i ∈ R } , we get x 1: N ≤ x 2: N ≤ ... ≤ x N : N , where x i : N is the i th smallest element in { x i : i ∈ { 1 , ..., N }} . Then, the trimmed-mean estimator removes a α × N elements from each tail of the sorted sequence.

Original: Assumption 3.
Revised: Definition 3.

Original: The variance of gradients is bounded: [Equation]
Revised: (Trimmed Mean Estimator) (Xie et al., 2020b)

Original: Assumption 4. The sample variance of reference gradients is greater than 0:
Revised: For α ∈ [0 , 1] , the α -trimmed mean of the set of scalars x i : N ∈ { 1 , ..., N } is defined as follows:

Original: [Equation]
Revised: [Equation]

Original: Assumption 5.
Revised: where ⌈ .

Original: The L 2 norm of expected and estimated gradients is bounded: [Equation] ∈ { 0 , 1 , ..., N } . [Figure]
Revised: ⌉ denotes the ceiling function.

Original: are two spherical contours around E [ µ ]
Revised: 4. O UR A PPROACH : I NVARIANT A GGREGATOR

Original: Condition c ≥ E holds in Case 1, where all the points outside the accept region are more than c · s R 2 away from E [ µ ] .
Revised: Algorithm 1 outlines the steps of our server-side defense that perform aggregation of invariant updates from the clients.

Original: In Case 2 and Case 3, there are points between the two contours that have the same distance to E [ µ ] but are not always inside accept region. 5. F ILTERING (S TAGE 1) 
Revised: The solution is composed of the AND-mask (treatment 1) and trimmed-mean estimator (treatment 2).

Original: The filtering phase removes all the gradients that are more than c · s R away from the reference m R in terms of Euclidean distance. However, the data distribution D i may vary across clients in a federated learning setting. Such non-i.i.d.ness increases the risk of filtering out a benign gradient. We start our analysis by considering the expected gradients:
Revised: Our defense applies the two components separately based on the sign consistency of each dimension with a threshold τ .

Original: Lemma 6. Suppose there are N R reference clients among N clients, and the sample variance of gradients is s R , under Assumption 2, let c ≥ 2 · ∥ E [ µ ] − ¯ µ ∥ s R . Then, for any i ∈ [1 , ..., N ] , with probability at most 4 · Var[ µ ] 2 c 2 · s 2 R , we have ∥ µ i − ¯ µ R ∥ ≥ c · s R , where ¯ µ R = (cid:80) Ri =1 1 N R · µ R i . Remark 7.
Revised: We show how these two components, i.e., the AND-mask and the trimmed-mean estimator, complement each other.

Original: Condition c ≥ 2 · ∥ E [ µ ] − ¯ µ ∥ Var[ µ ] guarantees that all µ i s outside the accept region are at least c · s R 2 away from E [ µ ] , as Figure 1 shows, enabling concentration argument. Otherwise, bounding the probability ofµ i appears outside the accept region is hard without additional assumptions on P ( µ ) . Lemma 6 suggests that the probability of filtering out a benign gradients decreases at a rate of 
Revised: Our analysis considers a single dimension and starts with the robustness of the trimmed mean estimator, which improves the robustness of the AND-mask against outliers.

Original: O ( 1 c 2 ) . The following lemma further shows the probability ofviolating the assumed condition c ≥ 2 · ∥ E [ µ ] − ¯ µ ∥ s R decreases at a rate of O ( 1 N 2 R ) .
Revised: The following theorem extends the robustness guarantee of a modified trimmed-mean estimator (Lugosi & Mendelson, 2021), which is shown in Appendix B, to the conventional trimmed-mean estimator (Definition 3).

Original: Lemma 8.
Revised: Theorem 4.

Original: With Assumptions 2 and 5, for a fixed accept radius c · s R , with probability at most 4 · Var[ µ ] 2 NR · c 2 · s 2 R , we have c ≤ 2 · ∥ E [ µ ] − ¯ µ ∥ s R . The following theorem combines and extends theresult on expected gradients to estimated gradients.
Revised: With the trimmed-mean estimator in Definition 3, for a given set of samples x 1 , ..., x N , with a corruption level η = N ′ N and a confidence level δ , set the trim level α = 8 η + 12 log( 4 δ ) N , let a = x αN : N and b = x N − αN : N following Definition 2, x be a random variable with variance σ and ¯ x be the estimated mean, with probability at least 0 .

Original: Theorem 9. Under Assumptions 2 and 3, with probability at least
Revised: 99 N − 2 αN c − 4 (1 − 4 e

Original: [Equation]
Revised: [Equation]

Original:  ) , we have ∥ g i − m R ∥ ≤ c · s R
Revised: 12 ) , we have:

Original: [Equation]
Revised: [Equation]

Original:  Theorem 9 shows that the risk offiltering out a benign gradient decreases quickly w.r.t.
Revised: The proof is in Appendix B. Theorem 4 bounds the estimation error of a trimmed-mean estimator, which can increase as the variable’s variance increases.

Original: c and N R , enabling our strategy of using a small number of reference clients and a conservative c .
Revised: Multiple factors can increase the variance, such as non-i.i.d. federated data distribution and the stochastic gradient estimation process. In practice, a threat analysis is necessary to specify the maximum number of malicious clients to be tolerated, when the number of malicious clients is unknown. Our goal is to prevent the outliers from misleading the sign.

Original: 5. R E - WEIGHTING (S TAGE 2)
Revised: Therefore, the estimation error to expectation ratio, | ¯ x − E [ x ] | E [ x ] , is particularly relevant. Since the estimator error for a given α depends on the variance, a high expectationvariance ratio, E [ x ] σ , is desirable.

Original: Suppose the adversary adopts the Mimic-Shift attack and bypasses the filtering defense. We would like to figure out the probability of assigning a higher scalar projection to the malicious gradient g ′ than a benign gradient g i , i ∈ { 1 , ..., N } .
Revised: Then, we show that AND-mask identifies the elements with high expectation-variance ratios, avoiding the robustness degradation of the trimmed-mean estimator. The following theorem suggests that dimensions with a higher expectation-variance ratio have high probabilities of passing AND-mask, and increasing the mask threshold τ increases the chance of filtering out dimensions with low expectation-variance ratios.

Original: Theorem 10.
Revised: Theorem 5.

Original: Suppose all the gradients pass the filtering phase. Let the aggregated gradient be ¯ g ∗ = w · ¯ g + w ′ · g ′ , where ( w, w ′ )
Revised: Given a non-zero expectation-variance ratio ϕ

Original: =
Revised: =

Original: (cid:18) (cid:80) Ni =
Revised: E

Original: 1 n i (cid:80) N + N ′ i = n i , (cid:80) N + N ′  i
Revised: [ x ] σ , N ′ malicious elements, with probability at most (cid:80) min( N,N − 2 N ′ + τ ) i

Original: [Equation] N +1 n i (cid:80) N + N ′ i =
Revised: = N − 2 N ′ − τ (cid:0) N − N

Original: n i (cid:19) . Let θ be the angle between ¯ g and [Figure]
Revised: i (cid:1) ϕ − 2 i , the sign consistency is below τ .

Original: g ′ , θ ∗ be the angle between ¯ g ∗ and g ′ . Assume ww ′ ≤ ∥ g ′ ∥− cos θ 2 ·∥ ¯ g ∥ ∥ ¯ g ∥− cos θ 2 · ∥ g ′ ∥ such that θ ∗ ≤ θ − θ ∗ .
Revised: Appendix B provides the proof.

Original: Then, under Assumptions 1- 5, let γ − θ ∗ = ∥ γ − ∥ · tanh( θ − 2 · θ ∗ ) , with probability at least [Figure] , we have
Revised: For the ϕ = 0 case, we may use the probability of the estimated gradient sign being the same as the malicious gradient to replace ϕ − 2 .

Original: [Table] [Figure] [Table] [Table]
Revised: We do not propose directly using the sample mean-variance ratio due to a potential issue: using the sample mean-variance ratio can be over-aggressive when the benign value has a consistent sign but varying magnitudes.

Original: [Figure] [Table] [Figure] Theorem 10 suggests that if 2 · θ ∗ ≤ θ , the probability of down-weighting malicious gradients more  than benign gradients (i.e., s ′ >
Revised: Our ablation study in Appendix D.2 shows that AND-mask can preserve more utility than the sample mean-variance ratio when combined with the trimmed-mean estimator.

Original:  s i ) increases as the adversary owns more clients and samples in a system (i.e. w ′ ↑ ). The main idea is: the minimum amount of deviation between g i and ¯ g for having s i > s ′ increases as w ′ increases because w ′ ↑ ⇝ θ ∗ ↓ . This specific deviation is shown in Figure 2a as the dark dash line with length at least ∥ γ − ∥ · tanh( θ − 2 · θ ∗ ) .
Revised:  R OBUSTNESS ANALYSIS FOR A SIMPLIFIED MODEL

Original: Then, applying concentration arguments yields the result. 5.2. A DDITIONAL D ISCUSSION
Revised: To further motivate our approach, we consider a more direct robustness analysis for a specific generative model.

Original: Theorem 10 only deals with the case where ¯ g ∗ has a smaller angle with g ′ than ¯ g . However, in practice, we find that our method also achieves a decent accuracy even the condition 2 · θ ∗ ≤ θ does not hold.
Revised: This analysis discusses how the features impact the gradient sign consistency, when the two failure modes in Section 4.1 appear, and why we need a combination of defenses guaranteed by Theorems 4 and 5.

Original: To gain some insight, suppose there is a probability density function f of benign gradients. Then, the probability of having s i > s ′ , ∀ i ∈ { 1 , ..., N } equals the integral of f over a cone, which is defined by spinning g ′ around ¯ g ∗ as Figure 2b shows. In a federated scenario, the estimated g i concentrates around its own µ i , which is not necessarily close to ¯ g or ¯ µ , resulting in a small integral of f over the aforementioned cone.
Revised: First, we outline some preliminaries useful for this analysis.

Original: 5. C ONVERGENCE A NALYSIS
Revised: 5. P RELIMINARIES

Original: So far, our analysis has focused on the robustness of one update.
Revised: We consider a binary prediction task y ∈ { 0 , 1 } with a linear model h ( x ) =

Original: The following extends the single step analysis to a convergence analysis. Before showing the theorem, we present a lemma that quantifies the impact of false-positive filtering. Lemma 11. Suppose F : N × R d −→ N ×{ 0 , 1 } is a filtering function, let mask M = F ( g 1 , ..., g N ) ,
Revised: w ⊤ x and a decision rule

Original: N ′ F = 1 − ∥ M ∥ 1 , ˆ g =[-
 ∥ M ∥-] 1 · (cid:80) N i =
Revised: ˆ y =[-
 ∥ M ∥-] 1 w ⊤ x ≥ 0 ( w ⊤ x ) .

Original: 1 M i · g i , and δ = ˆ g − ¯ g . Then, with Assumption 5, we have ∥ δ ∥ ≤ 2 · N ′ F N · γ + .
Revised: The training procedure uses a Sigmoid activation function s ( z ) = 1 1+ e − z and a logistic loss function ℓ ( z, y )

Original: In the following theorem, we use assumed conditions to ease the reading. Later , we shall connect the assumed conditions to Theorems 9 and 10 and further support the assumed conditions with empirical results in Appendix C. Theorem 12.
Revised: =

Original: Suppose the malicious gradient g ′ t is filtered out with probability 0, ∀ t ∈ { 1 , ..., T } , and down-weighted by s ′ t ≥ 1 .
Revised: − y · log( z ) − (1 − y ) · log(1 − z ) , where z = h ( x ) .

Original: Assume at most N ′ F benign gradient are filtered out at each iteration.
Revised: Data Model. We assume that the samples per class are balanced on benign clients.

Original: Define an aggregated gradient ¯ g ∗ t = w · ˆ g t + w ′ · g ′ t s ′ t = w · (¯ µ t + δ t + ϵ t ) + w ′ · g ′ t s ′ t , where ˆ g t and δ t follow the definitions in Lemma 11, ϵ t ∼ 1 N · (cid:80) Ni =1 N (0 , σ t,i ) as Assumption 2 shows. Let ζ ∈ Z be the model parameter, ζ ∗ be the optimum, and ¯ ζ = 1 T (cid:80) T t =1 ζ t .
Revised: The data samples come from a non-i.i.d.

Original: Assume sup ζ ∈ Z ∥ ζ ∥ ≤ D and F : Z −→ R is convex.
Revised: Gaussian distribution with a diagonal covariance matrix. On the i th client, for the k th feature, we have:

Original: Let C = w · 2 · N ′ F N · γ +2 +2 · w · γ +2 + w ′ · γ +2 , C ′ = 2 · w · C + C 2 + w · γ + , and η t =
Revised: [Equation] Backdoor Attack. We consider a single feature backdoor attack where the trigger is the k th feature x k . A backdoor using a specific feature is common (Wang et al., 2020).

Revised: To backdoor images, the adversary often selects a pixel pattern or semantic pattern (e.g., blue color on airplanes).

Revised: For text data, the trigger could be a dedicated set of characters .

Revised: The malicious clients attack in collusion using the same backdoor samples, meaning that µ i,k = µ i ′ ,k ̸ = 0 , ∀ i, i ′ ∈ { N − N ′ + 1 , N } .

Revised: To simplify the notation, we omit the subscript i for simplicity while focusing on a single client.

Revised: Since the trigger is not useful or does not appear on benign clients, we assume µ i,k = 0 , ∀ i ∈ { 1 , N − N ′ }

Revised: Objective.

Revised: The backdoor attack is effective if the model entangles the trigger-label correlation.

Revised: For a non-zero µ k of the trigger x k , w k µ k > 0 is a necessary condition for a model to entangle the trigger-label correlation.

Revised: Therefore, in the case when µ k > 0 , avoiding w k from increasing and enforcing w k to decrease while w k > 0 can mitigate the backdoor attack.

Revised: 5. C ONNECTING F EATURES TO G RADIENTS

Revised: We define the invariance of a feature by measuring its feature-label correlation (e.g., positive or negative) consistency across clients.

Revised: Definition 6.

Original: D √ T · C ′ , under Assumptions 2 and 5, we have:
Revised: For a given feature k , its invariance p is defined as: p = (cid:12) (cid:12) (cid:12) 1 N · (cid:80) N i =1 sign ( µ i,k ) (cid:12) (cid:12)(cid:12) . Similarly, we define the sign consistency of a gradient dimension k : Definition 7. With a linear model h ( x ) =

Revised: w ⊤ x , a Sigmoid activation function s , and N clients, a q -consistent gradient w.r.t.

Revised: w k satisfies: (cid:12)(cid:12)(cid:12) 1 N · (cid:80) Ni =1 sign (cid:16) E x ,y ∼D i (cid:2) ∇

Original: [Equation]
Revised: [Equation]

Revised: Under Definitions 6 and 7, we discuss the connection between feature invariance and dimensionwise gradient sign consistency.

Revised: First, we need to analyze the behavior of the gradient sign per client.

Original: The convergence analysis shows that our method converges to a neighborhood around the optimum, whose size shrinks with fewer false-positive filtering (Theorem 9) and more down-weighting (tuning k and τ in Algorithm 2) with higher probability (Theorem 10) on the malicious gradients.
Revised: Theorem 8. For a linear model with a Sigmoid activation function s and the logistic loss ℓ , under our Gaussian data model, on the k th feature with a non-zero µ k , if w k µ k ≤ 0 , we have sign (cid:16) E x ,y ∼D i (cid:2) ∇ w k ℓ ( s ( w ⊤ · x ) , y ) (cid:3) (cid:17) = sign( µ k ) . In addition, if µ k = 0 , we have sign (cid:16) E x ,y ∼D i (cid:2) ∇ w k ℓ ( s ( w ⊤ · x ) , y ) (cid:3) (cid:17)

Revised: = sign( w k ) .

Revised: The proof is provided in Appendix B.

Revised: The result in Theorem 8 is intuitive.

Revised: With a non-zero µ k , if sign( w k ) agrees with sign( µ k ) , the gradient sign can be indefinite because the weight w k can be either larger or smaller than the optimal w ∗ k .

Revised: Otherwise, the gradient has the same sign as µ k .

Revised: If µ k = 0 , w k shall shrink to 0.

Revised: Then, we outline the conditions that lead to inconsistent signs.

Revised: Corollary 9.

Revised: Under Theorem 8, suppose x k represents the trigger, µ k = 0 on the N − N ′ benign clients and the N ′ malicious clients share the same non-zero µ k , if w k = 0, the gradient w.r.t.

Original: Converging to a neighborhood is common in previous noisy gradient descent studies (Wang et al., 2021;
Revised: w k is N ′ N -consistent. In addition, if w k µ k > 0 , the gradient w.r.t. w k is at least (1 − N ′ N ) -consistent.

Revised: The gradient w.r.t.

Revised: w k is 1-consistent if w k µ k < 0 .

Revised: Counting the gradient signs yields the result.

Revised: Corollary 9 suggests that if w k µ k > 0 and the model entangles the backdoor, the expected gradient w.r.t.

Revised: w k can align with µ k on malicious clients and conflict with µ k on benign clients.

Original: He et al., 2022a).
Revised: Then, employing the trimmed-mean estimator to remove the malicious values can recover the invariant direction pointed by benign clients (Theorem 4) and thereby shrink µ k . If µ k = 0 , using AND-mask can mask out the gradients w.r.t. w k from the malicious clients (Theorem 5). If w k remains 0 , the model parameterized by w is robust to the trigger on x k .

Original: Our result also suggests scaling up the learning rate as the adversary owns more clients.
Revised: It is worth noting that the gradients w.r.t. w k is 1-consistent and align to µ k when w k µ k < 0 . Such a consistent gradient may overshoot and flip the sign of w k . Reducing the learning rate can alleviate overshooting and the trimmed-mean estimator will guarantee w k to shrink after the overshooting.

Revised: Connection to Failure Modes.

Original: In practice, this scaling is handled by our re-weighting phase where thebenign gradients are down-weighted by scalars less than 1.
Revised: If w k µ k > 0 , the gradients w.r.t. w k have a consistent but non-zero expectation among benign clients, causing the failure mode 1 in Section 4.1. On the other hand, the gradient variance can diversify the estimated gradient sign and cause the failure mode 2. Appendix D.1 further provides simulation results under the linear regime (Section 5.1), showing that the AND-mask can prevent the adversary from exploiting w k and the trimmed-mean estimator helps shrink w k .

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We first compare the Mimic-Shift attack with the Mimic attack, showing that Mimic-Shift is more effective and the improved effectiveness remains with a non-omniscient adversary (Mimic-ShiftPar). Then, we evaluate our defense strategy against the Mimic-Shift attack where our strategy outperforms strong baselines.
Revised: We evaluate our defense on three realistic tasks on three different data types: (1) object recognition with visual data, (2) sentiment analysis with text data, and (3) phishing email detection with tabular data. We employ the state-of-the-art edge-case backdoor attack (Wang et al., 2020) to generate backdoor samples and evaluate of defense and existing defenses against it.

Original: Additional Results Appendix C provides more results.
Revised: Additional Results.

Original: We evaluate our defense via a nonomniscient Mimic-Shift-Par attack, an improved Mimic-Shift-Var attack as well as other standard attacks (e.g., Gaussian and sign-flipping).

Original: There are also additional experiment with fewer clients and various attack strengths, plots of the re-weighting vector s as well as convergence, and studies on defense hyper-parameters.
Revised: The simulations, an ablation study, an evaluation of the hyper-parameter sensitivity, evaluations with additional attack strategies, and empirical verifications of the two failure modes can be found in the Appendix D.

Original: 6.  S ETUP
Revised: 6. E XPERIMENTAL S ETUP

Original: We use three datasets, FEMNIST (FM) (Caldas et al., 2018), CelebA (CA)
Revised: We briefly summarize our setup and report more details in Appendix C. Metrics. Our experiments employ two metrics: the main task accuracy (Acc M ) estimated on the benign samples and the backdoor task accuracy (Acc B ) over backdoor samples.

Original: (Liu et al., 2015), and 
Revised: A defense is designed to reduce the model’s accuracy on backdoor task and maintain the utility on the main task.

Original: Shakespeare (SS) (McMahan et al., 2017), with realistic non-i.i.d. partitions (Caldas et al., 2018) implemented in the FedML library (He et al., 2020).
Revised: Datasets. The visual data of the object detection task and text data of the sentiment analysis task are from CIFAR-10 (Krizhevsky, 2009; McMahan et al., 2017) and Twitter (Caldas et al., 2018), respectively.

Original: The number of benign clients ranges fromto 500.
Revised: Each phishing email data sample has 45 standardized numerical features of the sender that represent the sender reputation scores. A large reputation score may indicate a phishing email.

Original: We select 2 reference clients and 28 other benign clients at each round.
Revised: The reputation scores come from peer-reviewers in a reputation system (Jøsang et al., 2007).

Original: The number of adversarial clients is adjusted accordingly.
Revised: The adversary may use malicious clients to manipulate the reputation.

Original:  We report the detailed setup in Appendix B, including various hyper-parameters. We employ seven baselines, including federated averaging (FedAvg) (McMahan et al., 
Revised: Federated Learning Setup. We consider horizontal federated learning (Kairouz et al., 2021) where the clients share the same feature and label spaces.

Original: Appendix B lists the details of these baselines.
Revised: The number of clients are 100 for the three tasks.

Original: The ”Oracle” aggregator operates with benign gradients and serves as a reference. Five attacks are considered in our experiments, including Mimic-Shift-Var, particularly designed as a strong attack for our proposed defense. Gaussian Draw a random update g ′ i from an isotropic Gaussian distribution N (0 , 200) .
Revised: The server sample 20 clients at each round on the CIFAR-10 and phishing email experiments. We reduce the sampled client number to 15 on the Twitter experiment due to limited hardware memory.

Original: Sign-flipping Flip
Revised: Backdoor Attack Setup.

Original:  the sign of the estimated gradient
Revised: The adversary employs the edge-case backdoor attack, where it selects the data samples with low marginal probability in their data distribution to create backdoor samples.

Original: g i = −
Revised: The visual and text backdoor samples follow the previous work (Wang et al., 2020).

Original: g i and report g i
Revised: For the tabular data, we select the 38 th feature (reputation), whose value is 0 on most of the data samples.

Original:  to  the server. Mimic-Shift Report g =
Revised: Then, we let the adversary manipulate the 38 th feature to 0.2 that has a low probability density on phishing emails and flip the label to non-phishing.

Original: ¯ g R + (¯ g R − ¯ g ) to the server, as is shown in Section 3.1. Mimic-Shift-Par Randomly eavesdrop 20% clients per round and draw two clients as reference.
Revised: The adversary can control 20% clients on the CIFAR-10 and phishing email experiments and 10% clients on the Twitter experiment. Section 6.2 explains the different experiment configurations.

Original: Mimic
Revised: Such an adversary is considered strong in practice (Shejwalkar et al., 2022).

Original: -Shift-Var Mirror the local update using ¯ g R and report g ′ i = ¯ g R + (¯ g R − ¯ g i ) to the server
Revised: We consider a strong adversary because defending against strong adversary yields robustness against weak adversary, whose effectiveness is already shown (Wang et al., 2020). The adversary only uses backdoor samples during training.

Original: 6. M IMIC - TYPE A TTACK C OMPARISON 
Revised: 6. R ESULT AND C OMPARISON TO P RIOR W ORKS

Original:  Table 1 shows the accuracy of FedAvg aggregator under Mimic and Mimic-Shift attack.
Revised: Our results. Table 1 summarizes the performance of each defense on three tasks.

Original: Here, the adversary owns 80% of the system, and the Mimic adversary mimics the first client in the system. Mimic-Shift attack constantly outperforms Mimic and is116% more effective on average.
Revised: Our approach decreases the backdoor task accuracy by 61.6% on average. The edge-case backdoor attack on the text sentiment analysis task (Twitter) is more difficult to defend and our approach mitigates the accuracy increase on the backdoor task by 41.7%.

Original: Such advantages are preserved under a non-omniscient adversary.
Revised: We hypothesize that the text sentiment analysis task has few invariant and benign features. For example, the shape features (Sun et al., 2021) in object classification tasks can be invariant across objects. In contrast, the sentiment largely depends on the entire sentence instead of a few symbols or features. Then, we discuss the limitations of prior defenses.

Original: [Table]
Revised: Vector-wise.

Original: 6. D EFENSE AGAINST M IMIC -S HIFT A TTACK
Revised: Common vector-wise defenses such as Krum estimate pair-wise similarities in terms of Euclidean distance (Blanchard et al., 2017) (Krum and multi-Krum) of cosime similarity (Nguyen et al., 2021) (multi-Krum C ) between each gradient and others.

Original: This experiment considers three settings where the adversary owns 0% - 80% of a system. The hyper-parameters of each aggregator are selected based on the 80% adversary setting and directly applied to the other settings.
Revised: The gradients that are dissimilar to others are removed. The vector-wise view is insufficient for defending against backdoor attacks because backdoor attacks can succeed by manipulating a tiny subset of parameters (e.g. 5%)

Original: Table 2 shows the accuracy of our strategy and baselines under the
Revised: (Wu & Wang, 2021) without incurring much vector-wise difference.

Original: Mimic-Shift attack. Under a majority adversary setting
Revised: In practice, we observe that the malicious gradients can get high similarity scores and circumvent vector-wise defenses.

Original: [Table]
Revised: [Table]

Original: our method outperforms all the baselines by a large margin. The reason is that the median-based (CM) method picks the malicious gradient once the adversary becomes the majority.
Revised: Dimension-wise. Failure mode 2 in Section 4.1 shows the limitation of the trimmed-mean estimator, which was the most effective defense against the edge-case backdoor attack.

Original: Clustering-based (Krum and Krum-B) methods suffers from a similar issue.
Revised: We also include SignSGD with majority vote (Bernstein et al., 2019) as a defense, which binarizes the gradient and takes the majority vote as the aggregation result. However, Sign-SGD struggles to train a large federated model (e.g., Resnet-18 on CIFAR-10) and can suffer from failure mode 1 where the clients have diverse signs. Then, the adversary can put more weight on one side and mislead the voting result. Combination.

Original: Other reference client-assisted distance-based re-weighting strategy (CClip-R) andfiltering strategy (Zeno ′ ) is not effective because Mimic-Shift carefully calibrates the malicious gradients so that they are as close to  the reference gradients as the benign gradients.
Revised: A naive combination of multi-Krum and the trimmed-mean estimator fails to defend against the backdoor attack because neither multi-Krum nor the trimmed-mean estimator avoids the failure mode of the other. Weak-DP. The weak-DP defense (Sun et al., 2019) first bounds the gradient norms, then add additive noise (e.g., Gaussian noise) to the gradient vector.

Original: Using only reference clients (Ref) outperforms existing robust aggregators but suffers from low client utilization.
Revised: The edge-case backdoor attack can work without scaling up the gradients, circumventing the norm bounding.

Original: We also find that our proposed defense yields the best accuracy under a minority adversary setting, as discussed in Section 5.2.1. Under a no adversary setting, our strategy weights the  benign gradients nearly uniformly without interfering with the training.
Revised: For the additive noise, we hypothesize that in some dimensions, the difference between malicious and benign gradients can be too large for the Gaussian noise to blur their boundary. Freezing Layers. Since we employ a pre-trained Resnet-18 (He et al., 2016) on CIFAR-10, freezing the convolution layers may avoid entangling the trigger.

Original: Our method occationally outperforms the oracle.
Revised: However, this approach lacks empirical robustness, possibly because the adversary can use semantic features (e.g., blue color on airplanes) that the pre-trained model already learns as triggers. Advanced Defenses. FoolsGold (Fung et al., 2020) down-weights an update if that update has a high cosine similarity with another update. There are many ways to diversify updates.

Original: We hypothesize that the improved results can come from the up-weighting of the underrepresented clients (Li et al., 2020), whose influence score is small.
Revised: Malicious clients may leverage the stochastic gradient estimation process or mix backdoor samples with benign samples, whose distribution can differ across clients. RFA (Pillutla et al., 2022) computes geometric medians as the aggregation result, which is shown to be ineffective (Wang et al., 2020).

Original: [Table]
Revised: SparseFed (Panda et al., 2022) only accepts elements with large magnitude in the aggregation results. However, benign and malicious updates can contribute to large magnitudes.

Original:  C ONCLUSION AND F UTURE W ORK
Revised:  C ONCLUSION AND F UTURE W ORK

Original: This paper shows two methods for improving the adversarial robustness of federated learning under a majority adversary regime.
Revised: This paper shows how to defend against backdoor attacks by focusing on the invariant directions in the model optimization trajectory.

Original: Empirical results in various settings and against a broad class of attacks demonstrate the proposed methods’ effectiveness. Additional theoretical analysis is conducted under the Mimic-Shift attack regime, showing conditions under which the proposed method helps.
Revised: Enforcing the model to follow the invariant direction requires AND-mask to compute the sign-consistency of each gradient dimension, which estimates how invariant a dimension-wise direction can be, and use the trimmed-mean estimator to guarantee the model follows the invariant direction within each dimension. Both theoretical and empirical results demonstrate the combination of AND-mask and the trimmed-mean estimator is necessary and effective.

Original: Further exploring the limitations of learning with majority adversaries is a good next step.
Revised: Further defending against more advanced backdoor attacks such as invisible backdoors that add calibrated noise to all the features (Li et al., 2021b; Manoj & Blum, 2021) can be interesting.