Original: I NVARIANT A GGREGATOR FOR D EFENDING AGAINST F EDERATED B ACKDOOR A TTACKS
Revised: R OBUST F EDERATED L EARNING WITH M AJORITY A DVERSARIES VIA P ROJECTION - BASED R E - WEIGHTING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Federated learning is gaining popularity as it enables training of high-utility models across several clients without directly sharing their private data. As a downside, the federated setting makes the model vulnerable to various adversarial attacks in the presence of malicious clients.
Revised: Most robust aggregators for distributed or federated learning assume that adversarial clients are the minority in the system. In contrast, this paper considers the majority adversary setting. We first show that a filtering method using a few trusted clients can defend against many standard attacks.

Original: Specifically, an adversary can perform backdoor attacks to control model predictions via poisoning the training dataset with a trigger. In this work, we propose a mitigation for backdoor attacks in a federated learning setup. Our solution forces the model optimization trajectory to focus on the invariant directions that are generally useful for utility and avoid selecting directions that favor few and possibly malicious clients. Concretely, we consider the sign consistency of the pseudo-gradient (the client update) as an estimation of the invariance.
Revised: However, a new attack called Mimic-Shift can circumvent simple filtering.

Original: Following this, our approach performs dimension-wise filtering to remove pseudo-gradient elements with low sign consistency. Then, a robust mean estimator eliminates outliers among the remaining dimensions. Our theoretical analysis further shows the necessity of the defense combination and illustrates how our proposed solution defends the federated learning model.
Revised: To this end, we develop a re-weighting strategy that identifies and down-weights the potential adversaries under the majority adversary regime. We show that our aggregator converges to a neighborhood around the optimum under the Mimic-Shift attack.

Original: Empirical results on three datasets with different modalities and varying number of clients show that our approach mitigates backdoor attacks with a negligible cost on the model utility.
Revised: Empirical results further show that our aggregator achieves negligible accuracy loss with a majority of adversarial clients, outperforming strong baselines

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Federated learning enables multiple distrusting clients to jointly train a machine learning  model without sharing their private data directly.
Revised: Federated learning (FL) is a leading framework for collaboratively training a machine learning (ML) model over local datasets.

Original: However, a rising concern in this setting is the ability of potentially malicious clients to perpetrate backdoor attacks. To this end, it has been argued that conducting backdoor attacks in a federated learning setup is practical (Shejwalkar et al., 2022) and can be effective (Wang et al., 2020). For instance, the adversary can connect to a federated learning system as a legitimate user and conduct a backdoor attack that forces the model to mispredict.
Revised: The decentralized nature of FL systems has raised concerns about vulnerability – as adversaries can connect to an FL system like other benign users and corrupt the ML model while evading detection by standard means (Kairouz et al., 2021). To this end, there is growing literature on the adversarial robustness of FL (Blanchard et al., 2017; Chen et al., 2018;

Original: The impact of such attacks is quite severe in many mission-critical federated learning applications. For example, anomaly detection is a common federated learning task where multiple parties (e.g., banks or email users) collaboratively train a model that detects frauds or phishing emails.
Revised: Xie et al., 2019b; Rajput et al., 2019b; Xie et al., 2020; Karimireddy et al., 2021a; 2022; He et al., 2022b), particularly where adversaries can upload malicious updates. Most existing defenses assume that the adversarial clients are the minority in the system (Blanchard et al., 2017;

Original: Backdoor attacks allow the adversary to successfully circumvent these detection methods.
Revised: Chen et al., 2018; Rajput et al., 2019b; Karimireddy et al., 2021a; He et al., 2022b). However, in a federated scenario, the decentralized nature means that it is relatively straightforward for the adversary to be the majority and thus break existing defenses. We call such an adversary the “majority adversary”. Our work joins a growing literature on robustness with majority adversaries, e.g., Xie et al.

Original: The most common backdoor attack embeds triggers in the data samples and forces the model to make an adversary-specified prediction when the trigger is observed (Liu et al., 2018; Bagdasaryan et al., 2020). Thus, an adversary can conduct a backdoor attack by generating a trigger that statistically correlates with a particular label.
Revised: (2019b; 2020), motivated by noted practical vulnerabilities. Although Shejwalkar et al. (2021) argue that the number of registered clients in a production system (e.g., GBoard) may be too large for the adversary to compromise a majority of them, they neglect the client availability issue in FL. In particular, Kairouz et al.

Original: Once the adversary injects these trigger-embedded backdoor data samples into the training data, the model can entangle the trigger-label correlation and predict as the adversary specifies. Meanwhile, the backdoor attack often does not degrade the predictive accuracy on the benign samples, making backdoor detection difficult in practice (Wang et al., 2020). In federated learning, the server aggregates only the client-level updates (a.k.a.
Revised: (2021) suggests that, at any given time, only a subset ( < 1%) of clients are available for the server. Such a low client availability allows the adversary to become the majority and overwhelm the server utilizing compromised networked devices (e.g., IoT devices) in a similar way as the common distributed denial-of-service (DDoS) attack (Specht & Lee, 2003; Bonguet &

Original: pseudo-gradient or gradient for short) without control over the training procedure or any data samples.
Revised: Bella¨ıche, 2017). Some other settings such as crowd-sourced training (Ryabinin & Gusev, 2020) (a.k.a. volunteer computing) are perhaps even more vulnerable to majority adversaries because the crowd-sourcing systems do not implement access control – allowing the adversary to connect an arbitrary number of clients as volunteers.

Original: Such limited visibility of the  federated learningserver on the client-side training makes defending against backdoor attacks challenging.
Revised: We consider the adversarial robustness of federated learning against a class of attacks where an adversary aims to decrease the accuracy of the trained ML model by uploading malicious updates.

Original: Common defenses against backdoor attacks aim at identifying the backdoor data samples or poisoned model parameters and usually require access to at least a subset of the training data (Tran et al., 2018; Li et al., 2021a), which is prohibitive for a federated learning server.
Revised: In particular, we are interested in Mimic-type attacks (Karimireddy et al., 2022), as is discussed later in this section. A key assumption in our setup is the existence of a few trusted clients, e.g., with secure hardware support. We call these trusted clients “reference clients”.

Original: Other defense methods against untargeted poisoning attacks that degrade the model utility (Shejwalkar et al., 2022) are applicable but lack robustness against backdoor attacks, as discussed in Section 6.2.
Revised: In practice, the number of reference clients could be as small as two in each round. Similar approaches have been considered in existing works (Xie et al., 2019b; 2020).

Original: Our approach.
Revised: One option for secure hardware is the trusted execution environment (TEE) (Pinto & Santos, 2019), which guarantees that the program is not Byzantine. TEE is so far commercialized (e.g., on Google Pixel (GoogleBlog), Apple iPhone (AppleSupport), Sam- sung phones (SamsungDeveloper)).

Original:  Our  defense leverages the observation that learning from the poisonous data does not benefit the model on benign data and vice versa.
Revised: Recent works (Mo et al., 2021) bring TEE support to federated learning systems. We propose a combination of defenses – filtering and projection-based re-weighting. Our first defense is a filtering method that constructs a spherical accept region and excludes the updates outside it. The center of the accept region is an average update from a few trusted clients with secure hardware support.

Original: Therefore, focusing on the invariant directions that are generally beneficial in the model optimization trajectory helps defending against  the aforementioned backdoor attack (which often lead to non-invariant directions). To this end, we develop a defense by examining each dimension of the gradients on the server-side and checking whether the dimension-wise gradients point in the same direction across the clients.
Revised: The radius of the accept region is the sample variance of reference updates times a scaling factor. Although the filtering method is effective against several standard attacks (e.g., sign flipping attack, Gaussian attack), this filtering is easy to circumvent. Building on the recently proposed Mimic attack (Karimireddy et al., 2022), shown to break many existing defenses, we develop an improved attack method called Mimic-Shift.

Original: Here, a dimension-wise gradient can point to  a positive or negative direction, or have a zero value.
Revised: Mimic-Shift clients send malicious updates which slightly shift away from benign updates and mislead the aggregated updates away from the expected update. The slight shift makes Mimic-Shift hard to detect, and the calibrated shifting direction can corrupt the aggregated model. To perform the Mimic-Shift attack, we consider a man-in-the-middle (MITM) adversary capable of intercepting the message between the clients and the server. Computer security researchers have extensively studied the MITM adversary, but existing encryption solutions can be too expensive for resource-constrained client devices.

Original: In the case of small learning rates and for a specific dimension, two gradients pointing in the same direction means that taking the direction of one gradient can benefit the other. As such, the invariance of a direction depends on how many dimension-wise gradients align with that direction.
Revised: For example, the AES (advanced encryption standard) encryption only has a throughput of around 50 MB/s even with a powerful desktop CPU (Gleeson et al., 2014). With a 1 GB moderate size neural network, the encryption takes more than 20 seconds on the client-side, draining the computational resources and increasing the client dropout rate.

Original: Following this intuition, we define the sign consistency of a dimension by the average gradient sign. The higher the sign consistency is, the more invariant direction the gradient dimension may have. Designing such a method carefully selecting only the invariant gradient directions is non-trivial, especially given the non-i.i.d.
Revised: Our second defense is a projection-based re-weighting method to deal with the Mimic-Shift attack under a majority adversary regime. The main idea is to measure the influence of each update on the aggregated update, then down-weight the updates with high influence.

Original: gradient distributions across benign clients and the presence ofmalicious clients. Hence, our approach enforces two separate treatments for each gradient dimension. First, we employ an AND-mask (Parascandolo et al., 2021), a dimension-wise filter setting the gradient dimension with sign consistency below a given threshold to zero.
Revised: Specifically, we compute the scalar projection of the aggregated update on each client’s update.

Original: However, this alone is not enough: the malicious clients can still use outliers to mislead the aggregation result in the remaining highly consistent dimensions.
Revised: The intuition is that the majority adversarial clients can significantly mislead the aggregated update, resulting in a large scalar projection.

Original: To address this issue, we propose using the trimmed-mean estimator (Xie et al., 2020; Lugosi & Mendelson, 2021), as a means to remove the outliers. Our analysis suggests that the AND-mask complements the trimmed-mean estimator well, motivating their composition. We support the proposed approach witha theoretical analysis under a conventional linear regime (Rosenfeld et al., 2021; Wang et al., 2022; Zhou et al., 2022; Manoj & Blum, 2021), showing that the composition of the AND-mask and the trimmed-mean estimator is necessary for defending against backdoor attacks.
Revised: Note that the filtering and re-weighting methods complement each other because the re-weighting defense does not deal with the aforementioned standard attacks, as is discussed in

Original: Our analysis starts with feature invariance and discusses the connection between feature invariance and gradient sign consistency.
Revised: Section 4.

Original: Then, we outline conditions underwhich triggerbased backdoor attacks can lead to non-invariant directions and decrease the sign consistency of a dimension. Further analysis results demonstrate the necessity for the combination of both the 
Revised: We further provide some theoretical analysis of our methods under the Mimic-Shift attack. First, false-positives in filtering can eliminate a benign update and perturb the aggregated update.

Original: AND-mask and the trimmed-mean estimator.
Revised: Regarding this concern, we show that the false positive rate decreases quickly w.r.t. the number of reference clients and the scaling factor of the accept radius, suggesting that our filtering method can work with a few reference clients and a conservative scaling factor.

Original: Simulation results in Appendix D.1 further verify our theoretical results. Our empirical evaluation employs the strong edge-case backdoor attack (Wang et al., 2020), as detailed in Section 6.1, to test our defense. Empirical results on tabular (phishing emails), visual (CIFAR-10) (Krizhevsky, 2009; McMahan et al., 2017), and text (Twitter) (Caldas et al., 2018)
Revised: For the re-weighting phase, the probability of malicious updates having larger scalar projections than benign updates increases as the adversary takes more shares in a system, complementing existing results (He et al., 2022b) that guarantee more robustness as the share of the adversary decreases.

Original: datasets demonstrate that our method is effective in defending against backdoor attacks without degrading utility as compared to prior works.
Revised: Additionally, we discuss the performance of our method under a conventional minority adversary setting.

Original: On average, our approach decreases the model accuracy on backdoor samples by 61.6% and only loses 1.2% accuracy on benign samples compared to  the standard FedAvg aggregator (McMahan et al., 2017).
Revised: Finally, we show that, in a convex setting, our method converges with a rate of O ( 1 √ T ) to a neighborhood of the optimum.

Original: Contributions. Our contributions are  as follows:
Revised: Our contributions are summarized as follows:

Original: • We develop  a combination of defenses using an AND-mask and the trimmed-mean estimator against the backdoor attack by focusing on the dimension-wise invariant directions in the model optimization trajectory.
Revised: • We develop the Mimic-Shift attack and show that Mimic-Shift circumvents many defense methods in federated learning. • We develop a two-stage defense using filtering and re-weighting to defend against a broad class of attacks.

Original: • We theoretically analyze our strategy and demonstrate that a combination of an AND-mask and the trimmed-mean estimator is necessary in some conditions.
Revised: • We theoretically analyze our strategy and outline conditions under which it helps.

Original: • We empirically evaluate our method on three datasets with varying modality, model architecture, and client numbers, as well as comparing the performance to existing defenses.
Revised: Empirical results on FEMNIST (Caldas et al., 2018), CelebA (Liu et al., 2015) and Shakespeare (McMahan et al., 2017) datasets show that our aggregator recovers a near-optimal model under a majority adversary setting with Mimic-Shift attack, outperforming existing methods by a large margin. Also, our method only loses up to 2.4% accuracy under conventional minority adversary settings. Additional empirical results demonstrate that our method is robust to a broad class of attacks, including standard Gaussian and sign-flipping attacks as well as an improved Mimic-ShiftVar attack.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Backdoor Attack. Common backdoor attacks aim at misleading the model predictions using a trigger (Liu et al., 2018). The trigger can be digital (Bagdasaryan et al., 2020), physical (Wenger et al., 2021), semantic (Wang et al., 2020), or invisible (Li et al., 2021b). Recent works extended backdoor attacks to the federated learning setting and proposed effective improvements such as gradient scaling (Bagdasaryan et al., 2020) or generating edge-case backdoor samples (Wang et al., 2020).
Revised: Many existing works on Byzantine robust machine learning assume the adversary is the minority in the system, based on clustering (Blanchard et al., 2017), median (Yin et al., 2018), voting (Bernstein et al., 2019), bucketing (Karimireddy et al., 2022), and robust estimation (Data & Diggavi, 2021).

Original: The state-of-the-art edge-case backdoor attack shows that using backdoor samples with low probability density on benign clients (i.e., unlikely samples w.r.t. the training distribution) are hard to defend in the federated learning setting. Centralized Defense.
Revised: However, these methods can fail once the adversary becomes the majority.

Original: There is a line of work proposing centralized defenses against backdoor attacks where the main aim is either detecting the backdoor samples (Tran et al., 2018) or purifying the model parameters that are poisoned (Li et al., 2021a). However, applying such centralized defense to  federated learning systems is in practice infeasible due to limited access to  theclient data in many implementations. Federated Defenses.
Revised: A few papers (Xie et al., 2019b; 2020) leverage a validation dataset to improve Byzantine tolerance to arbitrary numbers of adversarial clients but constructing a global validation dataset in a federated scenario with local private datasets is infeasible. Other redundancy-based methods with more robustness guarantees do not apply to federated learning because they assume a centralized setting with control over the data allocation on each node (Chen et al., 2018; Rajput et al., 2019a).

Original: Several recent works have attempted to defend against backdoor attacks in  federated learning systems.
Revised: Recently, a clipping-based method (Karimireddy et al., 2021b; He et al., 2022b) showed success against minority adversary in a federated learning setting.

Original: Sun et al. (2019) shows that weak differential-private (weak-dp) federated averaging can mitigate the backdoor attack. However, the weak-dp defense is circumvented by the improved edge-case federated backdoor attack (Wang et al., 2020). Nguyen et al.
Revised: Although it is relatively straightforward to combine the clipping with our reference clients, we find that the training is unstable, as is discussed in Section 6.

Original: (2021) suggest that the vector-wise cosine similarity can help detect malicious clients performing backdoor attacks. The vector-wise cosine similarity is insufficient when the backdoor attacks can succeed with few poisoned parameters, incurring little vector-wise difference (Wu & Wang, 2021). Other defenses against untargeted poisoning attacks (Blanchard et al., 2017;  Xie et al., 2020) lack robustness against the backdoor attack.
Revised: Our paper focuses on model poisoning attacks, which aim to decrease the accuracy of the trained ML model by uploading malicious updates (He et al., 2022b). Other attacks, including data poisoning (Steinhardt et al., 2017; Wang et al., 2021) and backdoor (Wang et al., 2020; Xie et al., 2021), are beyond the scope of this work.

Original: Sign-SGD with majority vote (Bernstein et al., 2018; 2019) is similar to our approach, but it always takes the majority direction instead of focusing on  the invariant directions. Section 6.2 discusses the limitation of previous defenses in more detail, along with the empirical evaluation. Unlike existing works, our defense encourages the model to pursue invariant directions in the optimization procedure. Certification.
Revised: We focus only on training the centralized model.

Original: Unlike the above discussed defenses, certification (Xie et al., 2021) aims at extinguishing backdoor samples within a neighborhood of a benign sample. A direct comparison between certification and our defense is not meaningful due to  the different evaluation metrics.
Revised: Additional considerations such as personalization are known to be handled effectively using robust centralized training, followed by local fine-tuning Li et al. (2021), thus are beyond the scope of this paper.

Original: Certification considers the certification rate of benign samples as the metric, while our defense aims at reducing the accuracy of the backdoor samples.

Original: However, it would be interesting to investigate whether the proposed defense can ease the certification of a model.

Original:  P ROBLEM S ETUP
Revised:  P ROBLEM S ETUP

Original: Notation. We assume asynchronous federated learning system, where N  clients collaboratively train an ML model f : X −→ Y with  parameter w coordinated by a server. An input to the model are the data samples x ∈ X =
Revised: We assume a federated learning system where N benign clients collaboratively train a ML model f : X −→ Y with d -dimensional parameter ζ coordinated by a server. The N benign clients include

Original:  R d with d features indexed by k and a label y .
Revised: N R reference (trusted) clients.

Original: There are N ′ < Nadversarial clients aiming at corrupting the ML model during training (Shejwalkar et al., 2022).
Revised: There are N ′ adversarial clients who aim to corrupt the ML model during training.

Original: The i th , i ∈ [1 , ..., N  ] , client has n i data samples, being benign for i ∈ [1 , ..., N− N ′ ] or being adversarial for i ∈
Revised: The i th , i ∈ [1 , ..., N + N ′ ] , client has n i data samples, being benign for i ∈ [1 , ..., N ] or being adversarial for i ∈

Original: [ N− N ′ + 1 , ..., N  ] .
Revised: [ N + 1 , ..., N + N ′ ] .

Original: Thesynchronous federated learning is conducted in T rounds.
Revised: The federated learning is conducted in T rounds.

Original: Ineach round t ∈ [1 , ..., T ] , the server broadcasts a model parameterized by w t − 1 to all the participating clients.
Revised: In round t ∈ [1 , ..., T ] , the server broadcasts a model parameterized by ζ t − 1 to each client.

Original: We omit the subscript t while focusing on a single round.
Revised: We omit the subscript t while focusing on one round.

Original: Then, the i th client optimizes w t − 1 on their local data samplesindexed by j and report the locally optimized w t,i to the server.
Revised: Then, the i th client optimizes ζ t −with their local data samples and report ζ t,i to the server.

Original: We define pseudo-gradient g t,i = w t − 1 − w t,i being the difference between the locally optimized model and the broadcasted model from the previous round.
Revised: We define pseudo-gradient g t,i = ζ t − 1 − ζ t,i being the difference between the locally optimized model and the broadcasted model from the previous round.

Original: Note, for simplicity, that we  often use the term “gradient” to refer to the pseudo-gradient.
Revised: Note, for simplicity, that we will often use the term ”gradient” to refer to the updates.

Original: Once all  gradients are uploaded, the server aggregates them and produces a new model with parameters w t using the following rule: w t = w t − 1 − (cid:80) Ni = n i (cid:80) Ni =
Revised: Once all the gradients are reported in, the server aggregates the gradients and produce a new model with parameters ζ t using the following rule: ζ t = ζ t − 1 − (cid:80) N + N ′ i = n i (cid:80) N + N ′ i

Original: 1 n i g t,i .
Revised: = n i g t,i .

Original: The goalof federated learning is to minimize aweighted risk function over the N clients: F ( w ) 
Revised: Our goal is to minimize a risk function over the benign clients: F ( ζ ) =

Original:  = 
Revised: (cid:80) Ni = n i (cid:80) Ni = 1 n i F [Equation]

Original:  (cid:80) 
Revised: ( ζ ) = (cid:80) Ni =

Original: [Equation]
Revised: [Equation]

Original:  where ℓ : R × Y −→ R is a loss function.
Revised: [ ℓ ( f ( x ; ζ ) , y )] , where ℓ : R × Y −→ R is a loss function.

Original: ⊙ denotes the Hadamard product operator.
Revised: Following the standard practice (Blanchard et al., 2017), we consider an omniscient man-in-themiddle (MITM) adversary that knows g i , ∀ i ∈ { 1 , ..., N } and can tell which clients are reference users, e.g., by observing the device identifier in the message. An omniscient adversary helps explore the limits of the defense.

Original: Threat Model.
Revised: However, an omniscient adversary is not mandatory for our Mimic-Shift attack, which can work with partial information as Section 6 will show. The MITM adversary also owns a majority of clients in a system and adopts an attack from the Mimic family (Karimireddy et al., 2022). Specifically, we assume the following Mimic-Shift attack. Mimic-Shift At each round, the MITM adversary first intercepts the gradients from benign clients.

Original: The adversary generates a backdoor data sample x ′ by embedding a trigger in a benign data sample x and correlating the trigger with a label y ′ , which is different from the label y of the  benign data sample. We use D ′ to denote the distribution of backdoor data samples.
Revised: Then the adversary computes the average reference gradient ¯ g R = (cid:80) N R i = n Ri (cid:80) NRi =1 n Ri · g R i from the reference users indexed by R i and the average benign gradient ¯ g = (cid:80) N i = n i (cid:80) N i =1 n i · g i from all benign users, including the reference users.

Original: Then,  the malicious clients connect to the federated learning system and insert backdoor data samples into the training set.
Revised: Then, all the adversarial clients report g ′ = ¯ g R + (¯ g R − ¯ g ) to the server.

Original: Since the goal of federated learning is  to minimize the risk over all clients’ datasets, the model can entangle the backdoor while trying to minimize the risk over backdoor samples on the malicious clients.
Revised: The Mimic-Shift attack is effective because it tries to push the aggregated gradient away from the expected gradient. The reason is that ¯ g is estimated with more clients and data samples.

Original: Appendix A visualizes some backdoor samples.
Revised: A simple concentration argument suffices to show that ¯ g will be closer to E [¯ g ] than ¯ g R with high probability, resulting in a ¯ g R − ¯ g pointing away from E [¯ g ] .

Original: [Figure]
Revised: Empirical results in Section 6 show that Mimic-Shift decreases the accuracy twice as much as Mimic (Karimireddy et al., 2022), which sets g ′ =

Original: To simplify tedious notation, we assume all users have the same number of samples i.e., n i = n i ′ , ∀ i ̸
Revised: g i for an i ∈ { 1 , ..., N } . The Mimic-Shift attack is also difficult to detect because the malicious g ′ has the same distance to the reference ¯ g R as the benign ¯ g .

Original: = i ′ .
Revised: We will see how this property lets Mimic-Shift bypass distance-based defenses in Section 6.

Original:  M ETHOD
Revised:  M ETHOD

Original: This section presents the proposed server-side defense operating on the gradients. We start with an overview of the idea, then introduce the two complementary components of our defense and outline the invariant aggregator steps.
Revised: Our robust aggregator has two phases, filtering and re-weighting. The filtering phase, applied first, excludes the gradients outside the accept region defined by the reference gradients, defending against standard attacks.

Revised: Then, the re-weighting phase further down-weights the potential malicious gradients within the accepting region of the filtering phase, targeting the Mimic-Shift attack.

Original: 4. O VERVIEW
Revised: 4. P HASE 1: F ILTERING

Original: In the proposed defense, we aim at finding invariant directions to optimize the federated learning model, such that the model is generally utilitarian for most of clients and can exclude the directions that benefit a subset of potentially malicious clients. Since our defense operates with the gradients, we consider the invariant direction from a first-order perspective (Parascandolo et al., 2021). Expanding the loss function around the current weight w on client i , for a parameter update g , we have:
Revised: In the filtering phase (Algorithm 1), we first compute the sample mean of reference gradient m R = (cid:80) N R i =

Original: [Equation]
Revised: [Equation]

Original: where R 2 ( w + g ) is a second-order Taylor remainder. With a reasonably small ∥ g ∥ (achievable with small learning rates), the remainder term R 2 ( w + g ) is negligible and the change of the loss function mainly depends on the first-order gradient and the parameter update g . Since learning the trigger benefits the malicious clients exclusively, there exists at least one gradient dimension k and one benign client i ∈ { 1 , ..., N − N ′ } where E D ′ [ ∇ w k ℓ ( x , y ; w )]
Revised: 1 N R · g R i , making a guess of where the good gradients might be. Next, we set m R as the center of a spherical accept region.

Original: ̸ = 0 and the dimension-wise gradient have inconsistent signs, i.e.,
Revised: Then, we compute the sample variance of the reference gradients,

Original: [Equation]
Revised: [Equation]

Original: If the condition in Equation 2 isnot true, then learning the trigger would always benefit the model on benign data, thereby contradicting common empirical observation. A more detailed analysis of the condition in Equation 2 is in Section 5.2.
Revised: The sample variance s R is further scaled up by a tunable hyper-parameter c . The product c · s R is the radius of the spherical accept region.

Original: The proposed methods show how to treat the inconsistent signs and defend against the backdoor attack by only allowing invariant directions. A dimensionwise analysis is necessary because backdoor attacks can succeed by poisoning few inconsistent dimensions (Wu & Wang, 2021) without incurring much vector-wise difference, as  Section 6.2 will show.
Revised: Finally, we remove all the updates outside the spherical accept region. This filtering is effective against attacks that do not know where the expected gradient is (e.g., Gaussian attack) or do not carefully calibrate the adversarial gradient (e.g., sign flipping attack), as is shown in Section 6. Algorithm 1 Filtering.

Original: We consider two treatments for each gradient dimension to help the model avoid the direction specified by E D ′ [ ∇ w ℓ ( x ′ , y ′ ; w )] : (Treatment 1) setting the dimension with inconsistent signs to zero using an AND-mask (Parascandolo et al., 2021) such that no client benefits or (Treatment 2) employing a robust mean estimator (e.g., trimmed-mean (Xie et al., 2020)) to remove the malicious values that cause the inconsistent sign.
Revised: Input: A set of reference gradients, { g R i | i ∈ { 1 , ..., N R }} ; A set of reported gradients, { g i | i ∈ { 1 , ..., N 4. P HASE 2: R E - WEIGHTING

Original: To achieve a better result, we combine these two treatments to avoid their failure modes. The following examples illustrate the failure mode of each treatment and motivate the combination of defense. Failure Mode 1.
Revised: Suppose a majority adversary misleads the aggregated gradient. In that case, the malicious gradients likely have a stronger influence on the aggregated gradient compared to benign gradients.

Original: Figure 1a shows an example where the adversary may exploit a dimension by inserting outliers where the  benign values have a consistent sign.
Revised: Here, we measure the influence via scalar projections. The re-weighting defense is outlined in Algorithm 2. The scaling weights are designed to augment benign gradients and down-weight adversarial updates. This is implemented using a monotonic re-scaling 1 of the scalar projection between the aggregate and the gradients (step 4).

Original: The outliers can mislead the average toward the non-invariant direction.
Revised: The monotonic re-scaling amplifies the difference between the scalar projections, making large projections larger. Therefore, the potential malicious gradients would be down-weighted more. However, the monotonic re-scaling may also lead to an over-up-weighting on certain benign gradients, which are far from ¯ g ∗ . such up-weighting leads to instability in the training process.

Original:  The robust mean estimator (treatment 2) can trim the outliers and accurately estimate the mean. In contrast, treatment 1 can fail with a high sign consistency.
Revised: Thus, we also include a clipping operator (step 6). The clipping addresses the over-up-weighting issue by specifying a bound.

Original: Because it either lets the highly consistent dimension pass or has to be over-aggressive in zeroing out the dimensions, hurting the model’s accuracy on benign data.
Revised: An additional benefit of using the power function and clipping is that they prompt a set of uniform weights on benign gradients, stabilizing the training. If we set k to be sufficiently large such that the maximum s i is greater than N F · ( (cid:80)

Original: Failure Mode 2.
Revised: N F i =

Original: Figure 1b shows some values with inconsistent signs, which treatment 1 can handle.
Revised: s i − max i ∈{ 1 ,...,N F } s i ) , we have s i = τ, ∀ i ̸ = arg max i ∈{ 1 ,...,N F } s i . In practice, we set k to 10 and τ to 0.6.

Original: However, this example can fail the robust mean estimator (treatment 2), whose result has the same sign as the malicious values. The following sections shall detail the two treatments and discuss their complementary relationship. 4. AND-M ASK
Revised: Both hyper-parameters generalize well to the three datasets in our experiments. Note that the filtering phase complements the re-weighting phase because standard attacks can corrupt the scalar projections via uploading 0 or flipping the sign.

Original: The AND-mask (Parascandolo et al., 2021) computes a dimension-wise mask by inspecting the sign consistency of each dimension across clients. For dimension k , the sign consistency is: | 1 N
Revised: For a class of inner product-based attacks (Xie et al., 2019a), an additional clipping operator that prevent the scalar projection s i from being negative can help. Algorithm 2 Re-weighting. [Equation]

Original: [Equation]
Revised: [Equation]

Original: Ni =1 sign( g i,k ) | .
Revised: We first show that the probability of not filtering out a benign gradient increases at a rate of

Original: If the sign consistency is below a given threshold τ , the mask element m k is set to 0, otherwise, m k is set to 1.
Revised: O (1 − 1 N 2 R ) w.r.t. the number of reference clients N R and O (cid:0) (1 − 1 c 2 ) 2 (cid:1) w.r.t.

Original: The mask along dimension k is defined as: Definition 1.
Revised: the scaling factor c in the accept radius.

Original: (AND-Mask) For the k th dimension in the gradient vector, the corresponding mask m k is defined as: [Equation] Our defense then multiplies the mask m with the aggregated gradient ¯ g element-wise, setting the inconsistent dimension to zero.
Revised: Then, we discuss conditions for the probability of down-weighting the malicious gradients to increase as the adversary owns more clients in the system.

Original: 4. T RIMMED -M EAN
Revised: Additional discussion considers how our strategy may still help even if the aforementioned conditions do not hold, providing insights to certain of empirical results.

Original: To complement the AND-mask, our defense broadcasts the trimmed-mean estimator to each gradient dimension.
Revised: Finally, we study the convergence of our method under a convex setting.

Original: The trimmed-mean estimator alleviates the outlier issue by removing the subset of largest and smallest elements before computing the mean. The largest and smallest elements appear on the two tails of a sorted sequence. Next, we define order statistics and the trimmed mean estimator.
Revised: Before proceeding, we outline some additional assumptions. To simplify tedious notation, we assume all users have the same number of samples because the difference in sample size can be merged into the difference between gradients.

Original: Definition 2.
Revised: Assumption 1.

Original: (Order Statistics) (Xie et al., 2020) By sorting the scalar sequence { x i : i ∈ { 1 , ..., N } , x i ∈ R } , we get x 1: N ≤ x 2: N ≤ ... ≤ x N : N , where x i : N is the i th smallest element in { x i : i ∈ { 1 , ..., N }} . Then, the trimmed-mean estimator removes a α × N elements from each tail of the sorted sequence.
Revised: Uniform sample size: [Equation]

Original: Definition 3.
Revised: Assumption 2.

Original: (Trimmed Mean Estimator) (Xie et al., 2020) For α ∈ [0 , 1] , the α -trimmed mean of the set of scalars x i : N ∈ { 1 , ..., N } is defined as follows:
Revised: The gradients follow a hierarchical distribution in all rounds:

Original: [Equation]
Revised: [Equation]

Original: where ⌈ .

Original: ⌉ denotes the ceiling function. 4. O UR A PPROACH : I NVARIANT A GGREGATOR Algorithm 1outlines the steps of our server-side defense that perform aggregation of invariant updates from the clients. The solution is composed of the AND-mask (treatment 1) and trimmed-mean estimator (treatment 2).
Revised: where Stage 1 is a distribution where the random variable µ has finite expected value E [ µ ] and finite non-zero variance Var[ µ ] .

Original: Our defense applies the two components separately based on the sign consistency of each dimension with a  threshold τ . We show how these two components, i.e., the AND-mask and the trimmed-mean estimator, complement each other. Our analysis considers a single dimension and starts with the robustness of the trimmed mean estimator, which improves the robustness of the AND-mask against outliers. The following theorem extends the robustness guarantee of a modified trimmed-mean estimator (Lugosi & Mendelson, 2021), which is shown in Appendix B, to the conventional trimmed-mean estimator (Definition 3).
Revised: Stage 2 is truncated isotropic Gaussian distribution with a truncation threshold γ + on the L 2 norm.

Original: Theorem 4.
Revised: Assumption 3.

Original: With the trimmed-mean estimator in Definition 3, for a given set of samples x 1 , ..., x N , with η = N ′ N and a confidence level δ , set α = 8 η + 12 log( 4 δ ) N , let a = x αN : N and b = x N − αN : N following Definition 2, x be a random variable with variance σ and ¯ x be the estimated mean, with probability at least c − 4 (1 − 4 e − αN 12 ) , we have:
Revised: The variance of gradients is bounded:

Original: [Equation]
Revised: [Equation]

Original: The proof is in Appendix B. Theorem 4 bounds the estimation error of a trimmed-mean estimator, which can increase as the variable’s variance increases. Multiple factors can increase the variance, such as non-i.i.d. federated data distribution and the stochastic gradient estimation process. In practice, a threat analysis is necessary to specify the maximum number of malicious clients to be tolerated, when the number of malicious clients is unknown.
Revised: Assumption 4.

Original: Our goal is to prevent the outliers from misleading the sign. Therefore, the estimation error to expectation ratio is particularly relevant.
Revised: The sample variance of reference gradients is greater than 0:

Original: Since the estimator error for a given α depends on the variance, a high expectation-variance ratio is desirable. Then, we show that AND-mask identifies the elements with high expectation-variance ratios, avoiding the robustness degradation of the trimmed-mean estimator. The following theorem suggests that dimensions with a higher expectation-variance ratio have high probabilities of passing AND-mask, and increasing the mask threshold τ increases the chance of filtering out dimensions with low expectation-variance ratios.
Revised: [Equation]

Original: Theorem 5.
Revised: Assumption 5.

Original: Given a non-zero expectation-variance ratio ϕ , N ′ malicious elements, with probability at most (cid:80) min( N,N − 2 N ′ + τ ) i = N − 2 N ′ − τ (cid:0) N − N i (cid:1) ϕ − 2 i , the sign consistency is below τ .
Revised: The L 2 norm of expected and estimated gradients is bounded:

Original: Appendix B provides the proof. For the ϕ = 0 case, we may use the probability of the estimated gradient sign being the same as the malicious gradient to replace ϕ − 2 . We do not propose directly using the sample mean-variance ratio due to a potential issue: using the sample mean-variance ratio can be over-aggressive when the benign value has a consistent sign but varying magnitudes. Our ablation study in Appendix D.2 shows that AND-mask can preserve more utility than the sample mean-variance ratio when combined with the trimmed-mean estimator.
Revised: [Equation]

Original: R OBUSTNESS ANALYSIS FOR A SIMPLIFIED MODEL
Revised: ∈ { 0 , 1 , ..., N } .

Original: To further motivate our approach, we consider a more direct robustness analysis for a specific generative model. This analysis discusses how the features impact the gradient sign consistency, when the two failure modes in Section 4.1 appear, and why we need a combination of defenses guaranteed by Theorems 4 and 5. First, we outline some preliminaries useful for this analysis. 5. P RELIMINARIES
Revised: [Figure]

Original: We consider a binary prediction task y ∈ { 0 , 1 } with a linear model h ( x ) =
Revised: are two spherical contours around E [ µ ]

Original: w ⊤ x and a decision rule ˆ y = 1 w ⊤ x ≥0 ( w ⊤ x ) . The training procedure uses a Sigmoid activation function s ( z ) = 1 1+ e − z and a logistic loss function ℓ ( z, y ) =
Revised: Condition c ≥

Original: − y · log( z ) − (1 − y ) · log(1 − z ) , where z = h ( x ) . Data Model. We assume that the samples per class are balanced on benign clients.
Revised: E holds in Case 1, where all the points outside the accept region are more than c · s R 2 away from E [ µ ] .

Original: The data samples come from a non-i.i.d. Gaussian distribution with a diagonal covariance matrix. On the i th client, for the k th feature, we have: [Equation]
Revised: In Case 2 and Case 3, there are points between the two contours that have the same distance to E [ µ ] but are not always inside accept region.

Original: [Equation] Backdoor Attack.
Revised: 5. F ILTERING (S TAGE 1)

Original: We consider a single feature backdoor attack where the trigger is the k th feature x k . A backdoor using a specific feature is common (Wang et al., 2020). To backdoor images, the adversary often selects a pixel pattern or semantic pattern (e.g., blue color on airplanes).
Revised: The filtering phase removes all the gradients that are more than c · s R away from the reference m R in terms of Euclidean distance.

Original: For text data, the trigger could be a dedicated set of characters . The malicious clientsattack incollusion using the same backdoor samples, meaning that µ i,k = µ i ′ ,k ̸ = 0 , ∀ i, i ′ ∈ { N − N ′ + 1 , N } . To simplify the notation, we omit the subscript i for simplicity while focusing on a single client. Since the trigger is not useful or does not appear on benign clients, we assume µ i,k = 0 , ∀ i ∈ { 1 , N − N ′ }
Revised: However, the data distribution D i may vary across clients in a federated learning setting.

Original: Objective. The backdoor attack is effective if the model entangles the trigger-label correlation. For a non-zero µ k of the trigger x k , w k µ k > 0 is a necessary condition for a model to entangle the trigger-label correlation. Therefore, in the case when µ k > 0 , avoiding w k from increasing and enforcing w k to decrease while w k > 0 can mitigate the backdoor attack.
Revised: Such non-i.i.d.ness increases the risk of filtering out a benign gradient.

Original: We define the invariance of a feature by measuring its feature-label correlation (e.g., positive or negative) consistency across clients.
Revised: We start our analysis by considering the expected gradients:

Original: Definition 6.
Revised: Lemma 6.

Original: For a given feature k , its invariance p is defined as: [Equation] Similarly, we define the sign consistency of a gradient dimension k :
Revised: Suppose there are N R reference clients among N clients, and the sample variance of gradients is s R , under Assumption 2, let c ≥ 2 · ∥ E [ µ ] − ¯ µ ∥ s R . Then, for any i ∈ [1 , ..., N ] , with probability at most 4 · Var[ µ ] 2 c 2 · s 2 R , we have ∥ µ i − ¯ µ R ∥ ≥ c · s R , where ¯ µ R = (cid:80) Ri =1 1 N R · µ R i .

Original: Definition 7.
Revised: Remark 7.

Original: With a linear model h ( x ) = w ⊤ x , a Sigmoid activation function s  , and N clients, a q -consistent gradient w.r.t.
Revised: Condition c ≥ 2 · ∥ E [ µ ] − ¯ µ ∥ Var[ µ ] guarantees that all µ i s outside the accept region are at least c · s R 2 away from E [ µ ] , as Figure 1 shows, enabling concentration argument.

Original: w k satisfies:
Revised: Otherwise, bounding the probability of µ i appears outside the accept region is hard without additional assumptions on

Original: [Equation]
Revised: P ( µ ) .

Original: Under Definitions 6 and 7, we discuss the connection between feature invariance and dimensionwise gradient sign consistency. First, we need to analyze the behavior of  the gradient sign per client.
Revised: Lemma 6 suggests that the probability of filtering out a benign gradients decreases at a rate of O ( 1 c 2 ) . The following lemma further shows the probability of violating the assumed condition c ≥ 2 · ∥ E [ µ ] − ¯ µ ∥ s R decreases at a rate of O ( 1 N 2 R ) .

Original: Theorem 8.
Revised: Lemma 8.

Original: For a linear model with a Sigmoid activation function s and the logistic loss ℓ ,under our Gaussian data model, on the k th feature with a non-zero µ k , if w k µ k ≤ 0 , we have sign (cid:16) E x ,y ∼D i (cid:2) ∇ w k ℓ ( s ( w ⊤ · x ) , y ) (cid:3) (cid:17)
Revised: With Assumptions 2 and 5, for a fixed accept radius c · s R , with probability at most 4 · Var[ µ ] 2 N R · c 2 · s 2 R , we have c ≤ 2 · ∥ E [ µ ] − ¯ µ ∥ s R . The following theorem combines and extends the result on expected gradients to estimated gradients.

Original: = sign( µ k ) .
Revised: Theorem 9.

Original: In addition, if µ k = 0 , we have sign (cid:16) E x ,y ∼D i (cid:2) ∇ w k ℓ ( s ( w ⊤ · x ) , y ) (cid:3) (cid:17)
Revised: Under Assumptions 2 and 3, with probability at least [Equation] ) , we have ∥ g i − m R ∥ ≤ c · s R

Original: = sign( w k ) .
Revised: [Equation]

Original: The proof is provided in Appendix B. The result in Theorem 8 is intuitive.
Revised: Theorem 9 shows that the risk of filtering out a benign gradient decreases quickly w.r.t.

Original: With a non-zero µ k , if sign( w k ) agrees with sign( µ k ) , the gradient sign can be indefinite because the weight w k can be either larger or smaller than the optimal w ∗ k . 
Revised: c and N R , enabling our strategy of using a small number of reference clients and a conservative c . 5. R E - WEIGHTING (S TAGE 2)

Original: Otherwise, the gradient has the same sign as µ k . If µ k = 0 , w k shall shrink to 0. Then, we outline the conditions that lead to inconsistent signs.
Revised: Suppose the adversary adopts the Mimic-Shift attack and bypasses the filtering defense. We would like to figure out the probability of assigning a higher scalar projection to the malicious gradient g ′ than a benign gradient g i , i ∈ { 1 , ..., N } .

Original: Corollary 9.
Revised: Theorem 10.

Original: Under Theorem 8, suppose x k represents the trigger, µ k = 0 on the N − N ′ benign clients and the N ′malicious clients share the same non-zero µ k , if w k =0, the gradient w.r.t.
Revised: Suppose all the gradients pass the filtering phase. Let the aggregated gradient be ¯ g ∗ = w · ¯ g + w ′ · g ′ , where ( w, w ′ ) = (cid:18) (cid:80) Ni =

Original: w k is N ′ 
Revised: 1 n i (cid:80) N + N ′ i = n i , (cid:80) N + N ′ i

Original:  N -consistent.
Revised: [Equation] N +1 n i (cid:80) N + N ′ i = n i (cid:19) .

Original: In addition, if w k µ k > 0 ,  the gradient w.r.t.
Revised: Let θ be the angle between ¯ g and [Figure] g ′ , θ ∗ be the angle between ¯ g ∗ and g ′ . Assume ww ′ ≤ ∥ g ′ ∥− cos θ 2 ·∥ ¯ g ∥

Revised: ∥ ¯ g ∥− cos θ 2 ·

Revised: ∥ g ′ ∥ such that θ ∗ ≤ θ − θ ∗ .

Revised: Then, under Assumptions 1- 5, let γ − θ ∗ =

Revised: ∥ γ − ∥ · tanh( θ − 2 · θ ∗ ) , with probability at least

Revised: [Figure]

Original: w k is (1 − N ′ N ) -consistent.
Revised: , we have [Table] [Figure] [Table]

Revised: [Table]

Original: The gradient w.r.t.
Revised: [Figure] [Table] [Figure] Theorem 10 suggests that if 2 · θ ∗ ≤ θ , the probability of down-weighting malicious gradients more than benign gradients (i.e., s ′ >

Original:  w k is 1-consistent if w k µ k < 0 .
Revised: s i ) increases as the adversary owns more clients and samples in a system (i.e. w ′ ↑ ). The main idea is: the minimum amount of deviation between g i and ¯ g for having s i > s ′ increases as w ′ increases because w ′ ↑ ⇝ θ ∗ ↓ . This specific deviation is shown in Figure 2a as the dark dash line with length at least ∥ γ − ∥ · tanh( θ − 2 · θ ∗ ) .

Original: Counting the gradient signs yields the result.
Revised: Then, applying concentration arguments yields the result.

Original: Corollary 9 suggests that if w k µ k > 0 and the model entangles the backdoor, the expected gradient w.r.t.
Revised: Theorem 10 only deals with the case where ¯ g ∗ has a smaller angle with g ′ than ¯ g . However, in practice, we find that our method also achieves a decent accuracy even the condition 2 · θ ∗ ≤ θ does not hold.

Original: w k shall align with µ k on malicious clients and conflict with µ k on benign clients.
Revised: To gain some insight, suppose there is a probability density function f of benign gradients. Then, the probability of having s i > s ′ , ∀ i ∈ { 1 , ..., N } equals the integral of f over a cone, which is defined by spinning g ′ around ¯ g ∗ as Figure 2b shows. In a federated scenario, the estimated g i concentrates around its own µ i , which is not necessarily close to ¯ g or ¯ µ , resulting in a small integral of f over the aforementioned cone. 5. C ONVERGENCE A NALYSIS

Original: Then, employing the trimmed-mean estimator to remove the malicious values can recover the invariant direction pointed by benign clients (Theorem 4) and thereby shrink µ k . If µ k = 0 , using AND-mask can mask out the gradients w.r.t.
Revised: So far, our analysis has focused on the robustness of one update. The following extends the single step analysis to a convergence analysis. Before showing the theorem, we present a lemma that quantifies the impact of false-positive filtering.

Original: w k from the malicious clients (Theorem 5).
Revised: Lemma 11.

Original: If w k remains 0 , the model parameterized by w is robust to the trigger on x k . It is worth noting that the gradients w.r.t.{+a filtering function, let mask M = F ( g 1 , ..., g N ) , N ′ F = 1 − ∥ M ∥ 1 , ˆ g = 
 ∥ M ∥ 1 · (cid:80) N i =+}
Revised: Suppose F : N × R d −→ N ×{ 0 , 1 } is {+a filtering function, let mask M = F ( g 1 , ..., g N ) , N ′ F = 1 − ∥ M ∥ 1 , ˆ g = 
 ∥ M ∥ 1 · (cid:80) N i =+}

Original: w k is 1-consistent and align to µ k when w k µ k < 0 . Such a consistent gradient may overshoot and flip the sign of w k .
Revised: 1 M i · g i , and δ = ˆ g − ¯ g . Then, with Assumption 5, we have ∥ δ ∥ ≤ 2 · N ′ F N · γ + .

Original: Reducing the learning rate can alleviate overshooting and the trimmed-mean estimator will guarantee w k to shrink after the overshooting.
Revised: In the following theorem, we use assumed conditions to ease the reading. Later , we shall connect the assumed conditions to Theorems 9 and 10 and further support the assumed conditions with empirical results in Appendix C. Theorem 12. Suppose the malicious gradient g ′ t is filtered out with probability 0, ∀ t ∈ { 1 , ..., T } , and down-weighted by s ′ t ≥ 1 .

Original: Connection to Failure Modes.
Revised: Assume at most N ′ F benign gradient are filtered out at each iteration. Define an aggregated gradient ¯ g ∗ t = w · ˆ g t + w ′ · g ′ t s ′ t = w · (¯ µ t + δ t + ϵ t ) + w ′ · g ′ t s ′ t , where ˆ g t and δ t follow the definitions in Lemma 11, ϵ t ∼ 1 N · (cid:80) Ni =1 N (0 , σ t,i ) as Assumption 2 shows. Let ζ ∈ Z be the model parameter, ζ ∗ be the optimum, and ¯ ζ = 1 T (cid:80) T t =1 ζ t . Assume sup ζ ∈ Z ∥ ζ ∥ ≤ D and

Original: If w k µ k > 0 , the gradients w.r.t. w k have a consistent but non-zero expectation among benign clients, causing the failure mode 1 in Section 4.1.
Revised: F : Z −→ R is convex. Let C = w · 2 · N ′ F N · γ +2 +2 · w · γ +2 + w ′ · γ +2 , C ′ = 2 · w · C + C 2 + w · γ + , and η t = D √ T · C ′ , under Assumptions 2 and 5, we have:

Original: On the other hand, the gradient variance can diversify the estimated gradient sign and cause the failure mode 2.
Revised: [Equation] The convergence analysis shows that our method converges to a neighborhood around the optimum, whose size shrinks with fewer false-positive filtering (Theorem 9) and more down-weighting (tuning k and τ in Algorithm 2) with higher probability (Theorem 10) on the malicious gradients. Converging to a neighborhood is common in previous noisy gradient descent studies (Wang et al., 2021;

Original: Appendix D.1 further provides simulation results under the linear regime (Section 5.1), showing that the AND-mask can prevent the adversary from exploiting w k and the trimmed-mean estimator helps shrink w k .
Revised: He et al., 2022a). Our result also suggests scaling up the learning rate as the adversary owns more clients. In practice, this scaling is handled by our re-weighting phase where the benign gradients are down-weighted by scalars less than 1.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We evaluate our defense on three realistic tasks on three different data types: (1) object recognition with visual data, (2) sentiment analysis with text data, and (3) phishing email detection with tabular data. We employ the state-of-the-art edge-case backdoor attack (Wang et al., 2020) to generate backdoor samples and  evaluate of defense and existing defenses against it.
Revised: We first compare the Mimic-Shift attack with the Mimic attack, showing that Mimic-Shift is more effective and the improved effectiveness remains with a non-omniscient adversary (Mimic-ShiftPar). Then, we evaluate our defense strategy against the Mimic-Shift attack where our strategy outperforms strong baselines.

Original: Additional Results 
Revised: Additional Results Appendix C provides more results.

Revised: We evaluate our defense via a nonomniscient Mimic-Shift-Par attack, an improved Mimic-Shift-Var attack as well as other standard attacks (e.g., Gaussian and sign-flipping).

Original: The simulations, an ablation study and an evaluation of the hyper-parameter sensitivity can be found in the Appendix D.
Revised: There are also additional experiment with fewer clients and various attack strengths, plots of the re-weighting vector s as well as convergence, and studies on defense hyper-parameters.

Original: 6.E XPERIMENTAL S ETUP
Revised: 6. S ETUP

Original: We briefly summarize our setup and report more details in Appendix C. Metrics. Our experiments employ two metrics: the main task accuracy (Acc M ) estimated on the benign samples and the backdoor task accuracy (Acc B ) over backdoor samples.
Revised: We use three datasets, FEMNIST (FM) (Caldas et al., 2018), CelebA (CA)

Original: A defense is designed to reduce the model’s accuracy on backdoor task andmaintain the utility on the main task.
Revised: (Liu et al., 2015), and

Original: Datasets. The visual data of the object detection task and text data of the sentiment analysis task are from CIFAR-10 (Krizhevsky, 2009; McMahan et al., 2017) and Twitter (Caldas et al., 2018), respectively.
Revised: Shakespeare (SS) (McMahan et al., 2017), with realistic non-i.i.d. partitions (Caldas et al., 2018) implemented in the FedML library (He et al., 2020).

Original: Each phishing email data sample has 45 standardized numerical features of the sender that represent the sender reputation scores. A large reputation score may indicate a phishing email.
Revised: The number of benign clients ranges fromto 500.

Original: The reputation scores come from peer-reviewers in a reputation system (Jøsang et al., 2007).
Revised: We select 2 reference clients and 28 other benign clients at each round.

Original: The adversary may use malicious clients to manipulate the reputation.
Revised: The number of adversarial clients is adjusted accordingly.

Original: Federated Learning Setup. We consider horizontal federated learning (Kairouz et al.,2021) where the clients share the same feature and label spaces.
Revised: We report the detailed setup in Appendix B, including various hyper-parameters. We employ seven baselines, including federated averaging (FedAvg) (McMahan et al.,

Original: The number of clients are 100 for the three tasks.
Revised: Appendix

Original: The server sample 20 clients at each round on the CIFAR-10 and phishing email experiments. We reduce the sampled client number to 15 on the Twitter experiment due to limited hardware memory.
Revised: B lists the details of these baselines. The ”Oracle” aggregator operates with benign gradients and serves as a reference. Five attacks are considered in our experiments, including Mimic-Shift-Var, particularly designed as a strong attack for our proposed defense.

Original: Backdoor Attack Setup. The adversary employs the edge-case backdoor attack, where it selects the data samples with low marginal probability in their data distribution to create backdoor samples.
Revised: Gaussian Draw a random update g ′ i from an isotropic Gaussian distribution N (0 , 200) . Sign-flipping Flip

Original: The visual and text backdoor samples follow the previous work (Wang et al., 2020).
Revised: the sign of the estimated gradient g i =

Original: For the tabular data, we select the 38 th feature (reputation), whose value is 0 on most of the data samples.
Revised: − g i and report g

Original: Then, we let the adversary manipulate the 38 th feature to0.2 that has a low probability density on phishing emails and flip the label to non-phishing.
Revised: i to the server. Mimic-Shift Report g =

Original: The adversary can control 20% clients on the CIFAR-10 and phishing email experiments and 10% clients on the Twitter experiment. Section 6.2 explains the different experiment configurations.
Revised: ¯ g R + (¯ g R − ¯ g ) to the server, as is shown in Section 3.1. Mimic-Shift-Par Randomly eavesdrop 20% clients per round and draw two clients as reference.

Original: Such an adversary is considered strong in practice (Shejwalkar et al., 2022).
Revised: Mimic

Original: We consider a strong adversary because defending against strong adversary yields robustness against weak adversary, whose effectiveness is already shown (Wang et al., 2020). The adversary only uses backdoor samples during training.
Revised: -Shift-Var Mirror the local update using ¯ g R and report g ′ i = ¯ g R + (¯ g R − ¯ g i ) to the server

Original: 6. R ESULT AND C OMPARISONTO P RIOR W ORKS
Revised: 6. M IMIC - TYPE A TTACK C OMPARISON

Original: Our results. Table 1 summarizes the performance of each defense on three tasks.
Revised: Table 1 shows the accuracy of FedAvg aggregator under Mimic and Mimic-Shift attack.

Original: Our approach decreases the backdoor task accuracy by 61.6% on average. The edge-case backdoor attack on the text sentiment analysis task (Twitter) is more difficult to defend andour approach mitigates the accuracy increase on the backdoor task by 41.7%.
Revised: Here, the adversary owns 80% of the system, and the Mimic adversary mimics the first client in the system. Mimic-Shift attack constantly outperforms Mimic and is 116% more effective on average.

Original: We hypothesize that the text sentiment analysis task has few invariant and benign features. For example, the shape features (Sun et al., 2021) in object classification tasks can be invariant across objects. In contrast, the sentiment largely depends on the entire sentence instead of a few symbols or features. Then, we discuss the limitations of prior defenses.
Revised: Such advantages are preserved under a non-omniscient adversary.

Original: [Table]
Revised: [Table]

Original: Vector-wise. Common vector-wise defenses such as Krum estimate pair-wise similarities in terms of Euclidean distance (Blanchard et al., 2017)
Revised: 6. D EFENSE AGAINST M IMIC -S HIFT A TTACK

Original: (Krum and multi-Krum) of cosime similarity (Nguyen et al., 2021) (multi-Krum C ) between each gradient and others.
Revised: This experiment considers three settings where the adversary owns 0% - 80% of a system.

Original: The gradients that are dissimilar to others are removed. The vector-wise view is insufficient for defending against backdoor attacks because backdoor attacks can succeed by manipulating a tiny subset of parameters (e.g. 5%)
Revised: The hyper-parameters of each aggregator are selected based on the 80% adversary setting and directly applied to the other settings. Table 2 shows the accuracy of our strategy and baselines under the Mimic-Shift attack.

Original: (Wu & Wang, 2021) without incurring much vector-wise difference. In practice, we observe that the malicious gradients can get high similarity scores and circumvent vector-wise defenses.
Revised: Under a majority adversary setting

Original: Dimension-wise.
Revised: [Table]

Original: Failure mode 2 in Section 4.1 shows the limitation of the trimmed-mean estimator, which was the most effective defense against the edge-case backdoor attack. We also include SignSGD with majority vote (Bernstein et al., 2019) as a defense, which binarizes the gradient and takes the majority vote as the aggregation result. However, Sign-SGD struggles to train a large federated model (e.g., Resnet-18 on CIFAR-10) and can suffer from failure mode 1 where the clients have diverse signs.
Revised: our method outperforms all the baselines by a large margin.

Original: Then, the adversary can put more weight on one side and mislead the voting result. Combination.
Revised: The reason is that the median-based (CM) method picks the malicious gradient once the adversary becomes the majority.

Original: A naive combination of multi-Krum and the trimmed-mean estimator fails to defend against the backdoor attack because neither multi-Krum nor the trimmed-mean estimator avoids the failure mode of the other. Weak-DP. The weak-DP defense (Sun et al., 2019) first bounds the gradient norms, then add additive noise (e.g., Gaussian noise) to the gradient vector.
Revised: Clustering-based (Krum and Krum-B) methods suffers from a similar issue. Other reference client-assisted distance-based re-weighting strategy (CClip-R) and filtering strategy (Zeno ′ ) is not effective because Mimic-Shift carefully calibrates the malicious gradients so that they are as close to the reference gradients as the benign gradients.

Original: The edge-case backdoor attack can work without scaling up the gradients, circumventing the norm bounding.
Revised: Using only reference clients (Ref) outperforms existing robust aggregators but suffers from low client utilization.

Original: For the additive noise, we hypothesize that  in some dimensions, thedifference between malicious and benign gradients can be too large for the Gaussian noise to blur their boundary. Freezing Layers.
Revised: We also find that our proposed defense yields the best accuracy under a minority adversary setting, as discussed in Section 5.2.1. Under a no adversary setting, our strategy weights the benign gradients nearly uniformly without interfering with the training.

Original: Since we employ a pre-trained Resnet-18 (He et al., 2016) on CIFAR-10, freezing the convolution layers may avoid entangling the trigger. However, this approach lacks empirical robustness, possibly because the adversary can use semantic features (e.g., blue color on airplanes) that the pre-trained model already learns as triggers.
Revised: Our method occationally outperforms the oracle. We hypothesize that the improved results can come from the up-weighting of the underrepresented clients (Li et al., 2020), whose influence score is small. [Table]

Original:  C ONCLUSION AND F UTURE W ORK
Revised:  C ONCLUSION AND F UTURE W ORK

Original: This paper shows how to defend against backdoor attacks by focusing on the invariant directions in the model optimization trajectory.
Revised: This paper shows two methods for improving the adversarial robustness of federated learning under a majority adversary regime.

Original: Enforcing the model to follow the invariant direction requires AND-mask to compute the sign-consistency of each gradient dimension, which estimates how invariant a dimension-wise direction can be, and use the trimmed-mean estimator to guarantee the model follows the invariant direction within each dimension. Both theoretical and empirical results demonstrate the combination of AND-mask and the trimmed-mean estimator is necessary and effective.
Revised: Empirical results in various settings and against a broad class of attacks demonstrate the proposed methods’ effectiveness. Additional theoretical analysis is conducted under the Mimic-Shift attack regime, showing conditions under which the proposed method helps.

Original: Defending against more advanced backdoor attacks such as invisible backdoors that add calibrated noise to all the features (Li et al., 2021b; Manoj & Blum, 2021) can be interesting for future work.
Revised: Further exploring the limitations of learning with majority adversaries is a good next step.