Original: I NVARIANT A GGREGATOR FOR D EFENDING AGAINST F EDERATED B ACKDOOR A TTACKS
Revised: I NVARIANT A GGREGATOR FOR D EFENDING AGAINST F EDERATED B ACKDOOR A TTACKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Federated learning is gaining popularity as it enables training of high-utility models across several clients without directly sharing their private data.
Revised: Federated learning is gaining popularity as it enables training of high-utility models across several clients without directly sharing their private data.

Original: As a downside, the federated setting makes the model vulnerable to various adversarial attacks in the presence of malicious clients.
Revised: As a downside, the federated setting makes the model vulnerable to various adversarial attacks in the presence of malicious clients.

Original: Specifically, an adversary can perform backdoor attacks to control model predictions via poisoning the training dataset with a trigger.
Revised: Specifically, an adversary can perform backdoor attacks to control model predictions via poisoning the training dataset with a trigger.

Original: In this work, we propose a mitigation for backdoor attacks in a federated learning setup.
Revised: In this work, we propose a mitigation for backdoor attacks in a federated learning setup.

Original: Our solution forces the model optimization trajectory to focus on the invariant directions that are generally useful for utility and avoid selecting directions that favor few and possibly malicious clients.
Revised: Our solution forces the model optimization trajectory to focus on the invariant directions that are generally useful for utility and avoid selecting directions that favor few and possibly malicious clients.

Original: Concretely, we consider the sign consistency of the pseudo-gradient (the client update) as an estimation of the invariance.
Revised: Concretely, we consider the sign consistency of the pseudo-gradient (the client update) as an estimation of the invariance.

Original: Following this, our approach performs dimension-wise filtering to remove pseudo-gradient elements with low sign consistency.
Revised: Following this, our approach performs dimension-wise filtering to remove pseudo-gradient elements with low sign consistency.

Original: Then, a robust mean estimator eliminates outliers among the remaining dimensions.
Revised: Then, a robust mean estimator eliminates outliers among the remaining dimensions.

Original: Our theoretical analysis further shows the necessity of the defense combination and illustrates how our proposed solution defends the federated learning model.
Revised: Our theoretical analysis further shows the necessity of the defense combination and illustrates how our proposed solution defends the federated learning model.

Original: Empirical results on three datasets with different modalities and varying number of clients show that our approach mitigates backdoor attacks with a negligible cost on the model utility.
Revised: Empirical results on three datasets with different modalities and varying number of clients show that our approach mitigates backdoor attacks with a negligible cost on the model utility.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Federated learning enables multiple distrusting clients to jointly train a machine learning model without sharing their private data directly.
Revised: Federated learning enables multiple distrusting clients to jointly train a machine learning model without sharing their private data directly.

Original: However, a rising concern in this setting is the ability of potentially malicious clients to perpetrate backdoor attacks.
Revised: However, a rising concern in this setting is the ability of potentially malicious clients to perpetrate backdoor attacks.

Original: To this end, it has been argued that conducting backdoor attacks in a federated learning setup is practical (Shejwalkar et al., 2022) and can be effective (Wang et al., 2020).
Revised: To this end, it has been argued that conducting backdoor attacks in a federated learning setup is practical (Shejwalkar et al., 2022) and can be effective (Wang et al., 2020).

Original: For instance, the adversary can connect to a federated learning system as a legitimate user and conduct a backdoor attack that forces the model to mispredict.
Revised: For instance, the adversary can connect to a federated learning system as a legitimate user and conduct a backdoor attack that forces the model to mispredict.

Original: The impact of such attacks is quite severe in many mission-critical federated learning applications.
Revised: The impact of such attacks is quite severe in many mission-critical federated learning applications.

Original: For example, anomaly detection is a common federated learning task where multiple parties (e.g., banks or email users) collaboratively train a model that detects frauds or phishing emails.
Revised: For example, anomaly detection is a common federated learning task where multiple parties (e.g., banks or email users) collaboratively train a model that detects frauds or phishing emails.

Original: Backdoor attacks allow the adversary to successfully circumvent these detection methods.
Revised: Backdoor attacks allow the adversary to successfully circumvent these detection methods.

Original: The most common backdoor attack embeds triggers in the data samples and forces the model to make an adversary-specified prediction when the trigger is observed (Liu et al., 2018; Bagdasaryan et al., 2020).
Revised: The most common backdoor attack embeds triggers in the data samples and forces the model to make an adversary-specified prediction when the trigger is observed (Liu et al., 2018; Bagdasaryan et al., 2020).

Original: Thus, an adversary can conduct a backdoor attack by generating a trigger that statistically correlates with a particular label.
Revised: Thus, an adversary can conduct a backdoor attack by generating a trigger that statistically correlates with a particular label.

Original: Once the adversary injects these trigger-embedded backdoor data samples into the training data, the model can entangle the trigger-label correlation and predict as the adversary specifies.
Revised: Once the adversary injects these trigger-embedded backdoor data samples into the training data, the model can entangle the trigger-label correlation and predict as the adversary specifies.

Original: Meanwhile, the backdoor attack often does not degrade the predictive accuracy on the benign samples, making backdoor detection difficult in practice (Wang et al., 2020).
Revised: Meanwhile, the backdoor attack often does not degrade the predictive accuracy on the benign samples, making backdoor detection difficult in practice (Wang et al., 2020).

Original: In federated learning, the server aggregates only the client-level updates (a.k.a.
Revised: In federated learning, the server aggregates only the client-level updates (a.k.a.

Original: pseudo-gradient or gradient for short) without control over the training procedure or any data samples.
Revised: pseudo-gradient or gradient for short) without control over the training procedure or any data samples.

Original: Such limited visibility of the federated learning server on the client-side training makes defending against backdoor attacks challenging.
Revised: Such limited visibility of the federated learning server on the client-side training makes defending against backdoor attacks challenging.

Original: Common defenses against backdoor attacks aim at identifying the backdoor data samples or poisoned model parameters and usually require access to at least a subset of the training data (Tran et al., 2018;
Revised: Common defenses against backdoor attacks aim at identifying the backdoor data samples or poisoned model parameters and usually require access to at least a subset of the training data (Tran et al., 2018;

Original: Li et al., 2021a), which is prohibitive for a federated learning server.
Revised: Li et al., 2021a), which is prohibitive for a federated learning server.

Original: Other defense methods against untargeted poisoning attacks that degrade the model utility
Revised: Other defense methods against untargeted poisoning attacks that degrade the model utility

Original: (Shejwalkar et al., 2022) are applicable but lack robustness against backdoor attacks, as discussed in Section 6.2.
Revised: (Shejwalkar et al., 2022) are applicable but lack robustness against backdoor attacks, as discussed in Section 6.2.

Original: Our approach.
Revised: Our approach.

Original: Our defense leverages the observation that learning from the poisonous data does not benefit the model on benign data and vice versa.
Revised: Our defense leverages the observation that learning from the poisonous data does not benefit the model on benign data and vice versa.

Original: Therefore, focusing on the invariant directions that are generally beneficial in the model optimization trajectory helps defending against the aforementioned backdoor attack (which often lead to non-invariant directions).
Revised: Therefore, focusing on the invariant directions that are generally beneficial in the model optimization trajectory helps defending against the aforementioned backdoor attack (which often lead to non-invariant directions).

Original: To this end, we develop a defense by examining each dimension of the gradients on the server-side and checking whether the dimension-wise gradients point in the same direction across the clients.
Revised: To this end, we develop a defense by examining each dimension of the gradients on the server-side and checking whether the dimension-wise gradients point in the same direction across the clients.

Original: Here, a dimension-wise gradient can point to a positive or negative direction, or have a zero value.
Revised: Here, a dimension-wise gradient can point to a positive or negative direction, or have a zero value.

Original: In the case of small learning rates and for a specific dimension, two gradients pointing in the same direction means that taking the direction of one gradient can benefit the other.
Revised: In the case of small learning rates and for a specific dimension, two gradients pointing in the same direction means that taking the direction of one gradient can benefit the other.

Original: As such, the invariance of a direction depends on how many dimension-wise gradients align with that direction.
Revised: As such, the invariance of a direction depends on how many dimension-wise gradients align with that direction.

Original: Following this intuition, we define the sign consistency of a dimension by the average gradient sign.
Revised: Following this intuition, we define the sign consistency of a dimension by the average gradient sign.

Original: The higher the sign consistency is, the more invariant direction the gradient dimension may have.
Revised: The higher the sign consistency is, the more invariant direction the gradient dimension may have.

Original: Designing such a method carefully selecting only the invariant gradient directions is non-trivial, especially given the non-i.i.d.
Revised: Designing such a method carefully selecting only the invariant gradient directions is non-trivial, especially given the non-i.i.d.

Original: gradient distributions across benign clients and the presence of malicious clients.
Revised: gradient distributions across benign clients and the presence of malicious clients.

Original: Hence, our approach enforces two separate treatments for each gradient dimension.
Revised: Hence, our approach enforces two separate treatments for each gradient dimension.

Original: First, we employ an AND-mask (Parascandolo et al., 2021), a dimension-wise filter setting the gradient dimension with sign consistency below a given threshold to zero.
Revised: First, we employ an AND-mask (Parascandolo et al., 2021), a dimension-wise filter setting the gradient dimension with sign consistency below a given threshold to zero.

Original: However, this alone is not enough: the malicious clients can still use outliers to mislead the aggregation result in the remaining highly consistent dimensions.
Revised: However, this alone is not enough: the malicious clients can still use outliers to mislead the aggregation result in the remaining highly consistent dimensions.

Original: To address this issue, we propose using the trimmed-mean estimator (Xie et al., 2020b;
Revised: To address this issue, we propose using the trimmed-mean estimator (Xie et al., 2020b;

Original: Lugosi & Mendelson, 2021), as a means to remove the outliers.
Revised: Lugosi & Mendelson, 2021), as a means to remove the outliers.

Original: Our analysis suggests that the AND-mask complements the trimmed-mean estimator well, motivating their composition.
Revised: Our analysis suggests that the AND-mask complements the trimmed-mean estimator well, motivating their composition.

Original: We support the proposed approach with a theoretical analysis under a conventional linear regime (Rosenfeld et al., 2021; Wang et al., 2022; Zhou et al., 2022; Manoj & Blum, 2021), showing that the composition of the AND-mask and the trimmed-mean estimator is necessary for defending against backdoor attacks.
Revised: We support the proposed approach with a theoretical analysis under a conventional linear regime (Rosenfeld et al., 2021; Wang et al., 2022; Zhou et al., 2022; Manoj & Blum, 2021), showing that the composition of the AND-mask and the trimmed-mean estimator is necessary for defending against backdoor attacks.

Original: Our analysis starts with feature invariance and discusses the connection between feature invariance and gradient sign consistency.
Revised: Our analysis starts with feature invariance and discusses the connection between feature invariance and gradient sign consistency.

Original: Then, we outline conditions under which triggerbased backdoor attacks can lead to non-invariant directions and decrease the sign consistency of a dimension.
Revised: Then, we outline conditions under which triggerbased backdoor attacks can lead to non-invariant directions and decrease the sign consistency of a dimension.

Original: Further analysis results demonstrate the necessity for the combination of both the
Revised: Further analysis results demonstrate the necessity for the combination of both the

Original: AND-mask and the trimmed-mean estimator.
Revised: AND-mask and the trimmed-mean estimator.

Original: Simulation results in Appendix D.1 further verify our theoretical results.
Revised: Simulation results in Appendix D.1 further verify our theoretical results.

Original: Our empirical evaluation employs the strong edge-case backdoor attack (Wang et al., 2020), as detailed in Section 6.1, to test our defense.
Revised: Our empirical evaluation employs the strong edge-case backdoor attack (Wang et al., 2020), as detailed in Section 6.1, to test our defense.

Original: Empirical results on tabular (phishing emails), visual (CIFAR-10) (Krizhevsky, 2009; McMahan et al., 2017), and text (Twitter) (Caldas et al., 2018)
Revised: Empirical results on tabular (phishing emails), visual (CIFAR-10) (Krizhevsky, 2009; McMahan et al., 2017), and text (Twitter) (Caldas et al., 2018)

Original: datasets demonstrate that our method is effective in defending against backdoor attacks without degrading utility as compared to prior works.
Revised: datasets demonstrate that our method is effective in defending against backdoor attacks without degrading utility as compared to prior works.

Original: On average, our approach decreases the model accuracy on backdoor samples by 61.6% and only loses 1.2% accuracy on benign samples compared to the standard FedAvg aggregator (McMahan et al., 2017).
Revised: On average, our approach decreases the model accuracy on backdoor samples by 61.6% and only loses 1.2% accuracy on benign samples compared to the standard FedAvg aggregator (McMahan et al., 2017).

Original: Contributions.
Revised: Contributions.

Original: Our contributions are as follows:
Revised: Our contributions are as follows:

Original: • We develop a combination of defenses using an AND-mask and the trimmed-mean estimator against the backdoor attack by focusing on the dimension-wise invariant directions in the model optimization trajectory.
Revised: • We develop a combination of defenses using an AND-mask and the trimmed-mean estimator against the backdoor attack by focusing on the dimension-wise invariant directions in the model optimization trajectory.

Original: • We theoretically analyze our strategy and demonstrate that a combination of an AND-mask and the trimmed-mean estimator is necessary in some conditions.
Revised: • We theoretically analyze our strategy and demonstrate that a combination of an AND-mask and the trimmed-mean estimator is necessary in some conditions.

Original: • We empirically evaluate our method on three datasets with varying modality, model architecture, and client numbers, as well as comparing the performance to existing defenses.
Revised: • We empirically evaluate our method on three datasets with varying modality, model architecture, and client numbers, as well as comparing the performance to existing defenses.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Backdoor Attack.
Revised: Backdoor Attack.

Original: Common backdoor attacks aim at misleading the model predictions using a trigger (Liu et al., 2018).
Revised: Common backdoor attacks aim at misleading the model predictions using a trigger (Liu et al., 2018).

Original: The trigger can be digital (Bagdasaryan et al., 2020), physical (Wenger et al., 2021), semantic (Wang et al., 2020), or invisible (Li et al., 2021b).
Revised: The trigger can be digital (Bagdasaryan et al., 2020), physical (Wenger et al., 2021), semantic (Wang et al., 2020), or invisible (Li et al., 2021b).

Original: Recent works extended backdoor attacks to the federated learning setting and proposed effective improvements such as gradient scaling (Bagdasaryan et al., 2020) or generating edge-case backdoor samples (Wang et al., 2020).
Revised: Recent works extended backdoor attacks to the federated learning setting and proposed effective improvements such as gradient scaling (Bagdasaryan et al., 2020) or generating edge-case backdoor samples (Wang et al., 2020).

Original: The state-of-the-art edge-case backdoor attack shows that using backdoor samples with low probability density on benign clients (i.e., unlikely samples w.r.t.
Revised: The state-of-the-art edge-case backdoor attack shows that using backdoor samples with low probability density on benign clients (i.e., unlikely samples w.r.t.

Original: the training distribution) are hard to defend in the federated learning setting.
Revised: the training distribution) are hard to defend in the federated learning setting.

Original: Centralized Defense.
Revised: Centralized Defense.

Original: There is a line of work proposing centralized defenses against backdoor attacks where the main aim is either detecting the backdoor samples (Tran et al., 2018) or purifying the model parameters that are poisoned (Li et al., 2021a).
Revised: There is a line of work proposing centralized defenses against backdoor attacks where the main aim is either detecting the backdoor samples (Tran et al., 2018) or purifying the model parameters that are poisoned (Li et al., 2021a).

Original: However, applying such centralized defense to federated learning systems is in practice infeasible due to limited access to the client data in many implementations.
Revised: However, applying such centralized defense to federated learning systems is in practice infeasible due to limited access to the client data in many implementations.

Original: Federated Defenses.
Revised: Federated Defenses.

Original: Several recent works have attempted to defend against backdoor attacks in federated learning systems.
Revised: Several recent works have attempted to defend against backdoor attacks in federated learning systems.

Original: Sun et al.
Revised: Sun et al.

Original: (2019) shows that weak differential-private (weak-dp) federated averaging can mitigate the backdoor attack.
Revised: (2019) shows that weak differential-private (weak-dp) federated averaging can mitigate the backdoor attack.

Original: However, the weak-dp defense is circumvented by the improved edge-case federated backdoor attack (Wang et al., 2020).
Revised: However, the weak-dp defense is circumvented by the improved edge-case federated backdoor attack (Wang et al., 2020).

Original: Nguyen et al.
Revised: Nguyen et al.

Original: (2021) suggest that the vector-wise cosine similarity can help detect malicious clients performing backdoor attacks.
Revised: (2021) suggest that the vector-wise cosine similarity can help detect malicious clients performing backdoor attacks.

Original: The vector-wise cosine similarity is insufficient when the backdoor attacks can succeed with few poisoned parameters, incurring little vector-wise difference (Wu & Wang, 2021).
Revised: The vector-wise cosine similarity is insufficient when the backdoor attacks can succeed with few poisoned parameters, incurring little vector-wise difference (Wu & Wang, 2021).

Original: Other defenses against untargeted poisoning attacks (Blanchard et al., 2017; Xie et al., 2020b) lack robustness against the backdoor attack.
Revised: Other defenses against untargeted poisoning attacks (Blanchard et al., 2017; Xie et al., 2020b) lack robustness against the backdoor attack.

Original: Sign-SGD with majority vote (Bernstein et al., 2018; 2019) is similar to our approach, but it always takes the majority direction instead of focusing on the invariant directions.
Revised: Sign-SGD with majority vote (Bernstein et al., 2018; 2019) is similar to our approach, but it always takes the majority direction instead of focusing on the invariant directions.

Original: Section 6.2 discusses the limitation of previous defenses in more detail, along with the empirical evaluation.
Revised: Section 6.2 discusses the limitation of previous defenses in more detail, along with the empirical evaluation.

Original: Unlike existing works, our defense encourages the model to pursue invariant directions in the optimization procedure.
Revised: Unlike existing works, our defense encourages the model to pursue invariant directions in the optimization procedure.

Original: Certification.
Revised: Certification.

Original: Unlike the above discussed defenses, certification (Xie et al., 2021) aims at extinguishing backdoor samples within a neighborhood of a benign sample.
Revised: Unlike the above discussed defenses, certification (Xie et al., 2021) aims at extinguishing backdoor samples within a neighborhood of a benign sample.

Original: A direct comparison between certification and our defense is not meaningful due to the different evaluation metrics.
Revised: A direct comparison between certification and our defense is not meaningful due to the different evaluation metrics.

Original: Certification considers the certification rate of benign samples as the metric, while our defense aims at reducing the accuracy of the backdoor samples.
Revised: Certification considers the certification rate of benign samples as the metric, while our defense aims at reducing the accuracy of the backdoor samples.

Original: However, it would be interesting to investigate whether the proposed defense can ease the certification of a model.
Revised: However, it would be interesting to investigate whether the proposed defense can ease the certification of a model.

Original:  P ROBLEM S ETUP
Revised:  P ROBLEM S ETUP

Original: Notation.
Revised: Notation.

Original: We assume a synchronous federated learning system, where N clients collaboratively train an ML model f : X −→ Y with parameter w coordinated by a server.
Revised: We assume a synchronous federated learning system, where N clients collaboratively train an ML model f : X −→ Y with parameter w coordinated by a server.

Original: An input to the model are the data samples x ∈ X =
Revised: An input to the model are the data samples x ∈ X =

Original: R d with d features indexed by k and a label y .
Revised: R d with d features indexed by k and a label y .

Original: There are N ′ <
Revised: There are N ′ <

Original: Nadversarial clients aiming at corrupting the ML model during training (Shejwalkar et al., 2022).
Revised: Nadversarial clients aiming at corrupting the ML model during training (Shejwalkar et al., 2022).

Original: The i th , i ∈ [1 , ..., N ] , client has n i data samples, being benign for i ∈ [1 , ..., N − N ′ ] or being adversarial for i ∈
Revised: The i th , i ∈ [1 , ..., N ] , client has n i data samples, being benign for i ∈ [1 , ..., N − N ′ ] or being adversarial for i ∈

Original: [ N − N ′ + 1 , ..., N ] .
Revised: [ N − N ′ + 1 , ..., N ] .

Original: The synchronous federated learning is conducted in T rounds.
Revised: The synchronous federated learning is conducted in T rounds.

Original: In each round t ∈ [1 , ..., T ] , the server broadcasts a model parameterized by w t − 1 to all the participating clients.
Revised: In each round t ∈ [1 , ..., T ] , the server broadcasts a model parameterized by w t − 1 to all the participating clients.

Original: We omit the subscript t while focusing on a single round.
Revised: We omit the subscript t while focusing on a single round.

Original: Then, the i th client optimizes w t − 1 on their local data samples indexed by j and report the locally optimized w t,i to the server.
Revised: Then, the i th client optimizes w t − 1 on their local data samples indexed by j and report the locally optimized w t,i to the server.

Original: We define pseudo-gradient g t,i = w t − 1 − w t,i being the difference between the locally optimized model and the broadcasted model from the previous round.
Revised: We define pseudo-gradient g t,i = w t − 1 − w t,i being the difference between the locally optimized model and the broadcasted model from the previous round.

Original: Note, for simplicity, that we often use the term “gradient” to refer to the pseudo-gradient.
Revised: Note, for simplicity, that we often use the term “gradient” to refer to the pseudo-gradient.

Original: Once all gradients are uploaded, the server aggregates them and produces a new model with parameters w t using the following rule:
Revised: Once all gradients are uploaded, the server aggregates them and produces a new model with parameters w t using the following rule:

Original: [Equation]
Revised: [Equation]

Original: n
Revised: n

Original: [Equation]
Revised: [Equation]

Original: g t,i .
Revised: g t,i .

Original: The goal of federated learning is to minimize a weighted risk function over the N clients: F ( w )
Revised: The goal of federated learning is to minimize a weighted risk function over the N clients: F ( w )

Original: =
Revised: =

Original: (cid:80) Ni =
Revised: (cid:80) Ni =

Original: [Equation]
Revised: [Equation]

Original: where ℓ : R × Y −→ R is a loss function.
Revised: where ℓ : R × Y −→ R is a loss function.

Original: ⊙ denotes the Hadamard product operator.
Revised: ⊙ denotes the Hadamard product operator.

Original: Threat Model.
Revised: Threat Model.

Original: The adversary generates a backdoor data sample x ′ by embedding a trigger in a benign data sample x and correlating the trigger with a label y ′ , which is different from the label y of the benign data sample.
Revised: The adversary generates a backdoor data sample x ′ by embedding a trigger in a benign data sample x and correlating the trigger with a label y ′ , which is different from the label y of the benign data sample.

Original: We use D ′ to denote the distribution of backdoor data samples.
Revised: We use D ′ to denote the distribution of backdoor data samples.

Original: Then, the malicious clients connect to the federated learning system and insert backdoor data samples into the training set.
Revised: Then, the malicious clients connect to the federated learning system and insert backdoor data samples into the training set.

Original: Since the goal of federated learning is to minimize the risk over all clients’ datasets, the model can entangle the backdoor while trying to minimize the risk over backdoor samples on the malicious clients.
Revised: Since the goal of federated learning is to minimize the risk over all clients’ datasets, the model can entangle the backdoor while trying to minimize the risk over backdoor samples on the malicious clients.

Original: Appendix A visualizes some backdoor samples.
Revised: Appendix A visualizes some backdoor samples.

Original: [Figure]
Revised: [Figure]

Original: To simplify tedious notation, we assume all users have the same number of samples i.e., n i = n
Revised: To simplify tedious notation, we assume all users have the same number of samples i.e., n i = n

Original: i ′ , ∀ i ̸
Revised: i ′ , ∀ i ̸

Original: = i ′ .
Revised: = i ′ .

Original:  M ETHOD
Revised:  M ETHOD

Original: This section presents the proposed server-side defense operating on the gradients.
Revised: This section presents the proposed server-side defense operating on the gradients.

Original: We start with an overview of the idea, then introduce the two complementary components of our defense and outline the invariant aggregator steps.
Revised: We start with an overview of the idea, then introduce the two complementary components of our defense and outline the invariant aggregator steps.

Original: 4. O VERVIEW
Revised: 4. O VERVIEW

Original: In the proposed defense, we aim at finding invariant directions to optimize the federated learning model, such that the model is generally utilitarian for most of clients and can exclude the directions that benefit a subset of potentially malicious clients.
Revised: In the proposed defense, we aim at finding invariant directions to optimize the federated learning model, such that the model is generally utilitarian for most of clients and can exclude the directions that benefit a subset of potentially malicious clients.

Original: Since our defense operates with the gradients, we consider the invariant direction from a first-order perspective (Parascandolo et al., 2021).
Revised: Since our defense operates with the gradients, we consider the invariant direction from a first-order perspective (Parascandolo et al., 2021).

Original: Expanding the loss function around the current weight w on client i , for a parameter update g , we have:
Revised: Expanding the loss function around the current weight w on client i , for a parameter update g , we have:

Original: [Equation]
Revised: [Equation]

Original: where R 2 ( w + g ) is a second-order Taylor remainder.
Revised: where R 2 ( w + g ) is a second-order Taylor remainder.

Original: With a reasonably small ∥ g ∥ (achievable with small learning rates), the remainder term R 2 ( w + g ) is negligible and the change of the loss function mainly depends on the first-order gradient and the parameter update g .
Revised: With a reasonably small ∥ g ∥ (achievable with small learning rates), the remainder term R 2 ( w + g ) is negligible and the change of the loss function mainly depends on the first-order gradient and the parameter update g .

Original: Since learning the trigger benefits the malicious clients exclusively, there exists at least one gradient dimension k and one benign client i ∈ { 1 , ..., N − N ′ } where E D ′ [ ∇ w k ℓ ( x , y ; w )]
Revised: Since learning the trigger benefits the malicious clients exclusively, there exists at least one gradient dimension k and one benign client i ∈ { 1 , ..., N − N ′ } where E D ′ [ ∇ w k ℓ ( x , y ; w )]

Original: ̸ = 0 and the dimension-wise gradient have inconsistent signs, i.e.,
Revised: ̸ = 0 and the dimension-wise gradient have inconsistent signs, i.e.,

Original: [Equation]
Revised: [Equation]

Original: If the condition in Equation 2 is not true, then learning the trigger would always benefit the model on benign data, thereby contradicting common empirical observation.
Revised: If the condition in Equation 2 is not true, then learning the trigger would always benefit the model on benign data, thereby contradicting common empirical observation.

Original: A more detailed analysis of the condition in Equation 2 is in Section 5.2.
Revised: A more detailed analysis of the condition in Equation 2 is in Section 5.2.

Original: The proposed methods show how to treat the inconsistent signs and defend against the backdoor attack by only allowing invariant directions.
Revised: The proposed methods show how to treat the inconsistent signs and defend against the backdoor attack by only allowing invariant directions.

Original: A dimensionwise analysis is necessary because backdoor attacks can succeed by poisoning few inconsistent dimensions (Wu & Wang, 2021) without incurring much vector-wise difference, as Section 6.2 will show.
Revised: A dimensionwise analysis is necessary because backdoor attacks can succeed by poisoning few inconsistent dimensions (Wu & Wang, 2021) without incurring much vector-wise difference, as Section 6.2 will show.

Original: We consider two treatments for each gradient dimension to help the model avoid the direction specified by E D ′ [ ∇ w ℓ ( x ′ , y ′ ; w )] : (Treatment 1) setting the dimension with inconsistent signs to zero using an AND-mask (Parascandolo et al., 2021) such that no client benefits or (Treatment 2) employing a robust mean estimator (e.g., trimmed-mean (Xie et al., 2020b)) to remove the malicious values that cause the inconsistent sign.
Revised: We consider two treatments for each gradient dimension to help the model avoid the direction specified by E D ′ [ ∇ w ℓ ( x ′ , y ′ ; w )] : (Treatment 1) setting the dimension with inconsistent signs to zero using an AND-mask (Parascandolo et al., 2021) such that no client benefits or (Treatment 2) employing a robust mean estimator (e.g., trimmed-mean (Xie et al., 2020b)) to remove the malicious values that cause the inconsistent sign.

Original: To achieve a better result, we combine these two treatments to avoid their failure modes.
Revised: To achieve a better result, we combine these two treatments to avoid their failure modes.

Original: The following examples illustrate the failure mode of each treatment and motivate the combination of defense.
Revised: The following examples illustrate the failure mode of each treatment and motivate the combination of defense.

Original: Failure Mode 1.
Revised: Failure Mode 1.

Original: Figure 1a shows an example where the adversary may exploit a dimension by inserting outliers where the benign values have a consistent sign.
Revised: Figure 1a shows an example where the adversary may exploit a dimension by inserting outliers where the benign values have a consistent sign.

Original: The outliers can mislead the average toward the non-invariant direction.
Revised: The outliers can mislead the average toward the non-invariant direction.

Original: The robust mean estimator (treatment 2) can trim the outliers and accurately estimate the mean.
Revised: The robust mean estimator (treatment 2) can trim the outliers and accurately estimate the mean.

Original: In contrast, treatment 1 can fail with a high sign consistency.
Revised: In contrast, treatment 1 can fail with a high sign consistency.

Original: Because it either lets the highly consistent dimension pass or has to be over-aggressive in zeroing out the dimensions, hurting the model’s accuracy on benign data.
Revised: Because it either lets the highly consistent dimension pass or has to be over-aggressive in zeroing out the dimensions, hurting the model’s accuracy on benign data.

Original: Failure Mode 2.
Revised: Failure Mode 2.

Original: Figure 1b shows some values with inconsistent signs, which treatment 1 can handle.
Revised: Figure 1b shows some values with inconsistent signs, which treatment 1 can handle.

Original: However, this example can fail the robust mean estimator (treatment 2), whose result has the same sign as the malicious values.
Revised: However, this example can fail the robust mean estimator (treatment 2), whose result has the same sign as the malicious values.

Original: The following sections shall detail the two treatments and discuss their complementary relationship.
Revised: The following sections shall detail the two treatments and discuss their complementary relationship.

Original: 4. AND-M ASK
Revised: 4. AND-M ASK

Original: The AND-mask (Parascandolo et al., 2021) computes a dimension-wise mask by inspecting the sign consistency of each dimension across clients.
Revised: The AND-mask (Parascandolo et al., 2021) computes a dimension-wise mask by inspecting the sign consistency of each dimension across clients.

Original: For dimension k , the sign consistency is: | 1 N (cid:80) Ni =1 sign( g i,k ) | .
Revised: For dimension k , the sign consistency is: | 1 N [Equation] Ni =1 sign( g i,k ) | .

Original: If the sign consistency is below a given threshold τ , the mask element m k is set to 0, otherwise, m k is set to 1.
Revised: If the sign consistency is below a given threshold τ , the mask element m k is set to 0, otherwise, m k is set to 1.

Original: The mask along dimension k is defined as:
Revised: The mask along dimension k is defined as:

Original: Definition 1.
Revised: Definition 1.

Original: (AND-Mask)
Revised: (AND-Mask)

Original: For the k th dimension in the gradient vector, the corresponding mask m k is defined as:
Revised: For the k th dimension in the gradient vector, the corresponding mask m k is defined as:

Original: [Equation]
Revised: [Equation]

Original: Our defense then multiplies the mask m with the aggregated gradient ¯ g element-wise, setting the inconsistent dimension to zero.
Revised: Our defense then multiplies the mask m with the aggregated gradient ¯ g element-wise, setting the inconsistent dimension to zero.

Original: 4. T RIMMED -M EAN
Revised: 4. T RIMMED -M EAN

Original: To complement the AND-mask, our defense broadcasts the trimmed-mean estimator to each gradient dimension.
Revised: To complement the AND-mask, our defense broadcasts the trimmed-mean estimator to each gradient dimension.

Original: The trimmed-mean estimator alleviates the outlier issue by removing the subset of largest and smallest elements before computing the mean.
Revised: The trimmed-mean estimator alleviates the outlier issue by removing the subset of largest and smallest elements before computing the mean.

Original: The largest and smallest elements appear on the two tails of a sorted sequence.
Revised: The largest and smallest elements appear on the two tails of a sorted sequence.

Original: Next, we define order statistics and the trimmed mean estimator.
Revised: Next, we define order statistics and the trimmed mean estimator.

Original: Definition 2.
Revised: Definition 2.

Original: (Order Statistics) (Xie et al., 2020b)
Revised: (Order Statistics) (Xie et al., 2020b)

Original: By sorting the scalar sequence { x i : i ∈ { 1 , ..., N } , x i ∈ R } , we get x 1: N ≤ x 2: N ≤ ... ≤ x N : N , where x i : N is the i th smallest element in { x i : i ∈ { 1 , ..., N }} .
Revised: By sorting the scalar sequence { x i : i ∈ { 1 , ..., N } , x i ∈ R } , we get x 1: N ≤ x 2: N ≤ ... ≤ x N : N , where x i : N is the i th smallest element in { x i : i ∈ { 1 , ..., N }} .

Original: Then, the trimmed-mean estimator removes a α × N elements from each tail of the sorted sequence.
Revised: Then, the trimmed-mean estimator removes a α × N elements from each tail of the sorted sequence.

Original: Definition 3.
Revised: Definition 3.

Original: (Trimmed Mean Estimator) (Xie et al., 2020b)
Revised: (Trimmed Mean Estimator) (Xie et al., 2020b)

Original: For α ∈ [0 , 1] , the α -trimmed mean of the set of scalars x i : N ∈ { 1 , ..., N } is defined as follows:
Revised: For α ∈ [0 , 1] , the α -trimmed mean of the set of scalars x i : N ∈ { 1 , ..., N } is defined as follows:

Original: [Equation]
Revised: [Equation]

Original: where ⌈ .
Revised: where ⌈ .

Original: ⌉ denotes the ceiling function.
Revised: ⌉ denotes the ceiling function.

Original: 4. O UR A PPROACH : I NVARIANT A GGREGATOR
Revised: 4. O UR A PPROACH : I NVARIANT A GGREGATOR

Original: Algorithm 1 outlines the steps of our server-side defense that perform aggregation of invariant updates from the clients.
Revised: Algorithm 1 outlines the steps of our server-side defense that perform aggregation of invariant updates from the clients.

Original: The solution is composed of the AND-mask (treatment 1) and trimmed-mean estimator (treatment 2).
Revised: The solution is composed of the AND-mask (treatment 1) and trimmed-mean estimator (treatment 2).

Original: Our defense applies the two components separately based on the sign consistency of each dimension with a threshold τ .
Revised: Our defense applies the two components separately based on the sign consistency of each dimension with a threshold τ .

Original: We show how these two components, i.e., the AND-mask and the trimmed-mean estimator, complement each other.
Revised: We show how these two components, i.e., the AND-mask and the trimmed-mean estimator, complement each other.

Original: Our analysis considers a single dimension and starts with the robustness of the trimmed mean estimator, which improves the robustness of the AND-mask against outliers.
Revised: Our analysis considers a single dimension and starts with the robustness of the trimmed mean estimator, which improves the robustness of the AND-mask against outliers.

Original: The following theorem extends the robustness guarantee of a modified trimmed-mean estimator (Lugosi & Mendelson, 2021), which is shown in Appendix B, to the conventional trimmed-mean estimator (Definition 3).
Revised: The following theorem extends the robustness guarantee of a modified trimmed-mean estimator (Lugosi & Mendelson, 2021), which is shown in Appendix B, to the conventional trimmed-mean estimator (Definition 3).

Original: Theorem 4.
Revised: Theorem 4.

Original: With the trimmed-mean estimator in Definition 3, for a given set of samples x 1 , ..., x N , with a corruption level η = N ′ N and a confidence level δ , set the trim level α = 8 η + 12 log( 4 δ ) N , let a = x αN : N and b = x N − αN : N following Definition 2, x be a random variable with variance σ and ¯ x be the estimated mean, with probability at least 0 .
Revised: With the trimmed-mean estimator in Definition 3, for a given set of samples x 1 , ..., x N , with a corruption level η = N ′ N and a confidence level δ , set the trim level α = 8 η +12 log( 4 δ ) N , let a = x αN : N and b = x N − αN : N following Definition 2, x be a random variable with variance σ and ¯ x be the estimated mean, with probability at least (cid:80) N − N ′ i

Original: 99 N −2 αN c − 4 (1 − 4 e
Revised: = N − αN (cid:0) N − N ′ i (cid:1)

Original: [Equation]
Revised: [Equation]

Original: 12 ) , we have:
Revised: we have:

Original: [Equation]
Revised: [Equation]

Original: The proof is in Appendix B. Theorem 4 bounds the estimation error of a trimmed-mean estimator, which can increase as the variable’s variance increases.
Revised: The proof is in Appendix B. Theorem 4 bounds the estimation error of a trimmed-mean estimator, which can increase as the variable’s variance increases.

Original: Multiple factors can increase the variance, such as non-i.i.d.
Revised: Multiple factors can increase the variance, such as non-i.i.d.

Original: federated data distribution and the stochastic gradient estimation process.
Revised: federated data distribution and the stochastic gradient estimation process.

Original: In practice, a threat analysis is necessary to specify the maximum number of malicious clients to be tolerated, when the number of malicious clients is unknown.
Revised: In practice, a threat analysis is necessary to specify the maximum number of malicious clients to be tolerated, when the number of malicious clients is unknown.

Original: Our goal is to prevent the outliers from misleading the sign.
Revised: Our goal is to prevent the outliers from misleading the sign.

Original: Therefore, the estimation error to expectation ratio, | ¯ x − E
Revised: Therefore, the estimation error to expectation ratio, | ¯ x − E

Original: [ x ] | E [ x ] , is particularly relevant.
Revised: [ x ] | E [ x ] , is particularly relevant.

Original: Since the estimator error for a given α depends on the variance, a high expectationvariance ratio, E
Revised: Since the estimator error for a given α depends on the variance, a high expectationvariance ratio, E

Original: [ x ] σ , is desirable.
Revised: [ x ] σ , is desirable.

Original: Then, we show that AND-mask identifies the elements with high expectation-variance ratios, avoiding the robustness degradation of the trimmed-mean estimator.
Revised: Then, we show that AND-mask identifies the elements with high expectation-variance ratios, avoiding the robustness degradation of the trimmed-mean estimator.

Original: The following theorem suggests that dimensions with a higher expectation-variance ratio have high probabilities of passing AND-mask, and increasing the mask threshold τ increases the chance of filtering out dimensions with low expectation-variance ratios.
Revised: The following theorem suggests that dimensions with a higher expectation-variance ratio have high probabilities of passing AND-mask, and increasing the mask threshold τ increases the chance of filtering out dimensions with low expectation-variance ratios.

Original: Theorem 5.
Revised: Theorem 5.

Original: Given a non-zero expectation-variance ratio ϕ
Revised: Given a non-zero expectation-variance ratio ϕ

Original: =
Revised: =

Original: E
Revised: E

Original: [ x ] σ , N ′ malicious elements, with probability at most (cid:80) min( N,N − 2 N ′ + τ )i
Revised: [ x ] σ , N ′ malicious elements, with probability at most (cid:80) min( N,N − 2 N ′ + τN )

Original:  = N − 2 N ′ − τ (cid:0) N − N 
Revised: i = N − 2 N ′ − τN (cid:0) N − N ′

Original: i (cid:1) ϕ − 2 i , the sign consistency is below τ .
Revised: i (cid:1) ϕ − 2 i , the sign consistency is below τ .

Original: Appendix B provides the proof.
Revised: Appendix B provides the proof.

Original: For the ϕ = 0 case, we may use the probability of the estimated gradient sign being the same as the malicious gradient to replace ϕ − 2 .
Revised: For the ϕ = 0 case, we may use the probability of the estimated gradient sign being the same as the malicious gradient to replace ϕ − 2 .

Original: We do not propose directly using the sample mean-variance ratio due to a potential issue: using the sample mean-variance ratio can be over-aggressive when the benign value has a consistent sign but varying magnitudes.
Revised: We do not propose directly using the sample mean-variance ratio due to a potential issue: using the sample mean-variance ratio can be over-aggressive when the benign value has a consistent sign but varying magnitudes.

Original: Our ablation study in Appendix D.2 shows that AND-mask can preserve more utility than the sample mean-variance ratio when combined with the trimmed-mean estimator.
Revised: Our ablation study in Appendix D.2 shows that AND-mask can preserve more utility than the sample mean-variance ratio when combined with the trimmed-mean estimator.

Original:  R OBUSTNESS ANALYSIS FOR A SIMPLIFIED MODEL
Revised:  R OBUSTNESS ANALYSIS FOR A SIMPLIFIED MODEL

Original: To further motivate our approach, we consider a more direct robustness analysis for a specific generative model.
Revised: To further motivate our approach, we consider a more direct robustness analysis for a specific generative model.

Original: This analysis discusses how the features impact the gradient sign consistency, when the two failure modes in Section 4.1 appear, and why we need a combination of defenses guaranteed by Theorems 4 and 5.
Revised: This analysis discusses how the features impact the gradient sign consistency, when the two failure modes in Section 4.1 appear, and why we need a combination of defenses guaranteed by Theorems 4 and 5.

Original: First, we outline some preliminaries useful for this analysis.
Revised: First, we outline some preliminaries useful for this analysis.

Original: 5. P RELIMINARIES
Revised: 5. P RELIMINARIES

Original: We consider a binary prediction task y ∈ { 0 , 1 } with a linear model h ( x ) =
Revised: We consider a binary prediction task y ∈ { 0 , 1 } with a linear model h ( x ) =

Original: w ⊤ x and a decision rule
Revised: w ⊤ x and a decision rule

Original: ˆ y = 1 w ⊤ x ≥ 0 ( w ⊤ x ) .
Revised: ˆ y = 1 w ⊤ x ≥ 0 ( w ⊤ x ) .

Original: The training procedure uses a Sigmoid activation function s ( z ) = 1 1+ e − z and a logistic loss function ℓ ( z, y )
Revised: The training procedure uses a Sigmoid activation function s ( z ) = 1 1+ e − z and a logistic loss function ℓ ( z, y )

Original: =
Revised: =

Original: − y · log( z ) − (1 − y ) · log(1 − z ) , where z = h ( x ) .
Revised: − y · log( z ) − (1 − y ) · log(1 − z ) , where z = h ( x ) .

Original: Data Model.
Revised: Data Model.

Original: We assume that the samples per class are balanced on benign clients.
Revised: We assume that the samples per class are balanced on benign clients.

Original: The data samples come from a non-i.i.d.
Revised: The data samples come from a non-i.i.d.

Original: Gaussian distribution with a diagonal covariance matrix.
Revised: Gaussian distribution with a diagonal covariance matrix.

Original: On the i th client, for the k th feature, we have:
Revised: On the i th client, for the k th feature, we have:

Original: [Equation]
Revised: [Equation]

Original: Backdoor Attack.
Revised: Backdoor Attack.

Original: We consider a single feature backdoor attack where the trigger is the k th feature x k .
Revised: We consider a single feature backdoor attack where the trigger is the k th feature x k .

Original: A backdoor using a specific feature is common (Wang et al., 2020).
Revised: A backdoor using a specific feature is common (Wang et al., 2020).

Original: To backdoor images, the adversary often selects a pixel pattern or semantic pattern (e.g., blue color on airplanes).
Revised: To backdoor images, the adversary often selects a pixel pattern or semantic pattern (e.g., blue color on airplanes).

Original: For text data, the trigger could be a dedicated set of characters .
Revised: For text data, the trigger could be a dedicated set of characters .

Original: The malicious clients attack in collusion using the same backdoor samples, meaning that µ i,k = µ i ′ ,k ̸ = 0 , ∀ i, i ′ ∈ { N − N ′ + 1 , N } .
Revised: The malicious clients attack in collusion using the same backdoor samples, meaning that µ i,k = µ i ′ ,k ̸ = 0 , ∀ i, i ′ ∈ { N − N ′ + 1 , N } .

Original: To simplify the notation, we omit the subscript i for simplicity while focusing on a single client.
Revised: To simplify the notation, we omit the subscript i for simplicity while focusing on a single client.

Original: Since the trigger is not useful or does not appear on benign clients, we assume µ i,k = 0 , ∀ i ∈ { 1 , N − N ′ }
Revised: Since the trigger is not useful or does not appear on benign clients, we assume µ i,k = 0 , ∀ i ∈ { 1 , N − N ′ }

Original: Objective.
Revised: Objective.

Original: The backdoor attack is effective if the model entangles the trigger-label correlation.
Revised: The backdoor attack is effective if the model entangles the trigger-label correlation.

Original: For a non-zero µ k of the trigger x k , w k µ k > 0 is a necessary condition for a model to entangle the trigger-label correlation.
Revised: For a non-zero µ k of the trigger x k , w k µ k > 0 is a necessary condition for a model to entangle the trigger-label correlation.

Original: Therefore, in the case when µ k > 0 , avoiding w k from increasing and enforcing w k to decrease while w k > 0 can mitigate the backdoor attack.
Revised: Therefore, in the case when µ k > 0 , avoiding w k from increasing and enforcing w k to decrease while w k > 0 can mitigate the backdoor attack.

Original: 5. C ONNECTING F EATURES TO G RADIENTS
Revised: 5. C ONNECTING F EATURES TO G RADIENTS

Original: We define the invariance of a feature by measuring its feature-label correlation (e.g., positive or negative) consistency across clients.
Revised: We define the invariance of a feature by measuring its feature-label correlation (e.g., positive or negative) consistency across clients.

Original: Definition 6.
Revised: Definition 6.

Original: For a given feature k , its invariance p is defined as: p = (cid:12) (cid:12) (cid:12) 1 N · (cid:80) N i =1 sign ( µ i,k ) (cid:12) (cid:12)(cid:12) .
Revised: For a given feature k , its invariance p is defined as: p = (cid:12) (cid:12) (cid:12) 1 N · (cid:80) N i =1 sign ( µ i,k ) (cid:12) (cid:12)(cid:12) .

Original: Similarly, we define the sign consistency of a gradient dimension k :
Revised: Similarly, we define the sign consistency of a gradient dimension k :

Original: Definition 7.
Revised: Definition 7.

Original: With a linear model h ( x ) =
Revised: With a linear model h ( x ) =

Original: w ⊤ x , a Sigmoid activation function s , and N clients, a q -consistent gradient w.r.t.
Revised: w ⊤ x , a Sigmoid activation function s , and N clients, a q -consistent gradient w.r.t.

Original: w k satisfies: (cid:12)(cid:12)(cid:12) 1 N · (cid:80) Ni =1 sign (cid:16) E x ,y ∼D i (cid:2) ∇
Revised: w k satisfies: (cid:12)(cid:12)(cid:12) 1 N · (cid:80) Ni =1 sign (cid:16) E x ,y ∼D i (cid:2) ∇

Original: [Equation]
Revised: [Equation]

Original: Under Definitions 6 and 7, we discuss the connection between feature invariance and dimensionwise gradient sign consistency.
Revised: Under Definitions 6 and 7, we discuss the connection between feature invariance and dimensionwise gradient sign consistency.

Original: First, we need to analyze the behavior of the gradient sign per client.
Revised: First, we need to analyze the behavior of the gradient sign per client.

Original: Theorem 8.
Revised: Theorem 8.

Original: For a linear model with a Sigmoid activation function s and the logistic loss ℓ , under our Gaussian data model, on the k th feature with a non-zero µ k , if w k µ k ≤ 0 , we have sign (cid:16) E x ,y ∼D i (cid:2) ∇ w k ℓ ( s ( w ⊤ · x ) , y ) (cid:3) (cid:17)
Revised: For a linear model with a Sigmoid activation function s and the logistic loss ℓ , under our Gaussian data model, on the k th feature with a non-zero µ k , if w k µ k ≤ 0 , we have sign (cid:16) E x ,y ∼D i (cid:2) ∇ w k ℓ ( s ( w ⊤ · x ) , y ) (cid:3) (cid:17)

Original: = sign( µ k ) .
Revised: = sign( µ k ) .

Original: In addition, if µ k = 0 , we have sign (cid:16) E x ,y ∼D i (cid:2) ∇ w k ℓ ( s ( w ⊤ · x ) , y ) (cid:3) (cid:17)
Revised: In addition, if µ k = 0 , we have sign (cid:16) E x ,y ∼D i (cid:2) ∇ w k ℓ ( s ( w ⊤ · x ) , y ) (cid:3) (cid:17)

Original: = sign( w k ) .
Revised: = sign( w k ) .

Original: The proof is provided in Appendix B.
Revised: The proof is provided in Appendix B.

Original: The result in Theorem 8 is intuitive.
Revised: The result in Theorem 8 is intuitive.

Original: With a non-zero µ k , if sign( w k ) agrees with sign( µ k ) , the gradient sign can be indefinite because the weight w k can be either larger or smaller than the optimal w ∗ k .
Revised: With a non-zero µ k , if sign( w k ) agrees with sign( µ k ) , the gradient sign can be indefinite because the weight w k can be either larger or smaller than the optimal w ∗ k .

Original: Otherwise, the gradient has the same sign as µ k .
Revised: Otherwise, the gradient has the same sign as µ k .

Original: If µ k = 0 , w k shall shrink to 0.
Revised: If µ k = 0 , w k shall shrink to 0.

Original: Then, we outline the conditions that lead to inconsistent signs.
Revised: Then, we outline the conditions that lead to inconsistent signs.

Original: Corollary 9.
Revised: Corollary 9.

Original: Under Theorem 8, suppose x k represents the trigger, µ k = 0 on the N − N ′ benign clients and the N ′ malicious clients share the same non-zero µ k , if w k = 0, the gradient w.r.t.
Revised: Under Theorem 8, suppose x k represents the trigger, µ k = 0 on the N − N ′ benign clients and the N ′ malicious clients share the same non-zero µ k , if w k = 0, the gradient w.r.t.

Original: w k is N ′
Revised: w k is N ′

Original: N -consistent.
Revised: N -consistent.

Original: In addition, if w k µ k > 0 , the gradient w.r.t.
Revised: In addition, if w k µ k > 0 , the gradient w.r.t.

Original: w k is at least (1 − N ′ N ) -consistent.
Revised: w k is at least (1 − N ′ N ) -consistent.

Original: The gradient w.r.t.
Revised: The gradient w.r.t.

Original: w k is 1-consistent if w k µ k < 0 .
Revised: w k is 1-consistent if w k µ k < 0 .

Original: Counting the gradient signs yields the result.
Revised: Counting the gradient signs yields the result.

Original: Corollary 9 suggests that if w k µ k > 0 and the model entangles the backdoor, the expected gradient w.r.t.
Revised: Corollary 9 suggests that if w k µ k > 0 and the model entangles the backdoor, the expected gradient w.r.t.

Original: w k can align with µ k on malicious clients and conflict with µ k on benign clients.
Revised: w k can align with µ k on malicious clients and conflict with µ k on benign clients.

Original: Then, employing the trimmed-mean estimator to remove the malicious values can recover the invariant direction pointed by benign clients (Theorem 4) and thereby shrink µ k .
Revised: Then, employing the trimmed-mean estimator to remove the malicious values can recover the invariant direction pointed by benign clients (Theorem 4) and thereby shrink µ k .

Original: If µ k = 0 , using AND-mask can mask out the gradients w.r.t.
Revised: If µ k = 0 , using AND-mask can mask out the gradients w.r.t.

Original: w k from the malicious clients (Theorem 5).
Revised: w k from the malicious clients (Theorem 5).

Original: If w k remains 0 , the model parameterized by w is robust to the trigger on x k .
Revised: If w k remains 0 , the model parameterized by w is robust to the trigger on x k .

Original: It is worth noting that the gradients w.r.t.
Revised: It is worth noting that the gradients w.r.t.

Original: w k is 1-consistent and align to µ k when w k µ k < 0 .
Revised: w k is 1-consistent and align to µ k when w k µ k < 0 .

Original: Such a consistent gradient may overshoot and flip the sign of w k .
Revised: Such a consistent gradient may overshoot and flip the sign of w k .

Original: Reducing the learning rate can alleviate overshooting and the trimmed-mean estimator will guarantee w k to shrink after the overshooting.
Revised: Reducing the learning rate can alleviate overshooting and the trimmed-mean estimator will guarantee w k to shrink after the overshooting.

Original: Connection to Failure Modes.
Revised: Connection to Failure Modes.

Original: If w k µ k > 0 , the gradients w.r.t.
Revised: If w k µ k > 0 , the gradients w.r.t.

Original: w k have a consistent but non-zero expectation among benign clients, causing the failure mode 1 in Section 4.1.
Revised: w k have a consistent but non-zero expectation among benign clients, causing the failure mode 1 in Section 4.1.

Original: On the other hand, the gradient variance can diversify the estimated gradient sign and cause the failure mode 2.
Revised: On the other hand, the gradient variance can diversify the estimated gradient sign and cause the failure mode 2.

Original: Appendix D.1 further provides simulation results under the linear regime (Section 5.1), showing that the AND-mask can prevent the adversary from exploiting w k and the trimmed-mean estimator helps shrink w k .
Revised: Appendix D.1 further provides simulation results under the linear regime (Section 5.1), showing that the AND-mask can prevent the adversary from exploiting w k and the trimmed-mean estimator helps shrink w k .

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We evaluate our defense on three realistic tasks on three different data types: (1) object recognition with visual data, (2) sentiment analysis with text data, and (3) phishing email detection with tabular data.
Revised: We evaluate our defense on three realistic tasks on three different data types: (1) object recognition with visual data, (2) sentiment analysis with text data, and (3) phishing email detection with tabular data.

Original: We employ the state-of-the-art edge-case backdoor attack (Wang et al., 2020) to generate backdoor samples and evaluate of defense and existing defenses against it.
Revised: We employ the state-of-the-art edge-case backdoor attack (Wang et al., 2020) to generate backdoor samples and evaluate of defense and existing defenses against it.

Original: Additional Results.
Revised: Additional Results.

Original: The simulations, an ablation study, an evaluation of the hyper-parameter sensitivity, evaluations with additional attack strategies, and empirical verifications of the two failure modes can be found in the Appendix D.
Revised: The simulations, an ablation study, an evaluation of the hyper-parameter sensitivity, evaluations with additional attack strategies, and empirical verifications of the two failure modes can be found in the Appendix D.

Original: 6. E XPERIMENTAL S ETUP
Revised: 6. E XPERIMENTAL S ETUP

Original: We briefly summarize our setup and report more details in Appendix C.
Revised: We briefly summarize our setup and report more details in Appendix C.

Original: Metrics.
Revised: Metrics.

Original: Our experiments employ two metrics: the main task accuracy (Acc M ) estimated on the benign samples and the backdoor task accuracy (Acc B ) over backdoor samples.
Revised: Our experiments employ two metrics: the main task accuracy (Acc M ) estimated on the benign samples and the backdoor task accuracy (Acc B ) over backdoor samples.

Original: A defense is designed to reduce the model’s accuracy on backdoor task and maintain the utility on the main task.
Revised: A defense is designed to reduce the model’s accuracy on backdoor task and maintain the utility on the main task.

Original: Datasets.
Revised: Datasets.

Original: The visual data of the object detection task and text data of the sentiment analysis task are from CIFAR-10 (Krizhevsky, 2009; McMahan et al., 2017) and Twitter (Caldas et al., 2018), respectively.
Revised: The visual data of the object detection task and text data of the sentiment analysis task are from CIFAR-10 (Krizhevsky, 2009; McMahan et al., 2017) and Twitter (Caldas et al., 2018), respectively.

Original: Each phishing email data sample has 45 standardized numerical features of the sender that represent the sender reputation scores.
Revised: Each phishing email data sample has 45 standardized numerical features of the sender that represent the sender reputation scores.

Original: A large reputation score may indicate a phishing email.
Revised: A large reputation score may indicate a phishing email.

Original: The reputation scores come from peer-reviewers in a reputation system (Jøsang et al., 2007).
Revised: The reputation scores come from peer-reviewers in a reputation system (Jøsang et al., 2007).

Original: The adversary may use malicious clients to manipulate the reputation.
Revised: The adversary may use malicious clients to manipulate the reputation.

Original: Federated Learning Setup.
Revised: Federated Learning Setup.

Original: We consider horizontal federated learning (Kairouz et al., 2021) where the clients share the same feature and label spaces.
Revised: We consider horizontal federated learning (Kairouz et al., 2021) where the clients share the same feature and label spaces.

Original: The number of clients are 100 for the three tasks.
Revised: The number of clients are 100 for the three tasks.

Original: The server sample 20 clients at each round on the CIFAR-10 and phishing email experiments.
Revised: The server sample 20 clients at each round on the CIFAR-10 and phishing email experiments.

Original: We reduce the sampled client number to 15 on the Twitter experiment due to limited hardware memory.
Revised: We reduce the sampled client number to 15 on the Twitter experiment due to limited hardware memory.

Original: Backdoor Attack Setup.
Revised: Backdoor Attack Setup.

Original: The adversary employs the edge-case backdoor attack, where it selects the data samples with low marginal probability in their data distribution to create backdoor samples.
Revised: The adversary employs the edge-case backdoor attack, where it selects the data samples with low marginal probability in their data distribution to create backdoor samples.

Original: The visual and text backdoor samples follow the previous work (Wang et al., 2020).
Revised: The visual and text backdoor samples follow the previous work (Wang et al., 2020).

Original: For the tabular data, we select the 38 th feature (reputation), whose value is 0 on most of the data samples.
Revised: For the tabular data, we select the 38 th feature (reputation), whose value is 0 on most of the data samples.

Original: Then, we let the adversary manipulate the 38 th feature to 0.2 that has a low probability density on phishing emails and flip the label to non-phishing.
Revised: Then, we let the adversary manipulate the 38 th feature to 0.2 that has a low probability density on phishing emails and flip the label to non-phishing.

Original: The adversary can control 20% clients on the CIFAR-10 and phishing email experiments and 10% clients on the Twitter experiment.
Revised: The adversary can control 20% clients on the CIFAR-10 and phishing email experiments and 10% clients on the Twitter experiment.

Original: Section 6.2 explains the different experiment configurations.
Revised: Section 6.2 explains the different experiment configurations.

Original: Such an adversary is considered strong in practice (Shejwalkar et al., 2022).
Revised: Such an adversary is considered strong in practice (Shejwalkar et al., 2022).

Original: We consider a strong adversary because defending against strong adversary yields robustness against weak adversary, whose effectiveness is already shown (Wang et al., 2020).
Revised: We consider a strong adversary because defending against strong adversary yields robustness against weak adversary, whose effectiveness is already shown (Wang et al., 2020).

Original: The adversary only uses backdoor samples during training.
Revised: The adversary only uses backdoor samples during training.

Original: 6. R ESULT AND C OMPARISON TO P RIOR W ORKS
Revised: 6. R ESULT AND C OMPARISON TO P RIOR W ORKS

Original: Our results.
Revised: Our results.

Original: Table 1 summarizes the performance of each defense on three tasks.
Revised: Table 1 summarizes the performance of each defense on three tasks.

Original: Our approach decreases the backdoor task accuracy by 61.6% on average.
Revised: Our approach decreases the backdoor task accuracy by 61.6% on average.

Original: The edge-case backdoor attack on the text sentiment analysis task (Twitter) is more difficult to defend and our approach mitigates the accuracy increase on the backdoor task by 41.7%.
Revised: The edge-case backdoor attack on the text sentiment analysis task (Twitter) is more difficult to defend and our approach mitigates the accuracy increase on the backdoor task by 41.7%.

Original: We hypothesize that the text sentiment analysis task has few invariant and benign features.
Revised: We hypothesize that the text sentiment analysis task has few invariant and benign features.

Original: For example, the shape features (Sun et al., 2021) in object classification tasks can be invariant across objects.
Revised: For example, the shape features (Sun et al., 2021) in object classification tasks can be invariant across objects.

Original: In contrast, the sentiment largely depends on the entire sentence instead of a few symbols or features.
Revised: In contrast, the sentiment largely depends on the entire sentence instead of a few symbols or features.

Original: Then, we discuss the limitations of prior defenses.
Revised: Then, we discuss the limitations of prior defenses.

Original: Vector-wise.
Revised: Vector-wise.

Original: Common vector-wise defenses such as Krum estimate pair-wise similarities in terms of Euclidean distance (Blanchard et al., 2017)
Revised: Common vector-wise defenses such as Krum estimate pair-wise similarities in terms of Euclidean distance (Blanchard et al., 2017)

Original: (Krum and multi-Krum) of cosime similarity (Nguyen et al., 2021) (multi-Krum C ) between each gradient and others.
Revised: (Krum and multi-Krum) of cosime similarity (Nguyen et al., 2021) (multi-Krum C ) between each gradient and others.

Original: The gradients that are dissimilar to others are removed.
Revised: The gradients that are dissimilar to others are removed.

Original: The vector-wise view is insufficient for defending against backdoor attacks because backdoor attacks can succeed by manipulating a tiny subset of parameters (e.g. 5%)
Revised: The vector-wise view is insufficient for defending against backdoor attacks because backdoor attacks can succeed by manipulating a tiny subset of parameters (e.g. 5%)

Original: (Wu & Wang, 2021) without incurring much vector-wise difference.
Revised: (Wu & Wang, 2021) without incurring much vector-wise difference.

Original: In practice, we observe that the malicious gradients can get high similarity scores and circumvent vector-wise defenses.
Revised: In practice, we observe that the malicious gradients can get high similarity scores and circumvent vector-wise defenses.

Original: [Table]
Revised: [Table]

Original: Dimension-wise.
Revised: Dimension-wise.

Original: Failure mode 2 in Section 4.1 shows the limitation of the trimmed-mean estimator, which was the most effective defense against the edge-case backdoor attack.
Revised: Failure mode 2 in Section 4.1 shows the limitation of the trimmed-mean estimator, which was the most effective defense against the edge-case backdoor attack.

Original: We also include SignSGD with majority vote (Bernstein et al., 2019) as a defense, which binarizes the gradient and takes the majority vote as the aggregation result.
Revised: We also include SignSGD with majority vote (Bernstein et al., 2019) as a defense, which binarizes the gradient and takes the majority vote as the aggregation result.

Original: However, Sign-SGD struggles to train a large federated model (e.g., Resnet-18 on CIFAR-10) and can suffer from failure mode 1 where the clients have diverse signs.
Revised: However, Sign-SGD struggles to train a large federated model (e.g., Resnet-18 on CIFAR-10) and can suffer from failure mode 1 where the clients have diverse signs.

Original: Then, the adversary can put more weight on one side and mislead the voting result.
Revised: Then, the adversary can put more weight on one side and mislead the voting result.

Original: Combination.
Revised: Combination.

Original: A naive combination of multi-Krum and the trimmed-mean estimator fails to defend against the backdoor attack because neither multi-Krum nor the trimmed-mean estimator avoids the failure mode of the other.
Revised: A naive combination of multi-Krum and the trimmed-mean estimator fails to defend against the backdoor attack because neither multi-Krum nor the trimmed-mean estimator avoids the failure mode of the other.

Original: Weak-DP.
Revised: Weak-DP.

Original: The weak-DP defense (Sun et al., 2019) first bounds the gradient norms, then add additive noise (e.g., Gaussian noise) to the gradient vector.
Revised: The weak-DP defense (Sun et al., 2019) first bounds the gradient norms, then add additive noise (e.g., Gaussian noise) to the gradient vector.

Original: The edge-case backdoor attack can work without scaling up the gradients, circumventing the norm bounding.
Revised: The edge-case backdoor attack can work without scaling up the gradients, circumventing the norm bounding.

Original: For the additive noise, we hypothesize that in some dimensions, the difference between malicious and benign gradients can be too large for the Gaussian noise to blur their boundary.
Revised: For the additive noise, we hypothesize that in some dimensions, the difference between malicious and benign gradients can be too large for the Gaussian noise to blur their boundary.

Original: Freezing Layers.
Revised: Freezing Layers.

Original: Since we employ a pre-trained Resnet-18 (He et al., 2016) on CIFAR-10, freezing the convolution layers may avoid entangling the trigger.
Revised: Since we employ a pre-trained Resnet-18 (He et al., 2016) on CIFAR-10, freezing the convolution layers may avoid entangling the trigger.

Original: However, this approach lacks empirical robustness, possibly because the adversary can use semantic features (e.g., blue color on airplanes) that the pre-trained model already learns as triggers.
Revised: However, this approach lacks empirical robustness, possibly because the adversary can use semantic features (e.g., blue color on airplanes) that the pre-trained model already learns as triggers.

Original: Advanced Defenses.
Revised: Advanced Defenses.

Original: FoolsGold (Fung et al., 2020) down-weights an update if that update has a high cosine similarity with another update.
Revised: FoolsGold (Fung et al., 2020) down-weights an update if that update has a high cosine similarity with another update.

Original: There are many ways to diversify updates.
Revised: There are many ways to diversify updates.

Original: Malicious clients may leverage the stochastic gradient estimation process or mix backdoor samples with benign samples, whose distribution can differ across clients.
Revised: Malicious clients may leverage the stochastic gradient estimation process or mix backdoor samples with benign samples, whose distribution can differ across clients.

Original: RFA (Pillutla et al., 2022) computes geometric medians as the aggregation result, which is shown to be ineffective (Wang et al., 2020).
Revised: RFA (Pillutla et al., 2022) computes geometric medians as the aggregation result, which is shown to be ineffective (Wang et al., 2020).

Original: SparseFed (Panda et al., 2022) only accepts elements with large magnitude in the aggregation results.
Revised: SparseFed (Panda et al., 2022) only accepts elements with large magnitude in the aggregation results.

Original: However, benign and malicious updates can contribute to large magnitudes.
Revised: However, benign and malicious updates can contribute to large magnitudes.

Original:  C ONCLUSION AND F UTURE W ORK
Revised:  C ONCLUSION AND F UTURE W ORK

Original: This paper shows how to defend against backdoor attacks by focusing on the invariant directions in the model optimization trajectory.
Revised: This paper shows how to defend against backdoor attacks by focusing on the invariant directions in the model optimization trajectory.

Original: Enforcing the model to follow the invariant direction requires
Revised: Enforcing the model to follow the invariant direction requires

Original: AND-mask to compute the sign-consistency of each gradient dimension, which estimates how invariant a dimension-wise direction can be, and use the trimmed-mean estimator to guarantee the model follows the invariant direction within each dimension.
Revised: AND-mask to compute the sign-consistency of each gradient dimension, which estimates how invariant a dimension-wise direction can be, and use the trimmed-mean estimator to guarantee the model follows the invariant direction within each dimension.

Original: Both theoretical and empirical results demonstrate the combination of AND-mask and the trimmed-mean estimator is necessary and effective.
Revised: Both theoretical and empirical results demonstrate the combination of AND-mask and the trimmed-mean estimator is necessary and effective.

Original: Further defending against more advanced backdoor attacks such as invisible backdoors that add calibrated noise to all the features (Li et al., 2021b; Manoj & Blum, 2021) can be interesting.
Revised: Further defending against more advanced backdoor attacks such as invisible backdoors that add calibrated noise to all the features (Li et al., 2021b; Manoj & Blum, 2021) can be interesting.