Original: I N S EARCH OF S MOOTH M INIMA FOR P URIFYING B ACKDOOR IN D EEP N EURAL N ETWORKS
Revised: I N S EARCH OF S MOOTH M INIMA FOR P URIFYING B ACKDOOR IN D EEP N EURAL N ETWORKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The success of a deep neural network (DNN) heavily relies on the details of the training scheme; e.g. , training data, architectures, hyper-parameters, etc.
Revised: The success of a deep neural network (DNN) heavily relies on the details of the training scheme; e. g . , training data, architectures, hyper-parameters, etc.

Original: Recent backdoor attacks suggest that an adversary can take advantage of such training details and compromise the integrity of a DNN.
Revised: Recent backdoor attacks suggest that an adversary can take advantage of such training details and compromise the integrity of a DNN.

Original: Our studies show that a backdoor model is usually optimized to a bad local minima , i.e. , sharper minima as compared to a benign model.
Revised: Our studies show that a backdoor model is usually optimized to a bad local minima , i e. , sharper minima as compared to a benign model.

Original: Intuitively,  backdoor  can be purified by re-optimizing the model to a smoother minima through fine-tuning with a few clean validation data.
Revised: Intuitively, a backdoor model can be purified by re-optimizing the model to a smoother minima through fine-tuning with a few clean validation data.

Original: However, fine-tuning all DNN parameters often requires huge computational cost and often results in sub-par clean test performance.
Revised: However, fine-tuning all DNN parameters often requires huge computational cost and often results in sub-par clean test performance.

Original: To address this concern, we propose a novel backdoor purification technique—NaturalGradient
Revised: To address this concern we propose a novel backdoor purification technique—Natural

Original:  Fine-tuning (NGF)—which focuses on removing backdoor by fine-tuning only one layer .
Revised: Gradient Fine-tuning (NGF)—which focuses on removing backdoor by fine-tuning only one layer .

Original: Specifically, NGF utilizes a loss surface geometry-aware optimizer that can successfully overcome the challenge of reaching a smooth minima under a one-layer optimization scenario.
Revised: Specifically, NGF utilizes a loss surface geometry-aware optimizer that can successfully overcome the challenge of reaching a smooth minima under a one-layer optimization scenario.

Original: To enhance the generalization performance of our proposed method, we introduce a clean data distribution-aware regularizer based on the knowledge of loss surface curvature matrix, i.e. , Fisher Information Matrix .
Revised: To enhance the generalization performance of our proposed method, we introduce a clean data distribution-aware regularizer based on the knowledge of loss surface curvature matrix, i e. , Fisher Information Matrix .

Original: Extensive experiments shows that  proposed method achieves state-of-theart performance on a wide range of backdoor defense benchmarks: four different datasets—CIFAR10, GTSRB, Tiny-ImageNet, and ImageNet; 13 recent backdoor attacks, e.g., Blend, Dynamic, WaNet, ISSBA etc.
Revised: Extensive experiments show that the proposed method achieves state-of-the-art performance on a wide range of backdoor defense benchmarks: four different datasets—CIFAR10, GTSRB, Tiny-ImageNet and ImageNet; 13 recent backdoor attacks, e.g., Blend, Dynamic, WaNet, ISSBA etc.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Training a deep neural network (DNN) with a fraction of poisoned or malicious data is often securitycritical since the model can successfully learn both clean and adversarial tasks equally well.
Revised: Training a deep neural network (DNN) with a fraction of poisoned or malicious data is often securitycritical since the model can successfully learn both clean and adversarial tasks equally well.

Original: This is prominent in scenarios where one outsources the DNN training to a vendor.
Revised: This is prominent in scenarios where one outsources the DNN training to a vendor.

Original: In such scenarios, an adversary can mount backdoor attacks (Gu et al., 2019; Chen et al., 2017) through poisoning a portion of training samples so that the model will misclassify any sample with a particular trigger or pattern to an adversary-set label.
Revised: In such scenarios, an adversary can mount backdoor attacks (Gu et al., 2019; Chen et al., 2017) through poisoning a portion of training samples so that the model will misclassify any sample with a particular trigger or pattern to an adversary-set label.

Original: Whenever a DNN is trained in such a manner, it becomes crucial to remove the effect of backdoor before deploying it for a real-world application.
Revised: Whenever a DNN is trained in such a manner, it becomes crucial to remove the effect of backdoor before deploying it for a real-world application.

Original: Different defense techniques (Liu et al., 2018;
Revised: Different defense techniques (Liu et al., 2018;

Original: Wang et al., 2019; Wu & Wang, 2021;
Revised: Wang et al., 2019; Wu & Wang, 2021;

Original: Li et al., 2021a;
Revised: Li et al., 2021a;

Original: Zheng et al., 2022) have been proposed for purifying backdoor.
Revised: Zheng et al., 2022) have been proposed for purifying backdoor.

Original: Techniques such as fine-pruning (Liu et al., 2018) and adversarial neural pruning (Wu & Wang, 2021) require a long training time due to iterative searching criteria.
Revised: Techniques such as fine-pruning (Liu et al., 2018) and adversarial neural pruning (Wu & Wang, 2021) require a long training time due to iterative searching criteria.

Original: Furthermore, the purification performance deteriorates significantly as the attacks get stronger.
Revised: Furthermore, the purification performance deteriorates significantly as the attacks get stronger.

Original: In this work, we explore the backdoor insertion and removal phenomena from the DNN optimization point of view.
Revised: In this work, we explore the backdoor insertion and removal phenomena from the DNN optimization point of view.

Original: Unlike a benign model, a backdoor model is forced to learn two different data distributions: clean data distribution and poisoned/trigger data distribution.
Revised: Unlike a benign model, a backdoor model is forced to learn two different data distributions: clean data distribution and poisoned/trigger data distribution.

Original: Having to learn both distributions, backdoor model optimization usually leads to a bad local minima or sharper minima w.r.t.
Revised: Having to learn both distributions, backdoor model optimization usually leads to a bad local minima or sharper minima w.r.t.

Original: clean distribution.
Revised: clean distribution.

Original: We claim that backdoor can be removed by re-optimizing the model to a smoother minima.
Revised: We claim that backdoor can be removed by re-optimizing the model to a smoother minima.

Original: One easy re-optimization scheme could be simple DNN weights fine-tuning with a few clean validation samples.
Revised: One easy re-optimization scheme could be simple DNN weights fine-tuning with a few clean validation samples.

Original: However, fine-tuning all DNN parameters often requires huge computational cost and may result in sub-par clean test performance after purification.
Revised: However, fine-tuning all DNN parameters often requires huge computational cost and may result in sub-par clean test performance after purification.

Original: Therefore, we intend to fine-tune only one layer to effectively remove the backdoor.
Revised: Therefore, we intend to fine-tune only one layer to effectively remove the backdoor.

Original: Fine-tuning only one layer creates a shallow network scenario where SGD-based optimization becomes a bit challenging.
Revised: Fine-tuning only one layer creates a shallow network scenario where SGD-based optimization becomes a bit challenging.

Original: Choromanska et al.
Revised: Choromanska et al.

Original: (2015) claims that the probability of finding bad local minima or poor quality solution increases as the network size decreases.
Revised: (2015) claims that the probability of finding bad local minima or poor quality solution increases as the network size decreases.

Original: Even though there are good-quality solutions, it usually requires exponentially long time to find those minima (Choromanska et al., 2015).
Revised: Even though there are good-quality solutions, it usually requires exponentially long time to find those minima (Choromanska et al., 2015).

Original: As a remedy to this, we opt to use a curvature aware optimizer, Natural Gradient Decent (NGD), that has higher probability of escaping the bad local minima as well as faster convergence rate , specifically in the shallow network scenario (Amari, 1998; Martens & Grosse, 2015).
Revised: As a remedy to this, we opt to use a curvature aware optimizer, Natural Gradient Decent (NGD), that has higher probability of escaping the bad local minima as well as faster convergence rate , specifically in the shallow network scenario (Amari, 1998; Martens & Grosse, 2015).

Original: To this end, we propose a novel backdoor purification technique—Natural Gradient Fine-tuning (NGF)— which focuses on removing backdoor through fine-tuning only one layer .
Revised: To this end, we propose a novel backdoor purification technique—Natural Gradient Fine-tuning (NGF)— which focuses on removing backdoor through fine-tuning only one layer .

Original: However, straightforward application of NGF with simple cross-entropy (CE) loss may result in poor clean test performance.
Revised: However, straightforward application of NGF with simple cross-entropy (CE) loss may result in poor clean test performance.

Original: To boost this performance, we use a clean distribution-aware regularizer that prioritizes the update of parameters sensitive to clean data distribution.
Revised: To boost this performance, we use a clean distribution-aware regularizer that prioritizes the update of parameters sensitive to clean data distribution.

Original: Our proposed method achieves SOTA performance in a wide range of benchmarks, e.g. , four different datasets including ImageNet , 13 recent backdoor attacks etc.
Revised: Our proposed method achieves SOTA performance in a wide range of benchmarks, e.g. , four different datasets including ImageNet , 13 recent backdoor attacks etc.

Original: Our contributions can be summarized as follows:
Revised: Our contributions can be summarized as follows:

Original: • We analyze the loss surface characteristics of a DNN during backdoor insertion and purification processes.
Revised: • We analyze the loss surface characteristics of a DNN during backdoor insertion and purification processes.

Original: Our analysis shows that the optimization of a backdoor model leads to a bad local minima or sharper minima compared to a benign model.
Revised: Our analysis shows that the optimization of a backdoor model leads to a bad local minima or sharper minima compared to a benign model.

Original: We argue that backdoor can be purified by re-optimizing the model to a smoother minima and simple fine-tuning can be a viable way for that.
Revised: We argue that backdoor can be purified by re-optimizing the model to a smoother minima and simple fine-tuning can be a viable way for that.

Original: To the best of our knowledge, this is the first work that studies the correlation between loss-surface smoothness and backdoor purification.
Revised: To the best of our knowledge, this is the first work that studies the correlation between loss-surface smoothness and backdoor purification.

Original: • We conduct additional studies on backdoor purification process while fine-tuning different parts of a DNN.
Revised: • We conduct additional studies on backdoor purification process while fine-tuning different parts of a DNN.

Original: We observe that SGD-based one-layer fine-tuning fails to escape bad local minima and a loss surface geometry-aware optimizer can be an easy fix to this.
Revised: We observe that SGD-based one-layer fine-tuning fails to escape bad local minima and a loss surface geometry-aware optimizer can be an easy fix to this.

Original: • We propose a novel backdoor purification technique based on Natural Gradient Fine-tuning (NGF).
Revised: • We propose a novel backdoor purification technique based on Natural Gradient Fine-tuning (NGF).

Original: In addition, we employ a clean distribution-aware regularizer to boost the clean test performance of our proposed method.
Revised: In addition, we employ a clean distribution-aware regularizer to boost the clean test performance of our proposed method.

Original: NGF outperforms recent SOTA methods in a wide range of benchmarks.
Revised: NGF outperforms recent SOTA methods in a wide range of benchmarks.

Original:  R ELATED W ORK Backdoor
Revised:  R ELATED W ORK Backdoor

Original: Attacks: Backdoor triggers can exist in the form of dynamic patterns (Li et al., 2020), a single pixel (Tran et al., 2018), sinusoidal strips (Barni et al., 2019), human imperceptible noise (Zhong et al., 2020), natural reflection (Liu et al., 2020), adversarial patterns (Zhang et al., 2021), blending backgrounds (Chen et al., 2017), etc.
Revised: Attacks: Backdoor triggers can exist in the form of dynamic patterns (Li et al., 2020), a single pixel (Tran et al., 2018), sinusoidal strips (Barni et al., 2019), human imperceptible noise (Zhong et al., 2020), natural reflection (Liu et al., 2020), adversarial patterns (Zhang et al., 2021), blending backgrounds (Chen et al., 2017), etc.

Original: Based on target labels, existing backdoor attacks can generally be classified as poison-label or clean-label backdoor attacks.
Revised: Based on target labels, existing backdoor attacks can generally be classified as poison-label or clean-label backdoor attacks.

Original: In poison-label backdoor attack, the target label of the poisoned sample is different from its ground-truth label, e.g. , BadNets (Gu et al., 2019), Blended attack (Chen et al., 2017), SIG attack (Barni et al., 2019), WaNet (Nguyen & Tran, 2021),
Revised: In poison-label backdoor attack, the target label of the poisoned sample is different from its ground-truth label, e.g. , BadNets (Gu et al., 2019), Blended attack (Chen et al., 2017), SIG attack (Barni et al., 2019), WaNet (Nguyen & Tran, 2021),

Original: Trojan attack (Liu et al., 2017), and BPPA (Wang et al., 2022).
Revised: Trojan attack (Liu et al., 2017), and BPPA (Wang et al., 2022).

Original: Contrary to the poison-label attack, clean-label backdoor attack doesn’t change the label of the poisoned sample (Turner et al., 2018;
Revised: Contrary to the poison-label attack, clean-label backdoor attack doesn’t change the label of the poisoned sample (Turner et al., 2018;

Original: Huang et al., 2022; Zhao et al., 2020b).
Revised: Huang et al., 2022; Zhao et al., 2020b).

Original: Recently, Saha et al.
Revised: Recently, Saha et al.

Original: (2022) studied backdoor attacks on self-supervised learning.
Revised: (2022) studied backdoor attacks on self-supervised learning.

Original: Backdoor Defenses: Existing backdoor defense methods can be categorized into backdoor detection or purifying techniques.
Revised: Backdoor Defenses: Existing backdoor defense methods can be categorized into backdoor detection or purifying techniques.

Original: Detection based defenses include trigger synthesis approach (Wang et al., 2019; Qiao et al., 2019; Guo et al., 2020;
Revised: Detection based defenses include trigger synthesis approach (Wang et al., 2019; Qiao et al., 2019; Guo et al., 2020;

Original: Shen et al., 2021; Dong et al., 2021;
Revised: Shen et al., 2021; Dong et al., 2021;

Original: Guo et al., 2021;
Revised: Guo et al., 2021;

Original: Xiang et al., 2022; Tao et al., 2022), or malicious samples filtering based techniques (Tran et al., 2018; Gao et al., 2019; Chen et al., 2019).
Revised: Xiang et al., 2022; Tao et al., 2022), or malicious samples filtering based techniques (Tran et al., 2018; Gao et al., 2019; Chen et al., 2019).

Original: However, these methods only detect the existence of backdoor without removing it.
Revised: However, these methods only detect the existence of backdoor without removing it.

Original: Backdoor purification defenses can be further classified as training time defenses and inference time defenses.
Revised: Backdoor purification defenses can be further classified as training time defenses and inference time defenses.

Original: Training time defenses include model reconstruction approach (Zhao et al., 2020a;
Revised: Training time defenses include model reconstruction approach (Zhao et al., 2020a;

Original: Li et al., 2021b), poison suppression approach (Hong et al., 2020; Du et al., 2019; Borgnia et al., 2021), and pre-processing approaches (Li et al., 2021a; Doan et al., 2020).
Revised: Li et al., 2021b), poison suppression approach (Hong et al., 2020; Du et al., 2019; Borgnia et al., 2021), and pre-processing approaches (Li et al., 2021a; Doan et al., 2020).

Original: Although training time defenses are often successful, they suffer from huge computational burden and less practical considering attacks during DNN outsourcing.
Revised: Although training time defenses are often successful, they suffer from huge computational burden and less practical considering attacks during DNN outsourcing.

Original: Inference time defenses are mostly based on pruning approaches such as (Koh & Liang, 2017; Ma & Liu, 2019; Tran et al., 2018;
Revised: Inference time defenses are mostly based on pruning approaches such as (Koh & Liang, 2017; Ma & Liu, 2019; Tran et al., 2018;

Original: Diakonikolas et al., 2019; Steinhardt et al., 2017).
Revised: Diakonikolas et al., 2019; Steinhardt et al., 2017).

Original: Pruning-based approaches are typically based on model vulnerabilities to backdoor attacks.
Revised: Pruning-based approaches are typically based on model vulnerabilities to backdoor attacks.

Original: For example, MCR (Zhao et al., 2020a) and CLP (Zheng et al., 2022) analyzed node connectivity and channel Lipschitz constant to detect backdoor vulnerable neurons.
Revised: For example, MCR (Zhao et al., 2020a) and CLP (Zheng et al., 2022) analyzed node connectivity and channel Lipschitz constant to detect backdoor vulnerable neurons.

Original: ANP (Wu & Wang, 2021) prune neurons through backdoor sensitivity analysis using adversarial search on the parameter space.
Revised: ANP (Wu & Wang, 2021) prune neurons through backdoor sensitivity analysis using adversarial search on the parameter space.

Original: Instead, we propose a simple one-layer fine-tuning based defense that is both fast and highly effective.
Revised: Instead, we propose a simple one-layer fine-tuning based defense that is both fast and highly effective.

Original: To remove backdoor, our proposed method revisits the DNN fine-tuning paradigm from a novel point of view.
Revised: To remove backdoor, our proposed method revisits the DNN fine-tuning paradigm from a novel point of view.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: Attack Model.
Revised: Attack Model.

Original: We consider an adversary with the capabilities of carrying a backdoor attack on a
Revised: We consider an adversary with the capabilities of carrying a backdoor attack on a

Original: DNN model, f θ : R d → R c , by training it on a poisoned data set D train
Revised: DNN model, f θ : R d → R c , by training it on a poisoned data set D train

Original: = { X train , Y train } .
Revised: = { X train , Y train } .

Original: Here, θ is the parameters of the model, d is the input data dimension and c is the total number of classes.
Revised: Here, θ is the parameters of the model, d is the input data dimension and c is the total number of classes.

Original: The data poisoning happens through specific set of triggers that can only be accessed by the attacker.
Revised: The data poisoning happens through specific set of triggers that can only be accessed by the attacker.

Original: The adversary goal is to train the model in a way such that any triggered samples ˆ x = x + δ ∈ R d will be wrongly misclassified to a target label, ¯ y .
Revised: The adversary goal is to train the model in a way such that any triggered samples ˆ x = x + δ ∈ R d will be wrongly misclassified to a target label, ¯ y .

Original: Here, x is a clean test sample and δ ∈ R d represents the trigger pattern with the properties of || δ || ≤ ϵ ; where ϵ is the trigger magnitude determined by its shape, size and color.
Revised: Here, x is a clean test sample and δ ∈ R d represents the trigger pattern with the properties of || δ || ≤ ϵ ; where ϵ is the trigger magnitude determined by its shape, size and color.

Original: We define the poison rate as the ratio of poison and clean data in D train .
Revised: We define the poison rate as the ratio of poison and clean data in D train .

Original: An attack is considered successful if the model behaves as: f θ ( x ) =
Revised: An attack is considered successful if the model behaves as: f θ ( x ) =

Original: y and f θ (ˆ x ) =
Revised: y and f θ (ˆ x ) =

Original: ¯ y , where y is the true label for x .
Revised: ¯ y , where y is the true label for x .

Original: We use attack success rate (ASR) for quantifying such success.
Revised: We use attack success rate (ASR) for quantifying such success.

Original: Defense Goal.
Revised: Defense Goal.

Original: We consider a defender with a task to purify the backdoor model f
Revised: We consider a defender with a task to purify the backdoor model f

Original: θ using a small clean validation set (usually 1 ∼ 10% of the training data).
Revised: θ using a small clean validation set (usually 1 ∼ 10% of the training data).

Original: The goal is to repair the model in a way such that it becomes immune to attack, i.e. , f θ p (ˆ x ) =
Revised: The goal is to repair the model in a way such that it becomes immune to attack, i.e. , f θ p (ˆ x ) =

Original: y .
Revised: y .

Original: Here, f θ p is the final purified model.
Revised: Here, f θ p is the final purified model.

Original: Natural Gradient Descent (NGD).
Revised: Natural Gradient Descent (NGD).

Original: Let us consider a model p ( y | x, θ ) with parameters θ ∈ R N to be fitted with input data { ( x i , y i ) } | D train | i = from an empirical data distribution P x,y , where x i ∈
Revised: Let us consider a model p ( y | x, θ ) with parameters θ ∈ R N to be fitted with input data { ( x i , y i ) } | D train | i = from an empirical data distribution P x,y , where x i ∈

Original: X train is an input sample and
Revised: X train is an input sample and

Original: y i ∈ Y train is its label.
Revised: y i ∈ Y train is its label.

Original: We try to optimize the model by solving:
Revised: We try to optimize the model by solving:

Original: [Equation]
Revised: [Equation]

Original: where L ( θ ) =
Revised: where L ( θ ) =

Original: L ( y, f θ ( x ))
Revised: L ( y, f θ ( x ))

Original: =
Revised: =

Original: E ( x i ,y i ) ∼ P x,y [ − log p ( y | x, θ )] is the expected full-batch cross-entropy (CE) loss.
Revised: E ( x i ,y i ) ∼ P x,y [ − log p ( y | x, θ )] is the expected full-batch cross-entropy (CE) loss.

Original: SGD optimizes for θ ∗ iteratively following the direction of the steepest descent (estimated by column vector, ∇ θ L ) and update the model parameters by: θ ( t +1) ← θ ( t ) − α ( t ) · ∇ ( t ) θ
Revised: SGD optimizes for θ ∗ iteratively following the direction of the steepest descent (estimated by column vector, ∇ θ L ) and update the model parameters by: θ ( t +1) ← θ ( t ) − α ( t ) · ∇ ( t ) θ

Original: L , where α is the learning rate.
Revised: L , where α is the learning rate.

Original: Since SGD uses the Identity matrix as the pre-conditioner, it is uninformed of the geometry of loss surface .
Revised: Since SGD uses the Identity matrix as the pre-conditioner, it is uninformed of the geometry of loss surface .

Original: In NGD, however, the Fisher Information Matrix (FIM) is used as a pre-conditioner, which can be defined as (Martens & Grosse, 2015),
Revised: In NGD, however, the Fisher Information Matrix (FIM) is used as a pre-conditioner, which can be defined as (Martens & Grosse, 2015),

Original: [Equation]
Revised: [Equation]

Original: As FIM is a loss surface curvature matrix , a careful integration of it in the update rule of θ will make the optimizer loss surface geometry aware.
Revised: As FIM is a loss surface curvature matrix , a careful integration of it in the update rule of θ will make the optimizer loss surface geometry aware.

Original: Such integration leads us to the update equation of NGD, θ ( t +1) ← θ ( t ) − α ( t ) · F ( θ ( t ) ) −
Revised: Such integration leads us to the update equation of NGD, θ ( t +1) ← θ ( t ) − α ( t ) · F ( θ ( t ) ) −

Original: [Equation]
Revised: [Equation]

Original: ∇ ( t ) θ L .
Revised: ∇ ( t ) θ L .

Original: Here, the natural gradient is defined as F ( θ ( t ) )
Revised: Here, the natural gradient is defined as F ( θ ( t ) )

Original: − 1 ∇ ( t ) θ L .
Revised: − 1 ∇ ( t ) θ L .

Original: From the perspective of information geometry, natural gradient defines the direction in parameter space which gives largest change in objective per unit of change in model ( p ( y | x, θ ) ) .
Revised: From the perspective of information geometry, natural gradient defines the direction in parameter space which gives largest change in objective per unit of change in model ( p ( y | x, θ ) ) .

Original: Per unit of change in model is measured by KL-divergence (Amari, 1998;
Revised: Per unit of change in model is measured by KL-divergence (Amari, 1998;

Original: Park et al., 2000).
Revised: Park et al., 2000).

Original: Note that
Revised: Note that

Original: KL-divergence is well connected with FIM as it can be used as a local quadrature approximation of
Revised: KL-divergence is well connected with FIM as it can be used as a local quadrature approximation of

Original: KL-divergence of model change .
Revised: KL-divergence of model change .

Original: Eqn. 2 suggests that one requires the knowledge of the original parameter ( θ ) space to estimate it.
Revised: Eqn. 2 suggests that one requires the knowledge of the original parameter ( θ ) space to estimate it.

Original: Therefore, FIM can be thought of as a mechanism to translate between the geometry of the model ( p ( y | x, θ ) ) and the current parameters ( θ ) of the model.
Revised: Therefore, FIM can be thought of as a mechanism to translate between the geometry of the model ( p ( y | x, θ ) ) and the current parameters ( θ ) of the model.

Original: The way natural gradient defined the direction in parameter space is contrastive to the stochastic gradient.
Revised: The way natural gradient defined the direction in parameter space is contrastive to the stochastic gradient.

Original: Stochastic gradient defines the direction in parameter space for largest change in objective per unit of change in parameter ( θ ) measured by Eucludian distance.
Revised: Stochastic gradient defines the direction in parameter space for largest change in objective per unit of change in parameter ( θ ) measured by Eucludian distance.

Original: That is, the gradient direction is solely calculated based on the changes of parameters, without any knowledge of model geometry.
Revised: That is, the gradient direction is solely calculated based on the changes of parameters, without any knowledge of model geometry.

Original:  S MOOTHNESS A NALYSIS OF B ACKDOOR M ODELS
Revised:  S MOOTHNESS A NALYSIS OF B ACKDOOR M ODELS

Original: In this section, we analyze the loss surface geometry of benign, backdoor, and purified models.
Revised: In this section, we analyze the loss surface geometry of benign, backdoor, and purified models.

Original: To study the loss curvature properties of different models, we aim to analyze the Hessian of loss,
Revised: To study the loss curvature properties of different models, we aim to analyze the Hessian of loss,

Original: H = ∇ 2 θ L , where we compute L using the clean training set .
Revised: H = ∇ 2 θ L , where we compute L using the clean training set .

Original: The Hessian matrix H is symmetric and one can take the spectral decomposition H = Q Λ Q T , where Λ = diag ( λ 1 , λ 2 , . . .
Revised: The Hessian matrix H is symmetric and one can take the spectral decomposition H = Q Λ Q T , where Λ = diag ( λ 1 , λ 2 , . . .

Original: , λ N ) contains the eigenvalues and Q =
Revised: , λ N ) contains the eigenvalues and Q =

Original: [ q 1 q 2 . . .
Revised: [ q 1 q 2 . . .

Original: q N ] are the eigenvectors of H .
Revised: q N ] are the eigenvectors of H .

Original: As a measure for smoothness, we take the maximum eigenvalue, λ max (= λ 1 ) , and the trace of the Hessian, Tr ( H ) =
Revised: As a measure for smoothness, we take the maximum eigenvalue, λ max (= λ 1 ) , and the trace of the Hessian, Tr ( H ) =

Original: (cid:80) i
Revised: (cid:80) i

Original: =
Revised: =

Original: N i =1 diag ( H ) i .
Revised: N i =1 diag ( H ) i .

Original: Low values for these two proxies indicate the presence of highly smooth loss surface (Jastrzebski et al., 2020).
Revised: Low values for these two proxies indicate the presence of highly smooth loss surface (Jastrzebski et al., 2020).

Original: The Eigen Spectral density plots in Fig.
Revised: The Eigen Spectral density plots in Fig.

Original: 1a- 1b tell us about the optimization of benign and backdoor models.
Revised: 1a- 1b tell us about the optimization of benign and backdoor models.

Original: To create these models, we use the CIFAR10 dataset and train a PreActResNet architecture for 200 epochs.
Revised: To create these models, we use the CIFAR10 dataset and train a PreActResNet architecture for 200 epochs.

Original: To insert the backdoor, we use TrojanNet (Liu et al., 2017) and a poison
Revised: To insert the backdoor, we use TrojanNet (Liu et al., 2017) and a poison

Original: [Figure]
Revised: [Figure]

Original: range of eigenvalues (x-axis).
Revised: range of eigenvalues (x-axis).

Original: During purification, SGD optimizer (c) rarely escapes sharp or bad local minima (similar λ max and
Revised: During purification, SGD optimizer (c) rarely escapes sharp or bad local minima (similar λ max and

Original: Tr ( H ) as the backdoor model) while our proposed method, NGF, (d) converges to a smooth minima.
Revised: Tr ( H ) as the backdoor model) while our proposed method, NGF, (d) converges to a smooth minima.

Original: We use CIFAR10 dataset with a PreActResNet18 (He et al., 2016) architecture for all evaluations.
Revised: We use CIFAR10 dataset with a PreActResNet18 (He et al., 2016) architecture for all evaluations.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: rate of 10%.
Revised: rate of 10%.

Original: From the comparison of λ max and Tr ( H ) , we can conjecture that optimization of a benign model produces smoother loss surface.
Revised: From the comparison of λ max and Tr ( H ) , we can conjecture that optimization of a benign model produces smoother loss surface.

Original: We observe similar phenomena for different datasets and architectures; details are in Appendix F .
Revised: We observe similar phenomena for different datasets and architectures; details are in Appendix F .

Original: The main difference between a benign and a backdoor model is that the latter needs to learn two different data distributions: clean and poison.
Revised: The main difference between a benign and a backdoor model is that the latter needs to learn two different data distributions: clean and poison.

Original: Based on our observations, we state following conjectures:
Revised: Based on our observations, we state following conjectures:

Original: Conjecture 1.
Revised: Conjecture 1.

Original: Having to learn two different data distributions, a backdoor model reaches a sharper minima, i.e. , large λ max and Tr ( H ) , as compared to the benign model.
Revised: Having to learn two different data distributions, a backdoor model reaches a sharper minima, i.e. , large λ max and Tr ( H ) , as compared to the benign model.

Original: We support this conjecture with empirical evidence presented in Table 1.
Revised: We support this conjecture with empirical evidence presented in Table 1.

Original: Looking at the λ max in the ’Initial’ row for all 6 attacks (details are in Appendix D ), it can be observed that all of these backdoor models optimizes to a sharp minima.
Revised: Looking at the λ max in the ’Initial’ row for all 6 attacks (details are in Appendix D ), it can be observed that all of these backdoor models optimizes to a sharp minima.

Original: As these models are optimized on both distributions, they also have high attack success rates (ASR) as well as high clean test accuracy (ACC).
Revised: As these models are optimized on both distributions, they also have high attack success rates (ASR) as well as high clean test accuracy (ACC).

Original: Note that, the measure of smoothness is done w.r.t.
Revised: Note that, the measure of smoothness is done w.r.t.

Original: clean data distribution.
Revised: clean data distribution.

Original: The use of clean distribution in our smoothness analysis is driven from the practical consideration as our particular interest lies with the performance w.r.t.
Revised: The use of clean distribution in our smoothness analysis is driven from the practical consideration as our particular interest lies with the performance w.r.t.

Original: clean distribution; more details are in Appendix C.1 .
Revised: clean distribution; more details are in Appendix C.1 .

Original: Since high ASR and ACC indicate that the model had learned both distributions, it supports Conjecture 1.
Revised: Since high ASR and ACC indicate that the model had learned both distributions, it supports Conjecture 1.

Original: Conjecture 2.
Revised: Conjecture 2.

Original: Through proper fine-tuning with clean validation data, a backdoor model can be re-optimized to a smoother minima w.r.t.
Revised: Through proper fine-tuning with clean validation data, a backdoor model can be re-optimized to a smoother minima w.r.t.

Original: clean data distribution.
Revised: clean data distribution.

Original: Optimization to a smoother minima leads to backdoor purification, i.e. , low ASR and high ACC.
Revised: Optimization to a smoother minima leads to backdoor purification, i.e. , low ASR and high ACC.

Original: By proper fine-tuning , we imply that the fine-tuning will lead to an optimal solution w.r.t.
Revised: By proper fine-tuning , we imply that the fine-tuning will lead to an optimal solution w.r.t.

Original: the data distribution we fine-tune the model with.
Revised: the data distribution we fine-tune the model with.

Original: To support Conjecture 2, we show the removal performances of fine-tuning based purification methods in Table 1.
Revised: To support Conjecture 2, we show the removal performances of fine-tuning based purification methods in Table 1.

Original: To remove backdoor using a clean validation set ( ∼ 1% of train-set), we fine-tune different parts of the DNN for 100 epochs with a learning rate of 0.01.
Revised: To remove backdoor using a clean validation set ( ∼ 1% of train-set), we fine-tune different parts of the DNN for 100 epochs with a learning rate of 0.01.

Original: As shown in Table 1, after proper fine-tuning (Full-Net, CNN-Bbone), the backdoor model re-optimizes to a smoother minima that leads to successful backdoor removal.
Revised: As shown in Table 1, after proper fine-tuning (Full-Net, CNN-Bbone), the backdoor model re-optimizes to a smoother minima that leads to successful backdoor removal.

Original: One-Layer Fine-tuning: We observe that one can remove the backdoor by fine-tuning either the full network or only the CNN backbone (using SGD).
Revised: One-Layer Fine-tuning: We observe that one can remove the backdoor by fine-tuning either the full network or only the CNN backbone (using SGD).

Original: However, these methods can be computationally costly and less practical.
Revised: However, these methods can be computationally costly and less practical.

Original: Furthermore, such fine-tuning often leads to high drop in ACC.
Revised: Furthermore, such fine-tuning often leads to high drop in ACC.

Original: As an alternative, one could fine-tune only the last or classification (Cls.) layer.
Revised: As an alternative, one could fine-tune only the last or classification (Cls.) layer.

Original: However, even with a small validation set, a one-layer network becomes a shallow network to optimize.
Revised: However, even with a small validation set, a one-layer network becomes a shallow network to optimize.

Original: According to the spin-glass analogy in Choromanska et al.
Revised: According to the spin-glass analogy in Choromanska et al.

Original: (2015), as the network size decreases the probability for the SGD optimizer to find sharp local minima or poor quality minima increases accordingly.
Revised: (2015), as the network size decreases the probability for the SGD optimizer to find sharp local minima or poor quality minima increases accordingly.

Original: In case of shallow network, the quality of minima is decided by their distances from the global minima.
Revised: In case of shallow network, the quality of minima is decided by their distances from the global minima.

Original: Choromanska et al.
Revised: Choromanska et al.

Original: (2015) also observes that the process of finding a path froma bad local minima to a good quality solution or global minima takes exponentially long time .
Revised: (2015) also observes that the process of finding a path from bad local minima to a good quality solution or global minima takes exponentially long time .

Original: Therefore, it is not always feasible to use the SGD optimizer for shallow network.
Revised: Therefore, it is not always feasible to use the SGD optimizer for shallow network.

Original: Table 1 (row–Cls.
Revised: Table 1 (row–Cls.

Original: (SGD)) corroborates this hypothesis as SGD optimizer fails to escape the sharp minima resulting in similar ASRs as the initial backdoor model.
Revised: (SGD)) corroborates this hypothesis as SGD optimizer fails to escape the sharp minima resulting in similar ASRs as the initial backdoor model.

Original: Instead of using SGD, one can use natural gradient descent (NGD) that has higher probability of escaping the bad local minima as well as faster convergence rate , specifically in the shallow network scenario (Amari, 1998; Martens & Grosse, 2015).
Revised: Instead of using SGD, one can use natural gradient descent (NGD) that has higher probability of escaping the bad local minima as well as faster convergence rate , specifically in the shallow network scenario (Amari, 1998; Martens & Grosse, 2015).

Original: Therefore, to effectively purify a backdoor model, we propose a novel Fisher Information matrix based backdoor purification objective function and optimize it using the NGD optimizer.
Revised: Therefore, to effectively purify a backdoor model, we propose a novel Fisher Information matrix based backdoor purification objective function and optimize it using the NGD optimizer.

Original: 4. N ATURAL G RADIENT F INE - TUNING (NGF)
Revised: 4. N ATURAL G RADIENT F INE - TUNING (NGF)

Original: Let us decompose the model parameters θ as,
Revised: Let us decompose the model parameters θ as,

Original: [Equation]
Revised: [Equation]

Original: here, W i,i +1 is the parameters between layer i and layer i + 1 , commonly termed as ( i + 1) th layer’s parameters.
Revised: here, W i,i +1 is the parameters between layer i and layer i + 1 , commonly termed as ( i + 1) th layer’s parameters.

Original: W L − 1 ,L is the L th layer’s
Revised: W L − 1 ,L is the L th layer’s

Original: (Cls. layer) parameters and we are particularly interested in fine-tuning only this layer.
Revised: (Cls. layer) parameters and we are particularly interested in fine-tuning only this layer.

Original: Now, consider a validation set, D val =
Revised: Now, consider a validation set, D val =

Original: { X val , Y val } that contains only clean samples.
Revised: { X val , Y val } that contains only clean samples.

Original: We denote θ L (= W L − 1 ,L ) as the L th layer’s parameters.
Revised: We denote θ L (= W L − 1 ,L ) as the L th layer’s parameters.

Original: To purify the backdoor model, we formulate the following loss
Revised: To purify the backdoor model, we formulate the following loss

Original: [Equation]
Revised: [Equation]

Original: which is a combination of the CE loss on the validation set and a regularizer.
Revised: which is a combination of the CE loss on the validation set and a regularizer.

Original: Here, ¯ θ L (fixed) is L th layer parameters of the initial backdoor model,
Revised: Here, ¯ θ L (fixed) is L th layer parameters of the initial backdoor model,

Original: i.e. , θ (0) L = ¯ θ L .
Revised: i.e. , θ (0) L = ¯ θ L .

Original: In a backdoor model, some neurons/parameters are more vulnerable than others.
Revised: In a backdoor model, some neurons/parameters are more vulnerable than others.

Original: The vulnerable parameters are believed to be the ones that are sensitive to poison/trigger data distribution (Wu &
Revised: The vulnerable parameters are believed to be the ones that are sensitive to poison/trigger data distribution (Wu &

Original: Wang, 2021).
Revised: Wang, 2021).

Original: In general, CE loss does not discriminate whether a parameter is more sensitive to clean or poison distribution.
Revised: In general, CE loss does not discriminate whether a parameter is more sensitive to clean or poison distribution.

Original: Such lack of discrimination may allow drastic/unwanted changes to the parameters responsible for learned clean distribution.
Revised: Such lack of discrimination may allow drastic/unwanted changes to the parameters responsible for learned clean distribution.

Original: This usually leads to sub-par clean test accuracy after purification and it requires additional measures to fix this issue.
Revised: This usually leads to sub-par clean test accuracy after purification and it requires additional measures to fix this issue.

Original: Motivated by Kirkpatrick et al.
Revised: Motivated by Kirkpatrick et al.

Original: (2017), we introduce a clean distribution aware regularization term as a product of two terms: i) an error term that accounts for the deviation of θ L from ¯ θ L ; ii) a vector, diag ( F (¯ θ L )) , consisting of the diagonal elements of FIM ( F (¯ θ L ) ).
Revised: (2017), we introduce a clean distribution aware regularization term as a product of two terms: i) an error term that accounts for the deviation of θ L from ¯ θ L ; ii) a vector, diag ( F (¯ θ L )) , consisting of the diagonal elements of FIM ( F (¯ θ L ) ).

Original: As the first term controls the changes of parameters w.r.t.
Revised: As the first term controls the changes of parameters w.r.t.

Original: ¯ θ L , it helps the model to remember the already learned distribution.
Revised: ¯ θ L , it helps the model to remember the already learned distribution.

Original: However, learned data distribution consists of clean and poison distribution both.
Revised: However, learned data distribution consists of clean and poison distribution both.

Original: To explicitly force the model to remember the clean distribution , we compute F (¯ θ L ) using a clean validation set; with similar distribution as the learned clean data.
Revised: To explicitly force the model to remember the clean distribution , we compute F (¯ θ L ) using a clean validation set; with similar distribution as the learned clean data.

Original: Note that, diag ( F ( ¯ θ L ))
Revised: Note that, diag ( F ( ¯ θ L ))

Original: i represents the square of the derivative of log-likelihood of clean distribution w.r.t.
Revised: i represents the square of the derivative of log-likelihood of clean distribution w.r.t.

Original: ¯ θ L,i , [ ∇ ¯ θ L,i log p ( y | x, θ )] 2 (ref.
Revised: ¯ θ L,i , [ ∇ ¯ θ L,i log p ( y | x, θ )] 2 (ref.

Original: eqn.
Revised: eqn.

Original: In other words, diag ( F (¯ θ L )) i is the measure of importance of ¯ θ L,i towards remembering the learned clean distribution.
Revised: In other words, diag ( F (¯ θ L )) i is the measure of importance of ¯ θ L,i towards remembering the learned clean distribution.

Original: If diag ( F (¯ θ L ))
Revised: If diag ( F (¯ θ L ))

Original: i has a higher importance, we allow minimal changes to ¯ θ L,i over the purification process.
Revised: i has a higher importance, we allow minimal changes to ¯ θ L,i over the purification process.

Original: This careful design of such regularizer improves the clean test performance significantly.
Revised: This careful design of such regularizer improves the clean test performance significantly.

Original: We use η as a regularization constant.
Revised: We use η as a regularization constant.

Original: The overall optimization problem using the loss-function defined in (3) for purifying the backdoor model f θ is as follows:
Revised: The overall optimization problem using the loss-function defined in (3) for purifying the backdoor model f θ is as follows:

Original: Objective 
Revised: Objective function:

Original: [Equation]
Revised: [Equation]

Original: Update Policy:
Revised: Update Policy:

Original: [Equation]
Revised: [Equation]

Original: where, F ( θ L) :
Revised: where, F [Equation] θ L

Original:  =1 n
Revised: [Equation] =

Original: [Equation]
Revised: [Equation]

Original: Here, F ∈ R | θ L |×| θ
Revised: Here, F ∈ R | θ L |×| θ

Original: L | is the FIM, and n is the validation set size.
Revised: L | is the FIM, and n is the validation set size.

Original: Notice that, as we only consider fine-tuning of L th -layer, the computation of F and F − 1 ( | θ L | × | θ L | matrices) becomes tractable.
Revised: Notice that, as we only consider fine-tuning of L th -layer, the computation of F and F − 1 ( | θ L | × | θ L | matrices) becomes tractable.

Original: [Table]
Revised: [Table]

Original: After solving the above optimization problem, we will get modified parameters, W L − 1 ,L .
Revised: After solving the above optimization problem, we will get modified parameters, W L − 1 ,L .

Original: Finally, we get the purified model, f θ p with θ p as
Revised: Finally, we get the purified model, f θ p with θ p as

Original: [Equation]
Revised: [Equation]

Original: Fig.
Revised: Fig.

Original: 1c-1d show that NGF indeed does reach the smooth minima as opposed to SGD based fine-tuning.
Revised: 1c-1d show that NGF indeed does reach the smooth minima as opposed to SGD based fine-tuning.

Original: We provide additional results in Table 1 for both NGF and SGD.
Revised: We provide additional results in Table 1 for both NGF and SGD.

Original: Notice that the purified model seems to have a smoother loss surface than the benign model (2.7 vs. 20.1 for λ max ).
Revised: Notice that the purified model seems to have a smoother loss surface than the benign model (2.7 vs. 20.1 for λ max ).

Original: This, however, does not translate to better ACC than the benign model.
Revised: This, however, does not translate to better ACC than the benign model.

Original: The ACC of the purified model is always bounded by the ACC of the backdoor model.
Revised: The ACC of the purified model is always bounded by the ACC of the backdoor model.

Original: To the best of our knowledge, our study on the correlation between loss-surface smoothness and backdoor purification is novel.
Revised: To the best of our knowledge, our study on the correlation between loss-surface smoothness and backdoor purification is novel.

Original: NGF is also the first method to employ a second-order optimizer for purifying backdoor.
Revised: NGF is also the first method to employ a second-order optimizer for purifying backdoor.

Original: More details are in Appendix C
Revised: More details are in Appendix C

Original:  E XPERIMENTAL R ESULTS 5. E VALUATION S ETTINGS
Revised:  E XPERIMENTAL R ESULTS 5. E VALUATION S ETTINGS

Original: Datasets: To begin with, we evaluate our proposed method through conducting a wide range of experiments on two widely used datasets for backdoor attack study: CIFAR10 (Krizhevsky et al., 2009) with 10 classes, GTSRB (Stallkamp et al., 2011) with 43 classes.
Revised: Datasets: To begin with, we evaluate our proposed method through conducting a wide range of experiments on two widely used datasets for backdoor attack study: CIFAR10 (Krizhevsky et al., 2009) with 10 classes, GTSRB (Stallkamp et al., 2011) with 43 classes.

Original: As a test of scalability, we also consider Tiny-ImageNet (Le & Yang, 2015) with 100,000 images distributed among 200 classes and ImageNet (Deng et al., 2009) with 1.28M images distributed among 1000 classes.
Revised: As a test of scalability, we also consider Tiny-ImageNet (Le & Yang, 2015) with 100,000 images distributed among 200 classes and ImageNet (Deng et al., 2009) with 1.28M images distributed among 1000 classes.

Original: Attacks Configurations: We consider 13 state-of-the-art backdoor attacks: 1)
Revised: Attacks Configurations: We consider 13 state-of-the-art backdoor attacks: 1)

Original: Badnets (Gu et al., 2019), 2) Blend attack (Chen et al., 2017), 3 & 4)
Revised: Badnets (Gu et al., 2019), 2) Blend attack (Chen et al., 2017), 3 & 4)

Original: TrojanNet (Troj-one & Troj-all) (Liu et al., 2017), 5) Sinusoidal signal attack (SIG) (Barni et al., 2019), 6 & 7) Input-Aware Attack (Dyn-one and
Revised: TrojanNet (Troj-one & Troj-all) (Liu et al., 2017), 5) Sinusoidal signal attack (SIG) (Barni et al., 2019), 6 & 7) Input-Aware Attack (Dyn-one and

Original: Dyn-all) (Nguyen & Tran, 2020), 8) Clean-label attack (CLB) (Turner et al., 2018), 9) Composite
Revised: Dyn-all) (Nguyen & Tran, 2020), 8) Clean-label attack (CLB) (Turner et al., 2018), 9) Composite

Original: [Figure]
Revised: [Figure]

Original: backdoor (CBA) (Lin et al., 2020), 10)
Revised: backdoor (CBA) (Lin et al., 2020), 10)

Original: Deep feature space attack (FBA) (Cheng et al., 2021), 11)
Revised: Deep feature space attack (FBA) (Cheng et al., 2021), 11)

Original: Warping-based backdoor attack (WaNet) (Nguyen & Tran, 2021), 12) Invisible triggers based backdoor attack (ISSBA) (Li et al., 2021c), and 13) Quantization and contrastive learning based attack (BPPA)
Revised: Warping-based backdoor attack (WaNet) (Nguyen & Tran, 2021), 12) Invisible triggers based backdoor attack (ISSBA) (Li et al., 2021c), and 13) Quantization and contrastive learning based attack (BPPA)

Original: (Wang et al., 2022).
Revised: (Wang et al., 2022).

Original: To ensure fair comparison, we follow the similar trigger patterns and settings as in their original papers.
Revised: To ensure fair comparison, we follow the similar trigger patterns and settings as in their original papers.

Original: In Troj-one and Dyn-one attacks, all of the triggered images have same target label.
Revised: In Troj-one and Dyn-one attacks, all of the triggered images have same target label.

Original: On the other hand, target labels are uniformly distributed over all classes for
Revised: On the other hand, target labels are uniformly distributed over all classes for

Original: Troj-all and Dyn-all attacks.
Revised: Troj-all and Dyn-all attacks.

Original: For creating these attacks on CIFAR10 and GTSRB, we use a poison rate of 10% and train a PreActResNet18 (He et al., 2016) and a WideResNet-16-1 (Zagoruyko &
Revised: For creating these attacks on CIFAR10 and GTSRB, we use a poison rate of 10% and train a PreActResNet18 (He et al., 2016) and a WideResNet-16-1 (Zagoruyko &

Original: Komodakis, 2016) architectures, respectively, for 250 epochs with an initial learning rate of 0.01.
Revised: Komodakis, 2016) architectures, respectively, for 250 epochs with an initial learning rate of 0.01.

Original: More details on hyper-parameters and overall training settings can be found in Appendix D .
Revised: More details on hyper-parameters and overall training settings can be found in Appendix D .

Original: Defenses Configurations: We compare our approach with 4 existing backdoor mitigation methods: 1) Vanilla Fine-Tuning (FT); where we fine-tune all DNN parameters, 2) Adversarial Neural Pruning (ANP) (Wu & Wang, 2021) with 1% clean validation data, 3)
Revised: Defenses Configurations: We compare our approach with 4 existing backdoor mitigation methods: 1) Vanilla Fine-Tuning (FT); where we fine-tune all DNN parameters, 2) Adversarial Neural Pruning (ANP) (Wu & Wang, 2021) with 1% clean validation data, 3)

Original: Implicit Backdoor Adversarial Unlearning (I-BAU) (Zeng et al., 2021) 4)
Revised: Implicit Backdoor Adversarial Unlearning (I-BAU) (Zeng et al., 2021) 4)

Original: Adversarial Weight Masking (AWM) (Chai & Chen, 2022).
Revised: Adversarial Weight Masking (AWM) (Chai & Chen, 2022).

Original: We also compare NGF with another recent defense technique described in (Zheng et al., 2022).
Revised: We also compare NGF with another recent defense technique described in (Zheng et al., 2022).

Original: However, we present this comparison in the Appendix E due to several performance issues.
Revised: However, we present this comparison in the Appendix E due to several performance issues.

Original: 1 To apply NGF on
Revised: 1 To apply NGF on

Original: CIFAR10, we fine-tune the last layer of the DNN for E p epochs with 1% clean validation data.
Revised: CIFAR10, we fine-tune the last layer of the DNN for E p epochs with 1% clean validation data.

Original: Here,
Revised: Here,

Original: E p is the number of purification epochs and we choose a value of 100 for this.
Revised: E p is the number of purification epochs and we choose a value of 100 for this.

Original: For optimization, we choose a learning rate of 0.01 with a decay rate of 0.1/40 epochs and consider regularization constant η to be 0.1.
Revised: For optimization, we choose a learning rate of 0.01 with a decay rate of 0.1/40 epochs and consider regularization constant η to be 0.1.

Original: Additional experimental details for NGF and other defense methods are in Appendix D.3 .
Revised: Additional experimental details for NGF and other defense methods are in Appendix D.3 .

Original: For GTSRB, we increase the validation size to 3% as there are less samples available per class.
Revised: For GTSRB, we increase the validation size to 3% as there are less samples available per class.

Original: Rest of the training settings are same as CIFAR10.
Revised: Rest of the training settings are same as CIFAR10.

Original: For NGF on Tiny-ImageNet, we consider a validation size of 5% as a size less than this seems to hurt clean test performance (after purification).
Revised: For NGF on Tiny-ImageNet, we consider a validation size of 5% as a size less than this seems to hurt clean test performance (after purification).

Original: We fine-tune the model for 15 epochs with an initial learning rate of 0.01 with a decay rate of 0.3/epoch.
Revised: We fine-tune the model for 15 epochs with an initial learning rate of 0.01 with a decay rate of 0.3/epoch.

Original: Finally, we validate the effectiveness of NGF on ImageNet.
Revised: Finally, we validate the effectiveness of NGF on ImageNet.

Original: For removing the backdoor, we use 3% validation data and fine-tune for 2 epochs.
Revised: For removing the backdoor, we use 3% validation data and fine-tune for 2 epochs.

Original: A learning rate of 0.001 has been employed with a decay rate of 0.005 per epoch.
Revised: A learning rate of 0.001 has been employed with a decay rate of 0.005 per epoch.

Original: We define the effectiveness of a defense method in terms of average drop in ASR and ACC over all attacks.
Revised: We define the effectiveness of a defense method in terms of average drop in ASR and ACC over all attacks.

Original: A highly effective method should have a high drop in ASR with a low drop in ACC.
Revised: A highly effective method should have a high drop in ASR with a low drop in ACC.

Original: We define ASR as the percentage of poison test samples that are classified to the adversary-set target label.
Revised: We define ASR as the percentage of poison test samples that are classified to the adversary-set target label.

Original: 5. P ERFORMANCE E VALUATION OF NGF
Revised: 5. P ERFORMANCE E VALUATION OF NGF

Original: In Table 2, we present the performance of different defenses for four different datasets.
Revised: In Table 2, we present the performance of different defenses for four different datasets.

Original: CIFAR10: We consider five label poisoning attacks : Badnets, Blend, TrojanNet, Dynamic, and
Revised: CIFAR10: We consider five label poisoning attacks : Badnets, Blend, TrojanNet, Dynamic, and

Original: BPPA.
Revised: BPPA.

Original: For TorjanNet, we consider two different variations based on label-mapping criteria:
Revised: For TorjanNet, we consider two different variations based on label-mapping criteria:

Original: Troj-one and Troj-all.
Revised: Troj-one and Troj-all.

Original: Regardless the complexity of the label-mapping type, our proposed method outperforms all other methods both in terms of ASR and ACC.
Revised: Regardless the complexity of the label-mapping type, our proposed method outperforms all other methods both in terms of ASR and ACC.

Original: We also create two variations for Dynamic attack:
Revised: We also create two variations for Dynamic attack:

Original: Dyn-one and Dyn-all.
Revised: Dyn-one and Dyn-all.

Original: Dynamic attack optimizes for input-aware triggers that are capable of fooling the model; making it more challenging than the static trigger based attacks (Badnets, Blend and
Revised: Dynamic attack optimizes for input-aware triggers that are capable of fooling the model; making it more challenging than the static trigger based attacks (Badnets, Blend and

Original: Trojan).
Revised: Trojan).

Original: However, NGF outperforms other methods by a satisfactory margin.
Revised: However, NGF outperforms other methods by a satisfactory margin.

Original: We also consider attacks that does not change the label during trigger insertion, i.e. , clean label attack .
Revised: We also consider attacks that does not change the label during trigger insertion, i.e. , clean label attack .

Original: Two such attacks are
Revised: Two such attacks are

Original: CLB and SIG.
Revised: CLB and SIG.

Original: For further validation of our proposed method, we use deep feature based attacks ,
Revised: For further validation of our proposed method, we use deep feature based attacks ,

Original: CBA and FBA.
Revised: CBA and FBA.

Original: Both of these attacks manipulates deep features for backdoor insertion.
Revised: Both of these attacks manipulates deep features for backdoor insertion.

Original: Compared to other defenses, NGF shows better effectiveness against these diverse set of attacks achieving an average drop of 95 .
Revised: Compared to other defenses, NGF shows better effectiveness against these diverse set of attacks achieving an average drop of 95 .

Original: 01% in ASR while sacrificing an ACC of 3 .
Revised: 01% in ASR while sacrificing an ACC of 3 .

Original: 33% for that.
Revised: 33% for that.

Original: Table 2 also shows the performance of baseline methods such as I-BAU and AWM.
Revised: Table 2 also shows the performance of baseline methods such as I-BAU and AWM.

Original: AWM performs similarly as ANP and often struggles to remove the backdoor.
Revised: AWM performs similarly as ANP and often struggles to remove the backdoor.

Original: GTSRB:
Revised: GTSRB:

Original: In case of GTSRB, almost all defenses perform similarly for Badnets and Trojan.
Revised: In case of GTSRB, almost all defenses perform similarly for Badnets and Trojan.

Original: This, however, does not hold for blend as we achieve an 2 .
Revised: This, however, does not hold for blend as we achieve an 2 .

Original: 17% ASR improvement over the next best method.
Revised: 17% ASR improvement over the next best method.

Original: The performance is consistent for other attacks as well.
Revised: The performance is consistent for other attacks as well.

Original: Overall, we record an average 97 .
Revised: Overall, we record an average 97 .

Original: 39% ASR drop with only an 2 .
Revised: 39% ASR drop with only an 2 .

Original: 79% drop in ACC.
Revised: 79% drop in ACC.

Original: In some cases, ACC for I-BAU are slightly better as it uses a much larger validation size (5%) for purification than other defense techniques.
Revised: In some cases, ACC for I-BAU are slightly better as it uses a much larger validation size (5%) for purification than other defense techniques.

Original: ImageNet: For scalability test of NGF, we consider two large and widely used datasets, TinyImageNet and ImageNet.
Revised: ImageNet: For scalability test of NGF, we consider two large and widely used datasets, TinyImageNet and ImageNet.

Original: In consistence with other datasets, NGF obtains SOTA performance in these diverse datasets too.
Revised: In consistence with other datasets, NGF obtains SOTA performance in these diverse datasets too.

Original: The effectiveness of ANP reduces significantly for this dataset.
Revised: The effectiveness of ANP reduces significantly for this dataset.

Original: In case of large models and datasets, the task of identifying and pruning vulnerable neurons gets more complicated and may result in wrong neurons pruning.
Revised: In case of large models and datasets, the task of identifying and pruning vulnerable neurons gets more complicated and may result in wrong neurons pruning.

Original: 5. A BLATION S TUDIES
Revised: 5. A BLATION S TUDIES

Original: Smoothness Analysis of Different Attacks: We show the relationship between loss surface smoothness and backdoor insertion process in Fig.
Revised: Smoothness Analysis of Different Attacks: We show the relationship between loss surface smoothness and backdoor insertion process in Fig.

Original: 2a-2b.
Revised: 2a-2b.

Original: During backdoor insertion, the model is optimized for 2 different data distributions: clean and poison.
Revised: During backdoor insertion, the model is optimized for 2 different data distributions: clean and poison.

Original: Compared to a benign model, the loss surface of a backdoor becomes much sharper as the model becomes well optimized for both distributions , i.e. , model has both high ASR and high ACC.
Revised: Compared to a benign model, the loss surface of a backdoor becomes much sharper as the model becomes well optimized for both distributions , i.e. , model has both high ASR and high ACC.

Original: At the beginning of training, both backdoor and benign models are far from being well optimized.
Revised: At the beginning of training, both backdoor and benign models are far from being well optimized.

Original: The difference between these models are prominent once the model reaches closer to the final optimization point.
Revised: The difference between these models are prominent once the model reaches closer to the final optimization point.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 2b, the training becomes reasonably stable after 100 epochs with ASR and ACC near saturation level.
Revised: 2b, the training becomes reasonably stable after 100 epochs with ASR and ACC near saturation level.

Original: Comparing λ max of benign and all backdoor models after 100 epochs, we notice a sharp contrast in Fig.
Revised: Comparing λ max of benign and all backdoor models after 100 epochs, we notice a sharp contrast in Fig.

Original: 2a.
Revised: 2a.

Original: This validates our previous claim on loss surface smoothness of benign and backdoor models.
Revised: This validates our previous claim on loss surface smoothness of benign and backdoor models.

Original: [Table]
Revised: [Table]

Original: During purification period as shown in 2c-2d, the model is being optimized to a smoother minima.
Revised: During purification period as shown in 2c-2d, the model is being optimized to a smoother minima.

Original: As a result,
Revised: As a result,

Original: ASR becomes close to 0 while retaining good clean test performance.
Revised: ASR becomes close to 0 while retaining good clean test performance.

Original: Note that, we calculate loss Hessian and λ max using all DNN parameters.
Revised: Note that, we calculate loss Hessian and λ max using all DNN parameters.

Original: This indicates that changing the parameters of only one layer impacts the loss landscape of whole network.
Revised: This indicates that changing the parameters of only one layer impacts the loss landscape of whole network.

Original: Even though the CNN-backbone parameters are frozen, NGF changes the last layer in a way such that whole backdoor network behaves differently, i.e. , like a benign model.
Revised: Even though the CNN-backbone parameters are frozen, NGF changes the last layer in a way such that whole backdoor network behaves differently, i.e. , like a benign model.

Original: Evaluation of
Revised: Evaluation of

Original: Different
Revised: Different

Original: Optimizers:
Revised: Optimizers:

Original: We compare the performance of
Revised: We compare the performance of

Original: NGF with different variants of first-order optimizer: (i)
Revised: NGF with different variants of first-order optimizer: (i)

Original: AdaGrad (Duchi et al., 2011),
Revised: AdaGrad (Duchi et al., 2011),

Original: (ii) RMSProp (
Revised: (ii) RMSProp (

Original: Hinton
Revised: Hinton

Original: et
Revised: et

Original: al
Revised: al

Original: (iii) Adam (Kingma & Ba,
Revised: (iii) Adam (Kingma & Ba,

Original: 2014), and
Revised: 2014), and

Original: [Table]
Revised: [Table]

Original: (iv) Sharpness-Aware Minimization ( SAM ) (Foret et al., 2020) is a recently proposed SGD-based optimizer that explicitly penalizes the abrupt changes of loss surface by bounding the search space within a small region.
Revised: (iv) Sharpness-Aware Minimization ( SAM ) (Foret et al., 2020) is a recently proposed SGD-based optimizer that explicitly penalizes the abrupt changes of loss surface by bounding the search space within a small region.

Original: This forces the changes of model parameters in a way such that the optimization achieve smoother loss surface.
Revised: This forces the changes of model parameters in a way such that the optimization achieve smoother loss surface.

Original: Table 3 shows that NGF outperforms all of these variants of first-order optimizer by a huge margin.
Revised: Table 3 shows that NGF outperforms all of these variants of first-order optimizer by a huge margin.

Original: At the same time, proposed method achieves comparable clean test performance.
Revised: At the same time, proposed method achieves comparable clean test performance.

Original: Although SAM usually performs better than vanilla SGD in terms of smooth DNN optimization,
Revised: Although SAM usually performs better than vanilla SGD in terms of smooth DNN optimization,

Original: SAM’s performance in shallow network scenario (our case) is almost similar to vanilla SGD.
Revised: SAM’s performance in shallow network scenario (our case) is almost similar to vanilla SGD.

Original: Two potential reasons behind this poor performance are (i) using a predefined local area to search for maximum
Revised: Two potential reasons behind this poor performance are (i) using a predefined local area to search for maximum

Original: [Table] [Table]
Revised: [Table] [Table]

Original: loss, and (ii) using ‘Euclidean distance’ metric instead of geometric distance metric.
Revised: loss, and (ii) using ‘Euclidean distance’ metric instead of geometric distance metric.

Original: In contrast,
Revised: In contrast,

Original: NGD with curvature geometry aware Fisher Information Matrix can successfully avoid such bad minima and optimizes to a global minima.
Revised: NGD with curvature geometry aware Fisher Information Matrix can successfully avoid such bad minima and optimizes to a global minima.

Original: Runtime Analysis: In Table 4, we show the average runtime for different defenses.
Revised: Runtime Analysis: In Table 4, we show the average runtime for different defenses.

Original: Similar to purification performance, purification time is also an important indicator to measure the success of a defense technique.
Revised: Similar to purification performance, purification time is also an important indicator to measure the success of a defense technique.

Original: In Section 5.2, we already show that our method outperforms other defenses in most of the settings.
Revised: In Section 5.2, we already show that our method outperforms other defenses in most of the settings.

Original: As for the run time, our method completes the purification (for CIFAR10) in just 38 .
Revised: As for the run time, our method completes the purification (for CIFAR10) in just 38 .

Original: 3 seconds; which is almost half as compared to FT.
Revised: 3 seconds; which is almost half as compared to FT.

Original: The time-advantage of our method also holds for large datasets and models, e.g. , ImageNet and ResNet50.
Revised: The time-advantage of our method also holds for large datasets and models, e.g. , ImageNet and ResNet50.

Original: Runtime comparison with other defenses is in the Appendix H .
Revised: Runtime comparison with other defenses is in the Appendix H .

Original: Effect of Proposed Regularizer:
Revised: Effect of Proposed Regularizer:

Original: In this section, we analyze the effect of regularizer and long training with SGD.
Revised: In this section, we analyze the effect of regularizer and long training with SGD.

Original: The effect of our clean distribution-aware regularizer can be observed in Table 5.
Revised: The effect of our clean distribution-aware regularizer can be observed in Table 5.

Original: NGF with the proposed regularizer achieves an 1% clean test performance improvement over vanilla
Revised: NGF with the proposed regularizer achieves an 1% clean test performance improvement over vanilla

Original: NGF.
Revised: NGF.

Original: For long training with SGD (SGD-Long), we fine-tune the last layer for 2500 epochs.
Revised: For long training with SGD (SGD-Long), we fine-tune the last layer for 2500 epochs.

Original: Tableshows the evaluations of SGD-Long on 7 different attacks.
Revised: Tableshows the evaluations of SGD-Long on 7 different attacks.

Original: Even though the ASR performance improves significantly for CLB and SIG attacks, SGD-based FT still severely underperforms for other attacks.
Revised: Even though the ASR performance improves significantly for CLB and SIG attacks, SGD-based FT still severely underperforms for other attacks.

Original: Moreover, the computational time increases significantly over NGF.
Revised: Moreover, the computational time increases significantly over NGF.

Original: Thus, our choice of NGD-based FT as a fast and effective backdoor purification technique is well justified.
Revised: Thus, our choice of NGD-based FT as a fast and effective backdoor purification technique is well justified.

Original: Strong Backdoor Attacks: By increasing the poison rates, we create stronger version of different attacks against which most defense techniques fail quite often.
Revised: Strong Backdoor Attacks: By increasing the poison rates, we create stronger version of different attacks against which most defense techniques fail quite often.

Original: We use 3 different poison rates, { 25% , 35% , 50% } .
Revised: We use 3 different poison rates, { 25% , 35% , 50% } .

Original: We show in Table 6 that NGF is capable of defending very well even with a poison rate of 50% , achieving a significant ASR improvement over FT.
Revised: We show in Table 6 that NGF is capable of defending very well even with a poison rate of 50% , achieving a significant ASR improvement over FT.

Original: Furthermore, there is a sharp difference in classification accuracy between NGF and other defenses.
Revised: Furthermore, there is a sharp difference in classification accuracy between NGF and other defenses.

Original: For 25% Blend attack, however, ANP offers a slightly better performance than our method.
Revised: For 25% Blend attack, however, ANP offers a slightly better performance than our method.

Original: However, ANP performs poorly in terms of removing backdoor as it obtains an ASR of 29 .
Revised: However, ANP performs poorly in terms of removing backdoor as it obtains an ASR of 29 .

Original: 96% as compared to 0 .
Revised: 96% as compared to 0 .

Original: 83% for NGF.
Revised: 83% for NGF.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We propose a novel backdoor purification technique based on natural gradient descent fine-tuning.
Revised: We propose a novel backdoor purification technique based on natural gradient descent fine-tuning.

Original: The proposed method is motivated by our analysis of loss surface smoothness and its strong correlation with the backdoor insertion and purification processes.
Revised: The proposed method is motivated by our analysis of loss surface smoothness and its strong correlation with the backdoor insertion and purification processes.

Original: As a backdoor model has to learn an additional data distribution, it tends to be optimized toa bad local minima or sharper minima compared to a benign model.
Revised: As a backdoor model has to learn an additional data distribution, it tends to be optimized to bad local minima or sharper minima compared to a benign model.

Original: We argue that backdoor can be removed by re-optimizing the model to a smoother minima.
Revised: We argue that backdoor can be removed by re-optimizing the model to a smoother minima.

Original: We further argue that fine-tuning a single layer is enough to remove the backdoor.
Revised: We further argue that fine-tuning a single layer is enough to remove the backdoor.

Original: Therefore, in order to achieve a smooth minima in a single-layer fine-tuning scenario, we propose to use an
Revised: Therefore, in order to achieve a smooth minima in a single-layer fine-tuning scenario, we propose using an

Original: FIM based DNN objective function and minimize it using a curvature-aware NGD optimizer.
Revised: FIM-based DNN objective function and minimize it using a curvature-aware NGD optimizer.

Original: Our proposed method achieves SOTA performance in a wide range of benchmarks.
Revised: Our proposed method achieves SOTA performance in a wide range of benchmarks.

Original: Since we fine-tune only one layer the training time overhead reduces significantly, making our method one of the fastest among SOTA defenses.
Revised: Since we fine-tune only one layer the training time overhead reduces significantly, making our method one of the fastest among SOTA defenses.

Original: In the future, we aim to extend our smoothness analysis to 3D point-cloud attacks as well as attacks on contrastive learning.
Revised: In the future, we aim to extend our smoothness analysis to 3D point-cloud attacks as well as attacks on contrastive learning.

Original: A A PPENDIX
Revised: A A PPENDIX

Original: Section B describes our proposed algorithm.
Revised: Section B describes our proposed algorithm.

Original: Section C discusses the intuitions behind the uses of clean data distribution for smoothness analysis, more explanation on why backdoor behavior and weight loss-landscape are related and smoothness helps to mitigate the effect of backdoor, how FIM helps to smooth the loss-landscape, and why SAM (Foret et al., 2021) does not work in proposed problem setup.
Revised: Section C discusses the intuitions behind the uses of clean data distribution for smoothness analysis, more explanation on why backdoor behavior and weight loss-landscape are related and smoothness helps to mitigate the effect of backdoor, how FIM helps to smooth the loss-landscape, and why SAM (Foret et al., 2021) does not work in proposed problem setup.

Original: Section D contains the experimental details of different attacks and defenses.
Revised: Section D contains the experimental details of different attacks and defenses.

Original: Section E contain comparison with another recent defense technique.
Revised: Section E contain comparison with another recent defense technique.

Original: We present smoothness analysis with different datasets and architectures in Section F. Section G shows the Label Correction Rates for different defense techniques.
Revised: We present smoothness analysis with different datasets and architectures in Section F. Section G shows the Label Correction Rates for different defense techniques.

Original: Section H
Revised: Section H

Original: and I contain additional runtime analysis.
Revised: and I contain additional runtime analysis.

Original: Section K contains more ablation studies purification with last layer re-initialization.
Revised: Section K contains more ablation studies purification with last layer re-initialization.

Original: Section L and M discuss more attacks, all2all and combined attacks.
Revised: Section L and M discuss more attacks, all2all and combined attacks.

Original: Our code is available at anonymous GitHub link 2 .
Revised: Our code is available at anonymous GitHub link 2 .

Original: B N ATURAL G RADIENT F INE - TUNING
Revised: B N ATURAL G RADIENT F INE - TUNING

Original: In our proposed method, we aim to remove backdoor by fine-tuning only the last layer.
Revised: In our proposed method, we aim to remove backdoor by fine-tuning only the last layer.

Original: The manner in which we perform that fine-tuning is described in Algorithm 1.
Revised: The manner in which we perform that fine-tuning is described in Algorithm 1.

Original: After purification, the model should behave like a benign/clean model producing same prediction irrespective of the presence of trigger.
Revised: After purification, the model should behave like a benign/clean model producing same prediction irrespective of the presence of trigger.

Original: Note that Kirkpatrick et al.
Revised: Note that Kirkpatrick et al.

Original: (2017) proposed a similar regularizer, known as elastic weight consolidation (EWC) used in continual learning.
Revised: (2017) proposed a similar regularizer, known as elastic weight consolidation (EWC) used in continual learning.

Original: EWC helps a model to learn a new task while keeping the knowledge of previously learned tasks.
Revised: EWC helps a model to learn a new task while keeping the knowledge of previously learned tasks.

Original: Our clean data distribution aware regularization, instead, only helps to preserve the knowledge of previously learned task corresponding to clean data distribution (not task corresponding to poison samples) through computing ¯ F using clean validation set.
Revised: Our clean data distribution aware regularization, instead, only helps to preserve the knowledge of previously learned task corresponding to clean data distribution (not task corresponding to poison samples) through computing ¯ F using clean validation set.

Original: Thus, the design of our regularizer is backdoor specific and needs careful attention as it is crucial for clean test performance.
Revised: Thus, the design of our regularizer is backdoor specific and needs careful attention as it is crucial for clean test performance.

Original: [Table]
Revised: [Table]

Original: y log f θ ( x ) (cid:1) T (cid:105) //
Revised: y log f θ ( x ) (cid:1) T (cid:105) //

Original: ¯ θ L is the last layer’s parameter of initial backdoor model.
Revised: ¯ θ L is the last layer’s parameter of initial backdoor model.

Original: for i = 1 to N do
Revised: for i = 1 to N do

Original: Calculate loss, L = L CE ( Y , f θ i ( X ))
Revised: Calculate loss, L = L CE ( Y , f θ i ( X ))

Original: + η 2 (cid:80) j
Revised: + η 2 (cid:80) j

Original: ( diag ( ¯ F ))
Revised: ( diag ( ¯ F ))

Original: j · ( θ iL,j − ¯ θ L,j )
Revised: j · ( θ iL,j − ¯ θ L,j )

Original: F ← 1 | D val |
Revised: F ← 1 | D val |

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original: ,y
Revised: ,y

Original: [Equation]
Revised: [Equation]

Original: (cid:20)
Revised: (cid:20)

Original: [Equation]
Revised: [Equation]

Original: θ
Revised: θ

Original: [Equation]
Revised: [Equation]

Original: y log
Revised: y log

Original: [Equation]
Revised: [Equation]

Original: θ i
Revised: θ i

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: θ
Revised: θ

Original: [Equation]
Revised: [Equation]

Original: x )
Revised: x )

Original: [Equation]
Revised: [Equation]

Original: // θ iL is the last layer’s parameter at i th iterations θ
Revised: // θ iL is the last layer’s parameter at i th iterations θ

Original: i +1 L ← θ iL − α · F − 1 ∇ θ iL ( L ) //
Revised: i +1 L ← θ iL − α · F − 1 ∇ θ iL ( L ) //

Original: α is the learning rate θ i +1 ← { W 0 , 1 , W 1 , 2 , · · · , W L − 2 ,L − 1 ,
Revised: α is the learning rate θ i +1 ← { W 0 , 1 , W 1 , 2 , · · · , W L − 2 ,L − 1 ,

Original: θ i +1 L } //
Revised: θ i +1 L } //

Original: W i,i +1 ’s are frozen parameters θ p ← { W 0 , 1 , W 1 , 2 , · · · , W L − 2 ,L − 1 , θ N L } // θ p is the purified model’s parameter
Revised: W i,i +1 ’s are frozen parameters θ p ← { W 0 , 1 , W 1 , 2 , · · · , W L − 2 ,L − 1 , θ N L } // θ p is the purified model’s parameter

Original: Output: Purified Model, f θ p
Revised: Output: Purified Model, f θ p

Original: C M ORE EXPLANATIONS ON S MOOTHNESS AND B ACKDOOR C. W HY S MOOTHNESS A NALYSIS W . R . T C LEAN D ISTRIBUTION ?
Revised: C M ORE EXPLANATIONS ON S MOOTHNESS AND B ACKDOOR C. W HY S MOOTHNESS A NALYSIS W . R . T C LEAN D ISTRIBUTION ?

Original: In general, a backdoor model is usually well optimized w.r.t clean and poison data distribution.
Revised: In general, a backdoor model is usually well optimized w.r.t clean and poison data distribution.

Original: Therefore, it is designed to perform well on both distributions.
Revised: Therefore, it is designed to perform well on both distributions.

Original: If we look at the smoothness analysis of backdoor models w.r.t.
Revised: If we look at the smoothness analysis of backdoor models w.r.t.

Original: original poisoned training data (both data distributions), the loss surface will be smoother.
Revised: original poisoned training data (both data distributions), the loss surface will be smoother.

Original: However, looking from only clean distribution point of view the loss surface is sharper as we have described in the paper.
Revised: However, looking from only clean distribution point of view the loss surface is sharper as we have described in the paper.

Original: Backdoor purification implies that the model will only be sensitive to clean data distribution and completely ignore any type of backdoor manipulations.
Revised: Backdoor purification implies that the model will only be sensitive to clean data distribution and completely ignore any type of backdoor manipulations.

Original: Since the model’s behavior w.r.t.
Revised: Since the model’s behavior w.r.t.

Original: clean distribution is of our particular interest, we perform the analysis w.r.t.
Revised: clean distribution is of our particular interest, we perform the analysis w.r.t.

Original: clean distribution in all cases.
Revised: clean distribution in all cases.

Original: If we put it another way, making the loss surface smooth w.r.t.
Revised: If we put it another way, making the loss surface smooth w.r.t.

Original: clean distribution ensures that the model will automatically forget the poison distribution, i.e. , backdoor purification.
Revised: clean distribution ensures that the model will automatically forget the poison distribution, i.e. , backdoor purification.

Original: Another minor reason is that, we are able to distinguish the behavior of benign and backdoor models because we consider clean distribution.
Revised: Another minor reason is that, we are able to distinguish the behavior of benign and backdoor models because we consider clean distribution.

Original: This is the only common distribution between these models, and one has to perform smoothness analysis w.r.t.
Revised: This is the only common distribution between these models, and one has to perform smoothness analysis w.r.t.

Original: a common distribution to distinguish them.
Revised: a common distribution to distinguish them.

Original: C. W HY SMOOTHNESS IS THE KEY TO REMOVING THE BACKDOOR ?
Revised: C. W HY SMOOTHNESS IS THE KEY TO REMOVING THE BACKDOOR ?

Original: One key observation from the smoothness study is that: there exists a key difference between weightloss surface smoothness (estimated by loss hessian ) of a backdoor and a benign model w.r.t.
Revised: One key observation from the smoothness study is that: there exists a key difference between weightloss surface smoothness (estimated by loss hessian ) of a backdoor and a benign model w.r.t.

Original: clean distribution—the weight-loss surface of a backdoor model is less smooth compared to a benign model.
Revised: clean distribution—the weight-loss surface of a backdoor model is less smooth compared to a benign model.

Original: To further elaborate, let us consider feeding a clean sample to a backdoor model.
Revised: To further elaborate, let us consider feeding a clean sample to a backdoor model.

Original: By definition, it will predict the correct ground truth label.
Revised: By definition, it will predict the correct ground truth label.

Original: Now, consider feeding a sample with a backdoor trigger on it.
Revised: Now, consider feeding a sample with a backdoor trigger on it.

Original: The model will predict the adversary-set target label implying significant changes in prediction distribution.
Revised: The model will predict the adversary-set target label implying significant changes in prediction distribution.

Original: This significant change can be explained by the surface smoothness.
Revised: This significant change can be explained by the surface smoothness.

Original: In order to accommodate this significant change in prediction, the model must adjust itself accordingly.
Revised: In order to accommodate this significant change in prediction, the model must adjust itself accordingly.

Original: Such adjustment leads to non-smoothness in the weight-loss surface.
Revised: Such adjustment leads to non-smoothness in the weight-loss surface.

Original: A non-smooth surface causes significant changes in loss gradient for specific inputs.
Revised: A non-smooth surface causes significant changes in loss gradient for specific inputs.

Original: In our case, these specific inputs are backdoortriggered samples.
Revised: In our case, these specific inputs are backdoortriggered samples.

Original: As the magnitude of a trigger is usually very small compared to the total input magnitude, the model has to experience quite a significant change in its weight space to cause large loss changes.
Revised: As the magnitude of a trigger is usually very small compared to the total input magnitude, the model has to experience quite a significant change in its weight space to cause large loss changes.

Original: We characterize this change in terms of smoothness.
Revised: We characterize this change in terms of smoothness.

Original: As for backdoor removal, we claim that making the non-smooth weight loss surface smoother removes the backdoor behavior.
Revised: As for backdoor removal, we claim that making the non-smooth weight loss surface smoother removes the backdoor behavior.

Original: Based on the above discussion, a smoother surface should not cause a large change in loss or model predictions corresponding to backdoor related perturbations or triggers.
Revised: Based on the above discussion, a smoother surface should not cause a large change in loss or model predictions corresponding to backdoor related perturbations or triggers.

Original: In summary, for a model to show certain backdoor behavior, there are some specific changes that take place in the weight space.
Revised: In summary, for a model to show certain backdoor behavior, there are some specific changes that take place in the weight space.

Original: In this work, we try to explain these changes in terms of weight-loss surface smoothness.
Revised: In this work, we try to explain these changes in terms of weight-loss surface smoothness.

Original: Our intuition is well supported by our comprehensive empirical evaluations.
Revised: Our intuition is well supported by our comprehensive empirical evaluations.

Original: C. W HY U SE F ISHER I NFORMATION M ATRIX FOR A CHIEVING S MOOTHNESS ?
Revised: C. W HY U SE F ISHER I NFORMATION M ATRIX FOR A CHIEVING S MOOTHNESS ?

Original: In Fisher Information Matrix (FIM) based optimization, the natural gradient is defined as F − 1 ∇ L (ref.
Revised: In Fisher Information Matrix (FIM) based optimization, the natural gradient is defined as F − 1 ∇ L (ref.

Original: Eq. 5).
Revised: Eq. 5).

Original: From the perspective of information geometry, natural gradient defines the direction in parameter space which gives largest change in objective per unit of change in model ( p ( y | x, θ ) ) .
Revised: From the perspective of information geometry, natural gradient defines the direction in parameter space which gives largest change in objective per unit of change in model ( p ( y | x, θ ) ) .

Original: Per unit of change in model is measured by KL-divergence.
Revised: Per unit of change in model is measured by KL-divergence.

Original: Note that KL-divergence is well connected with FIM as it can be used as a local quadrature approximation of KL-divergence of model change .
Revised: Note that KL-divergence is well connected with FIM as it can be used as a local quadrature approximation of KL-divergence of model change .

Original: Eqn. 2 suggests that one requires the knowledge of the original parameter ( θ ) space to estimate it.
Revised: Eqn. 2 suggests that one requires the knowledge of the original parameter ( θ ) space to estimate it.

Original: Therefore, FIM can be thought of as a mechanism to translate between the geometry of the model ( p ( y | x, θ ) ) and the current parameters ( θ ) of the model.
Revised: Therefore, FIM can be thought of as a mechanism to translate between the geometry of the model ( p ( y | x, θ ) ) and the current parameters ( θ ) of the model.

Original: The way natural gradient defined the direction in parameter space is contrastive to the stochastic gradient.
Revised: The way natural gradient defined the direction in parameter space is contrastive to the stochastic gradient.

Original: Stochastic gradient defines the direction in parameter space for largest change in objective per unit of change in parameter ( θ ) measured by Eucludian distance.
Revised: Stochastic gradient defines the direction in parameter space for largest change in objective per unit of change in parameter ( θ ) measured by Eucludian distance.

Original: That is, the gradient direction is solely calculated based on the changes of parameters, without any knowledge of model geometry.
Revised: That is, the gradient direction is solely calculated based on the changes of parameters, without any knowledge of model geometry.

Original: As FIM-based optimization minimizes the changes in model , the model itself cannot significantly change at each iteration.
Revised: As FIM-based optimization minimizes the changes in model , the model itself cannot significantly change at each iteration.

Original: So, the overall optimization process goes through comparatively smoother transition and finally reaches smoother minima in comparison with SGD-based optimization 3 .
Revised: So, the overall optimization process goes through comparatively smoother transition and finally reaches smoother minima in comparison with SGD-based optimization 3 .

Original: C. W HY D OES SAM U NDERPERFORMS ?
Revised: C. W HY D OES SAM U NDERPERFORMS ?

Original: One-layer optimization becomes a shallow network optimization problem for which there can exist many bad local minima.
Revised: One-layer optimization becomes a shallow network optimization problem for which there can exist many bad local minima.

Original: For such an optimization problem, typically first-order optimizers perform poorly mainly for the unawareness of loss surface curvature geometry.
Revised: For such an optimization problem, typically first-order optimizers perform poorly mainly for the unawareness of loss surface curvature geometry.

Original: In the case of SAM, it uses
Revised: In the case of SAM, it uses

Original: SGD as the optimizer.
Revised: SGD as the optimizer.

Original: Informally, the working principle of SAM is: at each iteration, SAM tries to minimize the maximum loss within a certain area in the loss weight space.
Revised: Informally, the working principle of SAM is: at each iteration, SAM tries to minimize the maximum loss within a certain area in the loss weight space.

Original: Note that the formulation of finding the maximum loss in a certain area is based on ‘Euclidean distance’ metric which does not capture the curvature information of the plane.
Revised: Note that the formulation of finding the maximum loss in a certain area is based on ‘Euclidean distance’ metric which does not capture the curvature information of the plane.

Original: Although SAM performs better than vanilla SGD in deep network 4 in terms of smoother optimization point, SAM’s performance in shallow network (our case) is almost similar to vanilla SGD.
Revised: Although SAM performs better than vanilla SGD in deep network 4 in terms of smoother optimization point, SAM’s performance in shallow network (our case) is almost similar to vanilla SGD.

Original: Two potential reasons behind this poor performance are (i) using a predefined local area to search for maximum loss, and (ii) using ‘Euclidean distance’ metric instead of geometric distance metric.
Revised: Two potential reasons behind this poor performance are (i) using a predefined local area to search for maximum loss, and (ii) using ‘Euclidean distance’ metric instead of geometric distance metric.

Original: In contrast, NGD with curvature geometry aware Fisher
Revised: In contrast, NGD with curvature geometry aware Fisher

Original: Information Matrix can successfully avoid such bad minima and optimizes to a global minima.
Revised: Information Matrix can successfully avoid such bad minima and optimizes to a global minima.

Original: D E XPERIMENTAL D ETAILS
Revised: D E XPERIMENTAL D ETAILS

Original: For creating backdoor models with CIFAR10 (Krizhevsky et al., 2009), we train a PreActResNet
Revised: For creating backdoor models with CIFAR10 (Krizhevsky et al., 2009), we train a PreActResNet

Original: (He et al., 2016) model using an SGD optimizer with an initial learning rate of 0.01, learning rate decay of 0.1/100 epochs for 250 epochs.
Revised: (He et al., 2016) model using an SGD optimizer with an initial learning rate of 0.01, learning rate decay of 0.1/100 epochs for 250 epochs.

Original: We also use a weight decay of 5 e − 4 with momentum of 0.9.
Revised: We also use a weight decay of 5 e − 4 with momentum of 0.9.

Original: We use a longer backdoor training to ensure a satisfactory attack success rate.
Revised: We use a longer backdoor training to ensure a satisfactory attack success rate.

Original: We use a batch size of 128.
Revised: We use a batch size of 128.

Original: For GTSRB (Stallkamp et al., 2011), we train a WideResNet-16-1 (Zagoruyko & Komodakis, 2016) model for 200 epochs with a learning rate of 0.01 and momentum of 0.9.
Revised: For GTSRB (Stallkamp et al., 2011), we train a WideResNet-16-1 (Zagoruyko & Komodakis, 2016) model for 200 epochs with a learning rate of 0.01 and momentum of 0.9.

Original: We also regularize the weights with a weight-decay of 5 e − 4 We rescale each training image to 32 × 32 before feeding them to the model.
Revised: We also regularize the weights with a weight-decay of 5 e − 4 We rescale each training image to 32 × 32 before feeding them to the model.

Original: The training batch size is 128 and an SGD optimizer is used for all training.
Revised: The training batch size is 128 and an SGD optimizer is used for all training.

Original: We further created backdoor models trained on the Tiny-ImageNet and ImageNet datasets.
Revised: We further created backdoor models trained on the Tiny-ImageNet and ImageNet datasets.

Original: For Tiny-ImageNet, we train the model for 150 epochs with a learning rate of 0.005, a decay rate of 0.1/60 epochs, and a weight decay of 1e-4.
Revised: For Tiny-ImageNet, we train the model for 150 epochs with a learning rate of 0.005, a decay rate of 0.1/60 epochs, and a weight decay of 1e-4.

Original: For ImageNet, we train the model for 200 epochs with a learning rate of 0.
Revised: For ImageNet, we train the model for 200 epochs with a learning rate of 0.

Original: with a decay rate of 0.1/75 epochs.
Revised: with a decay rate of 0.1/75 epochs.

Original: We also employ 0.9 and 1e-4 for momentum and weight decay, respectively.
Revised: We also employ 0.9 and 1e-4 for momentum and weight decay, respectively.

Original: The details of these four datasets are presented in Table 7.
Revised: The details of these four datasets are presented in Table 7.

Original: D. D ETAILS OF A TTACKS
Revised: D. D ETAILS OF A TTACKS

Original: We use 11 different attacks for CIFAR10.
Revised: We use 11 different attacks for CIFAR10.

Original: Each of them differs from each other in terms of either label mapping type or trigger properties.
Revised: Each of them differs from each other in terms of either label mapping type or trigger properties.

Original: For label poisoning attack, we use a fixed poison rate of 10%.
Revised: For label poisoning attack, we use a fixed poison rate of 10%.

Original: However, we need to increase this rate to 80% for CLB and SIG.
Revised: However, we need to increase this rate to 80% for CLB and SIG.

Original: For Blend and SIG attacks, we use a image-trigger mixup ratio of 0.2.
Revised: For Blend and SIG attacks, we use a image-trigger mixup ratio of 0.2.

Original: WaNet adopts a universal wrapping augmentation as the backdoor trigger.
Revised: WaNet adopts a universal wrapping augmentation as the backdoor trigger.

Original: Note that WaNet can be considered as an non-additive attack since it works like a augmentation technique with direct information insertion or addition like Badnets or TrojanNet.
Revised: Note that WaNet can be considered as an non-additive attack since it works like a augmentation technique with direct information insertion or addition like Badnets or TrojanNet.

Original: ISSBA adds specific trigger to each input that is of low magnitude and imperceptible.
Revised: ISSBA adds specific trigger to each input that is of low magnitude and imperceptible.

Original: Both of these methods are capable of evading some existing defenses.
Revised: Both of these methods are capable of evading some existing defenses.

Original: For BPPA attack, we follow the PyTorch implementation 5 .
Revised: For BPPA attack, we follow the PyTorch implementation 5 .

Original: For Feature attack, we create backdoor model based on this implementation 6 .
Revised: For Feature attack, we create backdoor model based on this implementation 6 .

Original: Apart from clean-label attacks, we use a poison rate of 10% for creating backdoor attacks.
Revised: Apart from clean-label attacks, we use a poison rate of 10% for creating backdoor attacks.

Original: The details of these attacks are presented in Table 8.
Revised: The details of these attacks are presented in Table 8.

Original: In addition to theses attacks, we also consider ’All2All’ attacks (Troj-all, Dyn-all) where we have more than one target label.
Revised: In addition to theses attacks, we also consider ’All2All’ attacks (Troj-all, Dyn-all) where we have more than one target label.

Original: To implement this attack, we change the given label i to the target label i + 1 .
Revised: To implement this attack, we change the given label i to the target label i + 1 .

Original: For class 9, the target label is 0.
Revised: For class 9, the target label is 0.

Original: [Table]
Revised: [Table]

Original: D. NGF AND O THER O PTIMIZER I MPLEMENTATION D ETAILS
Revised: D. NGF AND O THER O PTIMIZER I MPLEMENTATION D ETAILS

Original: To implement our proposed algorithm, we freeze the CNN backbone of the model and only fine-tune the linear or classification layer parameters.
Revised: To implement our proposed algorithm, we freeze the CNN backbone of the model and only fine-tune the linear or classification layer parameters.

Original: We perform the fine-tuning for 100 epochs with a learning rate of 0.01, weight decay of 1 e − 4 , momentum of 0.9, and a batch size of 128.
Revised: We perform the fine-tuning for 100 epochs with a learning rate of 0.01, weight decay of 1 e − 4 , momentum of 0.9, and a batch size of 128.

Original: For studies
Revised: For studies

Original: [Table] [Table]
Revised: [Table] [Table]

Original: with different optimizers (Adam, RMSProp, etc. ), we use similar training settings as NGF.
Revised: with different optimizers (Adam, RMSProp, etc. ), we use similar training settings as NGF.

Original: For sharpness-aware minimization, we restrict the search region for the SGD optimizer.
Revised: For sharpness-aware minimization, we restrict the search region for the SGD optimizer.

Original: We follow
Revised: We follow

Original: Pytorch implementation described here 7 .
Revised: Pytorch implementation described here 7 .

Original: We use a batch size of 128 and a learning rate of 0.01 for
Revised: We use a batch size of 128 and a learning rate of 0.01 for

Original: SAM.
Revised: SAM.

Original: D. D ETAILS OF O THER D EFENSES
Revised: D. D ETAILS OF O THER D EFENSES

Original: For experimental results with ANP (Wu & Wang, 2021), we follow the source code implementation 8 .
Revised: For experimental results with ANP (Wu & Wang, 2021), we follow the source code implementation 8 .

Original: After creating each of the above mentioned attacks, we apply adversarial neural pruning on the backdoor model for 500 epochs with a learning rate of 0.02.
Revised: After creating each of the above mentioned attacks, we apply adversarial neural pruning on the backdoor model for 500 epochs with a learning rate of 0.02.

Original: We use the default settings for all attacks.
Revised: We use the default settings for all attacks.

Original: For vanilla FT, we perform simple DNN fine-tuning with a learning rate of 0.01 for 125 epochs.
Revised: For vanilla FT, we perform simple DNN fine-tuning with a learning rate of 0.01 for 125 epochs.

Original: We higher number of epochs for FT due to its poor clean test performance.
Revised: We higher number of epochs for FT due to its poor clean test performance.

Original: The clean validation size is 1% for both of these methods.
Revised: The clean validation size is 1% for both of these methods.

Original: For I-BAU (Zeng et al., 2021), we follow their PyTorch Implementationand purify the model for 10 epochs.
Revised: For I-BAU (Zeng et al., 2021), we follow their PyTorch Implementationand purify the model for 10 epochs.

Original: We use 5% validation data for I-BAU.
Revised: We use 5% validation data for I-BAU.

Original: For AWM (Chai &
Revised: For AWM (Chai &

Original: Chen, 2022), we train the model for 100 epochs and use Adam optimizer with a learning rate of 0.01 and a wight decay of 0.001.
Revised: Chen, 2022), we train the model for 100 epochs and use Adam optimizer with a learning rate of 0.01 and a wight decay of 0.001.

Original: We use the default hyper-parameter setting as described in their work α = 0 .
Revised: We use the default hyper-parameter setting as described in their work α = 0 .

Original: 9 , β = 0 .
Revised: 9 , β = 0 .

Original: 1 , γ = 108 , η = 1000 .
Revised: 1 , γ = 108 , η = 1000 .

Original: Above settings is for CIFAR10 and GTSRB only.
Revised: Above settings is for CIFAR10 and GTSRB only.

Original: For
Revised: For

Original: Tiny-ImageNet, we keep most of the training settings similar except reducing the number of epochs significantly.
Revised: Tiny-ImageNet, we keep most of the training settings similar except reducing the number of epochs significantly.

Original: We also increase the validation size to 5% for vanilla FT, ANP, and AWM.
Revised: We also increase the validation size to 5% for vanilla FT, ANP, and AWM.

Original: For I-BAU, we use a higher valiadtion size of 10%.
Revised: For I-BAU, we use a higher valiadtion size of 10%.

Original: For purification, we apply ANP and AWM for 30 epochs,
Revised: For purification, we apply ANP and AWM for 30 epochs,

Original: I-BAU for 5 epochs and Vanilla FT for 25 epochs.
Revised: I-BAU for 5 epochs and Vanilla FT for 25 epochs.

Original: For ImageNet, we use a 3% validation size for all defenses (except for I-BAU, we use 5% validation data) and use different number of purification
Revised: For ImageNet, we use a 3% validation size for all defenses (except for I-BAU, we use 5% validation data) and use different number of purification

Original: [Table] [Table]
Revised: [Table] [Table]

Original: epochs for different methods.
Revised: epochs for different methods.

Original: We apply I-BAU for 2 epochs.
Revised: We apply I-BAU for 2 epochs.

Original: On the other hand, we train the model for 3 epochs for ANP, AWM and vanilla FT.
Revised: On the other hand, we train the model for 3 epochs for ANP, AWM and vanilla FT.

Original: E C OMPARISON WITH A DDITIONAL D EFENSE
Revised: E C OMPARISON WITH A DDITIONAL D EFENSE

Original: In Table
Revised: In Table

Original: [Table]
Revised: [Table]

Original: we show the comparison of NGF with a recently proposed defense technique based on channel Lipschitzness Pruning (CLP) (Zheng et al., 2022) that works without any data.
Revised: we show the comparison of NGF with a recently proposed defense technique based on channel Lipschitzness Pruning (CLP) (Zheng et al., 2022) that works without any data.

Original: We follow the
Revised: We follow the

Original: Github link 10 .
Revised: Github link 10 .

Original: Based on the trigger-activated change on channel activation, CLP prunes channel.
Revised: Based on the trigger-activated change on channel activation, CLP prunes channel.

Original: One disadvantage of pruning based method is that in case of challenging scenarios, e
Revised: One disadvantage of pruning based method is that in case of challenging scenarios, e

Original: [Table]
Revised: [Table]

Original: g. ,, strong attacks, large datasets and models etc. , it prunes neurons abruptly.
Revised: g. ,, strong attacks, large datasets and models etc. , it prunes neurons abruptly.

Original: This creates high possibility of pruning neurons sensible to clean data distribution.
Revised: This creates high possibility of pruning neurons sensible to clean data distribution.

Original: In turn, the clean test accuracy may decrease significantly in some scenario.
Revised: In turn, the clean test accuracy may decrease significantly in some scenario.

Original: As shown in Table 9, clean accuracies (ACCs) for Blend, Trojan and CLB attacks are much lower compared to NGF.
Revised: As shown in Table 9, clean accuracies (ACCs) for Blend, Trojan and CLB attacks are much lower compared to NGF.

Original: Even though CLP performs reasonably well at removing backdoor,
Revised: Even though CLP performs reasonably well at removing backdoor,

Original: NGF still outperforms in that area.
Revised: NGF still outperforms in that area.

Original: F M ORE ON S MOOTHNESS A NALYSIS
Revised: F M ORE ON S MOOTHNESS A NALYSIS

Original: For smoothness analysis, we follow the PyHessian implementation 11 and modify it according to our needs.
Revised: For smoothness analysis, we follow the PyHessian implementation 11 and modify it according to our needs.

Original: We use a single batch with size 200 to calculate the loss Hessian for all attacks with CIFAR and GTSRB datasets.
Revised: We use a single batch with size 200 to calculate the loss Hessian for all attacks with CIFAR and GTSRB datasets.

Original: We conduct further smoothness analysis for ImageNet dataset and different architectures.
Revised: We conduct further smoothness analysis for ImageNet dataset and different architectures.

Original: In Fig.
Revised: In Fig.

Original: 5, we show the Eigen density plots for different 5 different attacks.
Revised: 5, we show the Eigen density plots for different 5 different attacks.

Original: We used
Revised: We used

Original: A40 GPUs with 96GB system memory.
Revised: A40 GPUs with 96GB system memory.

Original: However, it was not enough to calculate the loss hessian if we consider all 1000 classes of ImageNet.
Revised: However, it was not enough to calculate the loss hessian if we consider all 1000 classes of ImageNet.

Original: Due to GPU memory constraint, we consider ImageNet subset with 12 classes.
Revised: Due to GPU memory constraint, we consider ImageNet subset with 12 classes.

Original: We train a ResNet34 architecture with 5 different attacks.
Revised: We train a ResNet34 architecture with 5 different attacks.

Original: To calculate the loss hessian, we use a batch size of 50.
Revised: To calculate the loss hessian, we use a batch size of 50.

Original: Density plots before and after purification further confirms our proposed hypothesis.
Revised: Density plots before and after purification further confirms our proposed hypothesis.

Original: To test our hypothesis for larger architectures, we consider 5 different architectures for CIFAR10, i.e. , VGG19 (Simonyan & Zisserman, 2014), MobileNetV2 (Sandler et al., 2018), DenseNet121 (Huang et al., 2017), GoogleNet (Szegedy et al., 2014), Inception-V3 (Szegedy et al., 2016).
Revised: To test our hypothesis for larger architectures, we consider 5 different architectures for CIFAR10, i.e. , VGG19 (Simonyan & Zisserman, 2014), MobileNetV2 (Sandler et al., 2018), DenseNet121 (Huang et al., 2017), GoogleNet (Szegedy et al., 2014), Inception-V3 (Szegedy et al., 2016).

Original: Each of the architectures is deeper compared to the ResNet18 architecture we consider for CIFAR10.
Revised: Each of the architectures is deeper compared to the ResNet18 architecture we consider for CIFAR10.

Original: Due to their large size, showing the effectiveness of NGF in case of these architecture will strengthen our claim— one layer NGF based fine-tuning is enough for backdoor purification.
Revised: Due to their large size, showing the effectiveness of NGF in case of these architecture will strengthen our claim— one layer NGF based fine-tuning is enough for backdoor purification.

Original: In
Revised: In

Original: Fig.
Revised: Fig.

Original: 6, we show the performance of NGF when backdoor models are created using these architectures.
Revised: 6, we show the performance of NGF when backdoor models are created using these architectures.

Original: Our proposed one layer fine-tuning successfully removes the backdoor in all of these scenarios.
Revised: Our proposed one layer fine-tuning successfully removes the backdoor in all of these scenarios.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: G L ABEL C ORRECTION R ATE OF D IFFERENT D EFENSES
Revised: G L ABEL C ORRECTION R ATE OF D IFFERENT D EFENSES

Original: In standard removal measurement, it is sufficient for backdoored images to be classified as a nontarget class.
Revised: In standard removal measurement, it is sufficient for backdoored images to be classified as a nontarget class.

Original: As we calculate ASR after removal, our evaluation follows standard measurement.
Revised: As we calculate ASR after removal, our evaluation follows standard measurement.

Original: We define the correction rate (CR) as the percentage of poisonous samples correctly classified to their original classes.
Revised: We define the correction rate (CR) as the percentage of poisonous samples correctly classified to their original classes.

Original: We define the method with the highest value of CR as the best performing or SOTA method.
Revised: We define the method with the highest value of CR as the best performing or SOTA method.

Original: We use the CIFAR10 dataset and 4 different attacks for demonstration.
Revised: We use the CIFAR10 dataset and 4 different attacks for demonstration.

Original: It can be observed from Table 10 that our method obtains SOTA correction performance for most of these attacks.
Revised: It can be observed from Table 10 that our method obtains SOTA correction performance for most of these attacks.

Original: H R UNTIME A NALYSIS OF O THER D EFENSES
Revised: H R UNTIME A NALYSIS OF O THER D EFENSES

Original: In Table 11, we show the average runtime of other defenses.
Revised: In Table 11, we show the average runtime of other defenses.

Original: It can be observed that ANP is almost 6 x slower than NGF.
Revised: It can be observed that ANP is almost 6 x slower than NGF.

Original: Other defenses, NAD and MCR are also much slower than NGF.
Revised: Other defenses, NAD and MCR are also much slower than NGF.

Original: NAD uses transfer learning based distillation using a teacher-student framework.
Revised: NAD uses transfer learning based distillation using a teacher-student framework.

Original: However, the complexity of this method results in computational overhead.
Revised: However, the complexity of this method results in computational overhead.

Original: Instead, NGF revisits much simpler fine-tuning approach from one-layer optimization point of view.
Revised: Instead, NGF revisits much simpler fine-tuning approach from one-layer optimization point of view.

Original: Our simple and effective method leads to one of the fastest purification.
Revised: Our simple and effective method leads to one of the fastest purification.

Original: We take the average of all run times against 11 attacks on CIFAR10.
Revised: We take the average of all run times against 11 attacks on CIFAR10.

Original: Note that, for each epoch in NGF, we have to feed-forward all validation data.
Revised: Note that, for each epoch in NGF, we have to feed-forward all validation data.

Original: However, we only update the parameters of last layer through back-propagation.
Revised: However, we only update the parameters of last layer through back-propagation.

Original: The reason behind this is that we use different data augmentations while fine-tuning.
Revised: The reason behind this is that we use different data augmentations while fine-tuning.

Original: This does not allow us to save the CNN features one time and re-use it for upcoming all epochs.
Revised: This does not allow us to save the CNN features one time and re-use it for upcoming all epochs.

Original: I F INE - TUNING A LL L AYERS
Revised: I F INE - TUNING A LL L AYERS

Original: We have considered fine-tuning all layers fusing NGF and SGD.
Revised: We have considered fine-tuning all layers fusing NGF and SGD.

Original: Note that vanilla FT does fine-tune all layers.
Revised: Note that vanilla FT does fine-tune all layers.

Original: We report the performance of NGF for all layers in Table 12.
Revised: We report the performance of NGF for all layers in Table 12.

Original: While fine-tuning all layers seems to improve the performance, it takes almost 6 × more computational time than NGF on last layer.
Revised: While fine-tuning all layers seems to improve the performance, it takes almost 6 × more computational time than NGF on last layer.

Original: We perform NGF with the regularizer here.
Revised: We perform NGF with the regularizer here.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: J E FFECT OF C LEAN V ALIDATION D ATA S IZE
Revised: J E FFECT OF C LEAN V ALIDATION D ATA S IZE

Original: We also present how the total number of clean validation data can impact the purification performance.
Revised: We also present how the total number of clean validation data can impact the purification performance.

Original: In Table
Revised: In Table

Original: [Table]
Revised: [Table]

Original: we see the change in performance while gradually reducing the validation size from 1% to 0.1%.
Revised: we see the change in performance while gradually reducing the validation size from 1% to 0.1%.

Original: We consider Badnets attack on CIFAR10 dataset for this evaluation.
Revised: We consider Badnets attack on CIFAR10 dataset for this evaluation.

Original: Even with only(0.1%) data points, NGF can successfully remove the backdoor by bringing down the attack success rate (ASR) to 6.91%.
Revised: Even with only(0.1%) data points, NGF can successfully remove the backdoor by bringing down the attack success rate (ASR) to 6.91%.

Original: We also consider adversarial weight masking for this comparison.
Revised: We also consider adversarial weight masking for this comparison.

Original: For both
Revised: For both

Original: ANP and AWM, reducing the validation size has severe impact on the test accuracy (ACC).
Revised: ANP and AWM, reducing the validation size has severe impact on the test accuracy (ACC).

Original: K P URIFICATION U SING L AST L AYER R E -I NITIALIZATION
Revised: K P URIFICATION U SING L AST L AYER R E -I NITIALIZATION

Original: We also conduct studies on the behavioral difference of SGD and NGD while we re-initialize the last layer.
Revised: We also conduct studies on the behavioral difference of SGD and NGD while we re-initialize the last layer.

Original: Even though we re-initialize the last layer, one still has to properly fine-tune the backdoor model to remove the backdoor.
Revised: Even though we re-initialize the last layer, one still has to properly fine-tune the backdoor model to remove the backdoor.

Original: However, we see throughout our evaluations that SGD-based onelayer fine-tuning is not a proper fine-tuning method and unable to remove backdoor.
Revised: However, we see throughout our evaluations that SGD-based onelayer fine-tuning is not a proper fine-tuning method and unable to remove backdoor.

Original: In case of re-initialization, the shallow network optimization problem still stands as wells as the issue of bad local minima.
Revised: In case of re-initialization, the shallow network optimization problem still stands as wells as the issue of bad local minima.

Original: Therefore, SGD shows similar behavior in this scenario too.
Revised: Therefore, SGD shows similar behavior in this scenario too.

Original: Table 13 shows the performance of SGD and NGF in case one decides to re-initialize the layer than fine-tunes.
Revised: Table 13 shows the performance of SGD and NGF in case one decides to re-initialize the layer than fine-tunes.

Original: As usual,
Revised: As usual,

Original: NGF is able to reach smooth minima due to its ability to properly fine-tune the model.
Revised: NGF is able to reach smooth minima due to its ability to properly fine-tune the model.

Original: L M ORE A LL 2A LL A TTACKS
Revised: L M ORE A LL 2A LL A TTACKS

Original: Most of the defenses evaluate their methods on only All2One attack where we consider only one target label.
Revised: Most of the defenses evaluate their methods on only All2One attack where we consider only one target label.

Original: However, there can be multiple target classes in a practical attack scenario.
Revised: However, there can be multiple target classes in a practical attack scenario.

Original: We consider one such case: All2All attack where target classes are uniformly distributed among all available classes.
Revised: We consider one such case: All2All attack where target classes are uniformly distributed among all available classes.

Original: In Table 15, we show the performance under such settings for 3 different attacks with a poison rate of 10% .
Revised: In Table 15, we show the performance under such settings for 3 different attacks with a poison rate of 10% .

Original: It shows that All2All attack is more challenging to defend against as compared to All2One attack.
Revised: It shows that All2All attack is more challenging to defend against as compared to All2One attack.

Original: However, the performance of NGF seems to be consistently better than other defenses for both of these attack variations.
Revised: However, the performance of NGF seems to be consistently better than other defenses for both of these attack variations.

Original: For reference, we achieve an ASR improvement of 3 .
Revised: For reference, we achieve an ASR improvement of 3 .

Original: 12% over ANP while maintaining a lead in classification accuracy too.
Revised: 12% over ANP while maintaining a lead in classification accuracy too.

Original: M C OMBINING D IFFERENT B ACKDOOR A TTACKS
Revised: M C OMBINING D IFFERENT B ACKDOOR A TTACKS

Original: We also perform experiments with combined backdoor attacks.
Revised: We also perform experiments with combined backdoor attacks.

Original: To create such attacks, we poison some portion of the training data using 3 different attacks; Badnets, Blend, and Trojan.
Revised: To create such attacks, we poison some portion of the training data using 3 different attacks; Badnets, Blend, and Trojan.

Original: Each of these
Revised: Each of these

Original: [Figure]
Revised: [Figure]

Original: attacks have an equal share in the poison data.
Revised: attacks have an equal share in the poison data.

Original: As shown in Table 16, we use 4 different poison rates: 10% ∼ 50% .
Revised: As shown in Table 16, we use 4 different poison rates: 10% ∼ 50% .

Original: NGF outperforms other baseline methods (MCR and ANP) by a satisfactory margin.
Revised: NGF outperforms other baseline methods (MCR and ANP) by a satisfactory margin.

Original: N D ECISION H EATMAPS : H OW NGF REMOVES B ACKDOOR ?
Revised: N D ECISION H EATMAPS : H OW NGF REMOVES B ACKDOOR ?

Original: While inserting the backdoor behavior, the model, especially the linear classification layer, memorizes the poison data distribution.
Revised: While inserting the backdoor behavior, the model, especially the linear classification layer, memorizes the poison data distribution.

Original: By memorization, we mean it memorizes the simpler trigger pattern.
Revised: By memorization, we mean it memorizes the simpler trigger pattern.

Original: Whenever the model sees that pattern in the input, it prioritizes the trigger-specific feature instead of image-specific (clean part) feature and predicts the adversary-set target label.
Revised: Whenever the model sees that pattern in the input, it prioritizes the trigger-specific feature instead of image-specific (clean part) feature and predicts the adversary-set target label.

Original: When we re-train or fine-tune the classifier with clean validation data, the classifier forgets the poison distribution as finetuning reinforces the dominance of clean features in model prediction.
Revised: When we re-train or fine-tune the classifier with clean validation data, the classifier forgets the poison distribution as finetuning reinforces the dominance of clean features in model prediction.

Original: After fine-tuning, the model looks for image-specific features for prediction as it has almost no memory of the trigger-specific features.
Revised: After fine-tuning, the model looks for image-specific features for prediction as it has almost no memory of the trigger-specific features.

Original: We illustrate the decision heat-maps for clean, backdoor and purified model in Figure 4.
Revised: We illustrate the decision heat-maps for clean, backdoor and purified model in Figure 4.

Original: We show the decision heatmaps for clean an poison data.
Revised: We show the decision heatmaps for clean an poison data.

Original: As clean model is only trained on clean data, it is not sensitive to the trigger.
Revised: As clean model is only trained on clean data, it is not sensitive to the trigger.

Original: Our defense objective says that, a purified model should behave like a benign model, i.e. , the decision making process (for clean and poison data) should resemble a clean model.
Revised: Our defense objective says that, a purified model should behave like a benign model, i.e. , the decision making process (for clean and poison data) should resemble a clean model.

Original: As we can see for the poison data, NGF successfully removes the effect of the trigger.
Revised: As we can see for the poison data, NGF successfully removes the effect of the trigger.

Original: The purified model ignores the trigger while making decisions.
Revised: The purified model ignores the trigger while making decisions.

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table] 
Revised: [Table] [Figure] [Table] [Figure]

Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table] 
Revised: [Table] [Figure] [Table] [Figure]

Revised: [Table]

Original: [Figure] 
Revised: [Figure] [Table] [Figure] [Table]

Original:  [Table]
Revised: [Figure] [Table] [Figure] [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table] [Figure] 
Revised: [Table] [Figure] [Table]

Original: [Table] [Figure] [Table] [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure] [Table] [Figure] [Table]
Revised: [Table]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure][Table] [Figure] [Figure]
Revised: [Figure]