Original: Make Sharpness-Aware Minimization Stronger: A Sparsified Perturbation Approach
Revised: Make Sharpness-Aware Minimization Stronger: A Sparsified Perturbation Approach

Original: Abstract
Revised: Abstract

Original: Deep neural networks often suffer from poor generalization caused by complex and non-convex loss landscapes.
Revised: Deep neural networks often suffer from poor generalization caused by complex and non-convex loss landscapes.

Original: One of the popular solutions is Sharpness-Aware Minimization (SAM), which smooths the loss landscape via minimizing the maximized change of training loss when adding a perturbation to the weight.
Revised: One of the popular solutions is Sharpness-Aware Minimization (SAM), which smooths the loss landscape via minimizing the maximized change of training loss when adding a perturbation to the weight.

Original: However, we find the indiscriminate perturbation of SAM on all parameters is suboptimal, which also results in excessive computation, i.e. , double the overhead of common optimizers like Stochastic Gradient Descent (SGD).
Revised: However, we find the indiscriminate perturbation of SAM on all parameters is suboptimal, which also results in excessive computation, i.e. , double the overhead of common optimizers like Stochastic Gradient Descent (SGD).

Original: In this paper, we propose an efficient and effective training scheme coined as Sparse SAM (SSAM), which achieves sparse perturbation by a binary mask.
Revised: In this paper, we propose an efficient and effective training scheme coined as Sparse SAM (SSAM), which achieves sparse perturbation by a binary mask.

Original: To obtain the sparse mask, we provide two solutions which are based onFisher information and dynamic sparse training, respectively.
Revised: To obtain the sparse mask, we provide two solutions which are based onFisher information and dynamic sparse training, respectively.

Original: In addition, we theoretically prove that SSAM can converge at the same rate as SAM, i.e. , O (log T/ √ T ) .
Revised: In addition, we theoretically prove that SSAM can converge at the same rate as SAM, i.e. , O (log T/ √ T ) .

Original: Sparse SAM not only has the potential for training acceleration but also smooths the loss landscape effectively.
Revised: Sparse SAM not only has the potential for training acceleration but also smooths the loss landscape effectively.

Original: Extensive experimental results on CIFAR10, CIFAR100, and ImageNet-1K confirm the superior efficiency of our method to SAM, and the performance is preserved or even better with a perturbation of merely 50% sparsity.
Revised: Extensive experimental results on CIFAR10, CIFAR100, and ImageNet-1K confirm the superior efficiency of our method to SAM, and the performance is preserved or even better with a perturbation of merely 50% sparsity.

Original: Code is availiable at https: //github.com/
Revised: Code is available at https: //github.com/

Original: Mi-Peng/Sparse-Sharpness-Aware-Minimization .
Revised: Mi-Peng/Sparse-Sharpness-Aware-Minimization .

Original:  Introduction
Revised:  Introduction

Original: Over the past decade or so, the great success of deep learning has been due in great part to ever-larger model parameter sizes [13, 57, 53, 10, 40, 5].
Revised: Over the past decade or so, the great success of deep learning has been due in great part to ever-larger model parameter sizes [13, 57, 53, 10, 40, 5].

Original: However, the excessive parameters also make the model inclined to poor generalization.
Revised: However, the excessive parameters also make the model inclined to poor generalization.

Original: To overcome this problem, numerous efforts have been devoted to training algorithm [24, 42, 51], data augmentation [11, 58, 56], and network design [26, 28].
Revised: To overcome this problem, numerous efforts have been devoted to training algorithm [24, 42, 51], data augmentation [11, 58, 56], and network design [26, 28].

Original: One important finding in recent research is the connection between the geometry of loss landscape and model generalization [31, 17, 27, 44, 54].
Revised: One important finding in recent research is the connection between the geometry of loss landscape and model generalization [31, 17, 27, 44, 54].

Original: In general, the loss landscape of the model is complex and non-convex, which makes model tend to converge to sharp minima.
Revised: In general, the loss landscape of the model is complex and non-convex, which makes model tend to converge to sharp minima.

Original: Recent endeavors [31, 27, 44] show that the flatter the minima of convergence, the better the model generalization.
Revised: Recent endeavors [31, 27, 44] show that the flatter the minima of convergence, the better the model generalization.

Original: This discovery reveals the nature of previous approaches [24, 26, 11, 56, 58, 28] to improve generalization, i.e. , smoothing the loss landscape.
Revised: This discovery reveals the nature of previous approaches [24, 26, 11, 56, 58, 28] to improve generalization, i.e. , smoothing the loss landscape.

Original: Based on this finding, Foret et al.
Revised: Based on this finding, Foret et al.

Original: [17] propose a novel approach to improve model generalization called sharpness-aware minimization (SAM), which simultaneously minimizes loss value and loss sharpness.
Revised: [17] propose a novel approach to improve model generalization called sharpness-aware minimization (SAM), which simultaneously minimizes loss value and loss sharpness.

Original: SAM quantifies the landscape sharpness as the maximized difference of loss when a perturbation is added to the weight.
Revised: SAM quantifies the landscape sharpness as the maximized difference of loss when a perturbation is added to the weight.

Original: When the model reaches a sharp area, the perturbed gradients in
Revised: When the model reaches a sharp area, the perturbed gradients in

Original: SAM help the model jump out of the sharp minima.
Revised: SAM help the model jump out of the sharp minima.

Original: In practice, SAM requires two forward-backward computations for each optimization step, where the first computation is to obtain the perturbation and the second one is for parameter update.
Revised: In practice, SAM requires two forward-backward computations for each optimization step, where the first computation is to obtain the perturbation and the second one is for parameter update.

Original: Despite the remarkable performance [17, 34, 14, 7], This property makes SAM double the computational cost of the conventional optimizer, e.g. , SGD [3].
Revised: Despite the remarkable performance [17, 34, 14, 7], This property makes SAM double the computational cost of the conventional optimizer, e.g. , SGD [3].

Original: Since SAM calculates perturbations indiscriminately for all parameters, a question is arisen:
Revised: Since SAM calculates perturbations indiscriminately for all parameters, a question is arisen:

Original: Do we need to calculate perturbations for all parameters?
Revised: Do we need to calculate perturbations for all parameters?

Original: Above all, we notice that in most deep neural networks, only about 5% of parameters are sharp and rise steeply during optimization [31].
Revised: Above all, we notice that in most deep neural networks, only about 5% of parameters are sharp and rise steeply during optimization [31].

Original: Then we explore the effect of SAM in different dimensions to answer the above question and find out (i) little difference between SGD and SAM gradients in most dimensions (see Fig. 1); (ii) more flatter without SAM in some dimensions (see Fig.
Revised: Then we explore the effect of SAM in different dimensions to answer the above question and find out (i) little difference between SGD and SAM gradients in most dimensions (see Fig. 1); (ii) more flatter without SAM in some dimensions (see Fig.

Original: and Fig. 5).
Revised: and Fig. 5).

Original: Inspired by the above discoveries, we propose a novel scheme to improve the efficiency of SAM via sparse perturbation, termed Sparse SAM (SSAM).
Revised: Inspired by the above discoveries, we propose a novel scheme to improve the efficiency of SAM via sparse perturbation, termed Sparse SAM (SSAM).

Original: SSAM, which plays the role of regularization, has better generalization, and its sparse operation also ensures the efficiency of optimization.
Revised: SSAM, which plays the role of regularization, has better generalization, and its sparse operation also ensures the efficiency of optimization.

Original: Specifically, the perturbation in SSAM is multiplied by a binary sparse mask to determine which parameters should be perturbed.
Revised: Specifically, the perturbation in SSAM is multiplied by a binary sparse mask to determine which parameters should be perturbed.

Original: To obtain the sparse mask, we provide two implementations.
Revised: To obtain the sparse mask, we provide two implementations.

Original: The first solution is to use Fisher information [16] of the parameters to formulate the binary mask, dubbed SSAM-F.
Revised: The first solution is to use Fisher information [16] of the parameters to formulate the binary mask, dubbed SSAM-F.

Original: The other one is to employ dynamic sparse training to jointly optimize model parameters and the sparse mask, dubbed
Revised: The other one is to employ dynamic sparse training to jointly optimize model parameters and the sparse mask, dubbed

Original: SSAM-D.
Revised: SSAM-D.

Original: The first solution is relatively more stable but a bit time-consuming, while the latter is more efficient.
Revised: The first solution is relatively more stable but a bit time-consuming, while the latter is more efficient.

Original: In addition to these solutions, we provide the theoretical convergence analysis of SAM and SSAM in non-convex stochastic setting, proving that our SSAM can converge at the same rate as SAM, i.e. ,
Revised: In addition to these solutions, we provide the theoretical convergence analysis of SAM and SSAM in non-convex stochastic setting, proving that our SSAM can converge at the same rate as SAM, i.e. ,

Original: O (log T/ √ T ) .
Revised: O (log T/ √ T ) .

Original: At last, we evaluate the performance and effectiveness of SSAM on CIFAR10 [33],
Revised: At last, we evaluate the performance and effectiveness of SSAM on CIFAR10 [33],

Original: CIFAR100 [33] and ImageNet [8] with various models.
Revised: CIFAR100 [33] and ImageNet [8] with various models.

Original: The experiments confirm that SSAM contributes to a flatter landscape than SAM, and its performance is on par with or even better than
Revised: The experiments confirm that SSAM contributes to a flatter landscape than SAM, and its performance is on par with or even better than

Original: SAM with only about 50% perturbation.
Revised: SAM with only about 50% perturbation.

Original: These results coincide with our motivations and findings.
Revised: These results coincide with our motivations and findings.

Original: To sum up, the contribution of this paper is three-fold:
Revised: To sum up, the contribution of this paper is three-fold:

Original: • We rethink the role of perturbation in SAM and find that the indiscriminate perturbations are suboptimal and computationally inefficient.
Revised: • We rethink the role of perturbation in SAM and find that the indiscriminate perturbations are suboptimal and computationally inefficient.

Original: • We propose a sparsified perturbation approach called Sparse SAM (SSAM) with two variants,
Revised: • We propose a sparsified perturbation approach called Sparse SAM (SSAM) with two variants,

Original: i
Revised: i

Original: e. , Fisher SSAM (SSAM-F) and Dynamic SSAM (SSAM-D), both of which enjoy better efficiency and effectiveness than SAM.
Revised: e. , Fisher SSAM (SSAM-F) and Dynamic SSAM (SSAM-D), both of which enjoy better efficiency and effectiveness than SAM.

Original: We also theoretically prove that SSAM can converge at the same rate as SAM,
Revised: We also theoretically prove that SSAM can converge at the same rate as SAM,

Original: i
Revised: i

Original: e. , O (log T/ √ T ) .
Revised: e. , O (log T/ √ T ) .

Original: • We evaluate SSAM with various models on CIFAR and ImageNet, showing WideResNet with SSAM of a high sparsity outperforms SAM on CIFAR; SSAM can achieve competitive performance with a high sparsity; SSAM has a comparable convergence rate to SAM.
Revised: • We evaluate SSAM with various models on CIFAR and ImageNet, showing WideResNet with SSAM of a high sparsity outperforms SAM on CIFAR; SSAM can achieve competitive performance with a high sparsity; SSAM has a comparable convergence rate to SAM.

Original:  Related Work
Revised:  Related Work

Original: In this section, we briefly review the studies on sharpness-aware minimum optimization (SAM),
Revised: In this section, we briefly review the studies on sharpness-aware minimum optimization (SAM),

Original: Fisher information in deep learning, and dynamic sparse training.
Revised: Fisher information in deep learning, and dynamic sparse training.

Original: SAM and flat minima.
Revised: SAM and flat minima.

Original: Hochreiter et al.
Revised: Hochreiter et al.

Original: [27] first reveal that there is a strong correlation between the generalization of a model and the flat minima.
Revised: [27] first reveal that there is a strong correlation between the generalization of a model and the flat minima.

Original: After that, there is a growing amount of research based on this finding.
Revised: After that, there is a growing amount of research based on this finding.

Original: Keskar et al.
Revised: Keskar et al.

Original: [31] conduct experiments with a larger batch size, and in consequence observe the degradation of model generalization capability.
Revised: [31] conduct experiments with a larger batch size, and in consequence observe the degradation of model generalization capability.

Original: They [31] also confirm the essence of this phenomenon, which is that the model tends to converge to the sharp minima.
Revised: They [31] also confirm the essence of this phenomenon, which is that the model tends to converge to the sharp minima.

Original: Keskar et al.
Revised: Keskar et al.

Original: [31] and
Revised: [31] and

Original: Dinh et al.
Revised: Dinh et al.

Original: [12] state that the sharpness can be evaluated by the eigenvalues of the Hessian.
Revised: [12] state that the sharpness can be evaluated by the eigenvalues of the Hessian.

Original: However, they fail to find the flat minima due to the notorious computational cost of Hessian.
Revised: However, they fail to find the flat minima due to the notorious computational cost of Hessian.

Original: Inspired by this, Foret et al.
Revised: Inspired by this, Foret et al.

Original: [17] introduce a sharpness-aware optimization (SAM) to find a flat minimum for improving generalization capability, which is achieved by solving a mini-max problem.
Revised: [17] introduce a sharpness-aware optimization (SAM) to find a flat minimum for improving generalization capability, which is achieved by solving a mini-max problem.

Original: Zhang et al.
Revised: Zhang et al.

Original: [59] make a point that SAM [17] is equivalent to adding the regularization of the gradient norm by approximating Hessian matrix.
Revised: [59] make a point that SAM [17] is equivalent to adding the regularization of the gradient norm by approximating Hessian matrix.

Original: Kwon et al.
Revised: Kwon et al.

Original: [34] propose a scale-invariant SAM scheme with adaptive radius to improve training stability.
Revised: [34] propose a scale-invariant SAM scheme with adaptive radius to improve training stability.

Original: Zhang et al.
Revised: Zhang et al.

Original: [60] redefine the landscape sharpness from an intuitive and theoretical perspective based on SAM.
Revised: [60] redefine the landscape sharpness from an intuitive and theoretical perspective based on SAM.

Original: To reduce the computational cost in
Revised: To reduce the computational cost in

Original: SAM, Du et al.
Revised: SAM, Du et al.

Original: [14] proposed Efficient SAM (ESAM) to randomly calculate perturbation.
Revised: [14] proposed Efficient SAM (ESAM) to randomly calculate perturbation.

Original: However,
Revised: However,

Original: ESAM randomly select the samples every steps, which may lead to optimization bias.
Revised: ESAM randomly select the samples every steps, which may lead to optimization bias.

Original: Instead of the perturbations for all parameters, i.e. , SAM, we compute a sparse perturbation, i.e. , SSAM, which learns important but sparse dimensions for perturbation.
Revised: Instead of the perturbations for all parameters, i.e. , SAM, we compute a sparse perturbation, i.e. , SSAM, which learns important but sparse dimensions for perturbation.

Original: Fisher information (FI).
Revised: Fisher information (FI).

Original: Fisher information [16] was proposed to measure the information that an observable random variable carries about an unknown parameter of a distribution.
Revised: Fisher information [16] was proposed to measure the information that an observable random variable carries about an unknown parameter of a distribution.

Original: In machine learning, Fisher information is widely used to measure the importance of the model parameters [32] and decide which parameter to be pruned [50, 52].
Revised: In machine learning, Fisher information is widely used to measure the importance of the model parameters [32] and decide which parameter to be pruned [50, 52].

Original: For proposed SSAM-F, Fisher information is used to determine whether a weight should be perturbed for flat minima.
Revised: For proposed SSAM-F, Fisher information is used to determine whether a weight should be perturbed for flat minima.

Original: Dynamic sparse training.
Revised: Dynamic sparse training.

Original: Finding the sparse network via pruning unimportant weights is a popular solution in network compression, which can be traced back to decades [35].
Revised: Finding the sparse network via pruning unimportant weights is a popular solution in network compression, which can be traced back to decades [35].

Original: The widely used training scheme, i.e. , pretraining-pruning-fine-tuning, is presented by Han et.al.
Revised: The widely used training scheme, i.e. , pretraining-pruning-fine-tuning, is presented by Han et.al.

Original: Limited by the requirement for the pre-trained model, some recent research [15, 2, 9, 30, 43, 37, 38] attempts to discover a sparse network directly from the training process.
Revised: Limited by the requirement for the pre-trained model, some recent research [15, 2, 9, 30, 43, 37, 38] attempts to discover a sparse network directly from the training process.

Original: Dynamic Sparse Training (DST) finds the sparse structure by dynamic parameter reallocation.
Revised: Dynamic Sparse Training (DST) finds the sparse structure by dynamic parameter reallocation.

Original: The criterion of pruning could be weight magnitude [18], gradient [15] and Hessian [35, 49], etc .
Revised: The criterion of pruning could be weight magnitude [18], gradient [15] and Hessian [35, 49], etc .

Original: We claim that different from the existing
Revised: We claim that different from the existing

Original: DST methods that prune neurons, our target is to obtain a binary mask for sparse perturbation.
Revised: DST methods that prune neurons, our target is to obtain a binary mask for sparse perturbation.

Original:  Rethinking the Perturbation in SAM
Revised:  Rethinking the Perturbation in SAM

Original: In this section, we first review how SAM converges at the flat minimum of a loss landscape.
Revised: In this section, we first review how SAM converges at the flat minimum of a loss landscape.

Original: Then, we rethink the role of perturbation in SAM.
Revised: Then, we rethink the role of perturbation in SAM.

Original: 3.1 Preliminary
Revised: 3.1 Preliminary

Original: In this paper, we consider the weights of a deep neural network as w = ( w 1 , w 2 , ..., w d ) ⊆ W ∈ R d and denote a binary mask as m ∈ { 0 , 1 } d , which satisfies 1 T m =
Revised: In this paper, we consider the weights of a deep neural network as w = ( w 1 , w 2 , ..., w d ) ⊆ W ∈ R d and denote a binary mask as m ∈ { 0 , 1 } d , which satisfies 1 T m =

Original: (1 − s ) · d to restrict the computational cost.
Revised: (1 − s ) · d to restrict the computational cost.

Original: Given a training dataset as S ≜ { ( x i , y i ) } ni =1 i.i.d.
Revised: Given a training dataset as S ≜ { ( x i , y i ) } ni =1 i.i.d.

Original: drawn from the distribution D , the per-data-point loss function is defined by f ( w , x i , y i ) .
Revised: drawn from the distribution D , the per-data-point loss function is defined by f ( w , x i , y i ) .

Original: For the classification task in this paper, we use cross-entropy as loss function.
Revised: For the classification task in this paper, we use cross-entropy as loss function.

Original: The population loss is defined by f D
Revised: The population loss is defined by f D

Original: = E
Revised: = E

Original: ( x i , y i )
Revised: ( x i , y i )

Original: ∼D f ( w , x i , y i ) , while the empirical training loss function is f S ≜ 1 n (cid:80) ni =1 f ( w , x i , y i ) .
Revised: ∼D f ( w , x i , y i ) , while the empirical training loss function is f S ≜ 1 n (cid:80) ni =1 f ( w , x i , y i ) .

Original: Sharpness-aware minimization (SAM) [17] aims to simultaneously minimize the loss value and smooth the loss landscape, which is achieved by solving the min-max problem:
Revised: Sharpness-aware minimization (SAM) [17] aims to simultaneously minimize the loss value and smooth the loss landscape, which is achieved by solving the min-max problem:

Original: [Equation]
Revised: [Equation]

Original: SAM first obtains the perturbation ϵ in a neighborhood ball area with a radius denoted as ρ .
Revised: SAM first obtains the perturbation ϵ in a neighborhood ball area with a radius denoted as ρ .

Original: The optimization tries to minimize the loss of the perturbed weight w + ϵ .
Revised: The optimization tries to minimize the loss of the perturbed weight w + ϵ .

Original: Intuitively, the goal of SAM is that small perturbations to the weight will not significantly rise the empirical loss, which indicates that
Revised: Intuitively, the goal of SAM is that small perturbations to the weight will not significantly rise the empirical loss, which indicates that

Original: SAM tends to converge to a flat minimum.
Revised: SAM tends to converge to a flat minimum.

Original: To solve the mini-max optimization, SAM approximately calculates the perturbations ϵ using Taylor expansion around w :
Revised: To solve the mini-max optimization, SAM approximately calculates the perturbations ϵ using Taylor expansion around w :

Original: [Equation]
Revised: [Equation]

Original: In this way, the objective function can be rewritten as min w
Revised: In this way, the objective function can be rewritten as min w

Original: [Equation]
Revised: [Equation]

Original: ∇ w f ( w ) || 2 (cid:1) , which could be implemented by a two-step gradient descent framework in Pytorch or TensorFlow:
Revised: ∇ w f ( w ) || 2 (cid:1) , which could be implemented by a two-step gradient descent framework in Pytorch or TensorFlow:

Original: • In the first step, the gradient at w is used to calculate the perturbation ϵ by Eq.
Revised: • In the first step, the gradient at w is used to calculate the perturbation ϵ by Eq.

Original: Then the weight of model will be added to w + ϵ .
Revised: Then the weight of model will be added to w + ϵ .

Original: In the second step, the gradient at w + ϵ is used to solve min w f S ( w + ϵ ) ,
Revised: In the second step, the gradient at w + ϵ is used to solve min w f S ( w + ϵ ) ,

Original: i
Revised: i

Original: e. , update the weight w by this gradient.
Revised: e. , update the weight w by this gradient.

Original: 3.2 Rethinking the Perturbation Step of SAM
Revised: 3.2 Rethinking the Perturbation Step of SAM

Original: How does SAM work in flat subspace?
Revised: How does SAM work in flat subspace?

Original: SAM perturbs all parameters indiscriminately, but the fact is that merely about 5% parameter space is sharp while the rest is flat [31].
Revised: SAM perturbs all parameters indiscriminately, but the fact is that merely about 5% parameter space is sharp while the rest is flat [31].

Original: We are curious whether perturbing the parameters in those already flat dimensions would lead to the instability of the optimization and impair the improvement of generalization.
Revised: We are curious whether perturbing the parameters in those already flat dimensions would lead to the instability of the optimization and impair the improvement of generalization.

Original: To answer this question, we quantitatively and qualitatively analyze the loss landscapes with different training schemes in Section 5, as shown in Fig.
Revised: To answer this question, we quantitatively and qualitatively analyze the loss landscapes with different training schemes in Section 5, as shown in Fig.

Original: 4 and Fig.
Revised: 4 and Fig.

Original: The results confirm our conjecture that optimizing some dimensions without perturbation can help the model generalize better.
Revised: The results confirm our conjecture that optimizing some dimensions without perturbation can help the model generalize better.

Original: What is the difference between the gradients of SGD and SAM?
Revised: What is the difference between the gradients of SGD and SAM?

Original: We investigate various neural networks optimized with SAM and SGD on CIFAR10/100 and ImageNet, whose statistics are given in Fig.
Revised: We investigate various neural networks optimized with SAM and SGD on CIFAR10/100 and ImageNet, whose statistics are given in Fig.

Original: We use the relative difference ratio r , defined as r = log | ( g SAM − g SGD )
Revised: We use the relative difference ratio r , defined as r = log | ( g SAM − g SGD )

Original: /g SGD | , to measure the difference between the gradients of SAM and SGD.
Revised: /g SGD | , to measure the difference between the gradients of SAM and SGD.

Original: As showin in Fig.
Revised: As showin in Fig.

Original: 1, the parameters with r less than 0 account for the vast majority of all parameters, indicating that most SAM gradients are not significantly different from SGD gradients.
Revised: 1, the parameters with r less than 0 account for the vast majority of all parameters, indicating that most SAM gradients are not significantly different from SGD gradients.

Original: These results show that most parameters of the model require no perturbations for achieving the flat minima, which well confirms our the motivation.
Revised: These results show that most parameters of the model require no perturbations for achieving the flat minima, which well confirms our the motivation.

Original: Inspired by the above observation and the promising hardware acceleration for sparse operation modern GPUs, we further propose Sparse SAM, a novel sparse perturbation approach, as an implicit regularization to improve the efficiency and effectiveness of SAM.
Revised: Inspired by the above observation and the promising hardware acceleration for sparse operation modern GPUs, we further propose Sparse SAM, a novel sparse perturbation approach, as an implicit regularization to improve the efficiency and effectiveness of SAM.

Original:  Methodology
Revised:  Methodology

Original: In this section, we first define the proposed Sparse SAM (SSAM), which strengths SAM via sparse perturbation.
Revised: In this section, we first define the proposed Sparse SAM (SSAM), which strengths SAM via sparse perturbation.

Original: Afterwards, we introduce the instantiations of the sparse mask used in SSAM via Fisher information and dynamic sparse training, dubbed SSAM-F and SSAM-D, respectively.
Revised: Afterwards, we introduce the instantiations of the sparse mask used in SSAM via Fisher information and dynamic sparse training, dubbed SSAM-F and SSAM-D, respectively.

Original: 4.1 Sparse SAM
Revised: 4.1 Sparse SAM

Original: Motivated by the finding discussed in the introduction, Sparse SAM (SSAM) employs a sparse binary mask to decide which parameters should be perturbed, thereby improving the efficiency of sharpness-aware minimization.
Revised: Motivated by the finding discussed in the introduction, Sparse SAM (SSAM) employs a sparse binary mask to decide which parameters should be perturbed, thereby improving the efficiency of sharpness-aware minimization.

Original: Specifically, the perturbation ϵ will be multiplied by a sparse binary mask m , and the objective function is then rewritten as min w f S (cid:16) w +
Revised: Specifically, the perturbation ϵ will be multiplied by a sparse binary mask m , and the objective function is then rewritten as min w f S (cid:16) w +

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: To stable the optimization, the sparse binary mask m is updated at regular intervals during training.
Revised: To stable the optimization, the sparse binary mask m is updated at regular intervals during training.

Original: We provide two solutions to obtain the sparse mask m , namely Fisher information based Sparse SAM (SSAM-F) and dynamic sparse training based Sparse SAM (SSAM-D).
Revised: We provide two solutions to obtain the sparse mask m , namely Fisher information based Sparse SAM (SSAM-F) and dynamic sparse training based Sparse SAM (SSAM-D).

Original: The overall algorithms of
Revised: The overall algorithms of

Original: SSAM and sparse mask generations are shown in Algorithm 1 and Algorithm 2, respectively.
Revised: SSAM and sparse mask generations are shown in Algorithm 1 and Algorithm 2, respectively.

Original: According to the previous work [55] and Ampere architecture equipped with sparse tensor cores [47, 46, 45], currently there exists technical support for matrix multiplication with 50% fine-grained sparsity [55] 3 .
Revised: According to the previous work [55] and Ampere architecture equipped with sparse tensor cores [47, 46, 45], currently there exists technical support for matrix multiplication with 50% fine-grained sparsity [55] 3 .

Original: Therefore, SSAM of 50% sparse perturbation has great potential to achieve true training acceleration via sparse back-propagation.
Revised: Therefore, SSAM of 50% sparse perturbation has great potential to achieve true training acceleration via sparse back-propagation.

Original: 3 For instance, 2:4 sparsity for A100 GPU.
Revised: 3 For instance, 2:4 sparsity for A100 GPU.

Original: Input
Revised: Input

Original: Compute
Revised: Compute

Original: Generate mask m via if 9: m 10:
Revised: Generate mask m via if 9: m 10:

Original: 11: w w
Revised: 11: w w

Original: ) 12: end for 13: return Final w
Revised: ) 12: end for 13: return Final w

Original: [Table]
Revised: [Table]

Original: Information
Revised: Information

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: ArgTopK
Revised: ArgTopK

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: : m
Revised: : m

Original: [Table] [Table]
Revised: [Table] [Table]

Original: =
Revised: =

Original: [Table] [Table]
Revised: [Table] [Table]

Original: m
Revised: m

Original: [Table]
Revised: [Table]

Original: = {
Revised: = {

Original: [Table]
Revised: [Table]

Original: =
Revised: =

Original: [Table]
Revised: [Table]

Original: m
Revised: m

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: m
Revised: m

Original: [Table]
Revised: [Table]

Original: m
Revised: m

Original: [Table]
Revised: [Table]

Original: m
Revised: m

Original: [Table]
Revised: [Table]

Original: 14: return Sparse mask m
Revised: 14: return Sparse mask m

Original: 4.2 SSAM-F: Fisher information based Sparse SAM
Revised: 4.2 SSAM-F: Fisher information based Sparse SAM

Original: Inspired by the connection between Fisher information and Hessian [16], which can directly measure the flatness of the loss landscape, we apply Fisher information to achieve sparse perturbation, denoted as SSAM
Revised: Inspired by the connection between Fisher information and Hessian [16], which can directly measure the flatness of the loss landscape, we apply Fisher information to achieve sparse perturbation, denoted as SSAM

Original: -F.
Revised: -F.

Original: The Fisher information is defined by
Revised: The Fisher information is defined by

Original: [Equation]
Revised: [Equation]

Original: where p w ( y | x ) is the output distribution predicted by the model.
Revised: where p w ( y | x ) is the output distribution predicted by the model.

Original: In over-parameterized networks, the computation of Fisher information matrix is also intractable, i.e. , | w |×| w | .
Revised: In over-parameterized networks, the computation of Fisher information matrix is also intractable, i.e. , | w |×| w | .

Original: Following [50, 19, 41, 22], we approximate F w as a diagonal matrix, which is included in a vector in R | w | .
Revised: Following [50, 19, 41, 22], we approximate F w as a diagonal matrix, which is included in a vector in R | w | .

Original: Note that there are the two expectation in Eq.
Revised: Note that there are the two expectation in Eq.

Original: The first one is that the original data distribution p ( x ) is often not available.
Revised: The first one is that the original data distribution p ( x ) is often not available.

Original: Therefore, we approximate it by sampling N F data x 1 , x 2 , . . .
Revised: Therefore, we approximate it by sampling N F data x 1 , x 2 , . . .

Original: , x N F from x ∼ p ( x ) :
Revised: , x N F from x ∼ p ( x ) :

Original: [Equation]
Revised: [Equation]

Original: For the second expectation over p w ( y | x ) , it is not necessary to compute its explicit expectation, since the ground-truth y
Revised: For the second expectation over p w ( y | x ) , it is not necessary to compute its explicit expectation, since the ground-truth y

Original: i for each training sample x
Revised: i for each training sample x

Original: i is available in supervised learning.
Revised: i is available in supervised learning.

Original: Therefore we rewrite the Eq.
Revised: Therefore we rewrite the Eq.

Original: (4) as "empirical Fisher":
Revised: (4) as "empirical Fisher":

Original: [Equation]
Revised: [Equation]

Original: We emphasize that the empirical Fisher is a | w | -dimension vector, which is the same as the mask m .
Revised: We emphasize that the empirical Fisher is a | w | -dimension vector, which is the same as the mask m .

Original: To obtain the mask m , we calculate the empirical Fisher by Eq.
Revised: To obtain the mask m , we calculate the empirical Fisher by Eq.

Original: (5) over N F training data randomly sampled from training set S .
Revised: (5) over N F training data randomly sampled from training set S .

Original: Then we sort the elements of empirical Fisher in descending, and the parameters corresponding to the the top k Fisher values will be perturbed:
Revised: Then we sort the elements of empirical Fisher in descending, and the parameters corresponding to the the top k Fisher values will be perturbed:

Original: [Equation]
Revised: [Equation]

Original: (6) where ArgTopK( v , N ) returns the index of the top N largest values among v , m 1 is the set of values that are 1 in m .
Revised: (6) where ArgTopK( v , N ) returns the index of the top N largest values among v , m 1 is the set of values that are 1 in m .

Original: k is the number of perturbed parameters, which is equal to (1 − s ) · | w | for sparsity s .
Revised: k is the number of perturbed parameters, which is equal to (1 − s ) · | w | for sparsity s .

Original: After setting the rest values of the mask to 0, i.e. , m 0 = { m i = 

Revised: After setting the rest values of the mask to 0, i.e. , m 0 = { m i =

Original: | m i / ∈ m 1 } , we get the final mask m = m
Revised: | m i / ∈ m 1 } , we get the final mask m = m

Original: [Equation]
Revised: [Equation]

Original: The algorithm of SSAM-F is shown in Algorithm 2.
Revised: The algorithm of SSAM-F is shown in Algorithm 2.

Original: 4.3 SSAM-D: Dynamic sparse training mask based sparse SAM
Revised: 4.3 SSAM-D: Dynamic sparse training mask based sparse SAM

Original: Considering the computation of empirical Fisher is still relatively high, we also resort to dynamic sparse training for efficient binary mask generation.
Revised: Considering the computation of empirical Fisher is still relatively high, we also resort to dynamic sparse training for efficient binary mask generation.

Original: The mask generation includes the perturbation dropping and the perturbation growth steps.
Revised: The mask generation includes the perturbation dropping and the perturbation growth steps.

Original: At the perturbation dropping phase, the flattest dimensions of the perturbed parameters will be dropped, i.e. , the gradients of lower absolute values, which means that they require no perturbations.
Revised: At the perturbation dropping phase, the flattest dimensions of the perturbed parameters will be dropped, i.e. , the gradients of lower absolute values, which means that they require no perturbations.

Original: The update of the sparse mask follows
Revised: The update of the sparse mask follows

Original: [Equation]
Revised: [Equation]

Original: where N drop is the number of perturbations to be dropped.
Revised: where N drop is the number of perturbations to be dropped.

Original: At the perturbation growth phase, for the purpose of exploring the perturbation combinations as many as possible, several unperturbed dimensions grow, which means these dimensions need to compute perturbations.
Revised: At the perturbation growth phase, for the purpose of exploring the perturbation combinations as many as possible, several unperturbed dimensions grow, which means these dimensions need to compute perturbations.

Original: The update of the sparse mask for perturbation growth follows
Revised: The update of the sparse mask for perturbation growth follows

Original: [Equation]
Revised: [Equation]

Original: where Random S ( N ) randomly returns N indexes in S , and N growth is the number of perturbation growths.
Revised: where Random S ( N ) randomly returns N indexes in S , and N growth is the number of perturbation growths.

Original: To keep the sparsity constant during training, the number of growths is equal to the number of dropping, i.e. , N growth =
Revised: To keep the sparsity constant during training, the number of growths is equal to the number of dropping, i.e. , N growth =

Original: N drop .
Revised: N drop .

Original: Afterwards, we set the rest values of the mask to 0 , i.e. , m 0 = { m i =

Revised: Afterwards, we set the rest values of the mask to 0 , i.e. , m 0 = { m i =

Original: | m i / ∈ m 1 } , and get the final mask m = m 0 ∪ m 1 .
Revised: | m i / ∈ m 1 } , and get the final mask m = m 0 ∪ m 1 .

Original: The drop ratio α represents the proportion of dropped perturbations in the total perturbations s · | w | , i.e. , α = N drop / ( s · | w | ) .
Revised: The drop ratio α represents the proportion of dropped perturbations in the total perturbations s · | w | , i.e. , α = N drop / ( s · | w | ) .

Original: In particular, a larger drop rate means that more combinations of binary mask can be explored during optimization, which, however, may slightly interfere the optimization process.
Revised: In particular, a larger drop rate means that more combinations of binary mask can be explored during optimization, which, however, may slightly interfere the optimization process.

Original: Following [15, 9], we apply a cosine decay scheduler to alleviate this problem:
Revised: Following [15, 9], we apply a cosine decay scheduler to alleviate this problem:

Original: [Equation]
Revised: [Equation]

Original: where T denotes number of training epochs.
Revised: where T denotes number of training epochs.

Original: The algorithm of SSAM-D is depicted in Algorithm 2.
Revised: The algorithm of SSAM-D is depicted in Algorithm 2.

Original: 4.4 Theoretical analysis of Sparse SAM
Revised: 4.4 Theoretical analysis of Sparse SAM

Original: In the following, we analyze the convergence of SAM and SSAM in non-convex stochastic setting.
Revised: In the following, we analyze the convergence of SAM and SSAM in non-convex stochastic setting.

Original: Before introducing the main theorem, we first describe the following assumptions that are commonly used for characterizing the convergence of nonconvex stochastic optimization [39, 60, 1, 6, 4, 20].
Revised: Before introducing the main theorem, we first describe the following assumptions that are commonly used for characterizing the convergence of nonconvex stochastic optimization [39, 60, 1, 6, 4, 20].

Original: Assumption 1.
Revised: Assumption 1.

Original: (Bounded Gradient.)
Revised: (Bounded Gradient.)

Original: It exists G ≥ 

Revised: It exists G ≥

Original: s.t.
Revised: s.t.

Original: ||∇ f ( w ) || ≤ G .
Revised: ||∇ f ( w ) || ≤ G .

Original: Assumption 2.
Revised: Assumption 2.

Original: (Bounded Variance.)
Revised: (Bounded Variance.)

Original: It exists σ ≥ 0 s.t.
Revised: It exists σ ≥ 0 s.t.

Original: E
Revised: E

Original: [ || g ( w ) − ∇ f ( w ) || 2 ] ≤ σ 2 .
Revised: [ || g ( w ) − ∇ f ( w ) || 2 ] ≤ σ 2 .

Original: Assumption 3.
Revised: Assumption 3.

Original: (L-smoothness.)
Revised: (L-smoothness.)

Original: It exists L > 0 s.t.
Revised: It exists L > 0 s.t.

Original: ||∇ f ( w ) − ∇ f ( v ) ||
Revised: ||∇ f ( w ) − ∇ f ( v ) ||

Original: ≤ L ||
Revised: ≤ L ||

Original: w − v || , ∀ w , v ∈ R d .
Revised: w − v || , ∀ w , v ∈ R d .

Original: Theorem 1.
Revised: Theorem 1.

Original: Consider function f ( w ) satisfying the Assumptions 1-3 optimized by SAM.
Revised: Consider function f ( w ) satisfying the Assumptions 1-3 optimized by SAM.

Original: Let η t = η 0 √ t and perturbation amplitude ρ decay with square root of t , e.g. , ρ t = ρ 0 √ t .
Revised: Let η t = η 0 √ t and perturbation amplitude ρ decay with square root of t , e.g. , ρ t = ρ 0 √ t .

Original: With ρ 0 ≤ Gη 0 , we have
Revised: With ρ 0 ≤ Gη 0 , we have

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: C 1 = 2 η 0 ( f ( w 0 ) − E
Revised: C 1 = 2 η 0 ( f ( w 0 ) − E

Original: f ( w T )) and C 2 = 2( Lσ 2 η 0 + LGρ 0 ) .
Revised: f ( w T )) and C 2 = 2( Lσ 2 η 0 + LGρ 0 ) .

Original: Theorem 2.
Revised: Theorem 2.

Original: Consider function f ( w ) satisfying the Assumptions 1-3 optimized by SSAM.
Revised: Consider function f ( w ) satisfying the Assumptions 1-3 optimized by SSAM.

Original: Let η t = η 0 √ t and perturbation amplitude ρ decay with square root of t , e.g. , ρ t = ρ 0 √ t .
Revised: Let η t = η 0 √ t and perturbation amplitude ρ decay with square root of t , e.g. , ρ t = ρ 0 √ t .

Original: With ρ 0 ≤ Gη 0 / 2 , we have:
Revised: With ρ 0 ≤ Gη 0 / 2 , we have:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where C 3 = 2 η 0 ( f ( w 0 ) − E f ( w T ) + η 0 L 2 ρ 2 (1 + η 0 L ) π
Revised: where C 3 = 2 η 0 ( f ( w 0 ) − E f ( w T ) + η 0 L 2 ρ 2 (1 + η 0 L ) π

Original: 2 6 ) and C 4 = 

Revised: 2 6 ) and C 4 =

Original: ( Lσ 2 η 0 + LGρ 0 ) .
Revised: ( Lσ 2 η 0 + LGρ 0 ) .

Original: For non-convex stochastic optimization, Theorems 1&2 imply that our SSAM could converge the same rate as SAM, i.e. , O (log T/ √ T ) .
Revised: For non-convex stochastic optimization, Theorems 1&2 imply that our SSAM could converge the same rate as SAM, i.e. , O (log T/ √ T ) .

Original: Detailed proofs of the two theorems are given in Appendix .
Revised: Detailed proofs of the two theorems are given in Appendix .

Original:  Experiments
Revised:  Experiments

Original: In this section, we evaluate the effectiveness of SSAM through extensive experiments on CIFAR10,
Revised: In this section, we evaluate the effectiveness of SSAM through extensive experiments on CIFAR10,

Original: CIFAR100 [33] and ImageNet-1K [8].
Revised: CIFAR100 [33] and ImageNet-1K [8].

Original: The base models include ResNet [25] and WideResNet [57].
Revised: The base models include ResNet [25] and WideResNet [57].

Original: We report the main results on CIFAR datasets in Tables 1&

Revised: We report the main results on CIFAR datasets in Tables 1&

Original: and ImageNet-1K in Table 3.
Revised: and ImageNet-1K in Table 3.

Original: Then, we visualize the landscapes and Hessian spectra to verify that the proposed SSAM can help the model generalize better.
Revised: Then, we visualize the landscapes and Hessian spectra to verify that the proposed SSAM can help the model generalize better.

Original: More experimental results are placed in Appendix due to page limit.
Revised: More experimental results are placed in Appendix due to page limit.

Original: 5.1 Implementation details
Revised: 5.1 Implementation details

Original: Datasets.
Revised: Datasets.

Original: We use CIFAR10/CIFAR100 [33] and ImageNet-1K [8] as the benchmarks of our method.
Revised: We use CIFAR10/CIFAR100 [33] and ImageNet-1K [8] as the benchmarks of our method.

Original: Specifically, CIFAR10 and CIFAR100 have 50,000 images of 32 × 32 resolution for training, while 10,000 images for test.
Revised: Specifically, CIFAR10 and CIFAR100 have 50,000 images of 32 × 32 resolution for training, while 10,000 images for test.

Original: ImageNet-1K [8] is the most widely used benchmark for image classification, which has 1,281,167 images of 1000 classes and 50,000 images for validation.
Revised: ImageNet-1K [8] is the most widely used benchmark for image classification, which has 1,281,167 images of 1000 classes and 50,000 images for validation.

Original: Hyper-parameter setting.
Revised: Hyper-parameter setting.

Original: For small resolution datasets, i.e. , CIFAR10 and CIFAR100, we replace the first convolution layer in ResNet and WideResNet with the one of 3 × 3 kernel size, 1 stride andpadding.
Revised: For small resolution datasets, i.e. , CIFAR10 and CIFAR100, we replace the first convolution layer in ResNet and WideResNet with the one of 3 × 3 kernel size, 1 stride andpadding.

Original: The models on CIFAR10/CIFAR100 are trained with 128 batch size for 200 epochs.
Revised: The models on CIFAR10/CIFAR100 are trained with 128 batch size for 200 epochs.

Original: We apply the random crop, random horizontal flip, normalization and cutout [11] for data augmentation, and the initial learning rate is 0.05 with a cosine learning rate schedule.
Revised: We apply the random crop, random horizontal flip, normalization and cutout [11] for data augmentation, and the initial learning rate is 0.05 with a cosine learning rate schedule.

Original: The momentum and weight decay of SGD are set to 0.9 and 5 e -4, respectively.
Revised: The momentum and weight decay of SGD are set to 0.9 and 5 e -4, respectively.

Original: SAM and SSAM apply the same settings, except that weight decay is set to 0.001 [14].
Revised: SAM and SSAM apply the same settings, except that weight decay is set to 0.001 [14].

Original: We determine the perturbation magnitude ρ from { 0 . 01 , 0 . 02 , 0 . 05 , 0 . 1 , 0 . 2 , 0 . 5 } via grid search.
Revised: We determine the perturbation magnitude ρ from { 0 . 01 , 0 . 02 , 0 . 05 , 0 . 1 , 0 . 2 , 0 . 5 } via grid search.

Original: In CIFAR10 and CIFAR100, we set ρ as 0.1 and 0.2, respectively.
Revised: In CIFAR10 and CIFAR100, we set ρ as 0.1 and 0.2, respectively.

Original: For ImageNet-1K, we randomly resize and crop all images to a resolution of 224 × 224, and apply random horizontal flip, normalization during training.
Revised: For ImageNet-1K, we randomly resize and crop all images to a resolution of 224 × 224, and apply random horizontal flip, normalization during training.

Original: We train ResNet with a batch size of 256, and adopt the cosine learning rate schedule with initial learning rate 0.1.
Revised: We train ResNet with a batch size of 256, and adopt the cosine learning rate schedule with initial learning rate 0.1.

Original: The momentum and weight decay of SGD is set as 0.9 and 1 e -4.
Revised: The momentum and weight decay of SGD is set as 0.9 and 1 e -4.

Original: SAM and SSAM use the same settings as above.
Revised: SAM and SSAM use the same settings as above.

Original: The test images of both architectures are resized to 256 × 256 and then centerly cropped to 224 × 224.
Revised: The test images of both architectures are resized to 256 × 256 and then centerly cropped to 224 × 224.

Original: The perturbation magnitude ρ is set to 0.07.
Revised: The perturbation magnitude ρ is set to 0.07.

Original: 5.2 Experimental results
Revised: 5.2 Experimental results

Original: Results on CIFAR10/CIFAR100.
Revised: Results on CIFAR10/CIFAR100.

Original: We first evaluate our SSAM on CIFAR-10 and CIFAR100.
Revised: We first evaluate our SSAM on CIFAR-10 and CIFAR100.

Original: The models we used are ResNet-18 [25] and WideResNet-28-10 [57].
Revised: The models we used are ResNet-18 [25] and WideResNet-28-10 [57].

Original: The perturbation magnitude ρ for
Revised: The perturbation magnitude ρ for

Original: SAM and SSAM are the same for a fair comparison.
Revised: SAM and SSAM are the same for a fair comparison.

Original: As shown in Table 1, ResNet18 with SSAM of 50% sparsity outperforms SAM of full perturbations.
Revised: As shown in Table 1, ResNet18 with SSAM of 50% sparsity outperforms SAM of full perturbations.

Original: From Table 2, we can observe that the advantages of SSAM-F and SSAM-D on WideResNet28 are more significant, which achieve better performance than SAM with up to 95% sparsity.
Revised: From Table 2, we can observe that the advantages of SSAM-F and SSAM-D on WideResNet28 are more significant, which achieve better performance than SAM with up to 95% sparsity.

Original: Note that the parameter size of WideResNet28 is much larger than that of ResNet-
Revised: Note that the parameter size of WideResNet28 is much larger than that of ResNet-

Original: [Table]
Revised: [Table]

Original: and CIFAR is often easy to overfit.
Revised: and CIFAR is often easy to overfit.

Original: In addition, even with very large sparsity, both SSAM-F and SSAM-D can still obtain competitive performance against SAM.
Revised: In addition, even with very large sparsity, both SSAM-F and SSAM-D can still obtain competitive performance against SAM.

Original: [Table]
Revised: [Table]

Original: Results on ImageNet.
Revised: Results on ImageNet.

Original: Table 3 reports the result of SSAM-F and SSAM-D on the large-scale
Revised: Table 3 reports the result of SSAM-F and SSAM-D on the large-scale

Original: ImageNet-1K [8] dataset.
Revised: ImageNet-1K [8] dataset.

Original: The model we used is ResNet50 [25].
Revised: The model we used is ResNet50 [25].

Original: We can observe that SSAM-F and
Revised: We can observe that SSAM-F and

Original: SSAM-D can maintain the performance with 50% sparsity.
Revised: SSAM-D can maintain the performance with 50% sparsity.

Original: However, as sparsity ratio increases,
Revised: However, as sparsity ratio increases,

Original: SSAM-F will receive relatively obvious performance drops while SSAM-D is more robust.
Revised: SSAM-F will receive relatively obvious performance drops while SSAM-D is more robust.

Original: Training curves.
Revised: Training curves.

Original: We visualize the training curves of SGD, SAM and SSAM in Fig.
Revised: We visualize the training curves of SGD, SAM and SSAM in Fig.

Original: The training curves of SGD are more jittery, while SAM is more stable.
Revised: The training curves of SGD are more jittery, while SAM is more stable.

Original: In SSAM, about half of gradient updates are the same as SGD, but its training curves are similar to those of SAM, suggesting the effectiveness.
Revised: In SSAM, about half of gradient updates are the same as SGD, but its training curves are similar to those of SAM, suggesting the effectiveness.

Original: [Table]
Revised: [Table]

Original: R e s N e t-
Revised: R e s N e t-

Original: [Table] [Table]
Revised: [Table] [Table]

Original: ImageNet
Revised: ImageNet

Original: FLOPs
Revised: FLOPs

Original: ResNet
Revised: ResNet

Original: SGD / 76.67% 1 ×
Revised: SGD / 76.67% 1 ×

Original: SAM 0% 77.25% 2 ×
Revised: SAM 0% 77.25% 2 ×

Original: SSAM-F (Ours) 50% 77.31% (+0.06) 1.65 × 80% 76.81%(-0.44) 1.

Revised: SSAM-F (Ours) 50% 77.31% (+0.06) 1.65 × 80% 76.81%(-0.44) 1.

Original: × 90% 76.74%(-0.51) 1.36 ×
Revised: × 90% 76.74%(-0.51) 1.36 ×

Original: SSAM-D (Ours) 50% 77.25% (-0.00) 1.65 × 80% 77.00%(-0.25)
Revised: SSAM-D (Ours) 50% 77.25% (-0.00) 1.65 × 80% 77.00%(-0.25)

Original: 1.44 × 90% 77.00%(-0.25) 1.36 ×
Revised: 1.44 × 90% 77.00%(-0.25) 1.36 ×

Original:  
Revised:  

Original:    (SRFK  
Revised:    (SRFK  

Original:   
Revised:  

Original:  
Revised:  

Original: [Figure]
Revised: [Figure]

Original: $ FF
Revised: $ FF

Original: [Figure]
Revised: [Figure]

Original: D F \   
Revised: D F \  

Original: [Figure]
Revised: [Figure]

Original: 66$0'
Revised: 66$0'

Original: [Figure]
Revised: [Figure]

Original: 
Revised: 

Original: 
Revised: 

Original:  
Revised:  

Original: [Figure]
Revised: [Figure]

Original: (SRFK
Revised: (SRFK

Original: [Figure]
Revised: [Figure]

Original:     
Revised:     

Original: [Figure]
Revised: [Figure]

Original: FF
Revised: FF

Original: [Figure]
Revised: [Figure]

Original: F \
Revised: F \

Original: [Figure]
Revised: [Figure]

Original:  
Revised: 

Original: [Figure]
Revised: [Figure]

Original: 66$0' 66$0) (
Revised: 66$0' 66$0) (

Original: [Figure]
Revised: [Figure]

Original: ResNet
Revised: ResNet

Original: [Figure]
Revised: [Figure]

Original: CIFAR  
Revised: CIFAR  

Original: [Figure]
Revised: [Figure]

Original: (SRFK
Revised: (SRFK

Original: [Figure]
Revised: [Figure]

Original: 
Revised: 

Original: [Figure]
Revised: [Figure]

Original: 
Revised: 

Original: [Figure]
Revised: [Figure]

Original: F
Revised: F

Original: [Figure]
Revised: [Figure]

Original: 
Revised:

Original: [Figure]
Revised: [Figure]

Original: Sparsity vs. Accuracy.
Revised: Sparsity vs. Accuracy.

Original: We report the effect of sparsity ratios in SSAM, as depicted in Fig.
Revised: We report the effect of sparsity ratios in SSAM, as depicted in Fig.

Original: We can observe that on CIFAR datasets, the sparsities of SSAM-F and SSAM-D pose little impact on performance.
Revised: We can observe that on CIFAR datasets, the sparsities of SSAM-F and SSAM-D pose little impact on performance.

Original: In addition, they can obtain better accuracies than SAM with up to 99% sparsity.
Revised: In addition, they can obtain better accuracies than SAM with up to 99% sparsity.

Original: On the much larger dataset, i.e. , ImageNet, a higher sparsity will lead to more obvious performance drop.
Revised: On the much larger dataset, i.e. , ImageNet, a higher sparsity will lead to more obvious performance drop.

Original: [Figure]
Revised: [Figure]

Original:  
Revised: 

Original: 
Revised: 

Original: 
Revised: 

Original:   
Revised: 

Original:   $ FF X U D F \    
Revised:    $ FF X U D F \   

Original: [Figure]     
Revised: [Figure]     

Original: [Figure]
Revised: [Figure]

Original: 5.3 SSAM with better generalization
Revised: 5.3 SSAM with better generalization

Original: Visualization of landscape.
Revised: Visualization of landscape.

Original: For a more intuitive comparison between different optimization schemes, we visualize the training loss landscapes of ResNet18 optimized by SGD, SAM and SSAM as shown in Fig.
Revised: For a more intuitive comparison between different optimization schemes, we visualize the training loss landscapes of ResNet18 optimized by SGD, SAM and SSAM as shown in Fig.

Original: Following [36], we sample 50 × 50 points in the range of [ − 1 , 1] from random "filter normalized" [36] directions, i.e. , the x and y axes.
Revised: Following [36], we sample 50 × 50 points in the range of [ − 1 , 1] from random "filter normalized" [36] directions, i.e. , the x and y axes.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 4, the landscape of SSAM is flatter than both SGD and SAM, and most of its area is low loss (blue).
Revised: 4, the landscape of SSAM is flatter than both SGD and SAM, and most of its area is low loss (blue).

Original: This result indicates that
Revised: This result indicates that

Original: SSAM can smooth the loss landscape notably with sparse perturbation, and it also suggests that the complete perturbation on all parameters will result in suboptimal minima.
Revised: SSAM can smooth the loss landscape notably with sparse perturbation, and it also suggests that the complete perturbation on all parameters will result in suboptimal minima.

Original: [Figure]
Revised: [Figure]

Original: Hessian spectra.
Revised: Hessian spectra.

Original: In Fig.
Revised: In Fig.

Original: 5, we report the Hessian spectrum to demonstrate that SSAM can converge to a flat minima.
Revised: 5, we report the Hessian spectrum to demonstrate that SSAM can converge to a flat minima.

Original: Here, we also report the ratio of dominant eigenvalue to fifth largest ones, i.e. , λ 1 /λ 5 , used as the criteria in [17, 29].
Revised: Here, we also report the ratio of dominant eigenvalue to fifth largest ones, i.e. , λ 1 /λ 5 , used as the criteria in [17, 29].

Original: We approximate the Hessian spectrum using the Lanczos algorithm [21] and illustrate the Hessian spectra of ResNet18 using SGD, SAM and SSAM on
Revised: We approximate the Hessian spectrum using the Lanczos algorithm [21] and illustrate the Hessian spectra of ResNet18 using SGD, SAM and SSAM on

Original: CIFAR10.
Revised: CIFAR10.

Original: From this figure, we observe that the dominant eigenvalue λ 1 with SSAM is less than
Revised: From this figure, we observe that the dominant eigenvalue λ 1 with SSAM is less than

Original: SGD and comparable to SAM.
Revised: SGD and comparable to SAM.

Original: It confirms that SSAM with a sparse perturbation can still converge to the flat minima as SAM does, or even better.
Revised: It confirms that SSAM with a sparse perturbation can still converge to the flat minima as SAM does, or even better.

Original:       
Revised:       

Original:   (LJHQYODXH         ' H Q V L W \ 1 = 82.
Revised:   (LJHQYODXH         ' H Q V L W \ 1 = 82.

Original: 1 / 5 = 16.
Revised: 1 / 5 = 16.

Original: [Figure]
Revised: [Figure]

Original:        (LJHQYODXH         ' H Q V L W \ 1 = 26.
Revised:        (LJHQYODXH         ' H Q V L W \ 1 = 26.

Original: 1 / 5 = 2.
Revised: 1 / 5 = 2.

Original: [Figure]
Revised: [Figure]

Original:      (LJHQYODXH         ' H Q V L W \ 1 = 19.
Revised:      (LJHQYODXH         ' H Q V L W \ 1 = 19.

Original: 1 / 5 = 1.
Revised: 1 / 5 = 1.

Original: [Figure]
Revised: [Figure]

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we reveal that the SAM gradients for most parameters are not significantly different from the SGD ones.
Revised: In this paper, we reveal that the SAM gradients for most parameters are not significantly different from the SGD ones.

Original: Based on this finding, we propose an efficient training scheme called Sparse
Revised: Based on this finding, we propose an efficient training scheme called Sparse

Original: SAM, which is achieved by computing a sparse perturbation.
Revised: SAM, which is achieved by computing a sparse perturbation.

Original: We provide two solutions for sparse perturbation, which are based on Fisher information and dynamic sparse training, respectively.
Revised: We provide two solutions for sparse perturbation, which are based on Fisher information and dynamic sparse training, respectively.

Original: In addition, we also theoretically prove that SSAM has the same convergence rate as SAM.
Revised: In addition, we also theoretically prove that SSAM has the same convergence rate as SAM.

Original: We validate our SSAM on extensive datasets with various models.
Revised: We validate our SSAM on extensive datasets with various models.

Original: The experimental results show that retaining much better efficiency, SSAM can achieve competitive and even better performance than SAM.
Revised: The experimental results show that retaining much better efficiency, SSAM can achieve competitive and even better performance than SAM.

Original: Acknowledgement
Revised: Acknowledgement

Original: This work is supported by the Major Science and Technology Innovation 2030 “Brain Science and Brain-like Research” key project (No. 2021ZD0201405), the National Science Fund for Distinguished Young Scholars (No.
Revised: This work is supported by the Major Science and Technology Innovation 2030 “Brain Science and Brain-like Research” key project (No. 2021ZD0201405), the National Science Fund for Distinguished Young Scholars (No.

Original: , the National Natural Science Foundation of China (No.
Revised: , the National Natural Science Foundation of China (No.

Original: U21B2037, No. 62176222, No. 62176223, No. 62176226, No. 62072386, No. 62072387, No. 62072389, and No. 62002305), Guangdong Basic and
Revised: U21B2037, No. 62176222, No. 62176223, No. 62176226, No. 62072386, No. 62072387, No. 62072389, and No. 62002305), Guangdong Basic and

Original: Applied Basic Research Foundation (No. 2019B1515120049), and the Natural Science Foundation of Fujian Province of China (No. 2021J01002).
Revised: Applied Basic Research Foundation (No. 2019B1515120049), and the Natural Science Foundation of Fujian Province of China (No. 2021J01002).

Original: Checklist
Revised: Checklist

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contribu- tions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contribu- tions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [No] (c) Did you discuss any potential negative societal impacts of your work?
Revised: [No] (c) Did you discuss any potential negative societal impacts of your work?

Original: [No] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: [No] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 2.
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] See Section 4.4.
Revised: [Yes] See Section 4.4.

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] The detailed proof is in Appendix.
Revised: [Yes] The detailed proof is in Appendix.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: [Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] See Section 5.1.
Revised: [Yes] See Section 5.1.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [No] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: [No] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [No] 4.
Revised: [No] 4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [N/A] (b) Did you mention the license of the assets?
Revised: [N/A] (b) Did you mention the license of the assets?

Original: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A] (d) Did you discuss whether
Revised: [N/A] (d) Did you discuss whether

Original: and how consent was obtained from people whose data you’re us- ing/curating?
Revised: and how consent was obtained from people whose data you’re us- ing/curating?

Original: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable informa- tion or offensive content?
Revised: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable informa- tion or offensive content?

Original: [N/A] 5.
Revised: [N/A] 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include

Original: full
Revised: full

Original: (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]

Original: A Missing Proofs
Revised: A Missing Proofs

Original: In this section, we will proof the main theorem in our paper.
Revised: In this section, we will proof the main theorem in our paper.

Original: Before the detailed proof, we first recall the following assumptions that are commonly used for characterizing the convergence of non-convex stochastic optimization.
Revised: Before the detailed proof, we first recall the following assumptions that are commonly used for characterizing the convergence of non-convex stochastic optimization.

Original: Assumption 1 ′ .
Revised: Assumption 1 ′ .

Original: (Bounded Gradient.)
Revised: (Bounded Gradient.)

Original: It exists G ≥ 

Revised: It exists G ≥

Original: s.t.
Revised: s.t.

Original: ||∇ f ( w ) || ≤ G .
Revised: ||∇ f ( w ) || ≤ G .

Original: Assumption 2 ′ .
Revised: Assumption 2 ′ .

Original: (Bounded Variance.)
Revised: (Bounded Variance.)

Original: It exists σ ≥ 0 s.t.
Revised: It exists σ ≥ 0 s.t.

Original: E
Revised: E

Original: [ || g ( w ) − ∇ f ( w ) || 2 ] ≤ σ 2 .
Revised: [ || g ( w ) − ∇ f ( w ) || 2 ] ≤ σ 2 .

Original: Assumption 3 ′ .
Revised: Assumption 3 ′ .

Original: (L-smoothness.)
Revised: (L-smoothness.)

Original: It exists L > 0 s.t.
Revised: It exists L > 0 s.t.

Original: ||∇ f ( w ) − ∇ f
Revised: ||∇ f ( w ) − ∇ f

Original: [Equation]
Revised: [Equation]

Original: ) || ≤ L || w −
Revised: ) || ≤ L || w −

Original: v ||
Revised: v ||

Original: [Equation]
Revised: [Equation]

Original: ∀ w , v ∈ R d .
Revised: ∀ w , v ∈ R d .

Original: A.1 Proof of Theorem
Revised: A.1 Proof of Theorem

Original: Based on the objective function of Sharpness-Aware Minimization (SAM), suppose we can obtain the noisy observation gradient g ( w ) of true gradient ∇ f ( w ) , we can write the iteration of SAM:
Revised: Based on the objective function of Sharpness-Aware Minimization (SAM), suppose we can obtain the noisy observation gradient g ( w ) of true gradient ∇ f ( w ) , we can write the iteration of SAM:

Original: [Equation]
Revised: [Equation]

Original: Lemma 1.
Revised: Lemma 1.

Original: For any ρ > 0 , L > 0 and the differentiable function f , we have the following inequality:
Revised: For any ρ > 0 , L > 0 and the differentiable function f , we have the following inequality:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We first add and subtract a term ||∇ f ( w t )
Revised: We first add and subtract a term ||∇ f ( w t )

Original: || to make use of classical inequalities bounding ⟨∇
Revised: || to make use of classical inequalities bounding ⟨∇

Original: f
Revised: f

Original: ( w 1 ) − ∇ f ( w 2 ) , w 1 − w 2 ⟩ by || w 1 − w 

Revised: ( w 1 ) − ∇ f ( w 2 ) , w 1 − w 2 ⟩ by || w 1 − w

Original: || 2 for smooth.
Revised: || 2 for smooth.

Original: [Equation]
Revised: [Equation]

Original: where the first inequality is that
Revised: where the first inequality is that

Original: [Equation]
Revised: [Equation]

Original: and the second inequality is the Assumption 1.
Revised: and the second inequality is the Assumption 1.

Original: Lemma 2.
Revised: Lemma 2.

Original: For ρ > 0 , L > 0 , the iteration 12 satisfies following inequality:
Revised: For ρ > 0 , L > 0 , the iteration 12 satisfies following inequality:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We denote the deterministic values of w t
Revised: We denote the deterministic values of w t

Original: + 12 as ˆ w t + 12 =
Revised: + 12 as ˆ w t + 12 =

Original: w t + ρ ∇ f ( w t )
Revised: w t + ρ ∇ f ( w t )

Original: ||∇ f ( w t ) || in this section.
Revised: ||∇ f ( w t ) || in this section.

Original: After we add and subtract the term g ( ˆ w t + 12 ) , we have the following equation:
Revised: After we add and subtract the term g ( ˆ w t + 12 ) , we have the following equation:

Original: [Equation]
Revised: [Equation]

Original: For the first term, we bound it by using the smoothness of g ( w ) : −⟨∇
Revised: For the first term, we bound it by using the smoothness of g ( w ) : −⟨∇

Original: [Equation]
Revised: [Equation]

Original: t
Revised: t

Original: [Equation]
Revised: [Equation]

Original: |
Revised: |

Original: [Equation]
Revised: [Equation]

Original: f
Revised: f

Original: [Equation]
Revised: [Equation]

Original: t
Revised: t

Original: [Equation]
Revised: [Equation]

Original: ||
Revised: ||

Original: [Equation]
Revised: [Equation]

Original: For the second term, by using the Lemma 1, we have:
Revised: For the second term, by using the Lemma 1, we have:

Original: [Equation]
Revised: [Equation]

Original: Asesmbling the two inequalities yields to the result.
Revised: Asesmbling the two inequalities yields to the result.

Original: Lemma 3.
Revised: Lemma 3.

Original: For η ≤ 1 L , the iteration 12 satisfies for all t > 0 :
Revised: For η ≤ 1 L , the iteration 12 satisfies for all t > 0 :

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: By the smoothness of the function f
Revised: By the smoothness of the function f

Original: [Equation]
Revised: [Equation]

Original: we obtain
Revised: we obtain

Original: [Equation]
Revised: [Equation]

Original: Taking the expectation and using Lemma 2 we obtain
Revised: Taking the expectation and using Lemma 2 we obtain

Original: [Equation]
Revised: [Equation]

Original: Proposition 1.
Revised: Proposition 1.

Original: Let η
Revised: Let η

Original: [Equation]
Revised: [Equation]

Original: where C 1 = 2 η 0 (
Revised: where C 1 = 2 η 0 (

Original: ( w 0 ) − E η
Revised: ( w 0 ) − E η

Original: Proof.
Revised: Proof.

Original: By Lemma 3, we replace ρ and η with ρ t = ρ 0 √ t and η t = η 0 √ t , we have
Revised: By Lemma 3, we replace ρ and η with ρ t = ρ 0 √ t and η t = η 0 √ t , we have

Original: [Equation]
Revised: [Equation]

Original: Take telescope sum, we have
Revised: Take telescope sum, we have

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Under ρ 0 ≤ Gη 0 , the last term will be less than 0, which means:
Revised: Under ρ 0 ≤ Gη 0 , the last term will be less than 0, which means:

Original: [Equation]
Revised: [Equation]

Original: With η [Equation]
Revised: With η [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: ) log T.
Revised: ) log T.

Original: Finally, we achieve the result:
Revised: Finally, we achieve the result:

Original: [Equation]
Revised: [Equation]

Original: which shows that SAM can converge at the rate of O (log T/ √ T ) .
Revised: which shows that SAM can converge at the rate of O (log T/ √ T ) .

Original: A.2 Proof of Theorem
Revised: A.2 Proof of Theorem

Original: Suppose we can obtain the noisy observation gradient g ( w t ) of true gradient ∇ f ( w t ) , and the mask m , we can write the iteration of SAM: Consider the iteration of Sparse SAM:
Revised: Suppose we can obtain the noisy observation gradient g ( w t ) of true gradient ∇ f ( w t ) , and the mask m , we can write the iteration of SAM: Consider the iteration of Sparse SAM:

Original: [Equation]
Revised: [Equation]

Original: Let us denote the difference as w t + 12 − ˜ w t + 

Revised: Let us denote the difference as w t + 12 − ˜ w t +

Original: =
Revised: =

Original: e t .
Revised: e t .

Original: Lemma 4.
Revised: Lemma 4.

Original: With ρ > 0 , we have:
Revised: With ρ > 0 , we have:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Similar to Lemma 2, We denote the true gradient as ˆ w t + 12 =
Revised: Similar to Lemma 2, We denote the true gradient as ˆ w t + 12 =

Original: w t + ρ ∇ f ( w t )
Revised: w t + ρ ∇ f ( w t )

Original: ||∇ f ( w t ) || , and also add and subtract the item g ( ˜ w t + 12 ) :
Revised: ||∇ f ( w t ) || , and also add and subtract the item g ( ˜ w t + 12 ) :

Original: [Equation]
Revised: [Equation]

Original: For the first term, we bound it by using the smoothness of g ( w ) : −⟨∇
Revised: For the first term, we bound it by using the smoothness of g ( w ) : −⟨∇

Original: [Equation]
Revised: [Equation]

Original: For the second term, we do the same in Lemma 2:
Revised: For the second term, we do the same in Lemma 2:

Original: [Equation]
Revised: [Equation]

Original: Assembling the two inequalities yields to the result.
Revised: Assembling the two inequalities yields to the result.

Original: Lemma 5.
Revised: Lemma 5.

Original: For η ≤ 1 L , the iteration 13 satisfies for all t > 0 :
Revised: For η ≤ 1 L , the iteration 13 satisfies for all t > 0 :

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: By the smoothness of the function f , we obtain
Revised: By the smoothness of the function f , we obtain

Original: [Equation]
Revised: [Equation]

Original: Taking the expectation and using Lemma 4 we obtain
Revised: Taking the expectation and using Lemma 4 we obtain

Original: [Equation]
Revised: [Equation]

Original: Proposition 2.
Revised: Proposition 2.

Original: Let us η t = η 0 √ t and perturbation amplitude ρ decay with square root of t , e.g. , ρ t = ρ 0 √ t .
Revised: Let us η t = η 0 √ t and perturbation amplitude ρ decay with square root of t , e.g. , ρ t = ρ 0 √ t .

Original: With ρ 0 ≤ Gη 0 / 2 , we have:
Revised: With ρ 0 ≤ Gη 0 / 2 , we have:

Original: [Equation]
Revised: [Equation]

Original: where C 3 = 2 η 0 ( f ( w 0 − E f ( w T ) + η 0 L 2 ρ 2 (1 + η 0 L ) π 2 6 ) and C 4 = 2( Lσ 2 η 0 + LGρ 0 ) .
Revised: where C 3 = 2 η 0 ( f ( w 0 − E f ( w T ) + η 0 L 2 ρ 2 (1 + η 0 L ) π 2 6 ) and C 4 = 2( Lσ 2 η 0 + LGρ 0 ) .

Original: Proof.
Revised: Proof.

Original: By taking the expectation and using Lemma 4, and taking the schedule to be η t = η 0 √ t , ρ t = ρ 0 √ t , we obtain:
Revised: By taking the expectation and using Lemma 4, and taking the schedule to be η t = η 0 √ t , ρ t = ρ 0 √ t , we obtain:

Original: [Equation]
Revised: [Equation]

Original: By taking sum and bound ρ with Gη 0 2 , we have:
Revised: By taking sum and bound ρ with Gη 0 2 , we have:

Original: [Equation]
Revised: [Equation]

Original: Finally, we achieve the result:
Revised: Finally, we achieve the result:

Original: [Equation]
Revised: [Equation]

Original: So far, we have completed the proof of the theory in the main text.
Revised: So far, we have completed the proof of the theory in the main text.

Original: B More Experimets
Revised: B More Experimets

Original: VGG on CIFAR10.
Revised: VGG on CIFAR10.

Original: To further confirm the model-agnostic characteristic of our Sparse SAM, we test the VGG-style architecture on CIFAR10.
Revised: To further confirm the model-agnostic characteristic of our Sparse SAM, we test the VGG-style architecture on CIFAR10.

Original: Following [48], we test SSAM training the VGG11-BN on CIFAR10 and the results are shown in the following Table 4.
Revised: Following [48], we test SSAM training the VGG11-BN on CIFAR10 and the results are shown in the following Table 4.

Original: The perturbation magnitude ρ is set to 0.05.
Revised: The perturbation magnitude ρ is set to 0.05.

Original: [Table]
Revised: [Table]

Original: SAM with different perturbation magnitude ρ .
Revised: SAM with different perturbation magnitude ρ .

Original: We determine the perturbation magnitude ρ by using grid search.
Revised: We determine the perturbation magnitude ρ by using grid search.

Original: We choose ρ from the set { 0 . 01 , 0 . 02 , 0 .
Revised: We choose ρ from the set { 0 . 01 , 0 . 02 , 0 .

Original: 2 , 0 . 5 } for CIFAR, and choose ρ from { 0 . 01 , 0 . 02 , 0 . 05 , 0 . 07 , 0 . 1 , 0 . 2 } for ImageNet.
Revised: 2 , 0 . 5 } for CIFAR, and choose ρ from { 0 . 01 , 0 . 02 , 0 . 05 , 0 . 07 , 0 . 1 , 0 . 2 } for ImageNet.

Original: We show the results when varying ρ in Table 5 and Table 6.
Revised: We show the results when varying ρ in Table 5 and Table 6.

Original: From this table, we can see that the ρ = 0 .
Revised: From this table, we can see that the ρ = 0 .

Original: 1 , ρ = 0 .
Revised: 1 , ρ = 0 .

Original: 2 and ρ = 0 .
Revised: 2 and ρ = 0 .

Original: 07 is sutiable for CIFAR10, CIFAR100 and
Revised: 07 is sutiable for CIFAR10, CIFAR100 and

Original: ImageNet respectively.
Revised: ImageNet respectively.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: Ablations of Masking Strategy.
Revised: Ablations of Masking Strategy.

Original: For further verification of our masking strategy, we perform more ablations in this paragraph.
Revised: For further verification of our masking strategy, we perform more ablations in this paragraph.

Original: For the mask update in SSAM-F [Table]
Revised: For the mask update in SSAM-F [Table]

Original: the parameters with largest fisher information are selected.
Revised: the parameters with largest fisher information are selected.

Original: Compared with SSAM-F, we consider the random mask,
Revised: Compared with SSAM-F, we consider the random mask,

Original: i
Revised: i

Original: [Table]
Revised: [Table]

Original: e. , the mask is randomly generated to choose which parameters are perturbated.
Revised: e. , the mask is randomly generated to choose which parameters are perturbated.

Original: For the mask update in SSAM-D
Revised: For the mask update in SSAM-D

Original: [Table]
Revised: [Table]

Original: we first drop the flattest weights and then random grow some weights.
Revised: we first drop the flattest weights and then random grow some weights.

Original: Compared with SSAM-D
Revised: Compared with SSAM-D

Original: [Table]
Revised: [Table]

Original: we experiment the SSAM-D which drops randomly or drops the sharpest weights, i.e. , the weights with large gradients.
Revised: we experiment the SSAM-D which drops randomly or drops the sharpest weights, i.e. , the weights with large gradients.

Original: The results of ablations are shown in Table 7.
Revised: The results of ablations are shown in Table 7.

Original: The results show that random strategies are less effective than our SSAM.
Revised: The results show that random strategies are less effective than our SSAM.

Original: The performance of SSAM-D dropping sharpest weights drops a lot even worse than random strategy, which is consistent with our conjecture.
Revised: The performance of SSAM-D dropping sharpest weights drops a lot even worse than random strategy, which is consistent with our conjecture.

Original: [Table]
Revised: [Table]

Original: Influence of hyper-parameters We first examine the effect of the number of sample size N F of SSAM-F in Table 8.
Revised: Influence of hyper-parameters We first examine the effect of the number of sample size N F of SSAM-F in Table 8.

Original: From it we can see that a certain number of samples is enough for the approximation of data distribution in SSAM-F, e.g. , N F = 128 , which greatly saves the computational cost of SSAM-F.
Revised: From it we can see that a certain number of samples is enough for the approximation of data distribution in SSAM-F, e.g. , N F = 128 , which greatly saves the computational cost of SSAM-F.

Original: In Table 9, we also report the influence of the mask update interval on SSAM-F and
Revised: In Table 9, we also report the influence of the mask update interval on SSAM-F and

Original: SSAM-D.
Revised: SSAM-D.

Original: The results show that the performance degrades as the interval becom longer, suggesting that dense mask updates are necessary for our methods.
Revised: The results show that the performance degrades as the interval becom longer, suggesting that dense mask updates are necessary for our methods.

Original: Both of them are ResNet18 on CIFAR10.
Revised: Both of them are ResNet18 on CIFAR10.

Original: reported in is the
Revised: reported in is the

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: C Limitation and Societal Impacts
Revised: C Limitation and Societal Impacts

Original: [Table] method
Revised: [Table] method

Original: [Table]
Revised: [Table]

Original: is mainly based on sparse operation.
Revised: is mainly based on sparse operation.

Original: At present
Revised: At present

Original: [Table]
Revised: [Table]

Original: the sparse operation that has been implemented is only
Revised: the sparse operation that has been implemented is only

Original: [Table]
Revised: [Table]

Original: 4 sparse operation.
Revised: 4 sparse operation.

Original: The 2:4 sparse operation requires that there are at most two non-zero values in four contiguous memory, which does not hold for us.
Revised: The 2:4 sparse operation requires that there are at most two non-zero values in four contiguous memory, which does not hold for us.

Original: To sum up, there is currently no concrete implemented sparse operation to achieve training acceleration.
Revised: To sum up, there is currently no concrete implemented sparse operation to achieve training acceleration.

Original: But in the future, with the development of hardware for sparse operation, our method has great potential to achieve truly training acceleration.
Revised: But in the future, with the development of hardware for sparse operation, our method has great potential to achieve truly training acceleration.

Original: Societal Impacts.
Revised: Societal Impacts.

Original: In this paper, we provide a Sparse SAM algorithm that reduces computation burden and improves model generalization.
Revised: In this paper, we provide a Sparse SAM algorithm that reduces computation burden and improves model generalization.

Original: In the future, we believe that with the development of deep learning, more and more models need the guarantee of generalization and also the efficient training.
Revised: In the future, we believe that with the development of deep learning, more and more models need the guarantee of generalization and also the efficient training.

Original: Different from the work on sparse networks, our proposed Sparse SAM does not compress the model for hardware limited device, but instead accelerates model training.
Revised: Different from the work on sparse networks, our proposed Sparse SAM does not compress the model for hardware limited device, but instead accelerates model training.

Original: It’s helpful for individuals or laboratories which are lack computing resources.
Revised: It’s helpful for individuals or laboratories which are lack computing resources.