Sharpness-Aware Training for Free
Anonymous Author(s)
AffiliationAddress email
Abstract
Modern deep neural networks (DNNs) have achieved state-of-the-art performancesbut are typically over-parameterized.
The over-parameterization may result inundesirably large generalization error in the absence of other customized trainingstrategies.
Recently, a line of research under the name of Sharpness-Aware Minimization (SAM) has shown that minimizing a sharpness measure, which reflectsthe geometry of the loss landscape, can significantly reduce the generalization error.
However, SAM-like methods incur a two-fold computational overhead of the givenbase optimizer (e.g. SGD) for approximating the sharpness measure.
In this paper,we propose Sharpness-Aware Training for Free , or SAF , which mitigates the sharplandscape at almost zero additional computational cost over the base optimizer.
Intuitively, SAF achieves this by avoiding sudden drops in the loss in the sharplocal minima throughout the trajectory of the updates of the weights.
Specifically,we suggest a novel trajectory loss, based on the KL-divergence between the outputsof DNNs with the current weights and past weights, as a replacement of the SAM’ssharpness measure.
This loss captures the rate of change of the training loss alongthe model’s update trajectory.
By minimizing it, SAF ensures the convergenceto a flat minimum with improved generalization capabilities.
Extensive empirical results show that SAF minimizes the sharpness in the same way that SAMdoes, yielding better results on the ImageNet dataset with essentially the samecomputational cost as the base optimizer.
The code will be released.
Introduction
Despite achieving remarkable performances in many applications, powerful neural networks [5, 27, 28,2 , 22, 27, 36, 35] are typically over-parameterized.
Such over-parameterized deep neural networksrequire advanced training strategies to ensure that their generalization errors are appropriatelysmall [2, 33] and the adverse effects of the overfitting are alleviated.
Understanding how deep neuralnetworks generalize is perhaps the most fundamental and yet perplexing topic in deep learning.
Numerous studies expend significant amounts of efforts to understand the generalization capabilities ofdeep neural networks and mitigate this problem from a variety of perspectives, such as the informationperspective [19], the model compression perspective [1, 9], and the Bayesian perspective [23, 25], etc.
The loss surface geometry perspective, in particular, has attracted a lot of attention from researchersrecently [15, 4, 13, 14, 29].
These studies connect the generalization gap and the sharpness of theminimum’s loss landscape, which can be characterized by the largest eigenvalue of the Hessianmatrix ∇ 2 θ f θ [15] where f θ represents the input-output map of the neural network.
In other words, a(local) minimum that is located in a flatter region tends to generalize better than one that is locatedin a sharper one [4, 14].
The recent work [8] proposes an effective and generic training algorithm,named Sharpness-Aware Minimization (SAM), to encourage the training process to converge to a flatminimum.
SAM explicitly penalizes a sharpness measure to obtain flat minima, which has achievedstate-of-the-art results in several learning tasks [2, 37].
[Figure]
Unfortunately, SAM’s computational cost istwice that compared to the given base optimizer, which is typically stochastic gradient descent (SGD).
This prohibits SAM from being deployed extensively in highly over-parameterizednetworks.
Half of SAM’s computational overhead is used to approximate the sharpness measure in its first update step.
The other half isused by SAM to minimize the sharpness measure together with the vanilla loss in the secondupdate step.
As shown in Figure 1, SAM significantly reduces the generalization error at theexpense of double computational overhead.
Liu et al.
[21] and Du et al.
[6] recently addressed the computation issue of SAM andproposed LookSAM [21] and Efficient SAM(ESAM)
[6], respectively.
LookSAM only minimizes the sharpness measure once in the first ofevery five iterations.
For the other four iterations,
LookSAM reuses the gradients that minimizesthe sharpness measure, which is obtained by decomposing the SAM’s updated gradients in the first iteration into two orthogonal directions.
As aresult, LookSAM saves 40% computations compared to SAM but unfortunately suffers from performance degradation.
On the other hand, ESAM proposes two strategies to save the computationaloverhead in SAM’s two updating steps.
The first strategy approximates the sharpness by usingfewer weights for computing; the other approximates the updated gradients by only using a subsetcontaining the instances that contribute the most to the sharpness.
ESAM is reported to save up to30% computations without degrading the performance.
However, ESAM’s efficiency degrades inlarge-scale datasets and architectures (from 30% on CIFAR-10/100 to 22% on ImageNet).
LookSAMand ESAM
both follow SAM’s path to minimize SAM’s sharpness measure, which limits the potentialfor further improvement of their efficiencies.
In this paper, we aim to perform sharpness-aware training with zero additional computations andyet still retain superior generalization performance .
Specifically, we introduce a novel trajectoryloss to replace SAM’s sharpness measure loss.
This trajectory loss measures the KL-divergencebetween the outputs of neural networks with the current weights and those with the past weights.
We propose the Sharpness-Aware training for Free (SAF) algorithm to penalize the trajectory lossfor sharpness-aware training.
More importantly, SAF requires almost zero extra computations (SAF0.7% v.s. SAM 100%).
SAF memorizes the outputs of DNNs in the process of training as the targetsof the trajectory loss.
By minimizing it, SAF avoids the quick converging to a local sharp minimum.
SAF has the potential to result in out-of-memory issue on extremely large scale datasets, such as
ImageNet-21K [16] and JFT-300M [26].
We also introduce a memory-efficient variant of SAF, whichis Memory-Efficient Sharpness-Aware Training (MESA).
MESA adopts a DNN whose weights arethe exponential moving averages (EMA) of the trained DNN to output the targets of the trajectoryloss.
As a result, MESA resolves the out-of-memory issue on extremely large scale datasets, at thecost of 15% additional computations (v.s. SAM 100%).
As shown in Figure 2, SAF and MESAboth encourage the training to converge to flat minima similarly as SAM.
Besides visualizing theloss landscape, we conduct experiments on the CIFAR-10/100 [17] and the ImageNet [3] datasetsto verify the effectiveness of SAF.
The experimental results indicate that our proposed SAF and
MESA outperform SAM and its variants with almost twice the training speed; this is illustrated onthe ImageNet-1k dataset in Figure 1.
In a nutshell, we summarize our contributions as follows.
• We propose a novel trajectory loss to measure the sharpness to be used for sharpness-awaretraining.
Requiring almost zero extra computational overhead, this trajectory loss is a betterloss to quantify the sharpness compared to SAM’s loss.•
We address the efficiency issue of current sharpness-aware training, which generally incurstwice the computational overhead compared to regular training.
Based on our proposed
trajectory loss, we propose the novel SAF algorithm for improved generalization ability in
this paper.
SAF is demonstrated to outperform SAM on the ImageNet dataset, with the samecomputational cost as the base optimizer.
[Figure]
• We also propose the MESA algorithm as a memory-efficient variant of SAF.
MESA reduces
the extra memory-usage of SAF at the cost of 15% additional computations, which allows
SAF/MESA to be deployed efficiently (both in terms of memory and computation) on
extremely large-scale datasets (e.g. ImageNet-21K [16]).
[Figure]
Preliminaries
[Figure]
Throughout the paper, we use f θ to denote a neural network with weight parameters θ .
We are givena training dataset S that contains i.i.d.
samples drawn from a natural distribution D .
The trainingof the network is typically a non-convex optimization problem which aims to search for an optimalweight vector ˆ θ that satisfies
[Equation]
where ℓ can be an arbitrary loss function, and we use x i to denote the pair (inputs, targets) of the i -thelement in the training set.
In this paper, we take ℓ to be the cross entropy loss; We use ∥·∥ representsℓ 2 norm; we assume that L S ( f θ ) is continuous and differentiable, and its first-order derivation isbounded .
In each training iteration, optimizers randomly sample a mini-batch B t ⊂ S with a fixedbatch size.
Sharpness-Aware Minimization
The conventional optimization and training focuses on minimizing the empirical loss of a single weight vector ˆ θ over the training set S as stated in Equation 1.
Thistraining paradigm is known as empirical risk minimization , and tends to overfit to the training setand converges to sharp minima.
Sharpness-Aware Minimization (SAM) [8] aims to encourage thetraining to converge to a flatter region in which the training losses in the neighborhood around theminimizer ˆ θ are lower.
To achieve this, SAM proposes a training scheme that solves the followingmin-max optimization problem:
[Equation]
where ρ is a predefined constant that constrains the radius of the neighborhood; ϵ is the weightperturbation vector that maximizes the training loss within the ρ -constrained neighborhood.
Theobjective loss function of SAM can be rewritten as the sum of the vanilla loss and the loss associatedto the sharpness measure, which is the maximized change of the training loss within the ρ -constrainedneighborhood, i.e.,
[Equation]
The sharpness measure is approximated as R S ( f θ ) =
L S ( f θ +ˆ ϵ ) − L S ( f θ ) , where ˆ ϵ is the solution toan approximated version of the maximization problem where the objective is the first-order Taylorapproximation of L S ( f θ + ϵ ) around θ , i.e.,
[Equation]
Intuitively, SAM seeks flat minima with low variation in their training losses when the optimalweights are slightly perturbed.
Methodology
The fact that SAM’s computational cost is twice that compared to the base optimizer is its mainlimitation.
The additional computational overhead is used to compute the sharpness term R S ( f θ )in Equation 3.
We propose a new trajectory loss as a replacement of SAM’s sharpness loss R S ( f θ )with essentially zero extra computational overhead over the base optimizer.
Next, we introduce the
Sharpness-Aware Training for Free (SAF) algorithm whose pseudocode can be found in Algorithm 1.
We first start with recalling SAM’s sharpness measure loss.
Then we explain the intuition for thetrajectory loss as a substitute for SAM’s sharpness measure loss in Section 3.1.
Next, we present thecomplete algorithm of SAF in Section 3.2 and a memory-efficient variant of it in Section 3.3.
Input: Training set S ;
A network f θ with weights θ ; Learning rate η ; Epochs E ; Iterations T per epoch;
SAF starting epoch E start ; SAF coefficients λ ; Temperature τ ; SAF hyperparameter ˜ E ;
EMA decay factor β for MESA.
1: for e = 1 to E do ▷ e represents the current epoch 2: for t = 1 to T , Sample a mini-batch B ⊂ S do 3: if SAF then 4:
Record the outputs: ˆ y ei ← f θ ( x i ) , where x i ∈ B 5:
Load ˆ y ( e − ˜ E )
i saved in ˜ E epochs ago for each x i ∈ B 6:
Compute L tra B ( f θ , Y ( e − ˜ E ) )
▷ Defined in Equation 8; 7: else if MESA then 8:
Update EMA model weights: v t = βv
t − 1 + (1 − β ) θ 9:
Compute L tra B ( f θ , f v t )
▷ Defined in Equation10: if e > E start then ▷ Added the trajectory loss after E start epoch 11:
L = L B ( f θ ) + λL tra B 12: else 13:
L = L B ( f θ ) 14:
Update the weights: θ ← θ − η ∇ θ L
Output: A flat minimum solution ˜ θ .
3.1 General Idea: Leverage the trajectory of weights to estimate the sharpness
We first rewrite the sharpness measure R B ( f θ ) of SAM based on its first-order Taylor expansion.
Given a vector ˆ ϵ (Equation 4) whose norm is small, we have
[Equation]
We remark that minimizing the sharpness loss R B ( f θ ) is equivalent to minimizing the ℓ 2 -norm of thegradient ∇ θ L B ( f θ ) , which is the same gradient used to minimize the vanilla loss L B ( f θ ) .
The learning rate η in the standard training (using SGD) is typically smaller than ρ as suggested by
SAM [8].
Hence, if the mini-batch B is the same for the two consecutive iterations, the change of thetraining loss after the weights have been updated can be approximated as follow,
[Equation]
We also remark that the change of the training loss after the weights have been updated is proportionalto R B ( f θ ) 2 .
Hence, minimizing the loss difference is equal to minimizing the sharpness loss of SAMin this case.
[Figure]
This inspires us to leverage the update of the weights in the
[Figure]
standard training to approximate SAM’s sharpness mea-
[Figure]
sure.
Regrettably though, the samples in the mini-batches
B t and B t +1 in two consecutive iterations are different 152 with high probability, which does not allow the sharpnessto be computed as in Equation 6.
This is precisely thereason why SAM uses an additional step to compute ˆ ϵ forapproximating the sharpness.
To avoid these additionalcomputations completely, we introduce a novel trajectoryloss that makes use of the trajectory of the weights learnedin the standard training procedure to measure sharpness.
We denote the trajectory of the weights as the iterationsprogress as Θ = { θ 1 , θ 2 , . . .
, θ t } .
Hence, θ t representsthe weights in the t -th iteration, which is also the currentmodel update step.
We use SGD as the base optimizer to illustrate our ideas.
Recall that standard SGD updates the weights as θ t {++
+} =
θ t − η ∇ θ t L B t ( f θ t ) .
For the current iteration t , we observe that
[Equation]
where γ t =
η t ρ 2 cos(Φ t ) and Φ t is the angle between the gradients that are computed using themini-batches B i and B t , respectively.
The sharpness can therefore be alternatively estimated by
[Equation]
where θ i is is uniformly distributed over the set Θ .
We remark that minimizing the loss difference
L B t ( f θ 1 ) − L B t ( f θ t ) is equivalent to minimizing the SAM’s loss R B t ( f θ t ) .
Accordingly, requiringno additional computational overhead, the training loss difference is a good replacement of SAM’sloss R B t ( f θ t ) to quantify and penalize the sharpness.
3.2 Sharpness-Aware Training for Free (SAF)
We elaborate our proposed training algorithm SAF in this subsection.
To estimate the sharpnessprecisely, SAF only takes the update trajectory in the past ˜ E epochs into consideration.
Whensimultaneously minimizing the vanilla loss and the training loss difference (as in Equation 7), thesecond term − L B T ( f θ T ) will cancel out the vanilla loss.
Therefore, we replace the cross entropy losswith the KL divergence loss to decouple the vanilla loss.
As inspired by Knowledge distillation [12],we also soften the targets of the KL divergence loss using a temperature τ .
Accordingly, the trajectoryloss at e -th epoch is defined as follow
[Equation]
( e − ˜ E )
KL (cid:18)τ ˆ y
( e − ˜ E ) i ,τ f θ ( x i ) (cid:19) , (8) where Y ( e − ˜ E ) =
(cid:8) ˆ y ( e − ˜ E ) i =
f ( e − ˜ E )
θ ( x i ) :
x i ∈ B (cid:9) .
We remark that ˆ y ( e − ˜ E )
i is the network output ofthe instance x
i in ˜ E epochs ago, as illustrated in Line 4 of Algorithm 1.
The outputs of each instancex i will be recorded, and no additional computations are required during this procedure.
The trajectoryloss will be deployed after a predefined epoch E start
(Line 10 of Algorithm 1), because the outputsof the DNN are not stable and reliable at the beginning epochs.
Intuitively, the trajectory loss slowsdown the rate of change of the training loss to avoid convergence to sharp local minima, which isillustrated in Figure 3.
3.3 Memory-Efficient Sharpness-Aware Training (MESA)
The memory usage for recording the outputs is negligible for standard datasets such as CIFAR [17](57 MB).
However, SAF’s memory usage is proportional to the size of the training datasets, which may result in an out-of-memory issue on extremely large-scale datasets such as ImageNet-21k [16]and JFT-300M [26].
Another limitation of SAF is that the coefficient γ t will decay at the same rate asthe learning rate η t since γ t ∝ η t as shown in Equation 7.
Hence, the sharpness at the current weightsare quantified by smaller coefficients γ t .
For a SAF-like algorithm to be applicable on extremelylarge-scale datasets and to emphasize the sharpness of the most current/recent weights, we introduce
Memory-Efficient Sharpness-Aware Training (MESA), which adopts an exponential moving average(EMA) weighting strategy on the weights to construct the trajectory loss.
The EMA weight v t at thet -th iteration is updated as follows
[Equation]
where β ∈ (0 . 9 , 1) is the decay coefficient of EMA.
Given that v 1 = θ 1 , and θ t +1 =
θ t −η ∇ θ t L B t ( f θ t ) , the EMA weight in the t -th iteration, can be expressed as
[Equation]
More details can be found in Appendix.
Therefore, the trajectory from θ t to v t is collected in thevector W EMA = { w 1 , w 2 , . . .
, w t } , where w i =
w i − 1 − β t − i η ∇ θ i L B i ( f θ i ) , w 1 = v t , and w t = θ t .
If we regard the outputs of EMA model f v t as the targets of the trajectory loss, and substitute it into
Equation 7,
[Equation]
More details can be found in Appendix.
Hence, the EMA coefficients β t − i will place more emphasison the sharpness of the current and more recent weights since 0 < β < 1 .
The trajectory loss of
MESA is
[Equation]
We see that the difference between this and the trajectory loss L tra B ( f θ , Y ( e − ˜ E ) ) discussed in Section 3.2 is that the target ˆ y ( e − ˜ E ) i at the t -iteration has been replaced by f v t ( x i ) for x i ∈ B .
Experiments
We verify the effectiveness of our proposed SAF and MESA algorithms in this section.
We firstconduct experiments to demonstrate that our proposed SAF achieves better performance comparingto SAM with twice the training speed.
SAF is demonstrated to outperform SAM and its variants inlarge-scale datasets and models.
The main results are summarized into Tables 1 and 3.
Next, weevaluate the sharpness of SAF using the measurement proposed by SAM.
We demonstrate that SAFencourages the training to converge to a flat minimum.
We also visualize the loss landscape of theminima converged by SGD, SAM, SAF, and MESA in Figures 2 and 5, which shows that both SAF’sand MESA’s loss landscape are as flat as SAM.
4.1 Experiment Setup
Datasets We conduct experiments on the following image classification benchmark datasets: CIFAR10, CIFAR-100 [17], and ImageNet [3].
The 1000-class ImageNet dataset contains roughly 1.28million training images, which is the popular benchmark for evaluating large-scale training.
Models We employ a variety of widely-used DNN architectures to evaluate the performance andtraining speed.
We use ResNet-18 [11], Wide ResNet-28-10 [32], and PyramidNet-110 [10] for thetraining in CIFAR-10/100 datasets.
We use ResNet and Vision Transformer [5] models with varioussizes on the ImageNet dataset.
Baselines We take the vanilla (AdamW for ViT, SGD for the other models), SAM [8], ESAM [6],
GSAM [37], and LookSAM [21] as the baselines.
ESAM and LookSAM are SAM’s follow-up worksthat improve efficiency.
GSAM achieves the best performance among the SAM’s variants.
Implementation details We set all the training hyperparameters to be the same for a fair comparisonamong the baselines and our proposed algorithms.
The details of the training setting are displayed in
Appendix.
We follow the settings of [2, 21, 6, 37] on the ImageNet datasets which is different fromthe experimental setting of SAM [8].
The codes are implemented based on the TIMM framework[30].
The ResNets are trained with a batch size of 4096, 1.4 learning rate, 90 training epochs, and
SGD optimizer (momentum=0.9) over 8 Nvidia V-100 GPU cards.
The ViTs are trained with 300training epochs and AdamW optimizer ( β 1 = 0 .
9 , β 2 = 0 .
We only conduct the basic dataaugmentation for the training on both CIFAR and ImageNet (Inception-style data augmentation).
Thehyperparameters of SAF and MESA are obtained via grid searches on the CIFAR datasets, which areconsistent among various DNNs architectures and various datasets.
We set τ = 5 , ˜ E
= 3 , E start =
5for all the experiments on CIFAR, λ ∈ { 0 . 3 , 0 . 8 } for SAF and MESA , respectively.
[Table]
4.2 Experimental Results
ImageNet Our proposed SAF and MESA achieve better performance on the ImageNet datasetcompared to the other competitors.
We report the best test accuracies and the average training speedsin Table 1.
The experiment results demonstrate that SAF incurs no additional computational overheadto ensure that the training speed is the same as the base optimizer—SGD.
We observe that SAF and
MESA perform better in the large-scale datasets.
SAF trains DNN at a douled speed than SAM(SAF 99.2% vs SAM 50%).
MESA achieves a training speed which is 84% to 85% that of SGD.
Concerning the performance, both SAF and MESA outperform SAM up to (0.9%) on the ImageNetdataset.
More importantly, SAF and MESA achieve a new state-of-the-art results of SAM’s follow-upworks on the ImageNet datasets trained with ResNets.
CIFAR10 and CIFAR
We ran all the experiments three times with different random seeds forfair comparisons.
We summarize the results in Table 3, in the same way as we do for the experimentson the ImageNet dataset.
The training speed is consistent with the results on ImageNet.
Similarly,
SAF and MESA outperform SAM on large-scale models—Wide Resnet and PyradmidNets.
4.3 Discussion
Memory Usage We evaluate the additional memory usage of SAF and MESA on the ImageNetdataset with the ResNet-50 model.
We only present the results on the ImageNet in Table 2, because
200 Epochs 0.
T r a i n i n g L o ss )
Training Loss vs Epochs Vanilla (SGD)
SAM SAF
MESA
[Figure]
200 Epochs 0.
S h a r p n e ss ( S
A M )
Sharpness vs Epochs Vanilla (SGD)
[Figure]
the memory usage of SAF on the CIFAR dataset is negligible (57 Mb).
MESA saves 99 .
3% memoryusage compared to SAF, which allows MESA to be deployed on extremely large-scale datasets.
Algorithms
[Table]
Convergence Rate Intuitively, SAF minimizes the trajectory loss to control the rate of training loss change to obtainflat minima.
A critical problem of SAF may be SAF’s influence on the convergence rate.
We empirically show that thechange of the sharpness-inducing term in SAF and MESA(compared to SAM) will not affect the convergence rateduring training.
Figure 4a illustrates the change of the training loss in each epoch of SGD, SAM, SAF, and MESA .
Itshows that SAF and MESA converge at the same rate as SGD and SAM.
Sharpness We empirically demonstrate that the trajectory loss can be a reliable substitute of thesharpness loss proposed by SAM.
We plot the sharpness, which is the measurement of SAM in
Equation 6, in each epoch of SAM, SGD, SAF, and MESA during training.
As shown in Figure4b, SAF and MESA minimize the sharpness as SAM does throughout the entire training procedure.
Both SAF’s and MESA’s sharpnesses drop significantly at epoch 5, from which the trajectory lossstart to be minimized.
SAM’s sharpness is lower than SAF’s and MESA’s in the second half of thetraining.
A plausible reason is that SAM minimizes the sharpness directly.
However, SAF and MESAoutperform SAM in terms of the test accuracies, as demonstrated in Table 1.
Visualization of Loss Landscapes
We also demonstrate that minimizing the trajectory loss is an effective way to obtain flat minima.
We visualize the loss landscape of converged minima trainedby SGD, SAM, SAF, and MESA in Figures 2 and 5.
We follow the method to do the plottingproposed by [18], which has also been used in [6, 2].
The x - and y -axes represent two randomsampled orthogonal Gaussian perturbations.
We sampled 100 × 100 points with random Gaussianperturbations for visualization.
The visualized loss landscape clearly demonstrate that SAF canconverge to a region as flat as SAM does.
Other Related Works
The first work that revealed the relation between the generalization ability and the geometry of theloss landscape (sharpness) can be traced back to [13].
Following that, many studies verified therelation between the flat minima and the generalization error [15, 4, 14, 20, 18, 7, 24] .
Specifically,
Keskar et al.
[15] proposed a sharpness measure and indicated the negative correlation between the
[Table]
sharpness measure and the generalization abilities.
Dinh et al.
[4] further argued that the sharpnessmeasure can be related to the spectrum of the Hessian, whose eigenvalues encode the curvatureinformation of the loss landscape.
Jiang et al.
[14] demonstrated that one of the sharpness-basedmeasures is the most correlated one among 40 complexity measures by a large-scale empirical study.
SAM [8] solved the sharp minima problem by modifying training schemes to approximate andminimize a certain sharpness measure.
The concurrent works [31, 34] propose a model to adversarially perturb trained weights to bias the convergence.
The aforementioned methods lead theway for sharpness-aware training despite the computational overhead being doubled over that ofthe base optimizer.
Subsequently, LookSAM [21] and Efficient SAM (ESAM)
[6] were proposedto alleviate the computational issue of SAM.
Apart from efficiency issues, Surrogate Gap Guided
Sharpness-Aware Minimization (GSAM)
[37] was proposed to further improve SAM’s performance.
GSAM places more emphasis on the gradients that minimize the sharpness loss to achieve the bestgeneralization performance among all the research that followed SAM.
Conclusion and Future Research
In this work, we introduce a novel trajectory loss as an equivalent measure of sharpness, whichrequires almost no additional computational overhead and preserves the superior performance ofsharpness-aware training [8, 6, 37, 21].
In addition to deriving a novel trajectory loss that penalizessharp and sudden drops in the objective function, we propose a novel sharpness-aware algorithm SAF,which achieves impressive performances in terms of its accuracies on the benchmark CIFAR and
ImageNet datasets.
More importantly, SAF trains DNNs at the same speed as non-sharpness-awaretraining (e.g., SGD).
We also propose MESA as a memory-efficient variant of SAF to avail sharpnessaware training on extremely large-scale datasets.
In future research, we will further enhance SAFto automatically sense the current state of training and alter the training dynamics accordingly.
Wewill also evaluate SAF in more learning tasks (such as natural language processing) and enhance it tobecome a general-purpose training strategy.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
Yes
(b) Did you describe the limitations of your work?
Yes, in Section 3.3.
(c) Did you discuss any potential negative societal impacts of your work?
N.A.
Our
research work does not have any potential negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
Yes
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
Yes
(b) Did you include complete proofs of all theoretical results?
Yes
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
Yes, in Appendix.
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
Yes, in Section 4 and Appendix.
(c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
N.A.
(d) Did you include the total amount of compute and the type of resources used (e
g., typeof GPUs, internal cluster, or cloud provider)?
Yes, in Section 4.4.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
Yes, in Section 4.
(b) Did you mention the license of the assets?
N.A.
(c) Did you include any new assets either in the supplemental material or as a URL?
N.A.
(d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
N.A.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
N.A.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
N.A.
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
N.A.
(c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
N.A.