Random Sharpness-Aware Minimization
Paper ID:
Abstract
Currently, Sharpness-Aware Minimization (SAM) is proposed to seek the parameters that lie in a flat region to improve the generalization when training neuralnetworks.
In particular, a minimax optimization objective is defined to find themaximum loss value centered on the weight, out of the purpose of simultaneouslyminimizing loss value and loss sharpness.
For the sake of simplicity, SAM appliesone-step gradient ascent to approximate the solution of the inner maximization.
However, one-step gradient ascent may not be sufficient and multi-step gradientascents will cause additional training costs.
Based on this observation, we proposea novel random smoothing based SAM (R-SAM) algorithm.
To be specific, R-SAMessentially smooths the loss landscape, based on which we are able to apply theone-step gradient ascent on the smoothed weights to improve the approximation ofthe inner maximization.
Further, we evaluate our proposed R-SAM on CIFAR and
ImageNet datasets.
The experimental results illustrate that R-SAM can consistentlyimprove the performance on ResNet and Vision Transformer (ViT) training.
Introduction
Deep learning has seen remarkable progress in a variety of areas, with advanced development inover-parameterized neural networks.
However, over-parameterization usually leads to overfittingand suffers poor generalization for unseen data [31].
To this end, many studies have investigated thegeneralization of deep neural network to narrow down the gap [7, 17, 26, 29, 42].
In particular, theytry to analyse the local minima problem and connect it with loss geometry [17, 27, 31].
Amongstthem, Foret et al.
propose SAM, with an aim to solve a minimax optimization problem.
SAM initiallyfinds the maximum of loss function in the region of radius ρ centered on the w , regarding which tominimize the loss value.
This will prompt the model to converge to a flat region, where the entireneighborhood tends to have uniformly low training loss.
Inner maximization is a vital part of SAMprocedure as it essentially describes the loss landscape and plays an important role in whether wecan reduce the loss sharpness effectively.
The solution for inner maximization is NP-hard, and forsimplicity, SAM uses one-step gradient ascent directly to approximate the maximum loss.
However,one-step gradient ascent may not be sufficient. In addition, multi-step gradient ascents are not efficientand will cause additional training costs.
Therefore, how to improve the approximation of the innermaximization without additional computation cost is an important problem.
Based on our empirical analysis in Figure 1 and Theorem 1, we find that random initialization cansmooth the loss landscape and obtain a more stable gradient direction.
Noted that a more stablegradient can make a good approximation to the true maximum value.
Therefore, in this paper,we propose a novel random smoothing based sharpness-aware minimization
algorithm
(R-SAM).
Our proposed R-SAM consists of two steps.
First, we use a Gaussian noise to smooth the losslandscape and escape from the local sharp region to obtain a stable gradient for gradient ascent.
Second, we perform the one-step gradient ascent on the smoothed weights to eventually enhance theapproximation quality of inner maximization.
We summarize our contributions as follows:
• In this paper, we investigate the approximation quality of this one-step gradient ascent.
Ourevaluations show that unstable gradients might hurt the inner maximization of SAM and addingrandom smoothing can improve gradient stability.• Based on our observation, we propose a simple but novel algorithm, R-SAM, which leveragesa Gaussian noise in perturbation initialization and successfully enhance the inner maximizationwithout introducing additional cost.•
We conduct empirical studies on ResNet and ViT.
The experimental results demonstrate thatR-SAM can help model generalization ability and yield more remarkable performance comparedwith SAM across a wide range of computer vision tasks (e
g., CIFAR-10/100, ImageNet).
Background
2.1 Sharpness-Aware Minimization
Many studies focus on analyzing the generalization of deep neural networks.
For example, motivatedby the local geometry of the energy landscape, Chaudhari et al.
propose Entropy-SGD to constructa local-entropy-based objective function that favors well-generalizable solutions lying in large flatregions of the energy landscape, to address generalization problems in the training.
Izmailov et al.illustrate the benefits of Stochastic Weight Averaging (SWA) procedure in solving generalizationproblems and well apply SWA to generate much flatter solutions than SGD.
Keskar et al.
shows that deep neural network training is easy to converge to sharp local minima andcauses a generalization gap.
Therefore, how to make the model converge to a flat local minima andimprove the generalization is an important problem.
Foret et al.
propose SAM to seek the parametersthat lie in a flat region through optimizing loss value and loss sharpness.
Then we will overview
Sharpness-Aware Minimization (SAM).
Consider the training dataset as S = { ( x i , y i ) } ni =1 , where each sample ( x i , y i ) follows the distribution
D .
Define f
x ; w ) as the neural network model with trainable parameter w ∈ R p , with which the lossfunction w.r.t.
an input x i can be given by l ( f ( x i ; w ) , y i ) ∈ R + , shortened to l ( x i ) for convenience.
The empirical training loss is L = 1 n (cid:80) ni
=1 l ( f ( x i ; w ) , y i ) .
The objective of SAM [18] is to find theparameters with uniformly low loss value within the ℓ p ball.
The objective function is proposed asfollows.
[Equation]
where p ≥ 0 is defined as the radius of the ℓ p ball.
For simplicity, we ignore p when using ℓ 2 -norm.
As calculating the optimal solution of inner maximization is infeasible, SAM uses one-step gradientascent to approximate it:
[Equation]
Finally, SAM computes the gradient regarding the perturbed model w + ˆ δ for the update:
[Equation]
Although SAM can achieve a great performance in traditional computer vision [8] and NLP tasks [3],it still exists several drawbacks yet to be resolved.
To further improve its performance, Kwon et al.introduce the concept of adaptive sharpness of loss function to solve the scale-dependency problem.
Zhuang et al.
propose GSAM to seek the region with both small loss value and low sharpness basedon a novel optimization objective.
Zhou & Chen try to learn a dynamically reweighted perturbationfor each batch to obtain a better approximation to per-instance perturbation.
Noted that SAM needsto calculate two sequential gradients.
To improve its efficiency, Du et al.
propose ESAM, which usestochastic weight perturbation and sharpness-sensitive data selection to reduce the computation ofvanilla SAM.
Liu et al.
propose LookSAM to reuse the projected gradient that target to the flat regionevery k steps to accelerate ViT training.
SS-SAM is proposed to improve the efficiency by randomlyselecting SGD optimization or SAM at each step [51].
2.2 Generalization
There is a venerable line of important works on the problem of generalization in deep learning[5, 12,21, 28, 30].
For example, Hardt et al.
focus on generating generalization bounds for models learnedwith stochastic gradient descent.
Bisla et al.
investigate the sharpness of loss landscape and proposedsystematic mechanisms underlying the generalization abilities of deep learning models.
Jin et al.analyse the importance of perturbed gradient descent in efficiently escaping saddle points.
Further,
Jin et al.
investigate the use of a Hamiltonian function and propose a new framework to track thelong-term behavior of gradient-based optimization algorithms.
In addition, Dinh et al.
argue that theconcepts of flatness might be problematic and can not be directly applied to explain generalization,which requires rethinking what flatness actually means.
Adversarial training has achieved a great success in improving the robustness of model [13, 20,39, 40, 45].
Our approach was also inspired by recent works in random smoothing in adversarialtraining[1, 9, 44, 47, 50].
Wong et al.
demonstrate that applying random initialization in FGSMadversarial training can in fact be just as effective as the more costly PGD adversarial training.
Further,
Andriushchenko & Flammarion study the questions of when and why FGSM adversarial trainingworks, and how to enhance the solution of the inner maximization problem.
Zhang et al.
propose anovel bi-level optimization-based fast adversarial training framework, FAST-BAT to address the issueof overfitting and achieve better performance in generalization ability.
Proposed Method
[Figure]
3.1 Empirical Observation on Approximation of Inner Maximization
[Figure]
As shown in Section 2.1, SAM defines a min-max optimization objective to find the maximum lossvalue L SAM ( w ) near the current weight w and then minimize the value L SAM ( w ) .
However, for theinner maximization problem, finding the optimal solution is NP-hard, which means that it’s difficult toobtain the true maximum value.
Therefore, SAM uses a one-step gradient ascent to approximate theoptimal solution and obtain the maximum loss value.
In this paper, we investigate the approximationquality of this one-step gradient ascent and propose a simple way to improve the inner maximizationwithout introducing additional cost.
Inner maximization of SAM may be hurt by unstable gradient.
The adversarial weight w advwith the optimal solution of inner maximization can be defined as:
[Equation]
SAM uses a one-step gradient ascent to approximate the optimal adversarial weight:
[Equation]
However, the loss landscape of neural network is usually sharp [31, 35].
That means the directionof g ( w ) = ∇ w L ( w ) cannot represent the direction to maximize its neighbourhoods’ loss value.
However, a stable gradient direction in a continuous parameter space can narrow the approximationgap and help SAM obtain a better approximation of w ∗ adv .
Therefore, we try to conduct theexperiments in ResNet and WideResNet to analyse the smoothness of gradient g ( w ) .
As shownin Figure 1, we plot the cosine value between the g ( w ) and g ( w + ρ g ( w ) || g ( w ) || ) to visualize thesmoothness.
From this figure, we can find that the cosine value decreases significantly with the ρvalue increasing (the red line).
Intuitively, when the gradient is constant in a certain region (cosinesimilarity equals to 1), the one-step gradient ascent will get the exact maximizer; while when thegradient becomes unstable in a certain region, the one-step gradient ascent will lead to a poor solution.
Therefore, how to smooth the loss landscape and make the gradient stable during the gradient ascentprocess is an important problem.
Random smoothing improves gradient stability.
Some studies illustrate that random smoothingcan smooth the loss landscape [22, 46].
That motivates us to analyze the effects of random smoothingfor the smoothness of gradient g ( w ) .
In Figure 1, we sample
[Figure]
and obtain the weight ˆ w = w + δ 0 .
Next, we calculate the cosine value between g ( ˆ w ) andg ( ˆ w + ρ g ( ˆ w )
||
g ( ˆ w ) || ) to compare the smoothness with the gradient without random smoothing ( g ( w ) ).
From Figure 1, we can find that the cosine value with random smoothing (the green, blue andorange line) is higher than the vanilla cosine value on g ( w ) (the red line), which means that randomsmoothing can reduce the decay rate of cosine value.
In addition, the cosine value will be more stablewith the γ value increasing when using random smoothing (from γ 2 = 0 .
0075 to γ 2 = 0 . 0125 )
in
Figure 1.
The above observation illustrates that random smoothing can smooth the loss landscapeand obtain a more stable gradient.
We also provide the theoretical analysis for this observation.
Theorem 1.
Let L ( w ) be the vanilla loss function and L S ( w ) =
L ( w + δ 0 ) be the smoothed lossfunction.
Assuming vanilla loss function L ( w ) is α -Lipschitz
continuous
and the gradient ∇ w L ( w )is β -Lipschitz continuous.
Given the random noise δ 0 is from Gaussian distribution N (0 , γ 2 I ) .
Wecan obtain that the gradient ∇ w L S ( w ) = ∇ w L ( w + δ 0 ) is min { αγ , β } -Lipschitz continuous, whereγ is the standard deviation of δ 0 .
Theorem 1 is based on [5, 16], which implies that L S ( w ) =
L ( w + δ 0 ) is more smooth thanthe original loss function L ( w ) when αγ ≤ β .
In addition, with γ value (standard deviation of Gaussian noise) increasing, the loss function L S ( w ) become smoother.
That is also consistent withour observation in Figure 1, where the change of cosine value will be reduced with γ 2 increasing.
Therefore, based on the above intuition about random smoothing, we can use the Gaussian noise δ 0 toinitialize the one-step gradient ascent process.
That means we can firstly use Gaussian noise to escapefrom the local sharp region and smooth the loss landscape (initial perturbation), and then conduct theone-step gradient ascent step.
More specifically, the random initialization for inner maximizationcan be defined as ˆ w = w + δ 0 .
After that, we use the smoothed weight ˆ w to achieve the one-stepgradient ascent process and obtain the approximation of optimal adversarial weight:
[Equation]
Finally, we investigate whether the smoothed gradient can benefit the inner maximization and obtaina better approximated solution in practical applications.
We try to plot the perturbed loss value( L ( w + ρ g ( w ) ||
g ( w ) || ) and L ( w + δ
+ ρ g ( w + δ 0 )
||
g ( w + δ 0 ) || ) ) in Figure 2.
From this figure, we can find thatthe loss value is very close for different γ 2 value when the ρ value is small ( ρ ≤ 0 . 2 ).
However, withthe ρ value gradually increasing, the loss value grow slowly for vanilla weight w .
That means g ( w )cannot efficiently increase the loss value and is not applicable for solving the inner maximizationwith the region expanding.
That limited the region size that vanilla weight can select.
In addition, weobserve that the weight with Gaussian noise ( w + δ 0 ) shows a more stable pattern for the increasing
[Figure]
of loss value to solve the inner maximization problem.
Applying one-step gradient ascent based on thegradient of smoothed weight w + δ 0 can better maximize the loss value in the neighbouring region.
The above analysis motivates us to propose a random smoothing based SAM algorithm to obtain abetter solution for inner maximization.
We will introduce our proposed Random Sharpness-Aware Minimization algorithm
(R-SAM) in the next section.
3.2 Random Sharpness-Aware Minimization (R-SAM)
Based on the observation and intuition in section 3.1, we propose a novel random initialization based
Sharpness-Aware Minimization algorithm
(R-SAM) to benefit the solution of inner maximization in
SAM.
For R-SAM, we use Gaussian noise to initialize the perturbation for weight w and obtain thesmoothed weight ˆ w = w + δ 0 .
After that, we use the smoothed weight ˆ w to calculate the gradientg ( ˆ w ) for gradient ascent process: g ( ˆ w ) = ∇ ˆ w L ( ˆ w ) = ∇ w L ( w + δ 0 ) .
In this way, we can obtaina more stable gradient to solve the inner maximization problem.
More specifically, we will use thegradient g ( ˆ w ) to finish the gradient ascent step and obtain adversarial weight w adv .
A simple wayto obtain w adv is that we can directly add the gradient-based perturbation g ( ˆ w ) = ∇ ˆ w L ( ˆ w ) to thesmoothed weight ˆ w = w + δ 0 , which is defined as w adv = w + δ
+ ρ g ( ˆ w ) ||
g ( ˆ w ) || .
However, that willcause the norm of total perturbation δ
+ ρ g ( ˆ w ) ||
g ( ˆ w ) || larger than vanilla ρ .
To constrain the norm oftotal perturbation, we try to project the perturbation into the ball with radius ρ .
The final perturbationcan be defined as:
[Equation]
where δ 0 represents the Gaussian noise for random initialization and β is used to control the ratiobetween δ 0 and g ( ˆ w ) .
Noted that the selection of Gaussian noise plays an important role in R-SAM.
However, differentfrom the input image data that each pixel is from the same distribution, the distribution of weightsin each layer or kernel shows different distributions.
That makes the noise design be a challengingproblem.
Related work has illustrated that the weight in each kernel is from an independent Gaussiandistribution [25].
Inspired by this work, in this paper, we use kernel-wise Gaussian noise δ 0 ∼ 183 N (0 , γ 2 · diag ( || w j || kj )) as the initialization of perturbation in R-SAM, where || w j || kj represents theℓ 2 -norm of k -th kernel in the neural network.
Algorithm
Input: x ∈ R d , w represents vanilla weight, ˆ w is the smoothed weight, learning rate η t , k is the number of kernels in model.
for t ← 1 to T do
Sample Minibatch B = { ( x i , y i ) , · · · , ( x |B| , y |B| ) } from X .
δ
= N (0 , γ 2 · diag ( || w j || kj =1 ))
Obtain a stable gradient g ( ˆ w ) = ∇ ˆ w L ( ˆ w ) = ∇ w L ( w + δ 0 ) on minibatch B .
Compute gradient ascent w adv = w + ρ δ
+ λ g ( ˆ w ) ||
δ
+ λ g ( ˆ w ) ||
Compute SAM gradient: g sam = ∇ w L ( w ) | w adv
Update weights: w t +1 =
w t − η t g sam end for
The pseudo-code of R-SAM can be found in Algorithm 1.
Firstly, we need to sample a minibatch data
B = { ( x i , y i ) , · · · , ( x |B| , y |B| ) } from dataset X .
After that, we will calculate the norm of each kernel|| w j || kj =1 and sample kernel-wise Gaussian noise from the distribution N (0 , γ 2 · diag ( || w j || kj ))
asthe initialization of perturbation.
Then we can obtain g ( ˆ w ) =
∇ w L ( w + δ 0 ) as the stable gradientto better approximate the optimal adversarial weight w ∗ adv .
More specifically, we can use one-stepgradient ascent to calculate w adv = w + ρ δ + λ g ( ˆ w ) ||
δ + λ g ( ˆ w ) || .
In this way, we can calculate the gradienton w adv : g sam = ∇ w L ( w ) | w adv .
Finally, following the process in vanilla SAM, we can use thegradient g sam to update the vanilla weight w .
As shown in Algorithm 1, R-SAM is very easy toreproduce.
Experiments
4.1 Experimental Settings
In this section, we try to evaluate the performance of our proposed R-SAM optimizer on ResNet[23], WideResNet [48] with CIFAR [33] datasets and Vision Transformer (ViT) [14] with ImageNet[11] dataset.
First, compared with vanilla SAM, the experimental results illustrate that our proposed
R-SAM can improve the accuracy in CIFAR and ImageNet.
In addition, the ablation study andsensitivity analysis show that our proposed R-SAM is robust for hyper-parameters.
4.1.1 Datasets
To validate the performance of proposed method R-SAM, we firstly try to conduct the experimentson widely used CIFAR-10, CIFAR-100 [33] and ImageNet-1k [11] datasets.
To further validate theperformance and robustness of R-SAM, we also report the experiment results on ImageNet-Real [4],
ImageNet-V2 [41], ImageNet-R [24] and ImageNet-C [19].
4.1.2 Models
We firstly evaluate the performance of our proposed R-SAM in CNN-based models, such as ResNet18, ResNet-50 [23] and WideResNet (WRN-28-20) [48].
Currently, Transformer-based modelsreceive a great deal of attention in the field of computer vision [37, 43].
After that, we also try toevaluate our proposed algorithm R-SAM in Vision Transformer (ViT) [14].
4.1.3 Baselines
To illustrate that the random initialization in R-SAM can help to escape the non-smooth vicinity andbenefit to the approximation in inner maximization, the main baseline of this paper is vanilla SAM[18].
In addition, to show that SGD+M is more suitable for CIFAR training, we also try to comparethe accuracy of SGD+M with SGD, RMSProp and AdamW [38].
To further show the generality of
R-SAM, we also try to combine it with GSAM [53] and propose R-GSAM algorithm.
The experimentresults illustrate that R-GSAM can obtain a better performance compared with vanilla GSAM in ViT training.
That also illustrates the effectiveness of random initialization in R-SAM and R-GSAM
4.1.4 Implementation Details
The experiments in this paper are implemented with JAX [6] on Google TPU-V3 chips.
The hyperparameters and experimental setting is based on vanilla SAM [8, 18].
For ResNet training, weuse SGD with Momentum as the base optimizer of SAM and R-SAM.
For ViT training, the baseoptimizer of SAM and R-SAM is AdamW [38].
As for data augmentation, we select mixup [49] and
RandAugment [10] to conduct the experiments.
Noted that we do not use stochastic depth for ViTtraining.
The implementation details can be found in Appendix 6.2.
4.2 Training from Scratch on CIFAR-
To evaluate the performance of our proposed R-SAM, we conduct experiments of ResNet-18, ResNet50 and WideResNet (WRN-28-20) training on CIFAR-10 dataset.
Following the experimental settingin vanilla SAM [18], we use basic augmentation to preprocess the input image data and the baseoptimizer for SAM and R-SAM is SGD with Momentum (SGD+M).
The experimental results are shown in Table 1.
We can find that SAM-based optimizers (SAMand R-SAM) can achieve a better performance than traditional first-order optimizers (SGD and
AdamW).
For example, the accuracy of traditional first-order optimizers are around 96%, and thebest accuracy is 96.5% from SGD with Momentum (SGD+M).
However, SAM can improve theaccuracy from 96.5% to 97.3% for WRN-28-20.
That illustrates the power of SAM to improve theperformance of neural networks.
In addition, R-SAM can further improve the performance comparedwith vanilla SAM.
More specifically, R-SAM can improve the accuracy of SAM from 97.3% to97.5% on WRN-28-20 for CIFAR-10.
Noted that the accuracy of SAM for CIFAR-10 is pretty highand the improvement of R-SAM is also convincing.
-10 for 200 epoch.
The base optimizer for
SAM and R-SAM is SGD with Momentum (SGD+M).
We select batch size as 128.
[Table]
4.3 Training from Scratch on CIFAR-100
To further validate the performance of our proposed R-SAM [Table] we also try to evaluate its accuracy on
CIFAR-100 dataset.
We also follow the setting in SAM [18] and use basic augmentation to preprocessthe image data.
We select SGD+M as the base optimizer of SAM and R-SAM.
[Table]
The experimental results are shown in Table 2, we notice that the results show a similar pattern with
CIFAR-10 and can achieve a higher improvement.
In particular, SAM-based optimizers can obtainbetter performance compared with first-order optimizers.
For example, the best accuracy of first-orderoptimizers is 81.7% for WRN-28-20 model.
SAM can improve the accuracy from 81.7% to 84.6%for WRN-28-10 and from 81.4% to 83.3% for ResNet-50.
In addition, R-SAM can obtain a higheraccuracy compared (from 84.6% to 85.2% for WRN-28-20 and 83.3% to 84.0% for ResNet-50).
Theabove experimental results also illustrate that our proposed R-SAM can further help SAM to obtainbetter performance.
4.4 ViT Training from Scratch on ImageNet
Currently, Vision Transformer has been widely used in a variety of areas [37, 43].
Chen et al.illustrate that SAM can significantly improve the performance of ViT training.
Therefore, evaluatingthe performance of our proposed R-SAM on ViT is pretty important for verifying our contribution.
In this section, we select ViT-B-16 and ViT-S-16 to evaluate the performance of R-SAM.
Theexperimental setting follows the introduction in [8, 18].
Noted that the base optimizer of SAM and
R-SAM are both
AdamW.
We select batch size as 4096 for ViT training.
ViT Training.
The experimental results of ViT training on ImageNet are shown in Table 3.
Fromthis table, we can observe that SAM can also significantly improve the accuracy of ViT training.
Forexample, the accuracy of ViT-B-16 is improved from 74.7% (AdamW) to 79.8% (SAM).
In addition,when using SAM, the accuracy of ViT-S-16 can also be improved from 74.9% to 77.9%.
However,our proposed R-SAM can further improve the accuracy of ViT training compared with vanilla SAMand achieve better performance.
More specifically, R-SAM can improve the accuracy of ViT-B-16from 79.8% (vanilla SAM) to 80.7%.
Besides, ViT-S-16 can also obtain benefits from R-SAM andthe accuracy is increased from 77.9% (vanilla SAM) to 78.7%.
The above experimental results furthervalidate the effectiveness of our proposed R-SAM, especially when compared with vanilla SAM.
[Table]
Data Augmentation.
To further evaluate the generality of our proposed R-SAM, we also try tocombine R-SAM with strong augmentation methods and the results are shown in Table 3.
In thisexperiment, we select mixup [49] and RandAugment (RandAug) [10] as the augmentation methodsto conduct the experiments.
Firstly, we can find that augmentation methods can further improve theaccuracy of AdamW and vanilla SAM.
For instance, we can observe that mixup can improve theaccuracy of vanilla ViT-B-16 with AdamW from 74.7% to 75.7%.
In addition, when SAM is selected,the accuracy will further increase from 79.8% to 80.4% with mixup for ViT-B-16.
In addition, the experimental results illustrate that our proposed R-SAM can further improve theaccuracy of vanilla SAM even with the strong augmentations.
For example, R-SAM can improve theaccuracy of SAM from 80.4% to 81.1% even using strong augmentations.
These experiments furtherverify that our proposed R-SAM can be combined with strong augmentations and illustrate a greatgenerality.
Finally, we find that mixup and RandAug can work together to benefit the performanceof R-SAM.
As shown in Table 3, when RandAug is selected, the accuracy is increased to 81.6%compared to simply using only the mixup.
[Table]
Further Evaluations and Robustness.
Firstly, to further validate the performance of R-SAM,we report the experiment results on ImageNet-Real and ImageNet-v2.
As shown in Table 4, we
SAM
GSAM
R-GSAM 79.
A cc u r a c
y
(a) Comparison with GSAM 1.
A cc u r a c y (b) Sensitivity Analysis of ρ  3 79.
A cc u r a c y (c) Sensitivity Analysis of λ
We can find that R-SAM shows a stable pattern with different
SAM can observe that R-SAM can still improve the accuracy of ViT compared with vanilla SAM.
Forexample, R-SAM can improve the accuracy of ViT-B-16 from 85.2% to 86.1% on ImageNet-Real.
Inaddition, to evaluate the robustness of trained model with R-SAM, we also conduct the experimentsin ImageNet-R and ImageNet-C datasets.
The experimental results are also shown in Table 4 , whichillustrate that SAM can improve the robustness of trained model compared with AdamW.
In addition,
R-SAM can maintain the advantage of SAM and further strengthen its robustness.
For example, SAMcan improve the accuracy of ViT-B-16 from 20.1% to 26.4% on ImageNet-R. R-SAM can furtherimprove the accuracy to 26.9%.
4.5 Compared with Algorithms in the Literature
In section 2.1, we introduced some related work about SAM.
In this section, we try to evaluatethe performance of R-SAM compared with the algorithms in the literature.
We find that GSAMcan achieve the best accuracy for ViT training.
Therefore, we mainly focus on the comparisonwith GSAM.
The main reults are from ViT-B-16 training on ImageNet-1k dataset and are shown in
Figure 3(a).
We can find that R-SAM can obtain the similar accuracy compared with vanilla GSAM.
Howvever, when combining R-SAM and GSAM and named it as R-GSAM, we can obtain betterperformance.
For instance, the accuracy of R-SAM and GSAM are 80.7% and 80.8% respectively.
When using R-SAM for GSAM and obtain the results of R-GSAM, we can observe that the accuracyincrease from 80.8% to 81.2%.
The main purpose of this paper is to illustrate that random smoothingcan benefit the inner maximization and improve the performance of SAM.
4.6 Sensitivity Analysis
Reproducibility is critical for deep learning applications.
We notice that deep learning is usuallysensitive to hyper-parameters.
Therefore, in this section, we try to study the effects of hyperparametersfor the performance of ViT training on ImageNet-1k dataset.
More specifically, we mainly focus onthe effects of ρ and λ .
The reason is that ρ determines the radius of perturbation and λ controls theratio between Gaussian noise δ 0 and gradient-based perturbation g ( ˆ w ) ||
g ( ˆ w ) || .
The analysis results areshown in Figure 3.
We can find that ρ and λ shows a stable pattern for tuning.
That further improvesthe reproducibility of R-SAM and make it more applicable in real scenarios.
Conclusion
In this paper, we observe that simply applying one-step gradient ascent in the inner maximization of
SAM may suffer from poor approximation error due to unstable gradient.
To resolve this problem,we propose a novel algorithm R-SAM, which provably improves the gradient stability by initializingthe inner maximization of SAM with a Gaussian noise.
The experimental results on ResNet and
ViT illustrate that R-SAM can improve the generalization performance of the model and yield betterperformance compared with SAM on CIFAR-10, CIFAR-100 and ImageNet datasets.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes]
(c) Did you report error bars (e g., with respect to the random seed after running experiments multiple times)?
[No] It would be too computationally expensive for us.
We
conduct all the experiments at least three times and report the average value.
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
e ) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]