Random Sharpness-Aware Minimization
Abstract
Currently, Sharpness-Aware Minimization (SAM) is proposed to seek the parameters that lie in a flat region to improve the generalization when training neural networks.
In particular, a minimax optimization objective is defined to find the maximum loss value centered on the weight, out of the purpose of simultaneously minimizing loss value and loss sharpness.
For the sake of simplicity, SAM applies one-step gradient ascent to approximate the solution of the inner maximization.
However, one-step gradient ascent may not be sufficient and multi-step gradient ascents will cause additional training costs.
Based on this observation, we propose a novel random smoothing based SAM (R-SAM) algorithm.
To be specific, R-SAM essentially smooths the loss landscape, based on which we are able to apply the one-step gradient ascent on the smoothed weights to improve the approximation of the inner maximization.
Further, we evaluate our proposed R-SAM on CIFAR and
ImageNet datasets.
The experimental results illustrate that R-SAM can consistently improve the performance on ResNet and Vision Transformer (ViT) training.
Introduction
Deep learning has seen remarkable progress in a variety of areas, with advanced development in over-parameterized neural networks.
However, over-parameterization usually leads to overfitting and suffers poor generalization for unseen data [31].
To this end, many studies have investigated the generalization of deep neural network to narrow down the gap [7, 17, 26, 29, 42].
In particular, they try to analyse the local minima problem and connect it with loss geometry [17, 27, 31].
Amongst them, Foret et al.
propose SAM, with an aim to solve a minimax optimization problem.
SAM initially finds the maximum of loss function in the region of radius ρ centered on the w , regarding which to minimize the loss value.
This will prompt the model to converge to a flat region, where the entire neighborhood tends to have uniformly low training loss.
Inner maximization is a vital part of SAM procedure as it essentially describes the loss landscape and plays an important role in whether we can reduce the loss sharpness effectively.
The solution for inner maximization is NP-hard, and for simplicity, SAM uses one-step gradient ascent directly to approximate the maximum loss.
However, one-step gradient ascent may not be sufficient.
In addition, multi-step gradient ascents are not efficient and will cause additional training costs.
Therefore, how to improve the approximation of the inner maximization without additional computation cost is an important problem.
Based on our empirical analysis in Figure 1 and Theorem 1, we find that random initialization can smooth the loss landscape and obtain a more stable gradient direction.
Noted that a more stable gradient can make a good approximation to the true maximum value.
Therefore, in this paper, we propose a novel random smoothing based sharpness-aware minimization
algorithm
(R-SAM).
Our proposed R-SAM consists of two steps.
First, we use a Gaussian noise to smooth the loss landscape and escape from the local sharp region to obtain a stable gradient for gradient ascent.
Second, we perform the one-step gradient ascent on the smoothed weights to eventually enhance the approximation quality of inner maximization.
We summarize our contributions as follows:
• In this paper, we investigate the approximation quality of this one-step gradient ascent.
Our evaluations show that unstable gradients might hurt the inner maximization of SAM and adding random smoothing can improve gradient stability. Based on our observation, we propose a simple but novel algorithm, R-SAM, which leverages a Gaussian noise in perturbation initialization and successfully enhance the inner maximization without introducing additional cost.
• We conduct empirical studies on ResNet and ViT.
The experimental results demonstrate that R-SAM can help model generalization ability and yield more remarkable performance compared with SAM across a wide range of computer vision tasks (e
g., CIFAR-10/100, ImageNet).
Background 2.1 Sharpness-Aware Minimization
Many studies focus on analyzing the generalization of deep neural networks.
For example, motivated by the local geometry of the energy landscape, Chaudhari et al.
propose Entropy-SGD to construct a local-entropy-based objective function that favors well-generalizable solutions lying in large flat regions of the energy landscape, to address generalization problems in the training.
Izmailov et al. illustrate the benefits of Stochastic Weight Averaging (SWA) procedure in solving generalization problems and well apply SWA to generate much flatter solutions than SGD.
Keskar et al.
shows that deep neural network training is easy to converge to sharp local minima and causes a generalization gap.
Therefore, how to make the model converge to a flat local minima and improve the generalization is an important problem.
Foret et al.
propose SAM to seek the parameters that lie in a flat region through optimizing loss value and loss sharpness.
Then we will overview
Sharpness-Aware Minimization (SAM).
Consider the training dataset as S = { ( x i , y i ) } ni =1 , where each sample ( x i , y i ) follows the distribution
D .
Define f
x ; w ) as the neural network model with trainable parameter w ∈ R p , with which the loss function w.r.t.
an input x i can be given by l ( f ( x i ; w ) , y i ) ∈ R + , shortened to l ( x i ) for convenience.
The empirical training loss is L = 1 n (cid:80) ni
=1 l ( f ( x i ; w ) , y i ) .
The objective of SAM [18] is to find the parameters with uniformly low loss value within the ℓ p ball.
The objective function is proposed as follows.
[Equation]
where p ≥ 0 is defined as the radius of the ℓ p ball.
For simplicity, we ignore p when using ℓ 2 -norm.
As calculating the optimal solution of inner maximization is infeasible, SAM uses one-step gradient ascent to approximate it:
[Equation]
Finally, SAM computes the gradient regarding the perturbed model w + ˆ δ for the update:
[Equation]
Although SAM can achieve a great performance in traditional computer vision [8] and NLP tasks [3], there still exists several drawbacks yet to be resolved.
To further improve its performance, Kwon et al. introduce the concept of adaptive sharpness of loss function to solve the scale-dependency problem.
Zhuang et al.
propose GSAM to seek the region with both small loss value and low sharpness based on a novel optimization objective.
Zhou & Chen try to learn a dynamically reweighted perturbation for each batch to obtain a better approximation to per-instance perturbation.
Noted that SAM needs to calculate two sequential gradients.
To improve its efficiency, Du et al.
propose ESAM, which use stochastic weight perturbation and sharpness-sensitive data selection to reduce the computation of vanilla SAM.
Liu et al.
propose LookSAM to reuse the projected gradient that target to the flat region [Equation] every k steps to accelerate ViT training.
SS-SAM is proposed to improve the efficiency by randomly selecting SGD optimization or SAM at each step [51].
2.2 Generalization
There is a venerable line of important works on the problem of generalization in deep learning[5, 12, 21, 28, 30].
For example, Hardt et al.
focus on generating generalization bounds for models learned with stochastic gradient descent.
Bisla et al.
investigate the sharpness of loss landscape and proposed systematic mechanisms underlying the generalization abilities of deep learning models.
Jin et al. analyse the importance of perturbed gradient descent in efficiently escaping saddle points.
Further,
Jin et al.
investigate the use of a Hamiltonian function and propose a new framework to track the long-term behavior of gradient-based optimization algorithms.
In addition, Dinh et al.
argue that the concepts of flatness might be problematic and can not be directly applied to explain generalization, which requires rethinking what flatness actually means.
Adversarial training has achieved a great success in improving the robustness of model [13, 20, 39, 40, 45].
Our approach was also inspired by recent works in random smoothing in adversarial training[1, 9, 44, 47, 50].
Wong et al.
demonstrate that applying random initialization in FGSM adversarial training can in fact be just as effective as the more costly PGD adversarial training.
Further,
Andriushchenko & Flammarion study the questions of when and why FGSM adversarial training works, and how to enhance the solution of the inner maximization problem.
Zhang et al.
propose a novel bi-level optimization-based fast adversarial training framework, FAST-BAT to address the issue of overfitting and achieve better performance in generalization ability.
Proposed Method 3.1 Empirical Observation on Approximation of Inner Maximization
As shown in Section 2.1, SAM defines a min-max optimization objective to find the maximum loss value L SAM ( w ) near the current weight w and then minimize the value L SAM ( w ) .
However, for the inner maximization problem, finding the optimal solution is NP-hard, which means that it’s difficult to obtain the true maximum value.
Therefore, SAM uses a one-step gradient ascent to approximate the optimal solution and obtain the maximum loss value.
In this paper, we investigate the approximation quality of this one-step gradient ascent and propose a simple way to improve the inner maximization without introducing additional cost.
Inner maximization of SAM may be hurt by unstable gradient.
The adversarial weight w adv with the optimal solution of inner maximization can be defined as:
[Equation]
SAM uses a one-step gradient ascent to approximate the optimal adversarial weight:
[Equation]
However, the loss landscape of neural network is usually sharp [31, 35].
That means the direction of g ( w ) =
∇ w L ( w ) cannot represent the direction to maximize its neighbourhoods’ loss value.
However, a stable gradient direction in a continuous parameter space can narrow the approximation gap and help SAM obtain a better approximation of w ∗ adv .
Therefore, we try to conduct the experiments in ResNet and WideResNet to analyse the smoothness of gradient g ( w ) .
As shown in Figure 1, we plot the cosine value between the g ( w ) and g ( w + ρ g ( w ) || g ( w ) || ) to visualize the smoothness.
From this figure, we can find that the cosine value decreases significantly with the ρ value increasing (the red line).
Intuitively, when the gradient is constant in a certain region (cosine similarity equals to 1), the one-step gradient ascent will get the exact maximizer; while when the gradient becomes unstable in a certain region, the one-step gradient ascent will lead to a poor solution.
Therefore, how to smooth the loss landscape and make the gradient stable during the gradient ascent process is an important problem.
[Figure]
Random smoothing improves gradient stability.
Some studies illustrate that random smoothing can smooth the loss landscape [22, 46].
That motivates us to analyze the effects of random smoothing for the smoothness of gradient g ( w ) .
In Figure we sample a Gaussian noise δ 0 from N (0 , γ 2 I ) and obtain the weight ˆ w = w + δ 0 .
Next, we calculate the cosine value between g ( ˆ w ) and g ( ˆ w + ρ g ( ˆ w )
||
g ( ˆ w ) || ) to compare the smoothness with the gradient without random smoothing ( g ( w ) ).
From Figure 1, we can find that the cosine value with random smoothing (the green, blue and orange line) is higher than the vanilla cosine value on g ( w ) (the red line), which means that random smoothing can reduce the decay rate of cosine value.
In addition, the cosine value will be more stable with the γ value increasing when using random smoothing (from γ 2 = 0 .
0075 to γ 2 = 0 . 0125 )
in
Figure 1.
The above observation illustrates that random smoothing can smooth the loss landscape and obtain a more stable gradient.
We also provide the theoretical analysis for this observation.
Theorem 1.
Let L ( w ) be the vanilla loss function and L S ( w ) =
L ( w + δ 0 ) be the smoothed loss function.
Assuming vanilla loss function L ( w ) is α -Lipschitz
continuous
and the gradient ∇ w L ( w ) is β -Lipschitz continuous.
Given the random noise δ 0 is from Gaussian distribution N (0 , γ 2 I ) .
We can obtain that the gradient ∇ w L S ( w ) = ∇ w L ( w + δ 0 ) is min { αγ , β } -Lipschitz continuous, where γ is the standard deviation of δ 0 .
Theorem 1 is based on [5, 16], which implies that L S ( w ) =
L ( w + δ 0 ) is more smooth than the original loss function L ( w ) when α γ ≤ β .
In addition, with γ value (standard deviation of
Gaussian noise) increasing, the loss function L S ( w ) become smoother.
That is also consistent with our observation in Figure 1, where the change of cosine value will be reduced with γ 2 increasing.
Therefore, based on the above intuition about random smoothing, we can use the Gaussian noise δ 0 to initialize the one-step gradient ascent process.
That means we can firstly use Gaussian noise to escape from the local sharp region and smooth the loss landscape (initial perturbation), and then conduct the one-step gradient ascent step.
More specifically, the random initialization for inner maximization can be defined as ˆ w = w + δ 0 .
After that, we use the smoothed weight ˆ w to achieve the one-step gradient ascent process and obtain the approximation of optimal adversarial weight: [Equation]
[Equation]
Finally, we investigate whether the smoothed gradient can benefit the inner maximization and obtain a better approximated solution in practical applications.
We try to plot the perturbed loss value ( L ( w + ρ g ( w ) ||
g ( w ) || ) and L ( w + δ
+ ρ g ( w + δ 0 )
||
g ( w + δ 0 ) || ) ) in Figure 2.
From this figure, we can find that the loss value is very close for different γ 2 value when the ρ value is small ( ρ ≤ 0 . 2 ).
However, with [Figure] Noted that δ 0 is from the Gaussian distribution with three standard deviation ( We select the checkpoint of 200-th epoch on CIFAR-100 as the weight w to the the ρ value gradually increasing, the loss value grow slowly for vanilla weight w .
That means g ( w ) cannot efficiently increase the loss value and is not applicable for solving the inner maximization with the region expanding.
That limited the region size that vanilla weight can select.
In addition, we observe that the weight with Gaussian noise ( w + δ 0 ) shows a more stable pattern for the increase of loss value to solve the inner maximization problem.
Applying one-step gradient ascent based on the gradient of smoothed weight w + δ 0 can better maximize the loss value in the neighbouring region.
The above analysis motivates us to propose a random smoothing based SAM algorithm to obtain a better solution for inner maximization.
We will introduce our proposed Random Sharpness-Aware
Minimization algorithm
(R-SAM) in the next section.
3.2 Random Sharpness-Aware Minimization (R-SAM)
Based on the observation and intuition in section 3.1, we propose a novel random initialization based
Sharpness-Aware Minimization algorithm
(R-SAM) to benefit the solution of inner maximization in
SAM.
For R-SAM, we use Gaussian noise to initialize the perturbation for weight w and obtain the smoothed weight ˆ w = w + δ 0 .
After that, we use the smoothed weight ˆ w to calculate the gradient g ( ˆ w ) for gradient ascent process: g ( ˆ w ) = ∇ ˆ w L ( ˆ w ) = ∇ w L ( w + δ 0 ) .
In this way, we can obtain a more stable gradient to solve the inner maximization problem.
More specifically, we will use the gradient g ( ˆ w ) to finish the gradient ascent step and obtain adversarial weight w adv .
A simple way to obtain w adv is that we can directly add the gradient-based perturbation g ( ˆ w ) = ∇ ˆ w L ( ˆ w ) to the smoothed weight ˆ w = w + δ 0 , which is defined as w adv = w + δ
+ ρ g ( ˆ w ) ||
g ( ˆ w ) || .
However, that will cause the norm of total perturbation δ
+ ρ g ( ˆ w ) ||
g ( ˆ w ) || larger than vanilla ρ .
To constrain the norm of total perturbation, we try to project the perturbation into the ball with radius ρ .
The final perturbation can be defined as:
[Equation]
where δ 0 represents the Gaussian noise for random initialization and β is used to control the ratio between δ 0 and g ( ˆ w ) .
Noted that the selection of Gaussian noise plays an important role in R-SAM.
However, different from the input image data that each pixel is from the same distribution, the distribution of weights in each layer or kernel shows different distributions.
That makes the noise design be a challenging
Algorithm
Input: x ∈ R d , w represents vanilla weight, ˆ w is the smoothed weight, learning rate η t , k is the number of kernels in model.
for t ← 1 to T do
Sample Minibatch B = { ( x i , y i ) , · · · , ( x |B| , y |B| ) } from X .
δ
= N (0 , γ 2 · diag ( || w j || kj =1 ))
Obtain a stable gradient g ( ˆ w ) = ∇ ˆ w L ( ˆ w ) = ∇ w L ( w + δ 0 ) on minibatch B .
Compute gradient ascent w adv = w + ρ[-δ 
-]
[Equation]
[Equation] g ( ˆ w ) ||
[-δ 
-][Equation]
+ [Equation] ||
Compute SAM gradient: g sam = ∇ w L ( w ) | w adv
Update weights: w t +1 =
w t − η t g sam end for problem.
Related work has illustrated that the weight in each kernel is from an independent Gaussian distribution [25].
Inspired by this work, in this paper, we use kernel-wise Gaussian noise δ 0 ∼ N (0 , γ 2 · diag ( || w j || kj )) as the initialization of perturbation in R-SAM, where || w j || kj represents the ℓ 2 -norm of k -th kernel in the neural network.
The pseudo-code of R-SAM can be found in Algorithm 1.
Firstly, we need to sample a minibatch data
B = { ( x i , y i ) , · · · , ( x |B| , y |B| ) } from dataset X .
After that, we will calculate the norm of each kernel || w j || kj =1 and sample kernel-wise Gaussian noise from the distribution N (0 , γ 2 · diag ( || w j || kj )) as the initialization of perturbation.
Then we can obtain g ( ˆ w ) =
∇ w L ( w + δ 0 ) as the stable gradient to better approximate the optimal adversarial weight w ∗ adv .
More specifically, we can use one-step gradient ascent to calculate w adv = w + ρ δ + λ g ( ˆ w ) ||
δ + λ g ( ˆ w ) || .
In this way, we can calculate the gradient on w adv : g sam = ∇ w L ( w ) | w adv .
Finally, following the process in vanilla SAM, we can use the gradient g sam to update the vanilla weight w .
As shown in Algorithm 1, R-SAM is very easy to reproduce.
Experiments 4.1 Experimental Settings
In this section, we try to evaluate the performance of our proposed R-SAM optimizer on ResNet [23], WideResNet [48] with CIFAR [33] datasets and Vision Transformer (ViT) [14] with ImageNet [11] dataset.
First, compared with vanilla SAM, the experimental results illustrate that our proposed
R-SAM can improve the accuracy in CIFAR and ImageNet.
In addition, the ablation study and sensitivity analysis show that our proposed R-SAM is robust for hyper-parameters.
4.1.1 Datasets
To validate the performance of proposed method R-SAM, we firstly try to conduct the experiments on widely used CIFAR-10, CIFAR-100 [33] and ImageNet-1k [11] datasets.
To further validate the performance and robustness of R-SAM, we also report the experiment results on ImageNet-Real [4],
ImageNet-V2 [41], ImageNet-R [24] and ImageNet-C [19].
4.1.2 Models
We firstly evaluate the performance of our proposed R-SAM in CNN-based models, such as ResNet18, ResNet-50 [23] and WideResNet (WRN-28-20) [48].
Currently, Transformer-based models receive a great deal of attention in the field of computer vision [37, 43].
After that, we also try to evaluate our proposed algorithm R-SAM in Vision Transformer (ViT) [14].
4.1.3 Baselines
To illustrate that the random initialization in R-SAM can help to escape the non-smooth vicinity and benefit to the approximation in inner maximization, the main baseline of this paper is vanilla SAM [18].
In addition, to show that SGD+M is more suitable for CIFAR training, we also try to compare the accuracy of SGD+M with SGD, RMSProp and AdamW [38].
To further show the generality of
R-SAM, we also try to combine it with GSAM [53] and propose R-GSAM algorithm.
The experiment results illustrate that R-GSAM can obtain a better performance compared with vanilla GSAM in ViT training.
That also illustrates the effectiveness of random initialization in R-SAM and R-GSAM.
4.1.4 Implementation Details
The experiments in this paper are implemented with JAX [6] on Google TPU-V3 chips.
The hyperparameters and experimental setting is based on vanilla SAM [8, 18].
For ResNet training, we use SGD with Momentum as the base optimizer of SAM and R-SAM.
For ViT training, the base optimizer of SAM and R-SAM is AdamW [38].
As for data augmentation, we select mixup [49] and
RandAugment [10] to conduct the experiments.
Noted that we do not use stochastic depth for ViT training.
The implementation details can be found in Appendix 7.4.
4.2 Training from Scratch on CIFAR-
To evaluate the performance of our proposed R-SAM, we conduct experiments of ResNet-18, ResNet50 and WideResNet (WRN-28-20) training on CIFAR-10 dataset.
Following the experimental setting in vanilla SAM [18], we use basic augmentation to preprocess the input image data and the base optimizer for SAM and R-SAM is SGD with Momentum (SGD+M).
The experimental results are shown in Table 1.
We can find that SAM-based optimizers (SAM and R-SAM) can achieve a better performance than traditional first-order optimizers (SGD and
AdamW).
For example, the accuracy of traditional first-order optimizers are around 96%, and the best accuracy is 96.5% from SGD with Momentum (SGD+M).
However, SAM can improve the accuracy from 96.5% to 97.3% for WRN-28-20.
That illustrates the power of SAM to improve the performance of neural networks.
In addition, R-SAM can further improve the performance compared with vanilla SAM.
More specifically, R-SAM can improve the accuracy of SAM from 97.3% to 97.5% on WRN-28-20 for CIFAR-10.
Noted that the accuracy of SAM for CIFAR-10 is pretty high and the improvement of R-SAM is also convincing.
[Table]
[Table]
To further validate the performance of our proposed R-SAM
[Table]
we also try to evaluate its accuracy on
CIFAR-100 dataset.
We also follow the setting in SAM [18] and use basic augmentation to preprocess the image data.
We select SGD+M as the base optimizer of SAM and R-SAM.
[Table]
The experimental results are shown in Table 2, we notice that the results show a similar pattern with
CIFAR-10 and can achieve a higher improvement.
In particular, SAM-based optimizers can obtain better performance compared with first-order optimizers.
For example, the best accuracy of first-order optimizers is 81.7% for WRN-28-20 model.
SAM can improve the accuracy from 81.7% to 84.6% for WRN-28-10 and from 81.4% to 83.3% for ResNet-50.
In addition, R-SAM can obtain a higher accuracy compared (from 84.6% to 85.2% for WRN-28-20 and 83.3% to 84.0% for ResNet-50).
The above experimental results also illustrate that our proposed R-SAM can further help SAM to obtain better performance.
4.4 ViT Training from Scratch on ImageNet
Currently, Vision Transformer has been widely used in a variety of areas [37, 43].
Chen et al. illustrate that SAM can significantly improve the performance of ViT training.
Therefore, evaluating the performance of our proposed R-SAM on ViT is pretty important for verifying our contribution.
In this section, we select ViT-B-16 and ViT-S-16 to evaluate the performance of R-SAM.
The experimental setting follows the introduction in [8, 18].
Noted that the base optimizer of SAM and
R-SAM are both
AdamW.
We select batch size as 4096 for ViT training.
ViT Training.
The experimental results of ViT training on ImageNet are shown in Table 3.
From this table, we can observe that SAM can also significantly improve the accuracy of ViT training.
For example, the accuracy of ViT-B-16 is improved from 74.7% (AdamW) to 79.8% (SAM).
In addition, when using SAM, the accuracy of ViT-S-16 can also be improved from 74.9% to 77.9%.
However, our proposed R-SAM can further improve the accuracy of ViT training compared with vanilla SAM and achieve better performance.
More specifically, R-SAM can improve the accuracy of ViT-Bfrom 79.8% (vanilla SAM) to 80.7%.
Besides, ViT-S-16 can also obtain benefits from R-SAM and the accuracy is increased from 77.9% (vanilla SAM) to 78.7%.
The above experimental results further validate the effectiveness of our proposed R-SAM, especially when compared with vanilla SAM.
We use Inception-style preprocessing method for input image.
[Table]
Data Augmentation.
To further evaluate the generality of our proposed R-SAM, we also try to combine R-SAM with strong augmentation methods and the results are shown in Table 3.
In this experiment, we select mixup [49] and RandAugment (RandAug) [10] as the augmentation methods to conduct the experiments.
Firstly, we can find that augmentation methods can further improve the accuracy of AdamW and vanilla SAM.
For instance, we can observe that mixup can improve the accuracy of vanilla ViT-B-16 with AdamW from 74.7% to 75.7%.
In addition, when SAM is selected, the accuracy will further increase from 79.8% to 80.4% with mixup for ViT-B-16.
In addition, the experimental results illustrate that our proposed R-SAM can further improve the accuracy of vanilla SAM even with the strong augmentations.
For example, R-SAM can improve the accuracy of SAM from 80.4% to 81.1% even using strong augmentations.
These experiments further verify that our proposed R-SAM can be combined with strong augmentations and illustrate a great generality.
Finally, we find that mixup and RandAug can work together to benefit the performance of R-SAM.
As shown in Table 3, when RandAug is selected, the accuracy is increased to 81.6% compared to simply using only the mixup.
Further Evaluations and Robustness.
Firstly, to further validate the performance of R-SAM, we report the experiment results on ImageNet-Real and ImageNet-v2.
As shown in Table 4, we can observe that R-SAM can still improve the accuracy of ViT compared with vanilla SAM. For example, R-SAM can improve the accuracy of ViT-B-16 from 85.2% to 86.1% on ImageNet-Real. In addition, to evaluate the robustness of trained model with R-SAM, we also conduct the experiments in ImageNet-R and ImageNet-C datasets. The experimental results are also shown in Table 4 , which illustrate that SAM can improve the robustness of trained model compared with AdamW.
In addition, R-SAM can maintain the advantage of SAM and further strengthen its robustness. For example, SAM can improve the accuracy of ViT-B-16 from 20.1% to 26.4% on ImageNet-R. R-SAM can further improve the accuracy to 26.9%. SAM
GSAM
R-GSAM 79.
A cc u r a c
y
(a) Comparison with GSAM 1.
A cc u r a c y (b) Sensitivity Analysis of ρ  3 79.
A cc u r a c y (c) Sensitivity Analysis of
[Table]
4.5 Compared with Algorithms in the Literature
In section 2. [Table] we introduced some related work about SAM.
In this section [Table] we try to evaluate the performance of R-SAM compared with the algorithms in the literature.
We find that GSAM can achieve the best accuracy for ViT training.
Therefore, we mainly focus on the comparison with GSAM.
The main reults are from ViT-B-16 training on ImageNet-1k dataset and are shown in
Figure 3(a).
We can find that R-SAM can obtain the similar accuracy compared with vanilla GSAM.
Howvever, when combining R-SAM and GSAM and named it as R-GSAM, we can obtain better performance.
For instance, the accuracy of R-SAM and GSAM are 80.7% and 80.8% respectively.
When using R-SAM for GSAM and obtain the results of R-GSAM, we can observe that the accuracy increase from 80.8% to 81.2%.
The main purpose of this paper is to illustrate that random smoothing can benefit the inner maximization and improve the performance of SAM.
4.6 Sensitivity Analysis
Reproducibility is critical for deep learning applications.
We notice that deep learning is usually sensitive to hyper-parameters.
Therefore, in this section, we try to study the effects of hyperparameters for the performance of ViT training on ImageNet-1k dataset.
More specifically, we mainly focus on the effects of ρ and λ .
The reason is that ρ determines the radius of perturbation and λ controls the ratio between Gaussian noise δ 0 and gradient-based perturbation g ( ˆ w ) ||
g ( ˆ w ) || .
The analysis results are shown in Figure 3.
We can find that ρ and λ shows a stable pattern for tuning.
That further improves the reproducibility of R-SAM and make it more applicable in real scenarios.
Conclusion
In this paper, we observe that simply applying one-step gradient ascent in the inner maximization of
SAM may suffer from poor approximation error due to unstable gradient.
To resolve this problem, we propose a novel algorithm R-SAM, which provably improves the gradient stability by initializing the inner maximization of SAM with a Gaussian noise.
The experimental results on ResNet and
ViT illustrate that R-SAM can improve the generalization performance of the model and yield better performance compared with SAM on CIFAR-10, CIFAR-100 and ImageNet datasets.
Acknowledgements and Disclosure of Funding
We thank Google TFRC for supporting us to get access to the Cloud TPUs.
We thank CSCS (Swiss
National Supercomputing Centre) for supporting us to get access to the Piz Daint supercomputer.
We thank TACC (Texas Advanced Computing Center) for supporting us to get access to the Longhorn supercomputer and the Frontera supercomputer.
We thank LuxProvide (Luxembourg national supercomputer HPC organization) for supporting us to get access to the MeluXina supercomputer.
CJH and XC are partially supported by NSF IIS-2008173, NSF IIS-2048280 and research awards from
Google, Sony, Samsung and Okawa Foundation.
YY and YL are also partially supported by grants from Alibaba, Bytedance, Huawei, and Singapore Maritime Institute.