Escaping Saddle Points for Effective Generalization on Class-Imbalanced Data
Anonymous Author(s)
Affiliation
Abstract
Deep learning methods are primarily developed to work well on balanced dataacross benchmarks.
However, real-world datasets exhibit imbalance of varyingtype and degree.
Several techniques based on re-weighting and margin adjustmentof loss have been proposed to enhance the performance of models on minorityclasses.
In this work, we analyze the class-imbalanced learning problem throughthe lens of loss landscape.
Specifically, we examine the spectral density of Hessianof class-wise loss, in which we observe that for the tail class loss landscape, thesolution converges to a saddle point.
Due to this, optimization methods that escapesaddle points can effectively improve generalization of minority classes in classimbalanced learning.
We further theoretically and empirically demonstrate that
Sharpness Aware Minimization (SAM), a recent technique that aims to converge toflat minima, can be effectively used to escape saddle points for minority classes.
Using SAM results in a 6.2% increase in accuracy on the minority classes evenover the state-of-the-art Vector Scaling Loss, leading to an average increase of 4%across imbalanced datasets.
Introduction
In recent years, there has been a lot of progress in visual recognition thanks to the availability of wellcurated datasets [27, 34], which are artificially balanced in terms of the frequency of samples acrossclasses.
However, modern real-world datasets are often imbalanced ( i e .
long-tailed etc.)
[26, 37, 38]and suffer from various kinds of distribution shifts.
It has been shown that overparameterized modelslike Deep Neural Networks usually overfit classes with a high frequency of samples ignoring theminority (tail) ones [5, 38].
In such a scenario, when evaluated on metrics that focus on performanceon minority data, these models perform poorly.
These metrics are an essential and practical criterionfor evaluating models in various domains like fairness [10], medical imaging [43] etc.
Many approaches used for improving the generalization performance of models trained using imbalanced data are based on the re-weighting of loss [11].
The weights for samples of each class aredetermined so that the expected loss closely approximates the testing criterion objective.
In recentyears, re-weighting techniques such as Deferred Re-Weighting (DRW) [7], and Vector Scaling (VS)
Loss [25] have been introduced, which improve the classical re-weighting method of weighing lossproportional to the inverse of class frequency.
However, even these improved re-weighting techniqueslead to overfitting on the samples of tail classes.
Also, it has been shown that re-weighted loss fordeep networks lead to convergence to final solutions similar to the un-weighted loss, rendering itineffective in practice [6].
This work looks at the loss landscape in weight space around final solutions attained by trainingwith re-weighted loss.
We find that the generic Hessian-based analysis of the average loss does notuncover any insights into the suboptimal generalization on tail classes (Sec. 3).
As the frequencyis different for each class due to imbalance, we look at the Hessian of the loss for each class.
Thisnew way of analysis finds that re-weighting cannot prevent convergence to saddle points with high
[Figure]
negative curvature for tail classes, which eventually leads to poor generalization [13].
Whereas forhead classes, the solutions converge to a minima with almost no significant presence of negativecurvature, similar to networks trained on balanced data.
The problem of converging to saddle pointshas not received much traction in recent times, as the negative eigenvalues disappear when trained onbalanced datasets, indicating convergence to local minima [8, 16].
However, our analysis shows thatconvergence to saddle points is still a practical problem for neural networks when they are trained onimbalanced (long-tailed) data (Fig. 1).
In the community, a lot of effort has been made to develop optimization methods that are able toescape saddle points efficiently [15, 21, 22], which involve adding a component of isotropic noise togradients.
However, these methods have not been able to improve the performance of deep networkseffectively, as the implicit noise of SGD in itself mitigates the issue of saddle points when trainedon balanced data [12, 22].
In the case of imbalanced datasets, we find that the component of SGDalong negative curvature (i.e., implicit noise) is insufficient to escape saddle points for minorityclasses.
Thus, learning on imbalanced data can be used as a practical benchmark for optimizationalgorithms.
We further show that Sharpness-Aware Minimization (SAM) with re-weighting caneffectively enhance the gradient component along the negative curvature, allowing effective escapefrom saddle points.
We find that SAM can significantly improve the performance of various re-weighting and marginenhancing methods introduced for long-tailed learning.
The significant improvement is also similarlyseen on large-scale datasets of ImageNet-LT and iNaturalist 2018, demonstrating our observations’applicability at scale.
We summarize our contributions below:
• We propose class-wise Hessian analysis of loss which reveals convergence to saddle points
in the loss landscape for minority classes.
We find that even loss re-weighting solutionsconverge to saddle point, leading to sub-optimal generalization on the minority classes.
• We theoretically show that SAM with re-weighting and high regularization significantlyenhances the component of Stochastic Gradient along the direction of negative curvatureenabling effective escape from saddle points.
• We find that SAM can successfully enhance the performance of even state-of-the-art tech-
niques for learning on imbalanced datasets which have a re-weighting component ( e
g .
VS
Loss and LDAM) across various datasets and degrees of imbalance.
Related Work & Background
In this work, we use g ( x ) to denote the output of a model, ∇ g ( x ) to denote the gradients with respectto parameters, x and y denote the data and labels, respectively.
We review the re-weighting methodsused for training on imbalanced data with distribution shifts, followed by optimization techniquesrelated to our work.
2.1 Long-Tailed Learning
Re-sampling [5] and Re-weighting [18] are the most commonly used methods to train on classimbalanced datasets.
Oversampling the minority classes [9] and undersampling the majority classes[5] are two approaches to re-sampling.
Oversampling leads to overfitting on the tail classes, andundersampling discards a large amount of data, which inevitably results in poor generalization.
Kanget al.
[23] proposed to decouple representation learning and classifier training to improve performancewith the same.
Mixup Shifted Label-Aware Smoothing model (MiSLAS) [45] aims to improvethe calibration of models trained on long-tailed datasets by mixup and label-aware smoothing andthereby improve performance.
RIDE [40] and TADE [44] are ensemble-based methods that achievestate-of-the-art on the long-tailed visual recognition.
Samuel and Chechik [36] introduces a new loss,
DRO-LT, based on distributionally robust optimization for learning balanced feature representations.
We explore the problem of training class-imbalanced datasets through the lens of optimization andloss landscape.
We will now describe some representative recent effective methods in detail, whichwe will use as baselines.
LDAM [7] : LDAM introduces optimal margins for each class based on reducing the error through ageneralization bound.
It results in the following loss function where ∆ j is the margin for each class:
[Equation]
where ∆
[Equation] =
C n
[Equation]
j for j ∈ { 1 , . . . ,
[Equation]
[Equation] The core idea of LDAM is to regularize the classes with low frequency (low i .
e .
n j ) more, incomparison to the head classes with high frequency.
DRW [7] : Deferred Re-
Weighting refers to training the model with average loss till certain epochs(K), then introducing weight w j proportional to 1 /n j to loss term specific to each class j at alater stage.
This way of re-weighting has been shown to be effective for improving generalizationperformance when combined with various losses such as Cross Entropy (CE), LDAM etc.
We willbe using CE+DRW method as a representative re-weighting method for our analysis.
We define
CE+DRW loss below for completeness:
[Equation]
VS[25] : Vector Scaling loss is a recently proposed loss function which unifies the idea of multiplicative shift (CDT shift [42]), additive shift (i.e Logit Adjustment [32]) and loss re-weighting.
The finalloss has the following form:
[Equation]
Here the γ j and ∆ j are the multiplicative and additive logit hyperparameters, respectively.
2.2 Loss Landscape
Saddle Points :
Saddle points are regions in loss landscape that usually depict a plateau region withsome negative curvature.
In the non-convex setting, it has been shown that there is an existence of anexponential number of saddle points in loss landscape and convergence to these points demonstratepoor generalization [13].
There has been a lot of effort in developing methods for effectively escapingsaddle points which involve the addition of noise (e.g., Perturbed Gradient Descent (PGD) [15, 21,
[Figure]
-wise) on the head class (Class 0) and tail class (Class 8) with
SGD and SAM.
It can be observed that with the head classes, the validation accuracy with SGD (B) and SAM (E) are similar and the density of negative eigenvalues is not significant.
On the tail class,
SAM (F) escapes the saddle points (large density of negative eigenvalues) in SGD (C), leading to 20% increase in validation accuracy.
A and D show the overall spectral density calculated across all samples in the dataset.
Overall spectral density does not indicate the presence of saddles.
However, these algorithms have not received much attention in the deep learning communityas it has been shown that the implicit noise in SGD can escape saddles easily and converge tolocal minima [12].
Also, it has been empirically shown that negative eigenvalues from the Hessianspectrum disappear after a few steps of training, indicating escape from saddle points when neuralnetworks are trained on balanced datasets [1, 8, 35].
However, contrary to this, we demonstrate thatconvergence to saddle points is prevalent in minority class loss landscapes and is a practical problemthat can serve as a practical benchmark for the development of algorithms that escape saddle points.
Flat Minima based Optimization methods : Empirically, it has been shown that converging to a flatminima in loss landscape for a deep network leads to improved generalization in comparison to sharpminima [20, 24].
Recent works have tried to exploit this connection between the geometry of the losslandscape and generalization to achieve lower generalization error.
Sharpness-Aware Minimization(SAM)
[14] is one such algorithm that aims to simultaneously minimize the loss value and sharpnessof the loss surface.
Low-Pass Filtering SGD (LPF-SGD)
[4] is another recently proposed optimizationalgorithm that aims to recover flat minima from the optimization landscape.
LPF-SGD convolves theloss function with a Gaussian Kernel with variance proportional to the norm of the parameters ofeach filter in the network.
In this work, we aim to explore the effectiveness of such algorithms for thetask of escaping saddle points, which is a new direction for these algorithms.
Convergence to Saddle Points in Tail Class Loss Landscape
This section analyzes the dynamics of the loss landscape of neural networks trained on imbalanceddatasets.
We use the Cross Entropy (CE) loss ˆ L CE to denote the average cross entropy loss for eachclass.
For fine-grained analysis, we focus on average loss on each class ˆ L CE ( y ) .
We visualize the losslandscape of the head and tail classes through the computation of Hessian Eigenvalue Density [16].
The Hessian of the train loss for each class H = ∇ 2 w ˆ L CE ( y ) contains important properties aboutthe curvature of the classwise loss landscape.
The Hessian Eigenvalue Density provides all suitableinformation regarding the eigenvalues of H .
In this work, we focus on λ max (max eigenvalue) andλ min (min eigenvalue), which depict the extent of positive and negative curvature present.
We use the
Lanczos algorithm as introduced in Ghorbani et al.
[16] to compute the Hessian Eigenvalue Density(spectral density) tractably.
We further calculate the validation accuracy of a particular class y and itseigen spectral density for analysis.
We provide more details for these experiments in the Appendix E.
Does the proposed class-wise analysis of loss landscape offer any additional insights?
In priorworks [16, 17, 29], the Hessian of the average loss is used to characterize the nature of the convergedpoint in the loss landscape.
However, we find that when particularly trained on imbalanced datasets
[Figure]
like CIFAR-10 LT, the eigen spectral density of the Hessian of average loss (Fig.
2A) does not differfrom that of head class loss (Fig. 2B), indicating convergence to a local minima.
However, explicitlyanalyzing the Hessian for the tail class loss (Fig.
2C) gives the correct indication of the presenceof negative eigenvalues (i.e., curvature), which is in contrast to average loss.
Hence, our proposedclass-wise analysis of Hessian is essential for characterizing the nature of the converged solutionwhen the training data is imbalanced.
What happens when you train a neural network with CE-DRW method on CIFAR-10 LT?
Fig.2 shows the spectral density on samples from the head class (Class 0 with 5000 samples) and tailclass (Class 8 with 83 samples) at the checkpoint with the best validation accuracy.
The spectraldensity of the head class contains few negative eigenvalues.
Most of the eigenvalues are centeredaround zero, as also observed when training on a balanced dataset [16].
On the other hand, for the tailclass, there exists a large number of both negative and positive eigenvalues, indicating convergenceto a saddle point.
We find that at this point, the ˆ L CE ( y ) is low along with the norm of gradient,which indicates a stationary saddle point.
We also observe that the spectral density of the tail classcontains many outlier eigenvalues, and λ max is much larger compared to the head class indicatingsharp curvature.
These evidences show that the tail class solution converges to a saddle point insteadof a local minimum .
Merkulov and Oseledets [33] indicated the existence of stationary points withlow error but poor generalization in the loss landscape of neural networks.
Also, the existence ofsaddle points being associated with poor generalization has been observed for small networks [13].
However, in this work, we show that convergence to saddle points can specifically occur in the losslandscape of tail classes even for the popular ResNet [19] family of networks, which is an importantand novel observation to the best of our knowledge.
Dynamics of training on Long-Tailed Datasets :We analyze the | λ min /λ max | for the head, midand tail classes at various epochs (10, 50, 160, 170, 190, 200) across training to understand thedynamics of optimization with CE+DRW on long-tailed data (Fig. 3A).
| λ min /λ max | is a measure ofnon-convexity of the loss landscape [28], where a high value of | λ min /λ max | conveys non-convexityindicating convergence to points with significant negative curvature.
The network converges tonon-convex regions with negative curvature for tail classes, showing convergence to the saddle point.
Also, we find that for the certain tail (Class 7, 8) and mid classes (Class 5), the network startsconverging towards regions with negative curvature after applying loss re-weighting (DRW at 160thepoch).
This indicates that DRW leads to convergence to a saddle point rather than preventing it.
Escaping Saddle Points for Improved Generalization
In this section, we analyze the Sharpness-Aware Minimization technique for escaping from saddlepoints in tail class loss landscape.
In existing works [2, 30, 46], the effectiveness of SAM in escapingsaddle points has not been explored to the best of our knowledge.
Sharpness-Aware Minimization (SAM) :
Sharpness-Aware Minimization is a recent techniquewhich aims to flatten the loss landscape by first finding a sharp maximal point ϵ in the neighborhoodof current weight w .
It then minimizes the loss at the sharp point ( w + ϵ ).
[Equation]
Here, f is any objective (eg. CE or LDAM loss function) and ρ is the hyperparameter that controlsthe extent of neighborhood.
A high value of ρ leads to convergence to much flat loss landscape.
Theinner optimization in above objective is first approximated using a first order solution:
[Equation]
After finding ˆ ϵ ( w ) , the network weights are updated using the gradient ∇ f ( w ) | w +ˆ ϵ ( w ) .
In recentwork [2], it has been shown that the normalization of the norm of the gradient for ˆ ϵ ( w ) calculationabove leads to oscillation which implies non-convergence theoretically.
Also, it has been empiricallyshown that the unnormalized version of the gradient with adjusted ρ performs better than thenormalized version.
Hence, we use the approximation i.e. ˆ ϵ ( w )
=
ρ ∇ f ( w ) for our theoretical results.
As we will be using the stochastic version of the gradient, we use z as the stochasticity parameterand denote the gradient as ∇ f z ( w ) .
With this, we now define the gradient with respect to w that isassociated with SAM:
[Equation]
As we are using the same batch for obtaining the gradient to calculate the ˆ ϵ ( w ) and loss, we can usethe same z as the argument.
We now analyze the component of the SAM gradient in the direction ofnegative curvature, which is required for escaping saddle points [12].
4.1 Analysis of SAM for Escaping Saddle Points
Our analysis is based on the Correlated Negative Curvature (CNC) assumption [12] that states thatstochastic gradients have components along the direction of negative curvature, which helps themescape from saddle points.
This assumption has been shown to be theoretically valid for the problemof learning half-spaces and also has been empirically verified for a large number of neural networksof different sizes [12].
We now formally state the assumption below:
A SSUMPTION 1 (Correlated Negative Curvature [12]) .
Let v w be the minimum eigenvector corresponding to the minimum eigenvalue of the Hessian matrix ∇ 2 f ( w ) .
The stochastic gradient∇ f z ( w ) satisfies the CNC assumption if the second moment of the projection along the direction v wis uniformly bounded away from zero, i.e.
[Equation]
It has also been emphasized that the value of γ is shown to correlate with the magnitude of λ 2min .
This shows that with a high negative eigenvalue, there is a large component of gradient along thenegative curvature along v w .
This allows the SGD algorithms to escape the saddle points.
However,we find that in the case of class imbalanced learning (Fig. 1) even stochastic gradients may havean insufficient component in the direction of negative curvature to escape the saddle points.
Wenow show that SAM technique, which aims to reach a flat minima, further amplifies the gradientcomponent along negative curvature and can be effectively used to escape the saddle point.
We nowformally state our theorem based on the CNC assumption below:
T HEOREM 2.
Let v w be the minimum eigenvector corresponding to the minimum eigenvalue λ minof the Hessian matrix ∇ 2 f ( w ) .
The ∇ f SAM z ( w ) satisfies that it’s second moment of projection in v wis atleast (1 + ρλ min ) 2 times the original (component of ∇ f z ( w ) ):
[Equation]
R EMARK .
The above theorem adds the factor (1 + ρλ min ) 2 to increase the component in directionof negative curvature ( γ ) when λ min ≤ − 2 ρ .
Due to this increase, the model will be able to escapefrom directions with high negative curvature, leading to an increased λ min .
Also, as the factor − 2 ρ isinversely proportional to ρ , the high value of ρ aids in effectively increasing the minimum negativeeigenvalue.
To empirically verify this, we evaluate the Hessian spectrum for the CIFAR-10 LTdataset using CE-DRW method for different values of ρ (Fig.
3B).
We find that, as expected from thetheorem, in practice, the high values of ρ lead to less negative values of λ min .
This indicates escapingthe saddle points effectively, hence avoiding convergence to regions having negative curvature in losslandscape.
The proof of the above theorem and additional details is provided in Appendix B.
We also want to convey that theoretically, techniques like Perturbed Gradient Descent (PGD), and
LPF-SGD (Low-Pass Filter SGD), which add Gaussian noise into gradient to escape saddle points
[Table] [Table]
VS + SAM
[Table]
can also be used for mitigating negative curvature.
Also it has been found that SGD [12] can alsoescape the saddle points and converges to solutions with a flat loss landscape.
Also, theoreticallyaccording to Theorem 2 in Daneshmand et al.
[12] the SGD algorithm convergence to a second-orderstationary point depends on the γ as O ( γ − 4 ) under some assumptions on f .
As we find that as SAMwith high ρ enhances the component of SGD in direction of negative curvature ( γ ) by (1+ ρλ min ) 2 , itis reasonable to expect that SAM is able to escape saddle points effectively and converge to solutionswith significant less negative curvature implying better generalization.
We provide empirical evidencefor this in Fig.
3B and Sec. 5.2.
What happens when you train a neural network with SAM + DRW?
With SAM (high ρ ), thelarge negative eigenvalues present in the loss landscape of the tail class get suppressed (Fig.
2F).
In the spectral density for the tail class, it can be seen that λ min is much closer to zero for SAMcompared to its counterpart with SGD.
This aligns with the hypothesis that SAM escapes regions ofnegative curvature, leading to improved accuracy on the tail classes.
However, the spectral densityof the head class does not change significantly compared to that of Emperical Risk Minimization(ERM), although the λ max is much lower for SAM, indicating a flatter minima for the head class.
Experiments
5.1 Class-Imbalanced Learning
Datasets : We report our results on four long-tailed datasets: CIFAR-10 LT [7], CIFAR-
LT [7], ImageNet-LT [31], and iNaturalist 2018 [39].
a) CIFAR-10 LT and CIFAR-100 LT :
The original CIFAR-10 and CIFAR-100 datasets consist of 50,000 training images and 10,000validation images, spread across 10 and 100 classes, respectively.
We use two imbalance versions,i.e., long-tail imbalance and step imbalance, as followed in Cao et al.
The imbalance factor,β = N max N min , denotes the ratio between the number of samples in the most frequent ( N max ) and leastfrequent class ( N min ).
For both the imbalanced versions, we analyze the results with β = 100.b) ImageNet-LT and iNaturalist 2018 :
We use the ImageNet-LT version as proposed by [31],which is an class-imbalanced version of the large-scale ImageNet dataset [34].
It consists of 115.8Kimages from 1000 classes, with 1280 images in the most frequent class and 5 images in the least.iNaturalist 2018 [39] is a real-world long-tailed dataset that contains 437.5K images from 8,142categories.
In the case of long-tail imbalance, we segregate the classes of all the datasets into Head(Many), Mid (Medium), and Tail (Few) subcategories, as defined in [45].
For step imbalance experiments on CIFAR datasets, we split the classes into Head (Frequent) and Tail (Minority), as done in [7].255 Experimental Details : We follow the hyperparameters and setup as in Cao et al.
[7] for
CIFAR-10 LT and CIFAR-100 LT datasets.
We train a ResNet-32 architecture as the backbone and
SGD with a momentum of 0.9 as the base optimizer for 200 epochs.
A multi-step learning rateschedule is used, which drops the learning rate by 0.01 and 0.0001 at the 160th and 180th epoch,respectively.
For training with SAM, we set a constant ρ value of either 0.5 or 0.8 for most methods.
For ImageNet-LT and iNaturalist 2018 datasets, we use the ResNet-50 backbone similar to [45].
Aninitial learning rate of 0.1 and 0.2 is set for iNaturalist 2018 and ImageNet-LT, respectively, followedby a cosine learning rate schedule.
We initialize the ρ value with 0.05 and utilize a step schedule to
[Table]
[Figure] [Table] [Figure] [Table]
[Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
increase the ρ value during the course of training for SAM experiments.
We run every experiment onlong-tailed CIFAR datasets with three seeds and report the mean and standard deviation.
Additionalimplementation details are provided in the Appendix C.
Baselines : a) Cross-Entropy (CE) :
CE minimizes the average loss across all samples, and thus, theperformance of tail classes is much lower than that of head classes.
b) CE + Deferred Re-Weighting(DRW)
[7] : The re-weighting of CE loss inversely by class frequency is done in the later stage oftraining.
c) LDAM + DRW [7] : Label-Distribution-Aware Margin (LDAM) proposes a margin-basedloss that encourages larger margins for less-frequent classes.
d) Vector Scaling (VS) Loss [25] : VSloss incorporates both additive and multiplicative logit adjustments to modify inter-class margins.
Results : Table 1 summarizes our results on CIFAR-10 LT and CIFAR-100 LT with β of 100.
It canbe observed that SAM with re-weighting significantly improves the accuracy on mid and tail classeswhile preserving the accuracy on head classes.
DRW + SAM improves upon the overall performanceof CE+DRW by 5.1% on CIFAR-10 LT and 3.6% on CIFAR-100 LT datasets, with the tail classaccuracy increasing by 11.7% and 7.7% respectively.
As explained in Sec. 3, the reason for theincrease in performance can be attributed to the suppression of negative eigenvalues in the tail classspectral density.
[Table]
These results empirically show that escapingsaddle points with SAM leads to a notable increase in overall accuracy primarily due to themajor gain in the accuracy on the tail classes.
The addition of SAM to recently proposed longtail learning methods like LDAM and VS lossleads to a significant increase in performance,which indicates that the role of SAM is orthogonal to the margin-based methods.
On the otherhand, SAM without re-weighting (CE+SAM)improves accuracy on the head and mid classes rather than the tail class.
This can be attributed to thefact that standard ERM minimizes the average loss across all the samples without re-weighting andthe weightage of tail class samples in the overall loss is minimal.
Thus, the weightage of SAM in thetail classes is limited.
We also show results with step imbalance ( β = 100) on CIFAR-10 and CIFAR-100 datasets (Table 2).
With step imbalance on CIFAR-10, the first five classes have 5000 samples each, and the remainingclasses have 50 samples each.
The addition of SAM improves the overall performance of CE+DRWon CIFAR-10 by 7.1%, with the tail class accuracy increasing by 14.6%.
We observe that on mosttail classes, the density of negative eigenvalues in the spectral density is much lower with SAM.
Thisindicates that despite multiple classes with few samples, SAM with DRW can avoid the saddle points.
SAM systematically improves performance with LDAM and VS loss leading to state-of-the-artperformance on both CIFAR-10 and CIFAR-100 in the step imbalance setting.
Do these observations scale to large-scale datasets?
We report the results on ImageNet-LT datasetin Table 3.
The observations on CIFAR-10 LT and CIFAR-100 LT hold good even on ImageNet-LT.
For example, the accuracy on tail classes increases by 6.5% with the introduction of SAM on CE +
[Table]
DRW, which is similar to the gain observed in CIFAR-100 LT with CE + DRW.
These observationspoint out that the problem of saddle points also exists in large datasets and convey that SAM iseasily generalizable to large-scale imbalanced datasets without making any significant changes.
OniNaturalist 2018 [39] too, the accuracy on tail classes gets boosted by more than 3% with SAM (Table3).
Additional results are provided in the Appendix D.
Comparison with SOTA : VS loss [25] is a recently proposed margin-based method that achievesstate-of-the-art performance on class-imbalanced datasets with single-stage training without strongaugmentations [45], ensembles [44] or self-supervision [41].
SAM significantly improves upon theperformance of VS on both CIFAR-10 LT and CIFAR-100 LT.
For the practitioners, we suggest usinghigh ρ SAM with re-weighting or margin based methods for effective learning on long-tailed data.
5.2 Ablation Studies
A note on ρ value : We observe that as we increase the smoothness parameter ( ρ ) in SAM, theaccuracy on the tail classes increases significantly (Fig. 4).
The accuracy on tail classes increasesfrom 63% for ρ = 0.05 to 73% for ρ = 0.8 on CIFAR-10 LT with CE+DRW.
This can be ascribed to thecorrelation between λ min and ρ as discussed in Sec. 4.1.
As the ρ increases, the negative curvaturein the tail classes disappears because SAM aims to find a flat minima with a large neighborhood witha low loss value.
A very large ρ (0.8) leads to a drop in the head accuracy because it restricts thesolution space of the head class, resulting in a drop in the overall accuracy.
This also emphasizes thata high ρ is necessary for escaping saddle points and achieving the best results.
Other methods to escape saddle points : In Table 4, we show that other methods developed to escapesaddle points, such as PGD, can be used for improving generalization on tail classes.
LPF-SGD, analgorithm promoting convergence to flat landscape, inherently adds Gaussian noise to the networkparameters and could be considered similar to PGD.
We can see that the addition of PGD and
LPF-SGD to CE+DRW leads to a substantial gain in the performance of tail classes on CIFAR-
LT and CIFAR-100 LT.
It can also be observed that CE+DRW+SAM outperforms both PGD and
LPF-SGD by 2% on average.
This further highlights that various methods in literature developed toescape saddle points efficiently can be directly used to improve the performance of minority classeswhen training on class-imbalanced datasets.
Conclusion
In this work, we show that training on imbalanced datasets can lead to convergence to points withsufficiently large negative curvature in the loss landscape for minority classes.
We find that this isquite common when neural networks are trained with loss functions that are re-weighted or modifiedto enhance the focus on minority classes.
Due to the occurrence of saddle points, we observe thatthe network suffers from poor generalization on minority classes.
We propose to use SharpnessAware Minimization (SAM) with a high regularization factor ρ as an effective method to escaperegions of negative curvature and enhance the generalization performance.
We theoretically andempirically demonstrate that SAM with high ρ is able to escape saddle points faster than SGD and converge to better solutions, which is a novel observation to the best of our knowledge.
We show that combining SAM with state-of-the-art techniques for learning with imbalanced data leads to significant gains in performance on minority classes.
We hope that our work leads to further research in studying the effect of negative curvature in generalization as we show they are a practical issue for class-imbalanced learning using deep neural networks Checklist For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] Added in appendix.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
To the
best of our knowledge, we do not suspect any potential negative societal impacts of our
work.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section 4.
(b) Did you include complete proofs of all theoretical results?
[Yes] Proof is added in the
appendix.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper-
imental results (either in the supplemental material or as a URL)?
[Yes] Additional
information added in Appendix
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See
Section 5.
Additional information added in Appendix.
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
[Table]
ments multiple times)?
[Yes] See section 5.
We report error bars for CIFAR 10-LT andCIFAR 100-LT dataset.
For the large scale ImageNet-LT and iNaturalist 2018, we donot report error bars due to computational constraints.
[Table]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] Added in the appendix4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
[Table]
(a) If your work uses existing assets, did you cite the creators?
[Yes]
[Table]
(b) Did you mention the license of the assets?
[Yes] Added in appendix
[Table]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
[Table]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
[Table]
using/curating?
[N/A] We use publicly available assets for our work.
[Table]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
[Table]
information or offensive content?
[N/A]
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, if
[Table]
applicable?
[N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]
[Table]