CUDA: Curriculum of Data Augmentation for Long-tailed Recognition
Abstract
Class imbalance problems frequently occur in real-world tasks, and conventional deep learning algorithms are well known for performance degradation on imbalanced training datasets.
To mitigate this problem, many approaches have aimed to balance among given classes by re-weighting or re-sampling training samples.
These re-balancing methods increase the impact of minority classes and reduce the influence of majority classes on the output of models.
Despite extensive recent studies, no deep analysis has been conducted on determination of classes to be augmented and strength of augmentation has been conducted.
In this study, we propose a simple and efficient novel curriculum, which is designed to find the appropriate per-class strength of data augmentation, called CUDA : CU rriculum of
D ata
A ugmentation for long-tailed recognition.
CUDA can simply be integrated into existing long-tailed recognition methods.
We persent the results of experiments showing that CUDA effectively achieves better generalization performance compared to the state-of-the-art method on imbalanced datasets such as CIFAR-100-LT.
Introduction
Deep neural networks (DNNs) have significantly improved over the past few decades on a wide range of tasks [21, 45, 44].
This effective performance is made possible by come from well-organized datasets such as MNIST [30], CIFAR-10/100 [27] and ImageNet [48].
However, as Van Horn et al.
[51] indicated, gathering such balanced datasets is notoriously difficult in real-world applications.
In addition, the models perform poorly when trained on an improperly organized dataset, e g., in cases with class imbalance, because minority samples can be ignored due to their small portion.
The simplest solution to the class imbalance problem is to prevent the model from ignoring minority classes.
To improve generalization performance, many studies have aimed to emphasize minority classes or reduce the influence of the majority samples.
Reweighting [6, 40] or resampling [4, 52] are two representative methods that have been frequently applied to achieve this goal.
Although these elaborate rebalancing approaches have been adopted in some applications, limited information on minority classes due to fewer samples remains problematic.
To this end, we first consider that controlling the strength of class-wise augmentation can provide another dimension to mitigate the class imbalance problem.
In this paper, we use the number of augmentation operations and their magnitude to control the extent of the augmentation, which we refer to herein as its strength, e.g., a strength parameter of 2 means that two randomly sampled operations with a pre-defined magnitude index of 2 are used.
Our key finding is that class-wise augmentation improves performance in the non-augmented classes while that for the augmented classes may not be significantly improved, and in some cases, per-
[Figure]
in Appendix B. formances may even decrease. As described in Figure 1, regardless of whether a given dataset is class imbalanced, conventional class imbalance methods show similar trends: when only the major classes are strongly augmented ( e.g., strength 4 ), the performance of majority classes decreases, whereas that for the minority classes have better results.
This result motivates us to find the proper augmentation strength for each class to improve the performance for other classes while maintaining its own performance.
Contribution.
We propose a simple algorithm called CU rriculum of D ata
A ugmentation ( CUDA ) to find the proper class-wise augmentation strength for long-tailed recognition.
The proposed method consists of two modules, which compute a level-of-learning score for each class and leverage the score to determine the augmentation.
Therefore, CUDA increases and decreases the augmentation strength of the class that was successfully and wrongly predicted by the trained model.
We empirically examine performance of CUDA on synthetically imbalanced datasets, CIFAR-100LT [6].
With the high compatibility of CUDA , we apply our framework to various long-tailed recognition methods and achieve better performance compared to the existing long-tailed recognition methods.
We describe the further experimental results on ImagNet-LT [39] and iNaturalist 2018 [51] in Appendix D.
CUrriculum of Data Augmentation for Long-Tailed Recognition
The core philosophy of CUDA is to “generate an augmented sample that becomes the most difficult sample without losing its original information.”
In this section, we describe design of CUDA in terms of two parts: (1) a method to generate the augmented samples based on the given strength parameter, and (2) a method to measure a Level-of-Learning (LoL) score for each class.
2.1 Problem Formulation of Long-tailed Recognition
Suppose that the training dataset D = { ( x i , y i ) } Ni =1 is composed of images with size d , x i ∈ R d , and their corresponding labels y i ∈ { 1 , ..., C } .
D c ⊂ D is a set of class c , i.e., D c = { ( x, y ) | y
=
c, ( x, y ) ∈ D} .
Without loss of generality, we assume |D 1 | ≥ |D 2 | ≥ · · · ≥ |D C | , where |D| denotes the cardinality of the set D .
We denote the N max := |D 1 | and N min :=
|D C | .
LTR algorithms,
A LTR ( f θ , D ) , mainly focus on training the model f θ with parameter θ when the class distribution of training dataset P train ( y ) and test dataset P test ( y ) are not identical.
More precisely, P train ( y ) is highly imbalanced while P test ( y ) is balanced, i.e., uniform distribution.
[Table]
2.2 Curriculum of Data Augmentation
In this section, we describe our proposed DA with strength parameter, and the methods used to measured the LoL score.
Then, we integrate the two methods in a single framework to propose CUDA .
DA with a strength parameter.
Let us assume that there exist pre-defined K augmentation operations.
We utilize visual augmentation operations which is indiced as k ∈ { 1 , · · · , K } , e.g., Gaussian blur, Rotation, Horizontal flip.
Each augmentation operation
O m k
( s )
k : R d → R d has its own predefined augmentation magnitude function m k ( s ) where the strength parameter s ∈ { 0 , ..., S } .
These operations are described in detail along with each magnitude functions in Appendix E.
Given an augmentation strength parameter s and an input image x , we model a sequence of augmentation operations O ( x ; s ) as follows:
[Equation]
where, Cat ( · ) and U ( · ) denote categorical and discrete uniform distributions, respectively.
The sequential augmentation operation O ( x ; s ) samples s operations from the categorical distribution when the probability of seeing the operations follows uniform distribution.
Level-of-Learning (LoL).
To control the strength of augmentation properly, we check whether the model can correctly predict augmented versions without losing the original information.
To enable this, we define the LoL for each class c at epoch e , i.e., L ec , which is adaptively updated as the training continues as follows:
[Equation]
where
[Equation]
Here, γ ∈ [0 , 1] is threshold hyperparameter, T is coefficient of the number of samples used to updating LoL. V correct is a function which outputs the number of correctly predicted examples by the model f θ among l
+ 1 randomly augmented samples with strength l .
V correct is defined as:
[Equation]
c } where D ′ c ⊂ D c .
Note that D ′ c is a randomly sampled subset of D c with replacement and its size is T ( l + 1) .
The key philosophy of this criterion is two fold.
(1) If samples in the class c are trained sufficiently with an augmentation strength of L ec , the model is ready to learn a more difficult version with augmentation strength of L e +1 c ← L ec + 1 .
In contrast, if the model predicts incorrectly, it should re-learn the easier sample with an augmentation strength of L e +1 c ← L ec − 1 .
As the strength parameter increases, the number of candidates for the sequential augmentation operation O ( x ; L ) increases exponentially.
For example, the amount of increment is N L ( N − 1) when L is increases to
L + 1 .
To control the LoL in a large sequential augmentation operation space, we take more random samples to check as the strength parameter gets bigger.
In our experiments, linearly increasing the number of samples to evaluate corresponding to the strength with a small additional computation time was sufficient.
V LoL is described in Algorithm 2.
C E C E - DR
W
L D A M - DR
W
BS R I D E B C L C E +
CUD A
C E - DR W + CUD A
L D A M - DR
W + CUD
A
BS + CUD
A
R I D E +
CUD A
B C L +
CUD A A cc u r a cy       (a) All
C E C E - DR
W
L D A M - DR
W
BS R I D E B C L C E +
CUD A
C E - DR W + CUD A
L D A M - DR
W + CUD
A
BS + CUD
A
R I D E +
CUD A
B C L +
CUD A A cc u r a cy     (b) Many
C E C E - DR
W
L D A M - DR
W
BS R I D E B C L C E +
CUD A
C E - DR W + CUD A
L D A M - DR
W + CUD
A
BS + CUD
A
R I D E +
CUD A
B C L +
CUD A A cc u r a cy       (c) Median
C E C E - DR
W
L D A M - DR
W
BS R I D E B C L C E +
CUD A
C E - DR W + CUD A
L D A M - DR
W + CUD
A
BS + CUD
A
R I D E +
CUD A
B C L +
CUD A A cc u r a cy    (d) Few
Curriculum of DA.
By combining two components, including DA with a strength parameter and
LoL, our CUDA provides class-wise adaptive augmentation to enhance the performance of the others without losing its own information.
As shown in algorithm 1, we measure the LoL score L c for all classes in the training dataset to determine the augmentation strength for every epoch.
Based on L c , we generate the augmented version O ( x ; L c ) for x ∈ D c and train the model with the augmented samples.
Additionally, we randomly use the original sample instead of the augmented sample with probability p aug so that the trained models do not forget the original information.
In our experiments, this operation improved performance robustly on a wide range of p aug values.
Advantage of CUDA design.
Our proposed approach mainly has three advantages.
(1) CUDA adaptively finds proper augmentation strengths for each class without need for a validation set.
Following the spirits of existing curriculum learning methods [19, 62, 55], CUDA enables modeling by first presenting easier examples earlier during training to improve generalization.
This encourages the model to learn difficult samples ( i.e., within high augmentation strength) better.
(3) Moreover, owing to the universality of data augmentation, CUDA is easily compatible with other LTR algorithms, such as [6, 46, 54].
Experiments
In this section, we present empirical evaluation, the results of which demonstrate the superior performance of our proposed algorithm for class imbalance.
We describe the experimental results on
CIFAR-100-LT, a synthetic long-tailed benchmark.
CIFAR-100-LT is examined with imbalance ratio of 100, where an imbalance ratio is defined as N max /N min .
The related works, implementation setups, and additional results are detailed in Appendix.
In Figure 2, we report the performance when CUDA is applied to the various algorithms: CE, CEDRW [6], LDAM-DRW [6], BS [46], RIDE [54] with 3 experts, RIDE+CMO [42], and BCL [63].
We include four different categories of accuracy: all, many, med(ium), and few.
Each represents the average accuracy of all samples, classes containing more than 100 samples, 20 to 100 samples, and under 20 samples, respectively.
Compared to the case without CUDA cases, balanced validation performance is increased when we apply the proposed approach.
Conclusion
In this study, we proposed CUDA to address the class imbalance problem.
The proposed approach is also compatible with existing methods.
To design a proper augmentation for LTR, we first studied the impact of augmentation strength for LTR.
We found that the strength of augmentation for a specific type of class ( e.g., major class) could affect the performance of the other type ( e.g., minor class).
From this finding, we designed CUDA to adaptively find an appropriate augmentation strength without any further searching phase by measuring the LoL score for each epoch and determining the augmentation accordingly.
To verify the superior performance of the proposed approach, we examined each performance with various methods and obtained the best performance among the methods compared to CIFAR-100-LT, synthetically generated benchmarks. Acknowledgement
his work was supported by Institute of Information & communications Technology Planning &
Evaluation (IITP) grant funded by the Korea government(MSIT) (No.2019-0-00075, Artificial Intelligence Graduate School Program(KAIST), 10%) and the Institute of Information & communications Technology Planning & Evaluation(IITP) grant funded by the Korea government(MSIT) (No. 20220-00871, Development of AI Autonomy and Knowledge Enhancement for AI Agent Collaboration, 90%)
Long -tailed Learning
Owing to the page limitation of the main manuscript, we provide detailed information in this supplementary as follows.
(1) In Appendix A, we provide the related works for long-tailed recognition and data augmentation.
In Appendix B, we summarize the experimental setup of Figure 1, and further explain why augmentation on one side causes performance degradation on the opposite side. (3) In Appendix C, we describe in detail our experimental setting, including dataset configuration, data preprocessing, and training implementation.
In Appendix D, we show ImageNet-LT performance on different size and architecture networks, training time analysis, and accuracy on the balanced dataset case.
In Appendix E, we present in detail the augmentation operations that CUDA utilizes. (6) In Appendix F, we describe the experimental setting of Figure 9d.
A Related works
Long-tailed Recognition (LTR).
The datasets with class imbalances can lead DNNs to learn biases toward training data, and their performance may decrease significantly on the balanced test data.
To improve the robustness of such models to imbalance, LTR methods have been evolving in two main directions: (1) reweighting [15, 6, 43] methods that reweight the loss for each class by a factor inversely proportional to the number of data points, and (2) resampling methods [28, 7, 2] that balance the number of training samples for each class in the training set.
However, studies along these lines commonly sacrifice performance on majority classes to enhance that on minority classes, because the overfitting problem occurs with limited information on minority classes as a result of increasing the weight of a small number of minority samples.
Several methods have recently been developed to alleviate the overfitting issues in various categories: (1) two-stage training [6, 24, 39], (2) ensemble methods [61, 56, 54, 5], and (3) contrastive learning approach [23, 14, 63, 31, 34].
To re-balance the classifier layers after achieving a good representation on the imbalanced training dataset in an early phase, Cao et al.
[6] proposed deferred resampling (DRS) and reweighting (DRW) approaches.
Kang et al.
[24] decoupled the learning procedure into representation learning and training linear classifier, achieved higher performance than previous balancing methods.
Wang et al.
[54] and Cai et al.
[5] suggested efficient ensemble methods using multiple experts with a routing module and a shared architecture for experts to capture various representations.
Liu et al.
[38] found that self-supervised representations are more robust to class imbalance than supervised representations, and some works have developed supervised contrastive learning methods [25] for imbalanced datasets [14, 63, 34].
Another line of research has considered augmentation methods in terms of both input and feature spaces [26, 11, 33].
Recently, Park et al.
[42] mixed minority and majority images by using CutMix with different sampling strategies to enhance balancing and robustness simultaneously.
These methods commonly focus on utilizing the rich context of majority samples to improve the diversity of minority samples.
Moreover, these augmentation-based methods are relatitvely in easy to apply orthogonally with other LTR methods.
Data Augmentation (DA).
DA has been studied to mitigate overfitting which may occur due to a lack of data samples.
Some works have been proposed to erase random parts of images to enhance the generalization performance of neural networks [16, 60, 29, 9].
Recently, variants of MixUp [59] have been proposed; this method combines two images with specific weights [50, 18, 49, 16, 53].
By aggregating two approaches, CutMix [58] was proposed to erase and replace a small rectangular part of an image into another image.
In another line of research, methods have been proposed to automatically configure augmentation operations [12, 37, 35, 20, 17].
In addition, Cubuk et al.
[13] randomly selected augmentation operations using the given hyperparameters of the number of sampling augmentation and their magnitudes.
Recently, class-wise or per-sample auto-augmentation methods have also been proposed [8, 47].
B Detail for FigureB.1 Experimental Settings
Major and minor group decomposition.
To check the impact of augmentation on majority and minority classes, we split the training dataset into two clusters.
The majority cluster is the top classes by sorting through the number of samples for each class.
The bottom 50 classes are in the minority cluster.
For simplicity, we utilize class indices of 0 to 49 as the majority and 50 to 99 as the minority, respectively.
For the balanced case, we utilize 0 to 49 classes as cluster 1, and the others as cluster 2.
Controlling augmentation strength.
We set the augmentation strength as the number of augmentation and its augmentation magnitude by following the augmentation rule of CUDA .
For example, the samples in the majority classes with magnitude parameter 4 represents that they are augmented with randomly sampled 4 augmentations with their own pre-defined augmentation magnitude.
Training setting.
For heatmaps in Figure 1, we follow the training recipe of CIFAR-100-LT for CE case, e.g., ResNet-32, learning rate of 0 . 1 , and so on.
Further details, hyperparameters, and datasets are described in section 3 and Appendix C.
[Figure] [Figure]
T r a i n C o s - S
i m
T e s t C o s - S
i m
W e i gh t
N o r m
Without Augment
Partial Augment
All Augment
B.2 Analysis
Analysis for Figure 1.
To figure out the reason for the phenomena in Figure 1, we conduct further analysis as shown in Figure 3 and Figure 4.
Our experimental setups are as follows:
• Train the networks with three augmentation strategies, respectively (without, partial, and all), then measure the class-wise feature alignment and linear classifier weight norm for all networks.
(Experiment 1) • From a trained network without augmentation in Experiment 1, we freeze the feature extractor and train the linear classifier layer with augmenting partial classes.
Then, we measure the class-wise L1-norm for all linear classifiers.
(Experiment 2)
From the Figure 3 and Figure 4, we have three observations from Experiment 1:
When we conduct augmentation only for partial classes (0-49 classes), the feature alignment for augmented classes of the training dataset is degraded compared to the non-augmented classes.
This is because the augmentation classes have more diversified training data than non-augmentation classes, which leads to more diversification in feature space.
We observe the balance between alignment between classes in the cases of without augmentation and with all augmentation since all classes have similar diversity.
(See the first rows in Figure 3, 4) 2.
However, all three augmentation strategies have balanced class-wise feature alignment for the same test dataset.
This tendency can be observed in both balanced and imbalanced datasets.
This result is consistent with [24].
Furthermore, the values for feature alignment are increased when we conduct augmentation partially or all, compared to without augmentation.
This result shows that augmentation enhances the feature extraction ability, which is consistent with conventional studies.
(See the second rows in Figure 3, 4) 3.
When we conduct augmentation only for partial classes on a balanced dataset, the class-wise weight norm of the linear classifier is larger for non-augmentation classes.
This result incurs performance improvement for non-augmentation classes and reduction for augmentation classes since this linear classifier has a tendency to classify non-augmented classes with larger weight values.
However, we observe that class-wise weight norms are balanced in “without augmentation” and “all augmentation” cases.
(See the third row in Figure 3) 4.
We observe that the class-wise weight norm of the linear classifier is larger for majorities for all classes that have the same augmentation strength.
These results are consistent with previous works [24, 1].
However, when we conduct augmentation only for majorities, the class-wise weight norm is more balanced.
This phenomenon is similar to the balanced case in that partial augmentation incurs a reduction in the norm of the linear classifier for augmented classes.
(See the third row in Figure 4)
[Figure]
[Figure]
[Figure]
Our observations from Experiment 1 are highly consistent in both balanced and imbalanced datasets.
The results in Figure 1, Figure 3 and Figure 4 highly motivate the design of CUDA .
Moreover, our results for Experiment 2 can explain these observations as shown in Figure 5 and Figure 6.
We observe that in the presence of feature alignment degradation from augmentation, the corresponding norm is relatively small, as shown in Figure 5.
This is because in the class that has lower feature alignment, the variation of the gradient for the linear classifier is larger than in the class with high feature alignment.
As shown in Figure 6, from Experiment 2, we observe that ∥ ∆ w ∥ , the norm of class-wise difference of between current and initialize linear classifier parameters ∆ w :=
w − w 0 , have smaller value in augmented classes than non-augmented classes.
From our experimental analysis in Figure 3, 4, and 6, we can conclude that augmentation breaks the consistency of feature alignment and it makes the weight norm of the linear classifier decreases.
C Implementation detail in sectionC.1 Implementation Detail
Baselines.
We compare CUDA with previous long-tailed learning algorithms , including cross-entropy loss (CE), two-stage approaches: CE-DRW [6] and cRT [24], balanced loss approaches: LDAMDRW [6] and Balanced Softmax (BS; [46]), the ensemble method: RIDE with three experts [54], resampling algorithms: Remix [10] and CMO [42], and contrastive learning-based approach: BCL [63].
We integrate CUDA with CE, CE-DRW, LDAM-DRW, BS, RIDE, and BCL algorithms.
For longer epochs, we compare CUDA with PaCo [14], BCL, and NCL [31], by combining CUDA with BCL and
NCL.
For a fair comparison of the computational cost, we train the network with the official one-stage implementation of RIDE ( i.e., without distillation and routing).
Implementation.
For CIFAR-100-LT dataset, almost all implementations follow the general setting from [6], whereas cRT [24], BCL, NCL and RIDE follow the settings used in their original implementation.
Following [6], we use ResNet-32 [22] as a backbone network for CIFAR-100-LT.
The network is trained on SGD with a momentum of 0 .
9 and a weight decay of 2 ×
10 − 4 .
The initial learning rate is 0 .
1 and a linear learning rate warm-up is used in the first 5 epochs to reach the initial learning rate.
During training over 200 epochs, the learning rate is decayed at the 160 th and 180 th epochs by 0 .
For the ImageNet-LT and iNaturalist, the ResNet-50 is used as a backbone network and is trained for 100 epochs.
The learning rate is decayed at the 60 th and 80 th epochs by 0 .
As with CIFAR, for cRT, RIDE, and BCL, we follow the original experimental settings of the official released code.
For the hyperparameter values of CUDA , we apply a p aug of 0 .
5 and T of 10 for all experiments.
For γ , we set the values as 0 .
6 for CIFAR-100-LT and 0 .
4 for ImageNet-LT and iNaturalist 2018.
The detailed implementation for baselines are in Appendix C.
CIFAR-100-LT.
CIFAR-100-LT is a subset of CIFAR-100.
Following [54, 42, 63], we use the same long-tailed version for a fair comparison.
The number of samples of k th class is determined as follows: (1) Compute the imbalanced factor N max /N min , which reflects the degree of imbalance in the data.
(2) |D k | between |D 1 | =
N max and |D 100 | = N min follows an exponential decay ( i.e., |D k | =
|D 1 | × ( N max /N min ) k/ 100 ).
The imbalance factors used in the experiment are set to 100, 50, and 10.
ImageNet-LT.
ImageNet-LT [39] is a modified version of the large-scale real-world dataset [48].
Subsampling is conducted by following the Pareto distribution with power value α = 0 .
It consists of 115 .
8 K images of 1 , 000 classes in total.
The most common or rare class has 1 , 280 or 5 images, respectively. iNaturalist 2018.
iNaturalist [51] is a large-scale real-world dataset which consists of 437 .
5 K images from 8 , 142 classes.
It has long-tailed property by nature, with an extremely class imbalanced.
In addition to long-tailed recognition, this dataset is also used for evaluating the fine-grained classification task.
C.2 Data Preprocessing
For data preprocessing, we follow the default settings of [6].
For CIFAR-100-LT, each side of the image is padded with 4 pixels, and a 32 × 32 crop is randomly selected from the padded image or its horizontal flip.
For ImageNet-LT and iNaturalist 2018, after resizing each image by setting the shorter side to 256 pixels, a 224 × 224 crop is randomly sampled from an image or its horizontal flip.
For BCL and NCL, which use AutoAugment [12] or RandAugment [13] as default data augmentation, we apply them after random cropping by following their original papers [63, 31].
Then, we finally conduct CUDA after all default augmentation operations, and then normalize the image with following mean and standard deviation values sequentially: CIFAR-100-LT ((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010)), ImageNet-LT ((0.485, 0.456, 0.406), (0.229, 0.224, 0.225)), and iNaturalist 2019 ((0.466, 0.471, 0.380), (0.195, 0.194, 0.192)).
C.3 Detailed Implementation
Because some official codes do not open their entire implementations, we re-implement by following the rules.
For re-implementation, we reproduce the code based on their partial code and the authors’ responses.
RIDE.
We follow the officially offered code 2 .
Among various experimental configurations of official code ( e.g., one-stage RIDE, RIDE-EA, Distill-RIDE), for fair comparison (to leverage similar computation resources), we utilize one-stage training ( i.e., one-stage RIDE) for all cases.
We confirm that CMO [42] also utilizes this setup for RIDE + CMO from the response of the authors.
CMO.
We re-implement all CMO results from their official code 3 in our work.
However, the official code of CMO does not contain code for RIDE + CMO.
Therefore, we re-implement by injecting the CMO part for BS in the official code (weighted sampler and mixup part) into the RIDE code.
Furthermore, for iNaturalist 2018, we train the model for 100 epochs for a fair comparison with other methods (whereas the original RIDE + CMO is trained for 200 epochs on iNaturalist 2018).
BCL.
The officially released code 4 of BCL only contains ImageNet-LT and iNaturalist 2018.
Whereas the official code applies a cosine classifier for ImageNet-LT and iNaturalist 2018, we apply an ordinary linear classifier for CIFAR-100-LT from the author’s response.
All hyperparameters are the same as the experiment settings of the original work [63].
[Table] [Table]
[Table]
AA [12], FAA [37], DADA [35]
[Table]
and RA [13] with n = 1 , m =policies are used.
C, S, I represent CIFAR, SVHN, and ImageNet policy.
[Table]
D Analyses
CIFAR-100-LT.
In Figure 2, we report the performance when CUDA is applied to the various algorithms: CE, CE-DRW [6], LDAM-DRW [6], BS [46], RIDE [54] with 3 experts, RIDE+CMO [42],
[Figure] [Figure]
x
[Figure]
and BCL [63].
Compared to the case without CUDA cases, balanced validation performance is increased when we apply the proposed approach.
Recently, some works [14, 1, 63, 31] have shown impressive performances with diverse augmentation strategies and longer training epochs.
For a fair comparison with these methods, we examine CUDA using the same experimental setups from PaCo ([14]; 400 epochs with batch size of 64).
Table 3 shows that augmented image using CUDA can enhance LTR performance compare to the other baselines.
In particular, CUDA with NCL obtains the best performance over 400 epochs.
As noted by [31], the NCL algorithm utilizes six times as much memory compared to the vanilla architecture with three experts.
Hereinafter in large-scale benchmarks, we focus on the cases with similar network size.
ImageNet-LT and iNaturalist 2018.
To evaluate the performance of CUDA on larger datasets, we conduct experiments on ImageNet-LT [39] and iNaturalist 2018 [51].
Table 2 summarizes the performance of various LTR methods and the performance gain when integrated with CUDA .
Our proposed method consistently improves performance regardless of the LTR method and target dataset by simply adding class-wise data augmentation without complicated methodological modification.
Additionally, to evaluate the performance gain of CUDA on other architectures, we experiment with
CUDA on ImageNet-LT with ResNet-10 [39] and ResNeXt-50 [57], as reported in Appendix D.
D.1 Analysis
We design our analyses to answer the following questions.
(1) How does CUDA perform?
Does
CUDA perform better than other augmentation methods?
(3) How does LoL score change over training epochs when combined with various LTR methods?
(4) Which part of CUDA is important to improvet performance?
These analyses provide additional explanations to understand CUDA .
All experiments are conducted on CIFAR-100-LT with imbalance ratio of 100 .
How does CUDA mitigate the class imbalance problem?
To deeply understand CUDA , we observe two types of metrics: (1) variance of classifier weight L1-norm, (2) feature alignment gain ( i.e., cosine similarity with and without CUDA ) on validation dataset.
The classifier weight norm is usually used to measure how balanced the model consider the input from a class-wise perspective [24, 1].
Feature alignment, especially feature cosine similarity amongst samples belonging to the same class, is a measure of the extent to which the extracted features are aligned [41].
As shown in Figure 7, CUDA has two forces for alleviating imbalance.
For all cases, CUDA reduces the variance of the weight norm ( i.e., balance the weight norm), and thus the trained model consider the minority classes in a balanced manner.
Note that because LDAM-DRW and RIDE utilize a cosine classifier ( i.e., utilizing L normalized linear weight), their standard deviation scale is quite different from those other methods.
Because LDAM-DRW, BS, and RIDE include balancing logic in their loss function, they exibit lower
[Figure] u [Figure] cy
[Figure] [Figure] [Figure]
(a [Figure]
prob [Figure]
A cc u r a cy 50.0 50.
Num test (b) Num test T .
A cc u r a cy 49.649.850.
Accept rate γ0.3 0.5 0.
(c) Accept rate γ .
VanillaHyperOptDADA (Imb)
CUDA w/o Curr CUDA
A cc u r a cy ( % )
[Figure]
variance reduction compared to the CE and CE-DRW.
Second, as shown in the bottom row in Figure 7,
CUDA obtains feature alignment gains for almost all classes.
This shows that CUDA facilitates a network to learn to extract meaningful features.
Compared with other augmentations.
To verify the impact of CUDA , we examine the other augmentation methods as follows.
We compare five augmentation methods, including AutoAugment (AA, [12]), Fast AutoAugment (FAA, [37]), DADA [35], RandAugment (RA, [13]), and the proposed method CUDA .
Because AA, FAA, and DADA provide their policies searched by using CIFAR, SVHN (for AA), and ImageNet, we leverage their results.
Furthermore, RA suggests using their parameter ( n, m ) =
(1 , 2) for CIFAR, and we follow their guidelines.
As shown in Table 4, even though the augmentation policies use additional computation resources to search, CUDA outperforms the other pre-searched augmentations.
This shows that CUDA is computationally efficient.
Dynamics of LoL score.
We evaluate how LoL scores vary with algorithms: CE, CE-DRW, LDAMDRW, BS, and RIDE.
Note that we set a lower class index ( i.e., 0 ) as the most common class ( i.e., the number of samples is 500 ), while an index of 100 represents the rarest class ( i.e., with five samples).
As described in Figure 8, as training progressed, the LoL score of all algorithms increase.
After learning rate decay ( i.e., 160 epoch) all algorithms are able to learn to classify minority classes more easily than before.
In particular, except for BS, the majority classes of most algorithms show a steep increment.
The reason that BS exhibit a similar increasing speed for majority and minority classes is that it includes a module to balance the impact of majority and minority samples.
Moreover, because
RIDE can promote when all experts satisfying promotion criterion, its increasing speed is moderate.
Parameter sensitivity.
For further analysis, we conduct a sensitivity analysis of hyperparameters in CUDA .
More precisely, we study three kinds of parameters, including augmentation probability p aug
(Figure 9a), number of tests T (Figure 9b), and LoL update threshold γ (Figure 9c).
We examine each hyperparameter sensitivity on a CUDA case with RIDE and the remainder of the hyperparameters are fixed to the default values in Section ?? .
All results show that the performance gains of CUDA decreases if the parameters are adjusted to make the augmentation too strong or weak.
For example, the augmentation strength of all classes steeply increases when γ becomes small.
The strength cannot increase when γ becomes large, and thus it cannot improve the performance of the model.
Moreover, as shown in Figure 9b, the performance of CUDA increases as T increases.
However, larger T spends computational overhead, we set T as 10 and obtained cost-effective performance gain.
Impact of curriculum.
In addition to studying the impact of CUDA , we examine its performance component-wise.
In particular, we test the case where class-wise augmentation strength is searched based on the hyperparameter optimization algorithm.
We check five cases overall: vanilla algorithm, hyperparameter optimization (HO), re-searched DADA for CIFAR-100-LT, CUDA without curriculum, ( i.e., re-training utilizing the final augmentation strength of CUDA ), and CUDA .
As described in Figure 9d, CUDA finds better augmentation strengths compare to the hyperparameter search case.
This means that CUDA exhibits not only a lower searching time but also obtains better augmentation strength.
Moreover, by comparing the performance of with or without curriculum, the curriculum also can provide additional advance to the model to achieve better generalization.
Additionally, as
Figure 8, lower augmentation strength at the beginning of training is more effective than static higher augmentation strength.
These results are consistent with the results of previous studies on curriculum learning methods [19, 62].
Training Time Analysis.
CUDA requires additional computation for computing LoL score.
We measure the additional training time for adding CUDA on various algorithms.
As shown in Figure 11, when utilizing CUDA additional training time is spent.
However, the additional operation for searching
[Figure] [Figure]
the LoL score does not require a large value.
For example, BS with CUDA spends × 1 . 29 time to obtain adequate augmentation strength.
Network Architecture Analysis.
We also present our ResNet-10 [39] and ResNeXt-50 [57] experiments on the ImageNet-LT dataset in Figure 10, respectively.
These results show that CUDA consistently improves performance regardless of network sizes and corresponding LTR methods.
[Figure]
What if CUDA is ran on the balanced dataset.
We examine that if CUDA is applied to the balanced case, i.e., imbalance ratio is 1 .
As described in the Table 5 CUDA obtains 1 . 9% accuracy gain, which is lower than the other auto augmentation methods.
However, other autoaugmentation methods spend more computation time searching a good augmentation than CUDA .
Furthermore, as described in Figure 8, CUDA has higher performance than the others when the class imbalance dataset is given.
E Augmentation Preset E.1 Data augmentation operations used in CUDA .
There have been numerous data augmentation operations in vision tasks.
We used totally 22 augmentations for CUDA with their own parameter set.
Details of the operation set and parameters are described in Table 6.
For augmentation magnitude parameter m k ( s ) , we divide parameters into thirty values linearly.
For example of, ShearX case, its max and min values are 3 and 0 , respectively.
Therefore, m ShearX ( s ) =
(3 − 0) / 30 ∗ s , thus m ShearX (1) = 0 .
=
(3 − 0) / 30 ∗ 1 .
F Experimental setting of Figure 9d
To further analyze the impact of curriculum, we compare CUDA with the performance of previous hyper-parameter search algorithms and auto-augmentation methods, especially DADA [35].
We describe each setting in detail as follows.
Hyper-parameter search.
We utilize the strength score-based augmentation module in CUDA to verify the hyper-parameter search.
In other words, samples in each class utilize K augmentation operations.
Therefore, we search the class-wise augmentation on the search space K N where N is the number of classes.
We leverage the hyper-parameter searching open-source library, Ray [36], for search K N space efficiently.
Among various search modules, we utilize the HyperOptSearch module, which is the implementation of the Tree-structured Parzen Estimator [3].
Moreover, for fast search, we use the Asynchronous Successive Halving Algorithm (ASHA) [32].
We run 1 , 000 trials for each algorithms which spends almost 20 GPU hours ( i.e., × 80 overhead compare to CUDA).
[Figure]
[Table]
Edge Enhancement
[Table]
Increasing
[Table]
Detail
On/Off
Utilize convolutional kernel [[0 , − 1 , 0] , [ − 1 , 10 , − 1] , [0 , − 1 , 0]]
Smooth
On/Off
Utilize convolutional kernel [[1 , 1 , 1] , [1 , 5 , 1] , [1 , 1 , 1]]
AutoContrast
On/Off
Remove a specific percent of the lightest and darkest pixels
Equalize
On/Off apply non-linear mapping to make uniform distribution
Invert
On/Off
Negate the image
Gaussian Blur [0,2]
Blurring an image using Gaussian function
Resize Crop [1,1.3]
Resizing and center random cropping
Rotate [0,30]
Rotate the image
Posterize [0,4]
Reduce the number of bits for each channel
Solarize [0,256]
Invert all pixel values above a threshold
SolarizeAdd [0,110]
Adding value and run solarize
Color [0.1, 1.9]
Colorize gray scale values
Contrast [0.1,1.9]
Distance between the colors
Brightness [0.1,1.9]
Adjust image brightness
Sharpness [0.1,1.9]
Adjust image sharp
Shear X [0,0.3]
Shearing X-axis
Shear Y [0,0.3]
Shearing Y-axis
Translate X [0,100]
Shift X-axis
Translate Y [0,100]
Shifting Y-axis
Researched DADA operation on imbalanced CIFAR.
Because the officially offered policies on CIFAR by DADA [35] are searched for a balanced CIFAR dataset, we have to re-search the augmentation policy for the imbalanced dataset.
We utilize the official code of DADA and replace the dataloader to re-search the operations.
It spends 48 minutes for searching the augmentation policy ( × 8 .
6 than the overhead of CUDA).
Despite this additional overhead, DADA outputs worse performance than CUDA (even CUDA without curriculum case).
This is because (1) DADA does not consider class-wise augmentation and (2) it does not consider the impact of class imbalance.