C OMPOSITIONAL T RAINING FOR E ND - TO -E ND D EEP AUC M AXIMIZATION
A BSTRACT
Recently, deep AUC maximization (DAM) has achieved great success in different domains (e.g., medical image classiﬁcation).
However, the end-to-end training for deep AUC maximization still remains a challenging problem.
Previous studies employ an ad-hoc two-stage approach that ﬁrst trains the network by optimizing a traditional loss (e.g., cross-entropy loss) and then ﬁnetunes the network by optimizing an AUC loss.
This is because that training a deep neural network from scratch by maximizing an AUC loss usually does not yield a satisfactory performance.
This phenomenon can be attributed to the degraded feature representations learned by maximizing the AUC loss from scratch.
To address this issue, we propose a novel compositional training framework for end-to-end DAM, namely compositional DAM .
The key idea of compositional training is to minimize a compositional objective function, where the outer function corresponds to an AUC loss and the inner function represents a gradient descent step for minimizing a traditional loss, e.g., the cross-entropy (CE) loss.
To optimize the non-standard compositional objective, we propose an efﬁcient and provable stochastic optimization algorithm.
The proposed algorithm enhances the capabilities of both robust feature learning and robust classiﬁer learning by alternatively taking a gradient descent step for the CE loss and for the AUC loss in a systematic way.
We conduct extensive empirical studies on imbalanced benchmark and medical image datasets, which unanimously verify the effectiveness of the proposed method.
Our results show that the compositional training approach dramatically improves both the feature representations and the testing AUC score compared with traditional deep learning approaches, and yields better performance than the two-stage approaches for DAM as well.
The proposed method is implemented in our open-sourced library LibAUC ( www.libauc.org ) and code is available at https://github.com/Optimization-AI/LibAUC .
I NTRODUCTION
Deep AUC maximization (DAM) represents a new learning paradigm for deep learning, which maximizes the area under ROC curve (AUC) on a training dataset for learning a deep neural network.
It has received increasing attention recently due to the advancement in large-scale non-convex optimization algorithms for AUC maximization (Liu et al., 2019a; Yuan et al., 2021; Guo et al., 2020a;b).
Recently, DAM has been successfully applied to different domains with imbalanced data (Yuan et al., 2020; Wang et al., 2021b).
For example, Yuan et al.
(2020) has employed DAM for a variety of medical image classiﬁcation tasks, e.g., classiﬁcation of X-ray images, skin lesion images, mammograms, and microscopic images, and they observed great improvements with about 1% ∼ 5%
AUC increase over traditional deep learning approaches by optimizing a standard loss function, e.g., the cross-entropy (CE) loss.
These pioneering studies on DAM open a new direction for deep learning in the presence of imbalanced data but also raise many questions yet to be solved.
A particular question relates to how the network is trained by DAM.
Existing studies employ a two-stage approach for DAM, in which the ﬁrst stage pretrains the network on the training data by optimizing a traditional loss function (e.g. the CE loss) and the second stage ﬁnetunes the network by optimizing an AUC loss.
It was conjectured in (Yuan et al., 2020)
the feature extraction layers learned by directly optimizing AUC from scratch are not as good as optimizing the standard CE loss, similarly as optimizing a class-weighted loss for deep learning with imbalanced
Although the ad-hoc two-stage method of DAM has achieved some success, this approach leads to several undesirable consequences increasing the engineering costs in practice: (i) which layers should we ﬁnetune in the second stage?
Fine-tuning all layers increases the training costs but not necessarily improves the ﬁnal performance (Jamal et al., 2020; Qi et al., 2020); (ii) when do we stop the training for the ﬁrst-stage?
A long training time for the ﬁrst-stage increases the overall training costs but not necessarily increases the ﬁnal prediction performance, while a short training time for the ﬁrst-stage could harm the prediction performance (Kang et al., 2019).
Hence, the literature has suggested different tricks for the two-stage approach, including the decoupling method that simply optimizes the classiﬁer layer in the second-stage (Kang et al., 2019) and deferred re-weighting that only dedicates the iterations with the largest step size to the CE loss (Cao et al., 2019), which could be borrowed to DAM as well.
However, an important question remains open regarding DAM:
How can we conduct end-to-end training for deep AUC maximization?
To answer this question, we have examined the learned feature representations by optimizing an
AUC loss directly from scratch and conﬁrmed the conjecture in (Yuan et al., 2020) that the learned feature representations exhibit no advantage over optimizing the CE loss directly.
In Figure 1 we visualize the feature representations on an imbalanced training set for the CatvsDog classiﬁcation learned by different methods and visualized by t-SNE (van der Maaten & Hinton, 2008).
We can see that optimizing an AUC loss from scratch (2nd column) does not yield a cleaner feature representations for the two classes of data than optimizing the CE loss.
What makes end-to-end deep learning successful is its superb feature learning capability, i.e., the lower layers capture the low-level features and higher layers capture the high-level features.
In terms of feature learning, different examples roughly have equal weights regardless which classes they belong to.
From this perspective, we could understand why optimizing AUC loss alone bears worse feature learning capability.
The AUC loss assigns different weights to different examples from different classes for more robust classiﬁer learning.
In particular, the data in the positive class has a higher weight than data in the negative class.
These non-equal weights are important for learning a robust classiﬁer given that feature representations have been learned well but are not readily helpful for learning feature representations in an end-to-end fashion.
Can we achieve both effects in a uniﬁed and end-to-end learning framework, i.e., optimizing the CE loss with equal weights for robust feature learning and optimizing an AUC loss with uneven weights for robust classiﬁer learning?
A naive approach is to simply optimize a linear combination of the CE loss and an AUC loss.
However, this approach has a trade-off, meaning that AUC is not necessarily maximized due to the presence of the CE loss in the objective and the learned feature representations could be degraded by the AUC loss (Figure 1, 3rd column).
In this paper, we propose a better and novel end-to-end training method that not only achieves both beneﬁts of minimizing the CE loss for robust feature learning and minimizing an AUC loss for robust classiﬁer learning, but also achieves the effect of “1+1>2", i.e., achieves better performance than the naive linear combination approach.
The novel synthesis lies in how we composite the two training steps corresponding to the CE loss and the AUC loss.
The central idea is to minimize a two-level compositional objective, where the outer function is an AUC loss, and the inner function is a gradient descent step towards minimizing the CE loss, which represents a quick adaptation to the solution to minimizing the CE loss.
We propose a novel efﬁcient stochastic algorithm with provable convergence for minimizing the compositional objective, which performs alternating gradient-based updates that are ﬁrst based on the gradient of CE loss and then based on the gradient of an AUC loss at the point obtained in the ﬁrst step.
We summarize our contributions below.
• We propose a novel training framework for end-to-end deep AUC maximization, namely compositional DAM.
The novel compositional objective enables not only the robust feature learning of
lower layers from minimizing the standard loss function but also the robust learning of a classiﬁer from minimizing an AUC loss.
• Theoretically, we propose an efﬁcient stochastic optimization algorithm for solving compositional DAM, and establish the same convergence rate as standard SGD for optimizing a standard averaged loss.
Empirically, we conduct extensive studies on benchmark and medical image datasets, and observe that the proposed method not only improves the baseline methods including the naive linear combination approach but also improves ad-hoc two-stage approaches for DAM.
The learned feature representations of compositional DAM (e.g., Figure right ) are much better than those learned by minimizing the CE loss or an AUC loss alone and their combination.
R ELATED W ORK
Deep AUC Maximization.
AUC maximization has a history of two decades.
Most of the existing studies revolve around the design of efﬁcient optimization algorithms.
Earlier papers have focused on full batch methods (Herbrich et al., 1999; Yan et al., 2003; Ferri et al., 2002; Freund et al., 2003;
Joachims, 2005; Herschtal & Raskutti, 2004; Rakotomamonjy, 2004; Zhang et al., 2012) and online optimization methods (Zhao et al., 2011; Kar et al., 2014; 2013; Gao et al., 2013).
Recently, stochastic optimization for AUC maximization has become the dominating approach (Ying et al., 2016; Liu et al., 2018; Natole et al., 2018; 2019).
Ying et al.
(2016) propose a milestone work for stochastic optimization of AUC.
They consider optimizing the pairwise square loss and propose an equivalent min-max formulation that transforms the original non-decomposable objective into a decomposable one, which enables the design of efﬁcient stochastic methods based on mini-batch of data without explicitly constructing the pairs.
The min-max formulation also serves as the basis for recent works on DAM (Liu et al., 2019a; Yuan et al., 2021; Guo et al., 2020a;b).
(Liu et al., 2019a) is the ﬁrst work that explicitly considers DAM and develops the ﬁrst practical and provable stochastic algorithms for
DAM based on the min-max formulation of the pairwise square loss function.
However, this work only focuses on optimization and experiments are done on simple benchmark datasets.
Later, Yuan et al.
(2020) propose a new robust loss in the min-max form for DAM and evaluates the performance of DAM on various medical image classiﬁcation tasks, which demonstrates great success of DAM.
However, none of these works have addressed the problem of end-to-end training for DAM.
Deep Learning with Imbalanced data.
Deep learning in the presence of imbalanced data has recently attracted tremendous attention (Cui et al., 2019; Johnson & Khoshgoftaar, 2019; Masko &
Hensman, 2015; Lee et al., 2016; Khan et al., 2017; Dablain et al., 2021;
Ren et al., 2018;
Jamal et al., 2020; Qi et al., 2020; Lin et al., 2017; Cao et al., 2019; Kang et al., 2019; Liu et al., 2019b;
Zhu & Yang, 2020; Zhou et al., 2020; Xiang et al., 2020; Wang et al., 2021a; 2020; Menon et al., 2021).
Among these studies that are closely related to our work include (Lin et al., 2017; Cui et al., 2019; Cao et al., 2019;
Qi et al., 2020; Kang et al., 2019; Jamal et al., 2020), which focus on optimizing different objectives from the standard CE loss, including class-weighted loss, focal loss, individually weighted loss functions, etc.
Nevertheless, these works are not directly comparable to our method for maximizing AUC.
Two-stage Approaches.
However, directly optimizing a weighted loss for training a deep neural network from scratch does not work well (Cao et al., 2019;
Kang et al., 2019; Jamal et al., 2020; Qi et al., 2020; Yuan et al., 2020).
This phenomenon was ﬁrst observed in (Cao et al., 2019), which is attributed to the degraded feature representations.
To tackle this issue, Cao et al.
(2019) propose a deferred re-weighting/re-sampling trick.
It minimizes the standard average loss for the ﬁrst stage and switches to minimizing the class-weighted loss or re-sampling method in the second stage.
In their paper, the ﬁrst stage is deﬁned as the training period from the beginning to the iteration that the step size was reduced for the ﬁrst time in SGD or momentum methods.
Kang et al.
(2019) investigate a decoupling approach, where the ﬁrst stage learns the feature representations (i.e., a feature extraction network) by optimizing a standard loss with a large number of iterations, and the second stage learns a robust classiﬁer (i.e. the classiﬁer layer).
The authors show that the decoupling approach can achieve better performance than the deferred re-weighting trick in (Cao et al., 2019).
However, the decoupling approaches do not necessarily yield the best performance.
In particular, some studies have found that ﬁne-tuning some higher layers besides the classiﬁer layer in the second stage is beneﬁcial (Jamal et al., 2020;
Qi et al., 2020).
Recently, Yang & Xu (2020) propose to use self-supervised learning for learning the feature representations for the ﬁrst stage and to switch to re-weighting method in the second stage.
Different from these studies, our work is to design an elegant end-to-end training framework for deep learning with an AUC loss.
C OMPOSITIONAL T RAINING FOR D EEP AUC M AXIMIZATION
Notations.
We use ( x , y ) to denote an example, where x ∈ R d 0 denotes the input and
y ∈ Y denotes its corresponding label.
Let (cid:107) · (cid:107) denote the Euclidean norm of a vector, and let w ∈ R
d denote the weight parameters of a deep neural network.
A function F ( w ) is called L -smooth if its gradient is L -Lipschitz continuous, i.e., (cid:107)∇ F ( w ) − ∇ F ( w (cid:48) )
(cid:107) ≤ L (cid:107)
w − w (cid:48) (cid:107) .
Let f ( w ; x ) denote the prediction scores of a deep neural network parameterized by w on an input x , where f ( w ; x ) ∈ R for binary classiﬁcation with Y = { 1 , − 1 } .
Denote by D = { ( x 1 , y 1 ) , . .
, ( x n , y n ) } a set of n training examples.
Let (cid:96) ( w ; x , y ) denote a loss function on an individual data, e.g., cross-entropy loss.
We let L ( w ; S ) denote an aggregate loss function deﬁned on a set of samples S ⊆ D .
When
S =
D , we simply use the notation L ( w ) =
L ( w ; D ) .
Let Π Ω [ θ ] denotes an Euclidean projection on the set Ω .
Denote by n + ( n − ) the number of positive (negative) examples.
A standard approach of deep learning is to minimize an averaged loss on training examples, i.e.,
[Equation]
AUC losses.
AUC (area under the ROC curve) is a commonly used measure for evaluating classiﬁers for binary classiﬁcation with imbalanced data.
Recently, there emerge voluminous studies on optimizing AUC score for learning a predictive model (e.g., a deep neural network).
The idea is to optimize a surrogate loss for the AUC score.
A special surrogate loss is the AUC square loss (Gao & Zhou, 2015), which is deﬁned as:
[Equation]
where c is a margin parameter (e.g., 1 ).
Since directly optimizing the above pairwise loss is computationally expensive, existing works transform the above problem into an equivalent min-max optimization (Liu et al., 2019a), which is decomposable over individual examples:
[Equation]
where
[Equation]
Ω = R and p = n + /n .
From the above objective function φ , we can see that each example x i also has a class-level weight for their contributed loss, i.e., the data from the positive class is weighted by 1 − p and the data from the negative class is weighted by p .
It was recently shown that the AUC square loss is sensitive to noisy data and also has adverse effect when trained with easy data.
Hence, Yuan et al.
(2020) proposed the min-max AUC margin (AUCM) loss, whose optimization problem is (2) with Ω = { 0 ≤ θ ≤ u } for some u .
Let us deﬁne
L AUC ( w ) =
min a,b max θ ∈ Ω Φ ( w , a, b, θ ) as the AUC loss function.
3. C OMPOSITIONAL DAM: A C OMPOSITIONAL T RAINING METHOD FOR DAM
In this section, we present the proposed compositional training for DAM.
Our proposed objective for end-to-end deep learning is given by
[Equation]
where α is hyper-parameter.
Different from L AUC ( w ) , the above objective is a compositional function, where the inner component w − α ∇ L AVG ( w ) is another function of w .
We refer to the above objective as the compositional objective and a method for minimizing the above compositional objective as compositional training.
To understand the compositional objective (4), we take the second-order Taylor expansion of the compositional objective, which include three terms:
[Equation]
where C represents the Lipchitz continuity constant of ∇ L AUC ( · ) .
In order to understand how the three terms play their roles and evolve in the optimization process by our proposed algorithm presented in next subsection (Algorithm 1), we conduct some empirical studies on several benchmark datasets reported in Appendix A.6.
Here, we explain the result of the CatvsDog classiﬁcation shown in Figure 2.
Initially, the ﬁrst term L AUC ( w ) dominates the objective and the algorithm will focus on pushing this term to be smaller (1st column), once it reaches the same level of the third term the algorithm will shift its focus to push (cid:107)∇ L AVG ( w ) (cid:107) smaller (2nd column) while keeping ∇ L AUC ( w ) (cid:62) ∇ L AVG ( w ) to be positive (3rd column).
This process will continue by alternating between the efforts of pushing L AUC ( w ) smaller and of pushing (cid:107)∇ L AVG ( w ) (cid:107) 2 to be smaller while keeping ∇ L AUC ( w ) (cid:62) ∇ L AVG ( w ) to be non-negative.
We also notice that the ﬁnal AUC loss is close to zero.
The same phenomena are also observed on other datasets.
To further understand the compositional training intuitively, let us take a thought experiment by using a gradient descent method to optimize the compositional objective.
First, we evaluate the inner function by u = w − α ∇ L AVG ( w ) .
We can see that u is computed by a gradient descent step for minimizing the averaged loss L AVG ( w ) , which facilitates the learning of lower layers for feature extraction due to equal weights of all examples.
Then, we take a gradient descent step to update w for minimizing the outer function L AUC ( · ) by using the gradient ∇ L AUC ( u ) instead of ∇ L AUC ( w ) .
Because u is better than w in terms of feature extraction layers, taking a gradient descent step using ∇ L AUC ( u ) would be better than using ∇ L AUC ( w ) .
In addition, taking a gradient descent step for the outer function L AUC ( · ) will make the classiﬁer more robust to the minority class due to the higher weights of examples from the minority class.
Overall, we have two alternating conceptual steps , i.e., the inner gradient descent step u = w − α ∇ L AVG ( w ) acts as a feature puriﬁcation step , and the outer gradient descent step w − η
( I − α ∇ 2 L AVG ( w )) ∇ L AUC ( u ) acts as a classiﬁer robustiﬁcation step, where η is a step size.
VS.
linear combination approach.
It is notable that minimizing the compositional objective
L AUC ( w − α ∇ L AVG ( w )) is different from minimizing a linear combination of an AUC loss and the averaged loss, i.e., L AUC ( w )+ cL AVG ( w ) , where c > 0 is a combination weight.
First, minimizing the latter objective will push ∇ L AUC ( w ) + c ∇ L AVG ( w ) = 0 .
This makes ∇ L AUC ( w ) to have an opposite direction from ∇ L AVG ( w ) at the optimal solution, which is different from minimizing the compositional objective in light of the three terms in (5).
Second, if we take a gradient descent method for minimizing this objective, the update of w is given by w − η ( ∇ L AUC ( w )+ c ∇ L AVG ( w )) .
This update is fundamentally different from the two alternating steps of compositional training that use gradients of the AUC loss and the average loss at different points.
Third, minimizing the linear combination has a trade-off, meaning that the AUC score is not necessarily maximized due to the presence of the CE loss in the objective and the learned feature representations are degraded due to the presence of AUC loss.
More discussions.
Finally, we note that the inner gradient descent step w − α ∇ L AVG ( w ) is similar to the idea used in model-agnostic meta learning (MAML)
(Finn et al., 2017).
However, our compositional objective works fundamentally different from that for MAML.
In MAML, the outer loss function and the loss function for the inner gradient descent step is the same.
In contrast, the two loss functions in our objective are different.
But similar to MAML approaches, we can also run multiple gradient descent steps for the inner function, i.e, the inner function w − α ∇ L AVG ( w ) can be replaced by multiple gradient descent steps.
In our experiments, we also found this trick to be helpful for improving the performance.
3. S TOCHASTIC O PTIMIZATION A LGORITHMS
In this subsection, we develop efﬁcient stochastic optimization algorithms for optimizing the compositional objective (4) for DAM.
First, we argue the necessity for such development.
(i) the problem is a min-max form and the objective is a compositional function, which makes computing an unbiased stochastic gradient of the objective L AUC ( w −
α ∇ L AVG ( w )) impossible.
Existing algorithms of
1: Require Parameters: β 0 , β 1 , β 2 , α, G 0 , η 1 , η 2 , τ 2: Initialization: ¯ w 0 = ( w 0 ; a 0 ; b 0 ) ∈ R d
+2 , θ 0 , u 0 ∈ R d +2 , v 0 ∈ R 3: for t = 0 , 1 , ..., T do 4:
Sample two sets of examples denoted by S 1 , S5: u t +1 =
(1 − β 0 )
u t + β 0 h
( ¯ w t ; S 1 ) 6:
O t = ∇ ¯ w h ( ¯ w t ; S 1 ) (cid:62) [ ∇ u g 1 ( u t +1 ; S 2 )
+ θ t ∇ u g 2 ( u t +1 ; S 2 )] 7: z t +1 =
(1 − β 1 )
z t + β 1 O t 8: z 2 ,t +1 =
h t ( {O j , j = 0 , . . .
, t ) } (cid:5) h t can be implemented by that in Appendix B 9: ¯ w t +1 = ¯ w t − ηz t +1 √ z 2 ,t +1 + G(cid:5) with the simplest form h t =10: θ t +1 =
Π Ω [ θ t + η 2 ( g 2 ( u t +1 ; S 1 ∪ S 2 )
− ∇ g 3 ( θ t ))] 11: end for non-convex min-max optimization for DAM that focus on minimizing L AUC ( w ) (Liu et al., 2019a;
Yuan et al., 2021; Guo et al., 2020a;b) are not applicable due to the presence of inner function w − α ∇ L AVG ( w ) .
(ii)
Our objective is also different from that of MAML due to that L AUC ( · ) is a min-max form, which renders existing algorithms for MAML (Finn et al., 2017; Fallah et al., 2020) not applicable.
Hence, below we propose an efﬁcient stochastic algorithm for solving the compositional training for DAM whose objective is of the min-max compositional form, and establish its convergence rate similar to that of standard SGD for minimizing the standard averaged loss.
In particular, for the considered AUC loss, the compositional objective becomes:
[Equation]
We denote by a tuple ¯ w = ( w ; a ; b ) .
For simplicity of presentation, we write φ ( w , a, b, θ, x i , y i ) as
[Equation]
where {+g 1 ( ¯ w ; x i , y i ) =(1 − p ) ( f ( w ; x i ) − a ) 
 I [ y i =1] + p ( f [Equation]+}
w
[Equation]
x i
[Equation]
b
[Equation] 2 I [ y i = − 1] [Equation] ] , (7) and g 2 ( ¯ w ; x i , y i ) = 2 (cid:0) pf ( w ; x i ) I
[ y i = − 1] − (1 − p ) f
( w ; x i )
I
[ y i =1] (cid:1) and g 3 ( θ ) =
p (1 − p ) θ 2 .
Denote by g 1 ( ¯ w ; S ) =
[Equation]
Let h ( ¯ w ) =
( w − α ∇ L AVG ( w ); a ; b ) , ∇ ¯ w h ( ¯ w ) =
( I − α ∇ 2 w L AVG ( w ); 1; 1) , and h ( ¯ w ; S ) =
( w − α ∇ L AVG ( w ; S ); a ; b ) .
We propose a primal-dual stochastic algorithm shown in Algorithm 1, which is referred to as
PDSCA.
We provide some explanations of our algorithmic design.
First, the step 5 of updating u t +1 corresponds to feature puriﬁcation step.
We use a moving average technique to update u t + that takes all historical updates into account, which is inspired by existing stochastic algorithms for optimizing compositional functions (Wang et al., 2017).
This is important for us to prove the convergence rate of O (1 / √ T ) without using a large batch size at each iteration.
If we simply using u t +
= h ( ¯ w t ; S 1 ) (i.e., setting β 0 = 1 ) to estimate h ( ¯ w t ) , there will be a large error in estimating the gradient ∇ u g 1 ( u t +1 ; S 2 ) and ∇ u g 2 ( u t +1 ; S 2 ) in step 6.
Second, the step 6 is to estimate the gradient of the outer function.
We use two independent mini-batches S 1 , S 2 to ensure that O t is an unbiased estimator of ∇ h ( ¯ w t ) (cid:62) ∇ ¯ w φ ( u t +1 , θ ) .
Using two independent mini-batches is also helpful for improving generalization as demonstrated in experiments.
Third, the steps 7 - 9 are similar to the momentum and adaptive methods for updating the model parameter.
The step 8 is used for computing the adaptive step size 1 / √ z 2 ,t +1 + (cid:15) 0 , which is similar to adaptive methods used for deep learning, such as Adam, AMSGrad, AdaBound (Kingma & Ba, 2015; Reddi et al., 2018;
Luo et al., 2019).
We use a general function h t in the algorithm, which can be implemented by different methods corresponding to different adaptive step size choices.
We present different h t in the Appendix B.
The simplest one h t = 1 corresponds to that we do not use adaptive step size and only use the momentum update.
Fourth, the step 10 is for updating the dual variable θ using a stochastic gradient ascent method.
Finally, we point out that PDSCA is similar to some existing non-convex strongly-concave min-max optimization algorithms (Guo et al., 2021) but with additional care on the inner gradient descent step w − α ∇ L AVG ( w ) .
We present an informal convergence of PDSCA below.
Theorem 1.
(Informal) Under appropriate conditions on the functions L AVG , g 1 , g 2 and a boundness condition on θ t , w t , ∇ (cid:96) ( w t ; x , y ) , ∇ 2 (cid:96) ( w t ; x , y ) , with β 0 , β 1 , β 2 =
O (1 / √ T ) , η 1 , η 2 =
O (1 / √ T ) and a small constant τ , Algorithm 1 ensures that E (cid:104) 1 T +1 (cid:80) Tt =
(cid:107)∇ F ( ¯ w t ) (cid:107) 2 (cid:105) ≤
O ( 1 √ T ) .
where
F ( ¯ w ) =
max θ ∈ Ω Φ ( w − α ∇ L AVG ( w ) , a, b, θ ) .
Remark: We will present the detailed conditions in the supplement when proving the above theorem due to limit of space.
The above theorem indicates that we can optimize the compositional objective (6) with the same convergence rate as optimizing the averaged loss (1) for deep learning.
Practical Implementations.
It is notable that ∇ h ( ¯ w , S 1 )
=
( I − α ∇ 2 L ( w ; S 1 ); 1 , 1) (step 6) involves the Hessian matrix ∇ 2 L ( w ; S 1 ) .
Indeed, we only need to compute the Hessian vector product involving in step 6.
Similar computation occurs in the meta learning algorithms (Finn et al., 2017; Fallah et al., 2020).
Inspired by practical implementations of MAML (Finn et al., 2017) that simply ignore the second-order term, we use the same trick in our experiments.
An additional useful trick inspired by MAML is that we can take k ≥ 1 gradient descent steps for the inner function, correspondingly we maintain and update several u variables similar to step 5, i.e., using h ( ¯ w t , S 1 ) for updating the ﬁrst u (1) t +1 , and using h ( u (1) t +1 ; S 1 ) for updating second u (2) t +1 , and so on so forth.
In our experiments, we found that tuning k ∈ { 1 , 2 , 3 } is useful.
Finally, it is notable that although we focus on optimizing AUC loss for binary classiﬁcation in this work, our compositional training method can be also extended to optimize other weighted losses in an end-to-end fashion, and we include some discussion and results in the Appendix D.
E XPERIMENTS
In this section, we present some experimental results.
We choose ﬁve baselines: optimizing the
AUC loss from scratch (AUC sc ), optimizing the CE loss (CE), optimizing a linear combination of the AUC loss and the CE loss with a tuned weight (AUC-CE), the two-stage method with deferred re-weighting trick (Cao et al., 2019) (TS-DRW), the two-stage method by decoupling the learning of feature network by minimizing CE loss and the learning of a classiﬁer by minimizing the AUC loss (TS-DEC) (Kang et al., 2019).
We denote our method by CT (AUC).
For AUC loss, we use AUCM loss with the margin parameter ﬁxed to be 1 (Yuan et al., 2020).
We conduct experiments on four benchmark datasetes and four medical image datasets.
The statistics of these datasets are included in the Appendix A.1.
More training conﬁgurations can be found in Appendix A.2.
Benchmark datasets.
We choose four benchmark image classiﬁcation datasets, namely CatvsDog,
CIFAR10 (C10), CIFAR100 (C100), and STL10 (S10).
For AUC maximization, we construct imbalanced binary versions of these datasets by varying the imbalanced ratios (the ratio of positive examples to the total number of training examples) similar to (Yuan et al., 2020).
We use ResNet as the prediction network.
The weight decay is set to 1e-4 for all experiments.
For algorithms to maximize AUC, we use a batch size = 128 and train a total of 100 epochs, and we use step size 0.
and decrease it by 10 times at 50% and 75% of total training time.
We tune the beta parameters of our method in a range [0 . 1 , 0 . 99] with a grid search and ﬁnd that good values are around 0.9.
For linear combination methods, we tune the weight c of two losses in { 0 . 25 , 0 . 5 , 0 . 75 } .
We tune the number of inner gradient steps for CT in k ∈ { 1 , 2 , 3 } with α = 0 .
For all benchmark data, we run three times for different random seeds and compute the mean and standard deviations.
Medical image datasets.
We also conduct experiments on naturally imbalanced medical datasets.
We choose four medical image datasets, namely Melanoma data, CheXpert, DDSM+, and PatchCam data.
The Melanoma dataset is from the Kaggle 2020 competition (Rotemberg et al., 2021), which contains 33,126 labeled images in training set, including 584 positive samples and 32,542 negative samples.
We manually construct training, validation and testing datasets following 70/10/20 split.
For this dataset, we use the images with 256x256 resolution in the experiments.
CheXpert is a largescale chest X-ray dataset (Irvin et al., 2019), which has 224,316 images with 224 x 224 resolution.
The dataset contains 5 binary classiﬁcation tasks corresponding to 5 diseases, i.e., Cardiomegaly (C0), Edema , Consolidation, Atelectasis, Pleural Effusion.
We evaluate the performance on the ofﬁcial validation set consisting of 200 patient studies and report the averaged AUC scores of all 5 diseases.
The DDSM+ data is a combination of two datasets namely DDSM and CBIS-DDSM (Lee et al., 2017;
Bowyer et al., 1996; Heath et al., 1998), which consists of 55,890 mammographic training images (224×224) with an imratio of 13% and 15,364 images for testing with an imratio of 13%.
The PatchCamelyon dataset consists of 294,912 color images (96×96) extracted from
[Table]
histopathologic scans of lymph node section for training and 32,768 images for testing with balanced class ratio (Veeling et al., 2018; Bejnordi et al., 2017).
For PathCamelyon, we manually construct an imbalanced training dataset with an imratio of 1% and keep the testing set balanced.
For Melanoma data, we adopt a EfﬁcientNetV2-S (Tan & Le, 2021) as the network structure, and for
CheXpert, DDSM+, PatchCam, we use DenseNet121 (Huang et al., 2017).
We tune the number of inner gradient steps for CT in k ∈ { 1 , 2 , 3 } and also tune α in { 0 . 1 , 0 . 05 , 0 . 01 } for the inner steps.
Results.
The testing AUC results are reported in Table 1.
We can see that the proposed composition training method outperforms all baselines on all datasets for maximizing AUC.
In addition, we have the following observations: (i) optimizing an AUC loss from scratch does not necessarily yield a better performance than minimizing the standard CE loss; (ii) the CT (AUC) method dramatically improves the performance of optimizing the AUC loss from scratch, with about 2% ∼ 5% improvement on difﬁcult medical classiﬁcation tasks; (iii) the CT (AUC) method is generally better than the linear combination approach (AUC-CE), especially on the more difﬁcult medical image datasets.
It is notable that the reported results on some medical datasets are not comparable with that in (Yuan et al., 2020) because (i) we report the performance on ofﬁcial CheXpert validation data instead of the ofﬁcial testing data; (ii) we use a smaller resolution on Melanoma data; (ii) we do not tune the margin parameter in the AUCM loss.
We also plot the learned feature representations of training data of different datasets visualized by t-SNE in Figure 3.
We can see that the proposed CT (AUC) method obtains better feature representations than the baseline approaches of optimizing CE or AUC alone and the naive linear combination approach.
4. A BLATION S TUDY
We conduct some ablation study including (i) the comparison of convergence curves and the running time analysis of different methods; (ii) the veriﬁcation of our algorithmic design.
Convergence Curve.
Below, we compare the convergence speed of our CT (AUC) approach with other end-to-end learning baselines, i.e., CE, AUC, AUC-CE, with results on four benchmark datasets plotted in Figure 4.
The results indicate that our algorithm enjoys even faster convergence in terms of number of epochs.
The convergence curves of testing AUC are included in Appendix A.4.
Runtime analysis.
We notice our method has larger running time per-iteration than minimizing the
CE and the AUC loss alone because of several backpropagations, but with a reward of faster convergence in epochs and better testing performance.
For fair comparison, we have run the baselines with the same amount of time as our method and observed they are still worse than our method (cf
Appendix A.5).
For example, on CIFAR10 (10%), CT (AUC) can achieve an AUC of 0.944 with a running time of 1000s, in contrast, the baselines CE, AUC, AUC-CE, TS-DRW, and TS-DEC use same running time and achieve AUC scores of 0.925, 0.915, 0.943, 0.917, 0.917, respectively.
Veriﬁcation of Algorithmic Design.
We validate three algorithmic choices.
(i) Using two independent mini-batches S 1 (cid:54) =
S 2 is generally better than using the same mini-batch.
(ii) Using the momentum update for u t +1 (i.e., β 0 < 1 ) is better than without using momentum update ( β 0 = 1 ).
(iii) tuning the number of inner gradient steps k ∈ { 1 , 2 , 3 } is helpful for improving the performance.
The results are demonstrated in Table 2, where all results are averaged over three trials.
[Table]
C ONCLUSIONS
In this paper
[Table]
we have proposed a novel end-to-end compositional training framework for deep AUC maximization by optimizing a compositional objective.
We also proposed an efﬁcient stochastic optimization method for compositional training of deep AUC maximization.
We demonstrated the effectiveness of compositional training on multiple benchmark datasets and medical datasets for maximizing AUC.
In future work, we will investigate compositional training for other imbalanced loss functions more extensively.
A CKNOWLEDGEMENTS
We thank anonymous reviewers for their valuable comments.
This work was partially supported by
NSF Career Award # 1844403, NSF Award # 2110545, NSF Award # 1933212.
A S UPPLEMENT OF E XPERIMENTAL S ECTION A. D ATASET D ESCRIPTIONS
The detailed statistics of different datasetse are reported in Table 3.
Note that ”# of images” refers to the number of samples for the original training set.
"LT" denote long-tailed version of the datasets for multi-class tasks.
Imbalance ratio (imratio) of XXX (Binary) data means the ratio of number of positive examples to number of all examples.
Imbalance ratio (imratio) of XXX (LT) data means ration of the size of smallest class to the size of largest class.
The DDSM+ is slightly different from the standard version of CBIS-DDSM or DDSM (Lee et al.
; Bowyer et al.
; Heath et al.
We actually use the dataset from (Scuccimarra (2021)) constructed by Eric A. Scuccimarra, which consists of 55k images for training and 15k images for testing.
The details about the dataset construction can be found here (Scuccimarra (2021); Zheng et al.
; Fulton et al.
For constructing DDSM+, the positive samples (cancer cases) are from CBIS-DDSM and negative samples (normal cases) are from DDSM.
To increase the size of the training data, the author applies ofﬂine data augmentation and adds multiple augmented copies to the dataset.
In particular, each image (ROI) is randomly cropped three times into 598x598 images, with random ﬂips and rotations, and then the images are resized down to 299x299.
For the testing set, the same augmentation is also applied and thus the imbalance ratios remain the same as the training set.
The imbalance ratio is about 13% in both training and testing sets.
The train/test split in terms of patient ID follows the
CBIS-DDSM split, which do not have any overlap.
We will add these references in the revision and also correct the citation for CBIS-DDSM.
Please also note that this dataset (DDSM+) has also been used for some recent works (Zheng et al.
; Fulton et al.
[Table]
A. T RAINING C ONFIGURATIONS
All benchmark datasets are experimented by NVIDIA GTX-2080Ti and four medical datasets,
i
[Table]
e., CheXpert, Melanoma, DDSM+ and PatchCam, are experimented by NVIDIA V100.
For the datasets of binary classiﬁcation in Table 1, we use the dataloaders from (Yuan et al., 2020).
For the long-tailed datasets of multi-class classiﬁcation as in Section D, we uses the dataloaders from (Cui et al., 2019).
For TS-DRW, we train the models using cross-entropy loss at the ﬁrst stage and then switch to imbalanced losses at the later stages.
For TS-DEC, we ﬁrst conduct the regular training using cross-entropy loss and same settings and then we discard the trained classiﬁers and ﬁnetune the new classiﬁer for additional 10 epochs using imbalanced loss with the learning rate of 0.01.
For the proposed compositional training methods, we consider the non-adaptive version with h t ( · ) =
in all experiments.
For all datasets, we use the train/val split to do cross-validation for parameter tuning, except CheXpert as explained below.
For the benchmark datasets, we use 19k/1k, 45k/5k, 45k/5k.
4k/1k training/validation split on CatvsDog, CIFAR10, CIFAR100, STL10, respectively.
For melanoma dataset, we use 70/10/20 split for train/val/test.
For PatchCam, we use their ofﬁcial validation set for tuning parameters, which includes about 37k images with balanced positive and negative samples.
For DDSM+, we tune the parameters on 10% data sampled from the training set.
For CheXpert, since the ofﬁcial testing set is not released and it will take a long time to evaluate all methods on the ofﬁcial testing data, hence, we evaluate different methods only based on the ofﬁcial validation set with parameters tuned according to this set.
To make the experiment on Chexpert con- sistent with other datasets, we run experiments for all methods on CheXpert by following the same cross-validation procedure, i.e., by sampling 10% training data based on patient ID as the validation set to tune parameters and then we report the average scores of ﬁve diseases on the testing set (i.e., the ofﬁcial validation set as a testing set).
A. E VALUATIONS ON M EDICAL D ATASET WITH MULTIPLE RUNS
For medical datasets, we run all experiments and report the average performance over three runs.
We use batch size of 32 except for PatchCam that is 64, initial learning rate of 0.1 and weight decay of 1e-5.
We train Melanoma for 12 epochs, CheXpert for 2 epochs, DDSM+ for 5 epochs and
PatchCam for 5 epochs.
The learning rate is decayed at 50%, 75% of total training iterations by 10 times.
For compositional training, we tune the number of inner gradient steps in k ∈ { 1 , 2 , 3 } and also tune α ∈ { 0 . 1 , 0 . 05 , 0 .
01 } for the inner steps.
The results are summarized in the following table.
[Table]
A
T ESTING C ONVERGENCE C URVES .
The convergence curves of testing AUC on the benchmark datasets of different methods are plotted in Figure 5.
A. C OMPARISON WITH THE SAME R UNTIME
To compare the performance with the same running time
[Table]
we train ResNet20 on four benchmark datasets with an imbalance ratio of 10% on a GTX-2080Ti.
For each method, we train 1000 seconds and report the best achieved testing AUC.
The results are summarized in Table 5.
[Table]
A. E VOLUTION OF DIFFERENT TERMS OF THE COMPOSITIONAL OBJECTIVE .
To better understand the proposed compositional objective
[Table]
we plot the evolution curves of each term in the decomposition equation (5) and compare them with that of naive linear combination approach
(Linear Comb.).
We conduct experiments on CATvsDOG, CIFAR10, CIFAR100 and STL10 with imbalance ratio of 10% using ResNet20.
We start with initial learning rate of 0.1 and decay it at 50th, 75th epoch by 0.1.
In Figure 6, we plot the values of L AUC ( w ) , L AVG ( w ) , ∇ L AUC ( w ) (cid:62) ∇ L AVG ( w ) , (cid:107)∇ L AVG ( w ) (cid:107) 2 v.s. the number of epochs on training set.
For the calculations of L AUC ( w ) , we compute its values based on the optimal values of a, b, α according to (Yuan et al., 2020) for each epoch.
Regarding the calculations of w , we compute the mean values of all layers of models.
We defer results on other datasets to appendix.
From the results, we observe that initially L AUC ( w ) dominate the objective and keep decreasing at earlier iterations.
When it reaches similar level of the third term (cid:107)∇ L AVG ( w ) (cid:107) , the objective shifts its focus on third term and pushes it to smaller while maintaining second term positive.
Eventually, we can see that both AUC and CE losses of
CT method reach to a level close to zero.
In addition, comparing CT with the linear combination method, it is amazing to see that CT drives both the CE loss and the AUC loss decrease faster and to a smaller level than the linear combination method.
B I MPLEMENTATION OF z 2 ,t +1 = h t ( O 0 , O 1 , . . . , O t )
The h t is usually implemented by a recursion, where examples are given in Table 6.
Similar to (Guo et al., 2021), we make the following assumption for our analysis.
Assumption 1.
For the Adam-style algorithms in Table 6, we assume that s t
= 1 / ( √ z 2 ,t +1 + G 0 ) is upper bounded and lower bounded, i.e., there exists 0 <
c l < c u such that ∀ i, c l ≤ (cid:107) s t,i (cid:107) ≤ c u , where s t,i denotes the i -th element of s t .
Remark: Different implementations of h t that satisfy this assumption is presented in Table 6.
We need the following lemma to tackle the variance recursion.
Lemma 1 (Lemma 2, Ghadimi & Wang (2018)) .
Consider a moving average sequence z t +1 =
(1 − β )
z t + β t O h
( x t ) for tracking h ( x t ) ,
where E [ O h ( x t )]
= h ( x t ) and h is a L -Lipschitz continuous
[Table] [Table]
mapping.
Then we have
[Table]
− x t − 1 (cid:107) 2 β t , (8) where E t denotes the expectation conditioned on all randomness before O h ( x t ) .
C P ROOF OF T HEOREM
Denote η t =
η 1 s t , where s t = 1 / ( √ z 2 ,t +1 + G 0 ) .
We make the following assumptions regarding problem 6.
Assumption 2.
• ∇ L AVG ( w ) is C L AVG -Lipschitz continuous, g 1 ( ¯ w ) is C g 1 -Lipschitz continuous and g 2 ( ¯ w ) is C g 2Lipschitz continuous.
• ∇ 2 L AVG ( w ) is L L AVG -Lipschitz continuous, ∇ g 1 ( ¯ w ) is L g 1 -Lipschitz continuous, and ∇ g 2 ( ¯ w ) is L g 2 -Lipschitz continuous.
• E (cid:107) α ∇ L AVG ( w ) − α ∇ L AVG ( w ; S ) (cid:107) 2 ≤ σ 2 , E (cid:107) α ∇ 2 L AVG ( w ) −
α ∇ 2 L AVG ( w ; S ) (cid:107) 2 ≤ σ 2 , E (cid:107)∇ g 1 ( w ) − ∇ g 1 ( w ; S ) (cid:107) 2 ≤ σ 2 , E (cid:107) g 1 ( w ) − g 1 ( w ; S ) (cid:107) 2 ≤ σ 2 , E (cid:107)∇ g 2 ( w ) − ∇ g 2 ( w ; S ) (cid:107) 2 ≤ σ 2 , E (cid:107) g 2 ( w ) − g 2 ( w ; S ) (cid:107) 2 ≤ σ 2 .
• Ω is a bounded convex set with radius D .
It is notable that the last assumption can be replaced by a condition that the dual variables θ t are bounded.
We also know that g 3 ( θ ) is λ := 2 p (1 − p ) -strongly convex and also L g 3 = λ -smooth.
Denote θ ∗ ( ¯ w ) =
arg max θ ∈ Ω Φ( ¯ w , θ ) .
Based on Assumption 2, we have that h ( ¯ w ) is ( C h :
= 1 + αC L AVG ) -Lipschitz continuous, ∇ h ( ¯ w ) is ( L h :
= 1 + αL L AVG )
-Lipschitz continuous, E (cid:107)∇ h
( ¯ w ) − ∇ h ( ¯ w ; S ) (cid:107) 2 ≤ (1 + D 2 ) σ 2 , and E (cid:107) h
( ¯ w ) − h ( ¯ w ; S ) (cid:107) 2 ≤ (1 + D 2 ) σ 2 .
We also know that F ( ¯ w ) is
L F -smooth, where L F := ( L g 1 + DL g 2 + λ ) + ( L g 1 + DL g 2 + λ ) 2 λ
(Lemma 4.3 of (Lin et al., 2019)).
We present Theorem 1 formally in the following Theorem.
Theorem 2.
Assume F ( ¯ w 0 ) − F ∗ ≤ ∆ F where F ∗ =
min ¯ w F ( ¯ w ) .
Suppose Assumptions 1 andhold.
With {+β 
 = O (1 / √ T ) , β+} 1 = O (1 / √ T ) , β 2 =
[-β 
 =-] O (1 / √ T ) , β {+
+} = O (1 / √ T ) , β 1 =
O (1 / √ T ) , η 1 ≤ min { (cid:113) c l c 3 u C 3 β 1 2 , [Equation]
2 {+u L F } , and η 
+}
=
O (1 / √ T ) where C 1 and C 3 are proper constants speciﬁed in the proof, Algorithm 1 can ensure that
[Equation]
To prove this theorem, we ﬁrst need a couple of lemmas.
Lemma 2.
Suppose Assumption 1 and Assumption 2 hold.
Considering the update in Algorithm 1, we have
[Equation]
Proof of Lemma 2.
Due to the smoothness of F
[Equation]
we can prove that under η 1 L F ≤ c l / (2 c 2 u ) ,
[Equation]
Lemma 3.
Let θ t {++
+}
= Π Ω [ θ t + η 2 ( g 2 ( u t +1 ; S 1 ∪ S 2 )
− ∇ g 3 ( θ t ))] , we have
[Equation]
where L θ := C g 2 λ is the Lipschitz continuous constant of θ ∗ ( · ) .
Proof of Lemma 3.
Since θ ∗ ( h ( ¯ w t ))
=
Π Ω [ θ ∗ ( h ( ¯ w t ))
+ η 2 ( g 2 ( h ( ¯ w t ))
− ∇ g 3 ( θ ∗ ( h ( ¯ w t ))))] , we have E
[Equation] =
[Equation]
) + η 2 ( g 2 ( h ( ¯ w t ))
− ∇ g 3 ( θ ∗ ( h ( ¯ w t ))))] (cid:107)≤ E (cid:107) [ θ t + η 2 ( g 2 ( u t +1 ; S 1 ∪ S 2 ) − ∇ g 3 ( θ t )) ] − [ θ ∗ ( h ( ¯ w t ))
+ η 2 ( g 2 ( h ( ¯ w t ))
− ∇ g 3 ( θ ∗ ( h ( ¯ w t ))))] (cid:107)= E (cid:107) [ θ t + η 2 ( g 2 ( u t +1 ; S 1 ∪ S 2 ) − ∇ g 3 ( θ t )) − η 2 g 2 ( u t +1 ) + η 2 g 2 ( u t +1 )] − [ θ ∗ ( h ( ¯ w t ))
+ η 2 ( g 2 ( h ( ¯ w t ))
− ∇ g 3 ( θ ∗ ( h ( ¯ w t ))))] (cid:107) ≤ E (cid:107) [ θ t + η 2 ( g 2 ( u t +1 )
− ∇ g 3 ( θ t ))] − [ θ ∗ ( h ( ¯ w t )) + η 2 ( g 2 ( h ( ¯ w t ))
− ∇ g 3 ( θ ∗ ( h ( ¯ w t ))))]
(cid:107)+ η 22 E (cid:107) g 2 ( u t +1 ; S 1 ∪ S 2 ) − g 2 ( u t +1 ) (cid:107)≤ E (cid:107) [ θ t + η 2 ( g 2 ( u t +1 ) − ∇ g 3 ( θ t ))]
− [ θ ∗ ( h ( ¯ w t ))
+ η 2 ( g 2 ( h ( ¯ w t ))
− ∇ g 3 ( θ ∗ ( h ( ¯ w t ))))]
(cid:107) 2 + η 22 σ 2 , (11) where E (cid:107) [ θ t + η 2 ( g 2 ( u t +1 ) − ∇ g 3 ( θ t ))] − [ θ ∗ ( h ( ¯ w t )) + η 2 ( g 2 ( h ( ¯ w
[Equation]
where the ﬁrst inequality uses strong monotone inequality as g 3 ( · ) is λ -strongly convex and the second inequality uses η 2 ≤ min { λ 8 L 2 g 3 , 2 λ } .
Then,
E [Equation]
h
[Equation]
¯ w [Equation] + [Equation]
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation] [Table] [Equation] [Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation]
w
[Equation]
+1 )
[Equation]
2 , where the third inequality is because that θ ∗ ( · ) is L θ =
C g 2 λ
-Lipschitz Lin et al.
Proof of Theorem 2.
Denote by g 1 ( h ( ¯ w t ))
=
E x i ,y i [ g 1 ( h ( ¯ w t ); x i , y i )] and g 2 ( h ( ¯ w t )) =
E x i ,y i [ g 2 ( h ( ¯ w t ); x i , y i )] .
Note that ∇ h Φ( h ( ¯ w t ) , θ t ) = ∇ h g 1 ( h ( ¯ w t ))
+ θ t ∇ h g 2 ( h ( ¯ w t )) .
Denote by ∆ u,t = (cid:107) u t +1 − h
( ¯ w t ) (cid:107) 2 , ∆ z,t = (cid:107) z t +1 − ∇ ¯ w h
( ¯ w t ) (cid:62) ∇ h Φ( h ( ¯ w t ) , θ t ) (cid:107) 2 = (cid:107) z t +1 − ∇ h ( ¯ w t ) (cid:62) [ ∇ h g 1 ( h ( ¯ w t ))
+ θ t ∇ h g 2 ( h ( ¯ w t ))]
(cid:107) 2 , and δ t = (cid:107) θ t − θ ∗ ( h ( ¯ w t ))
(cid:107) 2 .
Applying Lemma 1 to u t , we have
[Equation]
Hence we have
E
[Equation]
C
[Equation] [Equation]
14) and
E
[Equation]
[Equation]
Deﬁne e [Equation] We have (cid:107) [Equation]
and E (cid:107) z [Equation] t
[Equation]
S 1 )
[Equation]
u
[Equation]
u t
[Equation]
[Equation]
¯ w
[Equation] ] + [Equation] [Equation]
≤
[Equation]
(cid:107)
[Equation]
[Equation]
)] +
[Equation]
[Equation]
+ β 21 E
(cid:107)∇
[Equation]
h (
[Equation]
[Equation]
∇
[Equation]
∇
[Equation]
∇
[Equation]
[Equation]
[Equation]
β
[Equation]
[Equation]
β 1 )
[Equation]
(cid:107)
[Equation]
∇
[Equation]
h (
[Equation]
∇
[Equation]
[Equation]
( ¯ w
[Equation]
[Equation]
[Equation]
( h ( ¯ w
[Equation]
))) (cid:107)+ (
[Equation]
2 β 1 )2 β 21 E (cid:107)∇ ¯ w h ( ¯ w t ) (cid:62) ∇ h
[Equation]
( u
[Equation]
, θ
[Equation]
) − ∇ ¯ w h ( ¯ w t )
[Equation]
∇ h Φ( h ( ¯ w t ) , θ t ) (cid:107)+ (1 + 2 β 1 )2 β 21 E (cid:107)∇ ¯ w h ( ¯ w t ) (cid:62) [Equation] where the last three terms can be bounded as below. First,
[Equation]
Second,
[Equation] Third, [Equation] It follows that
[Equation]
where C : = 32 C 2 h ( L 2 g 1 + L 2 g 2 ) , C 2 := 8 σ 2 (3 C 2 h +3 D 2 C 2 h +2 σ 2 ) and C 3 := 8[( C 2 g 1 + D 2 C 2 g 2 ) C 2 h
+ C 2 h ( L 2 g 1 + D 2 L 2 g 2 )] .
Thus,
[Equation]
Using Lemma 3, we have
[Equation]
and T +1 (cid:88) t =
[Equation]
Combining the upper bound of (cid:80) t ∆ z,t , (cid:80) t ∆ u,t , (cid:80) t δ t and Lemma 2, we have
[Equation]
≤
[Equation]
F
[Equation]
¯
[Equation]
Due to the setting
[Equation]
we have
[Equation]
Hence, we have
[Equation]
With β
= O (1 / √ T ) , β 1 =
O (1 / √ T ) , and η 2 =
O (1 / √ T ) , we have
[Equation]
[Table]
D C OMPOSITIONAL T RAINING WITH C LASS W EIGHTED LOSS
In this section
[Table]
we extend the compositional training method to deep learning with class weighted loss.
Let L CW ( w ) denote a class weighted loss written as:
[Equation]
where p y i ∈ (0 , 1) denotes a weight assigned to the i -th example that depends on the class the data belongs to.
There are different methods for determining the class-level weight.
A simple method is to set p i according to the reciprocal of its corresponding class size, i.e., p y i
=
1 /n y i .
Recently,
Cui et al.
(2019) proposed an improved variant of class-weighted loss by using an effective number of samples per-class instead of the class size to compute the individual weight, i.e., p y i = [Equation] − [Equation]
−
[Equation]
nyi , where γ ∈ (0 , 1) is a hyper-parameter.
We refer to the class weighted loss using these individual weights as L CB ( [Equation]
=
[Equation] [Equation]
With Class-Weighted Losses.
The corresponding compositional objective is a standard two-level compositional function, i.e.,
[Equation]
Assuming that the stochastic gradient of L CW can be easily computed, the optimization of the above problem is easier than that for AUC loss.
We present a simpliﬁed stochastic adaptive algorithm with an Adam-style adaptive step size in Algorithm 2 referred to as SCA, where h t ( O 0 , . . .
, O t ) denotes an appropriate mapping function, which can be implemented by using different methods, including Adam, AMSGrad, Adabound, etc.
Notice that when h t ( · ) = 1 Algorithm 2 becomes the NASA algorithm (Ghadimi et al., 2020) to stochastic compositional optimization.
We present an informal convergence of Algorithm 2 below.
Theorem 3.
(Informal) Under appropriate conditions on the loss functions L AVG , L CW and (cid:96) ( w ; x , y ) , with β 0 , β 1 , η
= O (1 / √ T )
, Algorithm 2 ensures that E (cid:104) 1 T +1 (cid:80) Tt =
(cid:107)∇ F ( w t ) (cid:107) 2 (cid:105) ≤
O ( 1 √ T ) .
Remark: The appropriate conditions include the Lipschitz continuous conditions on L AVG and
L CW and bounded variance conditions of ∇ (cid:96) ( w ; x i , y i ) and ∇ 2 (cid:96) ( w ; x i , y i ) .
We will present the detailed conditions below when proving the above theorem.
The above theorem indicates that we can optimize the compositional objective (24) with the same convergence rate as optimizing the averaged loss (1) for deep learning.
D. E XPERIMENTS WITH M ULTI - CLASS D ATASETS
We conduct experiments on three benchmark multi-class image classiﬁcation datasets, namely CIFAR10, CIFAR100, and STL10.
We construct imbalanced versions of these datasets by keep their classes but making the class sizes follow a long-tailed (LT) distribution with two imbalanced ratios (the ratio of the size of minority class to the size of majority class) similar to (Cui et al., 2019).
We use ResNet32 as the network stucture.
The weight decay is set to 2e-4 for all experiments.
For algorithms, we train a total of 200 epochs with a batch size
and we use step size 0.1 and decrease it by 10 times at at 80% and 90% of total training time.
We tune the beta parameters of our methods in a range [0 . 1 , 0 . 99] with a grid search and ﬁnd that good values are around 0.9.
For the class-weighted loss, we choose the class-weighted version of the LDAM loss (Cao et al., 2019).
For baselines, we compare with optimizing the CE loss (CE), optimizing the LDAM loss (LDAM), the two-stage method with the deferred re-weighting that optimizes the class balanced LDAM loss in the second stage (TS-DRW) (Cao et al., 2019), the two-stage method with the decoupling trick that optimizes the class balanced LDAM loss in the second stage (TS-DEC) (Kang et al., 2019).
The results are shown in Table 7.
We can see that the proposed CT method achieves the best accuracy on all datasets.
In addition, we conduct a large-scale experiment by following Jamal et al.
(2020) on
ImageNet-LT with ResNet32.
We use class-balanced loss (Cui et al., 2019) as outer loss function of our compositional objective.
We use an initial learning rate of 0.1 and run a total of 90 epochs decaying learning rate every 35 epochs by a factor of 10.
Eventually, we achieve the top1 accuracy of 26.61%, which is better than two baselines, i.e., CE(25.26%) and CBCE(26.59%).
D. A NALYSIS OF T HEOREM
In the section we analyze Algorithm 2.
An algorithm utilizing the adaptive step size and moving average for nonconvex optimization has been studied in (Guo et al., 2021).
But here we have to tailor the algorithm and analysis to the considered formulation with composition.
Denote s t
= 1 / ( z 2 ,t +1 + G 0 ) , η t = η s t .
We make the following assumptions regarding the problem 24.
Assumption 3.
• L CW ( u ) is C L CW -Lipschitz continuous, ∇ L AVG ( w ) is C L AVG -Lipschitz continuous.
• ∇ L CW ( u ) is L L CW -Lipschitz continuous, ∇ 2 L AVG ( w ) is L L AVG -Lipschitz continuous.
• The stochastic oracle satisﬁes E (cid:107) α ∇ L AVG ( w ) − α ∇ L AVG ( w ; S ) (cid:107) 2 ≤ σ 2 , E (cid:107) α ∇ 2 L AVG ( w ) − α ∇ 2 L AVG ( w ; S ) (cid:107) 2 ≤ σ 2 , E (cid:107)∇ L CW ( u ) − ∇ L CW ( u ; S ) (cid:107) 2 ≤ σ 2 . We formally present Theorem 3 as follows 1: Require Parameters: β 0 , β 1 , α, G 0 , η 2: Initialization: w 0 ∈ R d , z 0 , u3: for t = 0 , 1 , ..., T do 4:
Sample three sets of examples denoted by S 1 , S5:
[Equation]
[Equation] [Equation]
[Equation] (cid:5) h t can be implemented by that in Appendix B, 9: [Equation] (cid:5) with the simplest form h t =10: end for
Theorem 4.
Assume F ( x 0 ) − F ∗ ≤ ∆ F where F ∗ = min x F ( x ) .
Suppose Assumptions 1 andhold.
With η ≤ (cid:26) √ c l β 1 4 L F √ c 3 u , [Equation] β
[Equation]
(cid:27) ,
[Equation] [Equation] [Equation] ≤ O ( 1 √ T ) , and constants
L
[Equation]
=
[Equation]
[Equation]
[Equation]
[Equation]
L
[Equation]
) , Algorithm 2 can ensure that E
[Equation]
(cid:107)∇ F ( w
[Equation]
(cid:35)
[Equation]
Proof of Theorem 4.
By Assumption [Equation] we know that F ( w ) is smooth with coefﬁcient L F :=
[Equation]
[-1 + αC L AVG ) 
-]
We can prove that under ηL F ≤ c l / (2 c 2 u ) ,
[Equation]
[Equation]
Denote ∆ z,t = (cid:107) z t +1 − ∇ F ( w t ) (cid:107) =
(cid:107) z t +1 − ( I − α ∇
[Equation]
) (cid:107)and ∆ u,t = (cid:107) u t +1 − ( w t − α ∇ L AVG ( w t ))
(cid:107) 2 .
Applying Lemma 1 to u t , we have
[Equation]
Hence we have
[Equation]
[Equation]
Deﬁning
[Equation]
[Equation] 28) we get [Equation] [Equation]
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
where C {+4 := 4((
+}
+
[Equation]
2 + σ 2 + C 2 L CW ) , C 5 := 4(1 + αL L AVG ) 2 L 2 L CW .
It follows that
[Equation]
Thus
[Equation]
E
[Equation]
Combining this with (25), we obtain
[Equation]
Due to the setting η ≤
[Equation]
we have
[Equation]
Thus,
E
[Equation]
With {+β 0 = O ( 1 √ T ) , β 
 = O+}
[Equation]
1 √ T ) and
η = O
[Equation]
1 √ T ) ,
E
[Equation]