Learning to Re-weight Examples with Optimal Transport for Imbalanced Classification
Anonymous Author(s)
AffiliationAddress email
Abstract
Imbalanced data pose challenges for deep learning based classification models.
Oneof the most widely-used approaches for tackling imbalanced data is re-weighting,where training samples are associated with different weights in the loss function.
Most of existing re-weighting approaches treat the example weights as the learnableparameter and optimize the weights on the meta set, entailing expensive bileveloptimization.
In this paper, we propose a novel re-weighting method based onoptimal transport (OT) from a distributional point of view.
Specifically, we viewthe training set as an imbalanced distribution over its samples, which is transportedby OT to a balanced distribution obtained from the meta set.
The weights ofthe training samples are the probability mass of the imbalanced distribution andlearned by minimizing the OT distance between the two distributions.
Compared with existing methods, our proposed one disengages the dependence of the weightlearning on the concerned classifier at each iteration.
Experiments on image,text and point cloud datasets demonstrate that our proposed re-weighting methodhas excellent performance, achieving state-of-the-art results in many cases andproviding a promising tool for addressing the imbalanced classification issue.
Introduction
Deep neural networks (DNNs) have achieved remarkable success in various applications, which isundoubtedly inseparable from the high-quality large-scale datasets.
Usually, the number of samplesfor each class in these datasets are manually selected resulting in balanced datasets.
However, mostreal-world datasets are imbalanced, such as a few classes (a.k.a. head or majority class) occupy mostof the data while most classes (a.k.a.
tail or minority class) have a few samples.
A model trainedon the imbalanced training set but without considering such class imbalance would be significantlydominated by those majority classes and thus underperform on a balanced test dataset.
This can alsobe known as the long-tailed problem and exists in many domains, such as text classification [1, 2],object detection [3] and image classification [4–6].
There are rich research lines to solve the imbalance problem, including re-sampling [7–10], class-levelor instance-level re-weighting [1, 2, 4, 11–18], meta-learning [4, 5, 15, 16, 19], two-stage methods[4–6, 17] and post-hoc correction [20, 21].
Inspired by [2], re-weighting strategies can be roughlygrouped into empirical re-weighting and automatic re-weighting.
The former aims to design weightsmanually with the major insight that the minority class example will be assigned a larger weightvalue than that of the majority class [12–14].
However, manually setting weights can be less adaptiveto different datasets [2].
The latter aims to assign adaptive weights to the examples through learningmechanisms [1, 2, 4, 15, 16].
As the representative automatic re-weighting method, L2RW [15]optimizes the weight vector as a learnable parameter with an unbiased meta set ( i.e. , validationset).
Although L2RW and its followers have received widespread attention, most of them may belimited to optimizing the weights by the classification loss on the meta set: The gradient of weights is usually coupled with the to-be-learned classifier at each training iteration.
Since classifier is themajor concern in imbalanced issue [6], the dependence of weights on classifier at training stage maylead to inaccurate learning of the weights.
This paper develops a novel automatic re-weighting method for imbalanced classification basedon optimal transport (OT).
As discussed by Jamal et al.
[4], the major challenge for imbalancedclassification is essentially the mismatch between the imbalanced training dataset (seen by a machinelearning model) and the balanced test set (used to test the learned model).
To this end, we aimto view the learning of the weight vector as the distribution approximation problem.
We adoptthe two-stage learning manner motivated by [6], where stage 1 and stage 2 focus on learning thefeature extractor with the standard cross-entropy loss and the classifier with our proposed method,respectively.
Specifically, we represent the imbalanced training set as a discrete empirical distribution
P over all samples within it and view the to-be-learned weight vector w as its probability measure.
Then we represent the balanced meta set as a discrete empirical distribution Q over all sampleswithin it (in the same space with P ), which has a uniform probability measure for being balanced.
Therefore, the learning of a weight vector can be formulated as the process of learning the distribution
P to be as close to the balanced distribution Q as possible, a process facilitated by leveraging the OT distance [22].
Notably, the cost function plays a paramount role when learning the transportplan for OT, where we use the features and ground-truth labels of samples to design it.
Due to theflexibility of our method, we can also learn an explicit weight net directly from data like [16, 23]but with a different structure, optimized by OT loss instead of the classification loss on the meta set.
Generally, at each training iteration at stage 2, we minimize the OT loss to learn the weight vector(or weight net) for the current mini-batch, which is further used to re-weight the training loss foroptimizing the model.
As we can see, the gradient of weights only relies on the OT loss and thus isindependent of the classifier.
More importantly, our proposed method is robust to the distribution Q .
To save the memory consumption, we introduce the prototype-oriented OT loss by building anew distribution Q based on prototypes instead of samples (one prototype for each class).
Moreimportantly, our proposed method can achieve a reasonably good performance even if we randomlyselect a mini-batch from all prototypes to build Q , making our method applicable to datasets with alarge number of classes.
We summarize our main contributions as follows: (1) We formulate the learning of weight vector orweight net as the distribution approximation problem by minimizing the statistical distance betweento-be-learned distribution over samples from imbalanced training set and another balanced distributionover samples from the meta set.
(2) We leverage the OT distance between the distributions to guidethe learning of weight vector or weight net.
(3) We apply our method to imbalanced classificationtasks including image, text and point cloud.
Experiments demonstrate that introducing the OT loss tolearn the example weights can produce effective and efficient classification performance.  Related Work
Empirical Re-weighting A classic empirical re-weighting scheme is to provide the examples of eachclass with the same weight, such as inverse class frequency [11, 14].
It has been further improved bythe class-balanced loss [13], which calculates the effective number of examples as class frequency.
Focal Loss [12] uses the predicted probability to calculate higher weights for the hard examples anddynamically adjust the weights. LDAM-DRW [17] designs a label-distribution-aware loss functionand adopts a deferred class-level re-weighting method ( i.e. , inverse class frequency). Automatic Re-weighting The automatic re-weighting methods learn the weights with learningmechanisms. L2RW [15] adopts a meta-learning manner to learn the example weights, which areoptimized by the classification loss on the balanced meta set.
Hu et al.
[1] further improve L2RWby iteratively optimizing weights instead of re-estimation at each iteration.
Meta-weight-net [16]aims to learn an explicit weight net directly from data and optimize it by a meta-learning manner.
Meta-class-weight [4] defines the weight for each example as the combination of class-level weight(estimated by Cui et al.
[13]) and instance-level weight, optimized with a meta-learning approachsimilar to L2RW.
Influence-balanced loss (IB) is proposed to [18] re-weight samples by the magnitudeof the gradient.
Recently, Liu et al.
[2] propose to update the weights and model under a constraint.
Our method belongs to automatic re-weighting group, and the idea of building an explicit weightnet is similar to Shu et al.
However, the major difference is that we bypass the classificationloss on the meta set and use OT to learn the weights from the view of distribution approximation,disengaging the dependence of the weight learning on the concerned classifier at each iteration.
Meta Learning and Two-stage Learning Recently, researchers have proposed to tackle the imbalance issue with meta-learning, which can be applied to build a Balanced Meta-Softmax (BALMS)[19],learn weights [4, 15, 16] or transformed semantic directions for augmenting the minority classes in
MetaSAug [5].
Two-stage methods, where the first stage and second stage focus on representationlearning and classifier learning, respectively, have been proved effective for solving the imbalancedissue [5, 6, 16, 24].
BBN [25] unifies two stages with a specific cumulative learning strategy.
Optimal Transport Recently, OT has been used to solve the regression problem under the covariateshift [26], unsupervised domain adaption [27, 28], including sample-level, class-level or domain-levelweight vector.
Although they also adopt the re-weighting strategy and OT distance, they are distinctform ours in terms of task and technical detail.
Also, the dynamic importance weighting whichadopts MMD to re-weight samples for label-noise and class-prior-shift tasks [29] is also differentfrom ours, where we provide a more flexible way for learning the weights of samples and disengagethe dependence of the weight learning on the concerned classifier at each iteration.
To the best of ourknowledge, the works that solve imbalanced classification problem with OT are still very limited.
An oversampling method via OT (OTOS) [30] aims to make synthetic samples follow a similardistribution to that of minority class samples.
However, ours is a novel re-weighting method based on
OT, without augmenting samples.
Another recent work is Optimal Transport via Linear Mapping(OTLM)
[21], which performs the post-hoc correction from the OT perspective and proposes a linearmapping to replace the original exact cost matrix in OT problem.
Different from OTLM that belongsto the post-hoc correction group and aims to learn refined prediction matrix, ours falls into thetraining-aware group and aims to re-weight the training classification loss.
Imbalanced Classification Consider a training set D train =
{ ( x i , y i ) } Ni =1 , where ( x, y ) is the inputand target pair, x i the i -th sample, y i ∈ (0 , 1) K the one-hot associated label vector over K classes,and N the number of the entire training data.
Besides, consider a small balanced meta set D meta ={ ( x j , y j ) } Mj =1 , where M is the amount of total samples and M ≪ N .
Denote the model parameterizedwith θ as f ( x, θ ) , where θ is usually optimized by empirical risk minimization over the trainingset, i.e. , θ ∗ =
arg min θ 1 N (cid:80) Ni =1 ℓ ( y i , f ( x i ; θ )) .
For notational convenience, we denote l train i ( θ ) =
ℓ ( y i , f ( x i ; θ )) to represent the training loss function of pair ( x i , y i ) .
However, the model trained bythis method will prefer the majority class if the training dataset is imbalanced.
Learning to Re-Weight Examples To solve the imbalanced issue, a kind of re-weighting methods isto treat the weights as the learnable parameter and learn a fair model to the minority and the majorityclasses by optimizing the weighted training loss.
At each training iteration, the model is updated by
[Equation]
where w = ( w 1 , . . .
, w N ) T is the weight vector (usually with a simplex constraint) of all trainingexamples.
Then the optimal w is obtained by making the model parameter θ ∗ ( w ) from Eq.
(1)minimize the classification loss on a balanced meta set, formulated as
[Equation]
where l meta j is the loss function of pair ( x j , y j ) from meta set and the updated w ∗ is used to amelioratethe model.
Generally, model θ consists of two key components, feature extractor and classifier, wherethe classifier has been proved to be the major concerning part in imbalanced issue [6].
However,the gradient of weights in Eq.
(2) always depends on the to-be-concerned classifier at each trainingiteration, which may result in inaccurate learning of the weights.
Most automatic re-weightingmethods learn the weight vectors or weight-related parameters ( e.g. , weight net) following this line;see more details from the previous works [4, 15, 16].
Optimal Transport Theory OT has been widely used to calculate the cost of transporting oneprobability measure to another.
Among the rich theory of OT, this work presents a brief introductionto OT for discrete distributions; see Peyré and Cuturi [22] for more details.
Consider p = (cid:80) ni
=1 a i δ x iand q
=
(cid:80) mj
=1 b j δ y
j
as two probability distributions, where x i and y j live in the arbitrary same spaceand δ is the Dirac function.
Then, we can denote a ∈ ∆ n and b ∈ ∆ m as the probability simplex of R n and R m , respectively.
The OT distance between p and q can be expressed as:
[Equation]
where ⟨· , ·⟩ is the Frobenius dot-product and C ∈ R n × m ≥is the transport cost matrix constructedby C ij =
C ( x i , y j ) .
The doubly stochastic transport probability matrix T ∈ R n × m > 0 , which satisfiesΠ( p, q ) :
= { T | (cid:80) ni
=1 T ij =
b j , (cid:80) mj
=1 T ij = a i } , is learned by minimizing OT ( p, q ) .
Directlyoptimizing Eq.
(3) often comes at the cost of heavy computational demands, and OT with entropicregularization is introduced to allow the optimization at small computational cost in sufficientsmoothness [31].
Re-weighting Method with Optimal Transport
This work views a training set as a to-be-learned distribution, whose probability measure is set aslearnable weight vector w .
We use OT distance to optimize w for re-weighting the training loss.
4.1 Main Objective
Given the imbalanced training set D train , we can represent it as an empirical distribution over N pairs,where each pair ( x i , y i ) train has the sample probability w i ( i.e. , the weight), defined as:
[Equation] [Equation]
) where ( x i , y i ) train is the i -th pair from the training set and the learnable weight vector w of all trainingexamples means probability simplex of R N .
Since the meta set D meta is balanced for all classes andclosely related with the training set, it is reasonable to assume that meta set has already achieved thebalanced data distribution that the training set aims to approximate.
For meta set, we thus can sampleeach pair from it with equal probability and present it with an empirical distribution Q :
[Equation]
[Equation]
where ( x j , y j ) meta is the j -th pair from the meta set.
To learn w , different from most automaticre-weighting methods, which minimize the classification loss on the meta set, we aim to enforce theto-be-learned distribution P ( w ) to stay close to the balanced distribution Q .
Here, we explore there-weighting method by adopting the OT distance between P ( w ) and Q :
[Equation]
where cost matrix C ∈ R N × M ≥is described below and transport probability matrix T ∈ R N × M >shouldsatisfy Π( P ( w ) , Q ) :
= { T | (cid:80)
Ni =
1 T ij =1 /M, (cid:80) Mj =
[Equation]
ij = w i } .
4.2 Cost Function
For notation convenience, we reformulate the model as f ( x, θ ) =
f 2 ( f 1 ( x ; θ 1 ); θ 2 ) , where f 1 parameterized with θ 1 denotes the representation learning part before the classifier, and f 2 parameterizedwith θ 2 denotes the classifier.
Intuitively, the cost C ij measures the distance between pair i intraining set and pair j in meta set, which can be flexibly defined in different ways.
We explore a fewconceptually intuitive options of C ij , although other reasonable choices can also be used.
Label-aware Cost
As the first option, we can define C ij with the ground-truth labels of two samples:
[Equation]
where d Lab ( · , · ) also denotes a distance measure, and y train i , y meta j are the ground-truth label vectors ofthe two samples, respectively.
Intuitively, if we use the euclidean distance, then C is a 0 − 1 matrix(we can transfer the non-zero constant to 1 ), i.e. , C ij =0 if x train i and x meta j are from the same class,and C ij =1 otherwise.
Now the OT loss is influenced by neither feature extractor θ 1 nor classifier θ 2 .
Feature-aware Cost Besides
, we can define C ij purely based on the features of samples:
[Equation]
where z train i = f 1 ( [Equation] i ; θ 1 ) ∈ R E and z meta j =
[Equation]
[Equation]
; θ 1 ) ∈ R E denote the E -dimensionalrepresentation of x train i and x meta j , respectively.
d Fea ( · , · ) denotes any commonly used distance measureand we empirically find the cosine distance is a good choice.
It is easy to see that if x train i and x meta j ’sfeatures are close, their cost is small.
Here the OT loss is influenced by the feature extractor θ 1 .
Combined Cost
Finally, we can use both features and labels to define C ij , denoted as
[Equation]
Intuitively, C ij will be small if two samples have the same label and similar features.
Empirically,we find that using the d Fea =1 − cosine( · , · ) and euclidean distance for d Lab gives better performance.
Interestingly, given the feature-aware cost (8) or label-aware cost (7), the learned weight vector can beinterpreted as the instance-level or class-level re-weighting method, respectively.
The weight vectorlearned from the combined cost can be interpreted as the combination of class-level and instance-levelweights, although no specialized design for two-component weights like previous [4]; see Fig.
4.3 Learn the Weight Vector
Given the defined cost function, we adopt the entropy regularized OT loss [31] to learn the weightvector.
We thus rewrite (6) as the following optimization problem:
[Equation]
where λ > 0 is a hyper-parameter for the entropic constraint H ( T )
= − (cid:80) ij T ij ln T ij .
Note that 193 (10) provides us a new perspective to interpret the relationship between w and T , where w is theparameter of the leader problem and T is the parameter of the follower problem, which is of the lowerpriority.
Accordingly, when we minimize (10) with respect to w using gradient descent, we shoulddifferentiate through T .
Below we investigate the following two ways to optimize the weight vector.
Optimizing w directly Specifically, at each training iteration, we define P ( w ) with current w , usethe Sinkhorn algorithm [31] to compute OT loss, then optimize w by w ∗ =
arg min w L OT .
Amortizing the learning of w We also provide an alternative method by constructing an explicitweight net to output the example weights, whose structure can be designed flexibly.
For example, wecan build the following weight net and take the sample features as input:
[Equation]
where s i is the i -th element of s ∈ R N ,
w att ∈ R 1 × A and W
vz ∈ R
A × E are the learned parameters(we omit the bias for convenience), denoted as Ω = { w att , W vz } .
Denote S ( z ; Ω ) as the weightnet parameterized by Ω , which can be optimized by Ω ∗ =
arg min Ω L OT .
Overall Algorithm and Implementations
To integrate our proposed method with deep learning frameworks, we adopt a stochastic setting,i.e. , a mini-batch setting at each iteration.
Following [4, 5], we adopt two-stage learning, wherestage 1 trains the model f ( θ ) by the standard cross-entropy loss on the imbalanced training setand stage 2 aims to learn the weight vector w and meanwhile continue to update the model f
( θ ) .
Generally, at stage 2, calculating the optimal θ and w requires two nested loops of optimization,which is cost-expensive. Motivated by Hu et al.
[1], we optimize θ and w alternatively, correspondingto (1) and (10) respectively, where w is maintained and updated throughout the training, so thatre-estimation from scratch can be avoided in each iteration.
The implementation process of ourproposed method with w optimized directly is shown in Algorithm 1, where the key steps arehighlighted in Step (a), (b), and (c).
Specifically, at each training iteration t , in Step (a), we haveˆ θ ( t +1) ( w t ) =
{ ˆ θ ( t +1)( w t ) , ˆ θ ( t +1)( w t ) } and α is the step size for θ ; in Step (b), as the cost functionbased on features is related with ˆ θ ( t +1)( w t ) , the OT loss relies on ˆ θ ( t +1)( w t ) , and β is the step sizefor w ; in Step (c), we ameliorate model parameters θ ( t +1) .
We defer the learning of θ and Ω for theamortized learning of w . Discussion From Step (b), we find the gradient of w is unrelated to classifier θ 2 regardless of whichcost function we choose. If we use the label-aware cost or freeze the feature extractor parameterizedby θ 1 , which is trained in the first stage, the OT loss in Step (b) can be further reduced as L OT ( w t ) , Require: Datasets D train , D meta , initial model parameter θ and weight vector, hyper-parameters { α,β,λ } for t = 1 , 2 ,...,t 1 do
Sample a mini-batch B from the training set D train ;
Update θ ( t +1) ← θ ( t ) − α ∇ θ L B where L B = 1 | B | (cid:80) i ∈ B ℓ (cid:16)
y i ,f (cid:16) x i ; θ ( t ) (cid:17)(cid:17) ; end for for t = t 1 + 1 ,...,t 1 + t 2 do Sample a mini-batch B from the training set D train ;
Step (a): Update ˆ θ ( t +1) ( w ( t ) )
← θ ( t ) − α ∇ θ L B where L B = 1 | B | (cid:80) i ∈
B w ( t ) i ℓ (cid:16) y
i ,f (cid:16) x i ; θ ( t ) (cid:17)(cid:17)
Use D meta to build Q in (12) and B with w t to build P ( w t ) (4);
Step (b): Compute L OT (cid:16) ˆ θ ( t +1)( w t ) , w ( t ) (cid:17) with cost (9); Optimize w ( t +1) ← w ( t ) −
β ∇ w L OT (cid:16) ˆ θ ( t +1)( w t ) , w ( t ) (cid:17)
Step (c): Update θ ( t +1) ← θ ( t ) − α ∇ θ L B where L B = 1 | B | (cid:80) i ∈
B w ( t +1)
i ℓ (cid:16)
y i ,f (cid:16) x i ; θ ( t ) (cid:17)(cid:17) end for where we only need Steps (b)-(c) at each iteration.
This is different from most of automatic reweighting methods, where the gradient of w is always related with the to-be-learned model { θ 1 , θ 2 }or classifier θ 2 (when freezing θ 1 ) for minimizing the classification loss on meta set.
Prototype-oriented OT loss (POT) Recall that we represent a balanced meta set with M samplesas distribution Q in (5), where M/K is the number of data in each class and usually larger than 1.
Computing the OT loss requires to learn a B × M -dimensional transport matrix at each iteration.
Toimprove the efficiency of algorithm, we average all samples from each class in the meta set to achieveits prototype and propose a new Q distribution over K prototypes:
[Equation]
where POT loss only needs a B × K -dimensional transport matrix.
Due to the robustness of ourmethod to Q , when dealing with a large number of classes, we can randomly sample a mini-batchfrom K prototypes at each iteration to build Q .
Experiments
We conduct extensive experiments to validate the effectiveness of our proposed method on text,image, and point cloud imbalanced classification tasks.
Notably, different from the imbalanced imageand point cloud classification, we find that optimizing the weight net is better than optimizing theweight vector directly in the text classification.
Therefore, we optimize the weight vector for theimage and point cloud cases and build a weight net for text case.
Unless specified otherwise, weadopt the combined cost and set the hyper-parameter for the entropic constraint as λ = 0 .
1 and themaximum iteration number in the Sinkhorn algorithm as 200 .
We define the imbalance factor (IF) ofa dataset as the data point amount ratio between the largest and smallest classes.
6.1 Experiments on Imbalanced Image Classification
Datasets and Baselines We evaluate our method on CIFAR-LT-10, CIFAR-LT-100, ImageNet-LTand Places-LT.
We create CIFAR-LT-10 ( CIFAR-LT-100 ) from CIFAR-10 (CIFAR-100)[32] bydownsampling samples per class with IF ∈{ 200 , 100 , 50 , 20 } [5, 13].
ImageNet-LT is built from theclassic ImageNet with 1000 classes[33] and IF =1280 / 5 [5, 24].
Places-LT is created from Places-2[34] with 365 classes and IF =4980 / 5 [4, 24].
We randomly select 10 training images per class asmeta set [5]; see more details in Appendix B. We consider the following baselines: (1) Cross-entropy(CE) , the model trained on the imbalanced training set with CE loss.
(2) Empirical re-weightingmethods , like Focal loss [12], Class-balanced (CB) loss [13] and LDAM-DRW [17].
(3) Automaticre-weighting methods , including L2RW [15], IB [18], Meta-Weight-Net [16] and Meta-class-weight[4].
(4) Meta-learning methods , including MetaSAug [5] and above methods of [4, 15, 16, 19].
Two-stage methods , such as OLTR [24], cRT [6], LWS [6], BBN [25] and methods of [4, 5].
Experimental details and results on CIFAR-LT For a fair comparison, we use ResNet-32 [35] asthe backbone on CIFAR-LT-10 and CIFAR-LT-100.
Following Li et al.
[5], at stage 1, we use 200epochs, set the learning rate α of θ as 0.1, which is decayed by 1 e − 2 at the 160th and 180th epochs.
At stage 2, we use 40 epochs, set α as 2 e − 5 and learning rate β of weights as 1 e − 3 .
We use the SGDoptimizer with momentum 0.9, weight decay 5 e − 4 and set the batch size as 16 .
We list the recognitionresults of different methods on CIFAR-LT-10 and CIFAR-LT-100 with different imbalance factors in
Table 1.We report the average result of 5 random experiments without standard deviation which is of small scale(e.g., 1e-2).
We can see that our re-weighting method outperforms CE training by a largemargin and performs better than the empirical or automatic re-weighting methods.
Remarkably, ourproposed method outperforms competing MetaSAug that conducts a meta semantic augmentationapproach to learn appropriate class-wise covariance matrices when IF is 200, 100 and 50.
Importantly,as the training data becomes more imbalanced, our method is more advantageous.
Even thoughour proposed method is inferior to MetaSAug when the dataset is less imbalanced (IF =20 ), it canstill achieve competing results and surpasses related re-weighting methods.
This suggests that ourproposed method can be used to enhance the imbalanced classification, without the requirement ofdesigning complicated models or augmenting samples on purpose.
To more comprehensively understand our method, we provide a series of ablation studies on CIFARLT-100 with IF = 200 in Table 2.
Firstly, to explore the impact of cost function, we use differentcost functions for the OT loss.
We can see that the combined cost performs better than label-awarecost and feature-aware cost, confirming the validity of combining features and labels to define cost.
Besides, using either label-aware or feature-aware cost can still achieve acceptable performance,indicating the usefulness of OT loss in the imbalanced issue.
Secondly, to explore the robustness ofthe meta distribution Q , we adopt three ways to build Q : (1) using prototypes defined in Eq.
(12) ( Ksamples) ; (2) using all samples defined in Eq.
(5) ( 10 ∗ K samples) ; (3) randomly sampling onepoint from each class ( K samples) in meta set.
We find that prototype-based meta performs best, andthe performance with random-sample meta or whole meta is still competitive, which demonstratesthe robustness of our proposed method to the distribution Q and the benefit of using the prototypes tobuild Q .
Third, we compare two ways for learning w in each iteration, where one is re-estimating wfrom scratch and another one is maintaining and updating w throughout the training ( i.e. , iterativelyoptimizing weights).
We find that iteratively optimizing performs better
[Table] [Table]
Since cost function is essential in optimizing the OT loss, we are interested in examining the learnedweight vectors given by different cost functions.
Here, we use CIFAR-LT-10, randomly choose{ 10 , 9 , ..., 1 } training samples from class { 1 , 2 , ..., 10 } and obtain 55 samples, which are used tobuild distribution P .
Besides, the 10 prototypes from meta set are used to build the distribution Q .
Given the different cost functions, we show the learned weight vectors of 55 training samples in Fig.1, which have very different properties.
Specifically, the label-aware cost and feature-aware cost leadto class-level weights and sample-level weights, respectively.
It is reasonable that label-aware costonly decides whether the two samples (from the meta set and training set) belong to the same class, resulting in class-level measure.
However, feature-aware cost measures the distance between samplesfrom the sample-level, where each sample has its own feature.
More interestingly, the learned weightswith the combined cost own the characteristics of class-level and sample-level weights simultaneously,where example weights of different classes are far away and example weights of the same class areclose.
Coincidentally, using the combined cost to define the OT loss can reach the same goal of [4],which explicitly considers class-level and sample-level weight.
Besides, we find that the learnedexample weights of the minority class are usually more prominent than those of the majority classes.
[Table]
[Table] [Table]
To verify whether our method ameliorates the performance on minority classes, we plot the confusionmatrices of CE, MetaSAug, and ours on CIFAR-LT-10 with IF =200 in Fig.
As expected, although
CE training can almost perfectly classify the samples in majority classes, it suffers severe performancedegeneration in the minority classes.
MetaSAug improves the accuracies of the minority classes,where is still a big gap between the performance on the minority classes and the majority classes.
Incontrast, ours does not show a very clear preference for a certain class and outperforms the strongbaseline on the overall performance, which is the goal of on an imbalanced classification task.
Experimental details and results on Places-LT and ImageNet-LT Following [6], we employ
ResNet-152 pre-trained on the full ImageNet as the backbone on Places-LT.
For stage 1, we set theinitial learning rate as 0.01, which is decayed by 1 e − 1 every 10 epochs.
In the stage 2 of our method,we only fine-tune the last fully-connected layer for training efficiency and set α as 1 e − 4 and β as1 e − 3 within 50 epochs.
The mini-batch size is 32 and the optimizer is SGD with momentum 0 .
9 andweight decay 5 e − 3 .
As shown in Table 3, our method outperforms all baselines.
It further suggeststhat our method has excellent performance in the extreme imbalance setting with IF =4980 / 5 .
For afair comparison, we implement our method on ImageNet-LT with the same experimental conditionsof [5], from which we have taken the results of other comparison methods.
We consider ResNet-50[35] as the backbone on ImageNet-LT.
In stage 1, we run 200 epochs and decay the learning rate by0.1 at the 60th and 80th epochs.
In stage 2, we implement our method for 50 epochs, set learningrate α as 2 e − 5 and β as 1 e − 2 , and only fine-tune the last fully-connected layer for training efficiency.
We use the SGD optimizer with momentum 0.9, weight decay 5 e − 4 and set the batch size as 128 .
The results on ImageNet-LT of different models reported in Table 4 indicates the effectiveness of ourproposed method on ImageNet-LT when comparing with strong baseline MetaSAug.
Besides, wefurther consider randomly sampling a mini-batch of size 100 from all prototypes at each iterationto build Q , whose performance is comparable to the Q from all prototypes.
Thus, with a stochasticsetting for Q , our proposed method can be used to the imbalanced training set with a large number ofclasses.
We defer the time computational complexity, additional quantitative results and qualitativeresults on different image datasets to Appendix B.
6.2 Experiments on Imbalanced Text Classification
Datasets and settings Following [1, 2], we adopt the popular SST-2 for 2-class and SST-5 for 5-classsentence sentiment [37].
For a fair comparison, we use the same imbalanced datasets and settings with [2].
Specifically, we set class 1 as the minority class and the rest as the majority classes, wherethe number of examples in the majority class is fixed as 1000 (SST-2) and 500 (SST-5) and we achievedifferent imbalance settings by varying the number of examples in the minority class.
Besides, thenumber of samples in the meta set is 10 for each class.
We use the BERT (base, uncased) model[38] as feature extractor and a simple 3-layer fully-connected network (FCN) with the structure in
Appendix C as classifier.
To make subsequent experiments on strong models, following [2], we usean additional balanced training set (500 samples in each class) to fine-tune the BERT model, which israndomly selected from the remaining examples in each dataset except the imbalanced training set,meta set and to-be evaluated test set.
Based on the fine-tuned BERT, we adopt the two-stage mannerfor the imbalanced text datasets, where we train the BERT + FCN in the first stage with CE loss andtrain the FCN with our proposed method by freezing the BERT in the second stage.
The settings ofthe training process are deferred to Appendix C.
Baselines We consider the following methods: (1) vanilla BERT , the vanilla pretrained languagemodel.
(2) Fine-tuned BERT , where the pretrained BERT is fine-tuned on an additional balancedtraining set.
(3) Fine-tuned BERT + CE , the fine-tuned BERT model followed by the FCN whichis further trained by the CE loss on the imbalanced training set following [1, 2].
(4) Automaticre-weighting methods , including the method of Hu et al.
[1] and constraint-based re-weighting[2].
Since few works consider imbalanced text classification, we further consider (5) Empirical reweighting methods , including re-weighting with inverse class frequency ( i.e. , Proportion) [11, 14])and LDAM-DRW [17] and (6) Logit adjustment [20] using their official codes and settings 1 2 .
Werepeat all experiments 10 times and report the mean and standard deviation.
Experimental details and results on SST-2 and SST-5 We report the text classification resultsof compared methods under different imbalance factors in Table 5.
We find that our proposedmethod outperforms all competing methods in all imbalance factor settings, which demonstrates theeffectiveness of our proposed method.
Although all methods could achieve acceptable performancein a slight imbalance, the performance of three baselines (Vanilla BERT, Fine-Tuned BERT and FineTuned BERT+CE) drop dramatically, indicating the importance of proposing specialized methods forhandling imbalanced training datasets.
Logit adjustment (post-hoc correction), is very competitive toours on SST-2, which, however, only produces similar results to the three above-mentioned baselineson SST-5.
In contrast, ours is robust to not only the imbalance factors but also the number of classes,where the results are consistent with the image case.
We provide more results in Appendix C.4.
Inaddition to 1D text and 2D image, we further investigate the robustness of our method on 3D pointcloud data, where we use the popular ModelNet10 [39] and defer the experiments to Appendix D. [Table]
Conclusion
This paper introduces a novel automatic re-weighting method for imbalance classification basedon optimal transport (OT).
This method presents the imbalanced training set as a to-be-learneddistribution over its training examples, each of which is associated with a probability weight.
Similarly,our method views another balanced meta set as a balanced distribution over the examples.
Byminimizing the OT distance between the two distributions in terms of the defined cost function,the learning of weight vector is formulated as a distribution approximation problem.
Our proposedre-weighting method bypasses the commonly-used classification loss on the meta set and uses OT tolearn the weights, disengaging the dependence of the weight learning on the concerned classifier ateach iteration.
This is an approach different from most of the existing re-weighting methods and mayprovide new thoughts for future work.
Experimental results on a variety of imbalanced datasets ofboth images and texts validate the effectiveness and flexibility of our proposed method.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Section 6.1.
When handling imbalanced datasets of image classification, our proposed method is inferior to some
competitive baseline method when the dataset is less imbalanced.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See Appendix D.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes] We’ve read the guidelines and make sure our submission adheres to the
ethical standards.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] Code will be
submitted in the supplemental material.
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See Section 6.
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] In Section 6.1, we report the average result of 5 randomimage experiments and omit standard deviation for the limited space, which is usually
small (e.g., 1e-2).
In Section 6.2, we repeat all text experiments 10 times and report themean and standard deviation.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix B.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[N/A]
[Table] (c) Did you include any new assets either in the supplemental material or as a URL?
[No]
[Table] (d) Did you discuss whether and how consent was obtained from people whose data you’re [Table]
using/curating?
[N/A]
[Table] (e) Did you discuss whether the data you are using/curating contains personally identifiable [Table]
information or offensive content?
[N/A]
[Table] If you used crowdsourcing or conducted research with human subjects... [Table]
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
[Table] (b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
[Table]
A Amortizing the learning of weight vector by introducing a weight net
To integrate our proposed method with deep learning frameworks, we adopt a stochastic setting,i.e. , a mini-batch setting at each iteration.
We adopt two-stage learning, where stage 1 trains themodel f ( θ ) by the standard cross-entropy loss on the imbalanced training set and stage 2 aims tolearn the weight vector w and meanwhile continue to update the model f
( θ ) .
Generally, at stage 2,calculating the optimal θ and w requires two nested loops of optimization, which is cost-expensive.
We optimize θ and w alternatively, corresponding to (1) and (10) respectively, where w is maintainedand updated throughout the training, so that re-estimation from scratch can be avoided in eachiteration.
We summarize the amortized learning of w in Algorithm 2, where the key steps arehighlighted in Step (a), (b), and (c).
Specifically, at each training iteration t , in Step (a), we haveˆ θ ( t +1) ( Ω t ) =
{ ˆ θ ( t +1)
( Ω t ) , ˆ θ ( t +1)( Ω t ) } and α is the step size for θ ; in Step (b), as the cost functionbased on features is related with ˆ θ ( t +1)
( Ω t ) , the OT loss relies on ˆ θ ( t +1)
( Ω t ) , and β is the step sizefor Ω ; in Step (c), we ameliorate model parameters θ ( t +1) .
Require: Datasets D train , D meta , initial model parameter θ and weight net Ω , hyper-parameters { α, β, λ } for t = 1 , 2 , ..., t 1 do
Sample a mini-batch B from the training set D train ;
Update θ ( t +1) ← θ ( t ) − α ∇ θ L B where L B = 1 | B | (cid:80) i ∈ B ℓ (cid:16) y i ,
; end do
; Compute
Q
P
Compute
Optimize Ω t + ← Ω t −
B ℓ (cid:16) i ,
; θ t we
end for
B More details and results about the imbalanced image classification
B.1 Details about imbalanced training datasets
CIFAR-LT is the long-tailed version of CIFAR dataset, where the original CIFAR-10 (CIFAR-100)dataset [32] has 5000 (500) images per class and falls into 10 (100) classes.
Following [5], we createlong-tailed training sets from CIFAR-10 and CIFAR-100 by discarding some training samples, wherewe vary the imbalance factor IF ∈ { 200 , 100 , 50 , 20 } .
We do not change the balanced test sets andwe randomly select ten training images per class as our meta set.
We build ImageNet-LT based on the classic ImageNet [33] following [24].
After discarding sometraining examples, ImageNet-LT has 115.8K training examples in 1,000 classes, with an imbalancefactor IF = 1280 / 5 .
The authors also provided a small balanced validation set with 20 imagesper class, from which we sample 10 images to construct our meta set as [5].
Besides, the originalbalanced ImageNet validation set is adopted as the test set (50 images per class).
Places-LT is created from Places-2 [34] by [24] with the same strategy as above.
The Places-LTcontains 62.5K images from 365 categories with = 4980 / 5 .
This factor means that Places-LT is morechangeling than ImageNet-LT dataset.
Similarly, there is an official balanced validation dataset with20 images per class, where we random select 10 images to build our meta set following [4].
iNaturalist 2018 3 is collected from real world which contains 435,713 training samples in 8142categories with the imbalance factor IF of 1000 / 2 .
Following [4, 5], we use the original validationdataset to evaluate our method and random select 2 samples from every category in training datasetto construct our meta set.
B.2 Experimental details and results on iNaturalist 2018
Following [4], we use ResNet-50 pre-trained on ImageNet plus iNaturalist 2017 as the backbonenetwork on iNaturalist 2018.
For stage 1, we set the initial learning rate as 0.01, which is decay by1 e − 1 every 20 epochs.
In the stage 2 of our method, we only fine-tune the last fully connected layerand set α as 1 e − 4 and β as 1 e − 3 within 200 epochs.
The mini-batch size is 64, the optimizer is SGDwith momentum 0 .
9 and weight decay is 5 e − 3 .
Considering the label vector from the iNaturalist 2018 dataset is high-dimensional and sparse, weonly use the features of samples to define the cost function, i.e. , Feature-aware cost .
For trainingefficiency, we use Prototype to construct our meta set.
As summarized in Table 6, our proposedmethod outperforms all the baselines.
This observation demonstrates that our proposed re-weightingmethod can effectively deal with the extremely imbalanced large-scale training set with a largenumber of classes.
B.3 Learned weights for different classes with varying imbalanced settings
To explore the learned example weights, we average them within each class and visualize the averagedweight for each class with varying imbalanced factors in Fig.
We see that the learned weights ofthe class 10 (tail class) are more prominent than those of the majority classes.
This phenomenonbecomes increasingly evident as the training set becomes more imbalanced.
B.4 Convergence and time complexity
In this section, we investigate the convergence and time complexity.
The optimization of the OTdistance between a discrete distribution of size n and a discrete distribution of size m , is a linearprogram and can be solved with combinatorial algorithms such as the simplex methods, whichdampens the utility of the method when handling large datasets.
Recently, the regularization schemeproposed by Cuturi in [31] allows a very fast computation of a transportation plan, denoted as
Sinkhorn algorithm.
The Sinkhorn algorithm requires the computational cost of n 2 log( n ) /ε 2 toreach ε -accuracy, which is adopted by us to solve the OT problem.
As shown in Fig.
4, we plot the variation of training loss and test classification performance as theincrease of training time at the second stage.
The training loss of our proposed method can convergesat around 10th epoch, where 20 epochs are usually enough for our method to yield satisfactoryperformance.
This means our method can reach fast convergence.
[Table]
To investigate the time complexity, we conduct experiments on CIFAR 10, CIFAR100, ImageNet-LTand Places-LT with different imbalanced factor for ten epoches and average over the time.
Modelson CIFAR are evaluated with batch size of 16 on a single RTX 3080 GPU.
Models on ImageNet-LTare evaluated with batch size of 128 on 4 RTX 2080Ti GPU.
Models on Places-LT and iNaturalist2018 are evaluated with batch size of 32 and 64 on 4 Tesla V100 GPU, respectively.
The results aresummarized in Table 7.
We observe that our proposed method has an acceptable time complexitycompared with the cross entropy.
It means that our proposed method achieves a better performancewith an acceptable cost by introducing the OT loss for re-weighting.
B.5 Learned Transport plan matrix and weight vector
To explain the effectiveness of learning the weight vector with OT, we give a toy example.
Theimbalanced training mini-batch consists of { 10 , 9 , ..., 1 } training samples from class { 1 , 2 , ..., 10 } andthe balanced meta set consists of { 10 , 10 , ..., 10 } validation samples from class { 1 , 2 , ..., 10 } , whichare used to compute the prototype for each class.
The process of learning the weight vector is shownin Fig.
5, where we use feature-aware cost, label-aware cost and combined cost in (a)-(c), respectively.
For all experiments, we fix the probability measure of meta set as uniform distribution, initialize the tobe-learned probability measure ( i.e. , w ) of training mini-batch with a uniform measure, and we thenoptimize w by minimizing the OT loss with given cost function.
Given different cost functions, wefind that the learned weight vectors and transport probability matrices show different characteristics.
Given the feature-aware cost function, we can assign weight to examples instance-wisely.
However,the inaccurate discrimination of features may mislead the learning of weight.
Under the label-awarecost function, the learned weights are class-level and inversely related to the class frequency.
Giventhe combined cost function, the results combine the benefits of label-aware and feature-aware costfunctions.
Interestingly, the weights assigned to examples of tail classes are more prominent thanthose of the head classes, where the weights of head classes are below the initialization 1 / 55 andthose of tail classes are above 1 / 55 .
Such results demonstrate our re-weighting method can pay moreattention to the tail classes.
[Table]
[Figure]
[Table]
[Table]
[Table]
C More details about the text classificationC.1 FCN structure
We give the FCN structure in Table 9.
C.2 Weight net structure on text classification task
We give the weight net structure used in text classification task in Fig. 6.
Different that used in imageand point cloud classification tasks, we build a self attention-similar weight net and take the samplefeatures z train i as input:
[Equation]
Specifically, the output of dimension d f from feature extractor will be encoded to the same dimensiond l by two separate linear layers W 1 and W 2 .
We compute their dot products, divide by a scaledfactor √ d l and apply a softmax operator to obtain expected instance weights.
The hidden shape inour experiments is shown in Table 8.
C.3 Experimental settings
For fining tune BERT, we set the initial learning rate as 5 e − 6 , where epochs is 5 and batch size is 8.
For the first stage, we use SGD optimizer with learning rate 1 e − 4 .
For the second stage, we use SGDoptimizer with α learning rate 1 e − 4 and β learning rate 4 e − 2 .
For both two stages, our minibatchsize is 50 and epochs are 15 and 10, respectively.
The settings of the training process on the STT-2and
SST-5 are listed in Table 10.
C.4 Ablation Study
To prove the robustness of our method and its variants on text data, we execute an ablation study on
SST-2 dataset, which is shown in Table 11.
We can find that combined cost achieves best performanceand using label or feature cost can still obtain an acceptable result, which indicates the OT cost isflexible and robustness.
When comparing the influence of different ways in constructing meta set, wecan see that both whole meta set and prototype-based meta set achieve a close performance, whilerandom sample meta cause a noticeable performance drop.
D More details about the point cloud classificationD.1 Experiments on Imbalanced Point Cloud Classification
Dataset and Baselines In addition to 1D text and 2D image, we further investigate the robustnessof our method on 3D point cloud data, where we use the popular ModelNet10 [39].
We select
[Table]
PointNet++ [40] as our backbone without other additional layers.
We use 5 examples in each class formeta set.
Specifically, we down-sample the original points to 512 followed by a random point dropout,scale and shift to process the input data, which is a common setting for point cloud classification.
We conduct the same process on all the experiments to evaluate our method fairly.
Since we are thefirst to conduct imbalance classification task on point cloud data, we consider following methods: (1)
Pointnet++ baseline 4 , a classic network used in point cloud classification [40].
(2) Focal loss [12].(3) LDAM-DRW [17].
(4) Logit Adjustment [20].
Experimental details and results on ModelNet10 The classification results of different methodsare summarized in Table 12, where we evaluate our method from the aspects of instance accuracy(IA) and class accuracy (CA).
We can see that ours outperforms all baselines on IA and CA.
Theseexperimental results further demonstrate the effectiveness of integrating OT loss into imbalanced pointcloud classification, showing the generalization ability of our proposed method to data modalities.
D.2 Experimental settings
For point cloud classification, we select bathtub as minority class and the rest as majority, where thenumber of training sample is 100 in every class.
We train the baseline model from scratch wherewe use Adam optimizer with learning rate 1 e − 4 .
Besides, our minibatch size is 128 and epochs are40.
For the first and second stage, we still use Adam optimizer, where learning rate is 1 e − 4 in the1-st stage, α learning rate 1 e − 4 and β learning rate 5 e − 3 .
For both two stages, we keep the samenumber of epochs and minibatch size as them in training the baseline.
For training and meta dataset,we choice 100 samples from every majority class in original training set, where we then randomchoice 5 samples from the rest training set to build our meta set.
For test dataset, we random sample40 instances in every class from the original test dataset to construct our balanced test dataset.
[Table]
[Table]
D.3 Ablation study
Similarly, we execute an ablation on point cloud to investigate the influence of OT loss, which isshown Table 14.
We can see that the best performance is obtained when we use the setting of weight
[Table]
vector, combined cost and whole meta set.
Besides, performance are also acceptable when we uselabel cost and feature cost.
The performance is also competitive when we use the setting of weight
[Table]
vector, combined cost and prototype-based meta set.
D.4 Negative Societal Impacts
This paper introduces a re-weighting method based on optimal transport to solve the imbalancedproblem.
The proposed re-weighting method is very different from existing re-weighting methodsand may have the potential to provide new and better thoughts for imbalanced problems in the future.
Notably, once a malicious imbalanced task is selected, our re-weighting method may produce anegative societal impact, similar to the progress in other machine learning methods.
E Effect of Different Meta-sets
To explore whether our proposed method is effective even without an additional meta set fromthe validation set
[Table]
we build a dynamic meta set from the given imbalanced training set.
Notably,“dynamic” means that we randomly choose the samples from the training set to build a meta set ateach training stage.
Due to the limited rebuttal time
[Table]
we report experiment results in SST-2 dataset in
Table 15.
The results show our method could still reach a comparable performance even with themeta set built from the imbalance training dataset dynamically.
[Table]
Our method with additional meta set
[Table]
Our method with dynamic meta set
[Table]
F Label-awareness Experiments
In this section, we report label-aware performance of our method in image, text and point cloudclassification task.
In this setting, label-aware represents that we only use label information in metadataset to constrain the OT loss and then obtain the updated sample weights.
our best performancerepresents that we use both feature and label information in meta dataset.
As summarized in 16, 17and 18, all results show that our method could obtain a comparable performance under the label- aware setting, where this label information is free to be obtained and we could reach the goal by onlyrequiring the labels in training dataset obeying a uniform distribution.
[Table]
[Table]
[Table]