Learning to Re-weight Examples with Optimal Transport for Imbalanced Classification
Anonymous Author(s)
AffiliationAddress email
Abstract
Imbalanced data pose challenges for deep learning based classification models.
Oneof the most widely-used approaches for tackling imbalanced data is re-weighting,where training samples are associated with different weights in the loss function.
Most of existing re-weighting approaches treat the example weights as the learnableparameter and optimize the weights on the meta set, entailing expensive bileveloptimization.
In this paper, we propose a novel re-weighting method based onoptimal transport (OT) from a distributional point of view.
Specifically, we viewthe training set as an imbalanced distribution over its samples, which is transportedby OT to a balanced distribution obtained from the meta set.
The weights ofthe training samples are the probability mass of the imbalanced distribution andlearned by minimizing the OT distance between the two distributions.
Compared with existing methods, our proposed one disengages the dependence of the weightlearning on the concerned classifier at each iteration.
Experiments on image,text and point cloud datasets demonstrate that our proposed re-weighting methodhas excellent performance, achieving state-of-the-art results in many cases andproviding a promising tool for addressing the imbalanced classification issue.
Introduction
Deep neural networks (DNNs) have achieved remarkable success in various applications, which isundoubtedly inseparable from the high-quality large-scale datasets.
Usually, the number of samplesfor each class in these datasets are manually selected resulting in balanced datasets.
However, mostreal-world datasets are imbalanced, such as a few classes (a.k.a. head or majority class) occupy mostof the data while most classes (a.k.a.
tail or minority class) have a few samples.
A model trainedon the imbalanced training set but without considering such class imbalance would be significantlydominated by those majority classes and thus underperform on a balanced test dataset.
This can alsobe known as the long-tailed problem and exists in many domains, such as text classification [1, 2],object detection [3] and image classification [4–6].
There are rich research lines to solve the imbalance problem, including re-sampling [7–10], classlevel or instance-level re-weighting [1, 2, 4, 11–17], meta-learning [4, 5, 15, 16], two-stage methods[4–6, 17] and post-hoc correction [18, 19].
Inspired by [2], re-weighting strategies can be roughlygrouped into empirical re-weighting and automatic re-weighting.
The former aims to design weightsmanually with the major insight that the minority class example will be assigned a larger weightvalue than that of the majority class [12–14].
However, manually setting weights can be less adaptiveto different datasets [2].
The latter aims to assign adaptive weights to the examples through learningmechanisms [1, 2, 4, 15, 16].
As the representative automatic re-weighting method, L2RW [15]optimizes the weight vector as a learnable parameter with an unbiased meta set ( i.e. , validationset).
Although L2RW and its followers have received widespread attention, most of them may be limited to optimizing the weights by the classification loss on the meta set: The gradient of weightsis usually coupled with the to-be-learned classifier at each training iteration.
Since classifier is themajor concern in imbalanced issue [6], the dependence of weights on classifier at training stage maylead to inaccurate learning of the weights.
This paper develops a novel automatic re-weighting method for imbalanced classification basedon optimal transport (OT).
As discussed by Jamal et al.
[4], the major challenge for imbalancedclassification is essentially the mismatch between the imbalanced training dataset (seen by a machinelearning model) and the balanced test set (used to test the learned model).
To this end, we aimto view the learning of the weight vector as the distribution approximation problem.
We adoptthe two-stage learning manner motivated by [6], where stage 1 and stage 2 focus on learning thefeature extractor with the standard cross-entropy loss and the classifier with our proposed method,respectively.
Specifically, we represent the imbalanced training set as a discrete empirical distribution
P over all samples within it and view the to-be-learned weight vector w as its probability measure.
Then we represent the balanced meta set as a discrete empirical distribution Q over all sampleswithin it (in the same space with P ), which has a uniform probability measure for being balanced.
Therefore, the learning of a weight vector can be formulated as the process of learning the distribution
P to be as close to the balanced distribution Q as possible, a process facilitated by leveraging the OTdistance [20].
Notably, the cost function plays a paramount role when learning the transport plan for OT, where we use the features and ground-truth labels of samples to design it.
Due to the flexibility ofour method, we can also learn an explicit weight net directly from data like [16] but with a differentstructure, optimized by OT loss instead of the classification loss on the meta set.
Generally, at eachtraining iteration at stage 2, we minimize the OT loss to learn the weight vector (or weight net) forthe current mini-batch, which is further used to re-weight the training loss for optimizing the model.
As we can see, the gradient of weights only relies on the OT loss and thus is independent of theclassifier.
More importantly, our proposed method is robust to the distribution Q .
To save the memoryconsumption, we introduce the prototype-oriented OT loss by building a new distribution Q basedon prototypes instead of samples (one prototype for each class).
More importantly, our proposedmethod can achieve a reasonably good performance even if we randomly select a mini-batch from allprototypes to build Q , making our method applicable to datasets with a large number of classes.
We summarize our main contributions as follows: (1) We formulate the learning of weight vector orweight net as the distribution approximation problem by minimizing the statistical distance betweento-be-learned distribution over samples from imbalanced training set and another balanced distributionover samples from the meta set.
(2) We leverage the OT distance between the distributions to guidethe learning of weight vector or weight net.
(3) We apply our method to imbalanced classificationtasks including image, text and point cloud.
Experiments demonstrate that introducing the OT loss tolearn the example weights can produce effective and efficient classification performance.
Background
Imbalanced Classification Consider a training set D train =
{ ( x i , y i ) } Ni =1 , where ( x, y ) is the inputand target pair, x i the i -th sample, y i ∈ (0 , 1) K the one-hot associated label vector over K classes,and N the number of the entire training data.
Besides, consider a small balanced meta set D meta ={ ( x j , y j ) } Mj =1 , where M is the amount of total samples and M ≪ N .
Denote the model parameterizedwith θ as f ( x, θ ) , where θ is usually optimized by empirical risk minimization over the trainingset, i.e. , θ ∗ =
arg min θ 1 N (cid:80) Ni =1 ℓ ( y i , f ( x i ; θ )) .
For notational convenience, we denote l train i ( θ ) =
ℓ ( y i , f ( x i ; θ )) to represent the training loss function of pair ( x i , y i ) .
However, the model trained bythis method will prefer the majority class if the training dataset is imbalanced.
Learning to Re-Weight Examples To solve the imbalanced issue, a kind of re-weighting methods isto treat the weights as the learnable parameter and learn a fair model to the minority and the majorityclasses by optimizing the weighted training loss.
At each training iteration, the model is updated by
[Equation]
where w = ( w 1 , . . .
, w N ) T is the weight vector (usually with a simplex constraint) of all trainingexamples.
Then the optimal w is obtained by making the model parameter θ ∗ ( w ) from Eq.
(1)minimize the classification loss on a balanced meta set, formulated as
[Equation]
where l meta j is the loss function of pair ( x j , y j ) from meta set and the updated w ∗ is used to amelioratethe model.
Generally, model θ consists of two key components, feature extractor and classifier, wherethe classifier has been proved to be the major concerning part in imbalanced issue [6].
However,the gradient of weights in Eq.
(2) always depends on the to-be-concerned classifier at each trainingiteration, which may result in inaccurate learning of the weights.
Most automatic re-weightingmethods learn the weight vectors or weight-related parameters ( e.g. , weight net) following this line;see more details from the previous works [4, 15, 16].
Optimal Transport Theory OT has been widely used to calculate the cost of transporting oneprobability measure to another.
Among the rich theory of OT, this work presents a brief introductionto OT for discrete distributions; see Peyré and Cuturi [20] for more details.
Consider p = (cid:80) ni
=1 a i δ x iand q
=
(cid:80) mj
=1 b j δ y
j
as two probability distributions, where x i and y j live in the arbitrary same spaceand δ is the Dirac function.
Then, we can denote a ∈ ∆ n and b ∈ ∆ m as the probability simplexof R n and R m , respectively.
The OT distance between p and q can be expressed as:
[Equation]
where ⟨· , ·⟩ is the Frobenius dot-product and C ∈ R n × m ≥is the transport cost matrix constructedby C ij =
C ( x i , y j ) .
The doubly stochastic transport probability matrix T ∈ R n × m > 0 , which satisfiesΠ( p, q ) :
= { T | (cid:80) ni
=1 T ij =
b j , (cid:80) mj
=1 T ij = a i } , is learned by minimizing OT ( p, q ) .
Directlyoptimizing Eq.
(3) often comes at the cost of heavy computational demands, and OT with entropicregularization is introduced to allow the optimization at small computational cost in sufficientsmoothness [21].
Re-weighting Method with Optimal Transport
This work views a training set as a to-be-learned distribution, whose probability measure is set aslearnable weight vector w .
We use OT distance to optimize w for re-weighting the training loss.
3.1 Main Objective
Given the imbalanced training set D train , we can represent it as an empirical distribution over N pairs,where each pair ( x i , y i ) train has the sample probability w i ( i.e. , the weight), defined as:
[Equation]
(4) where ( x i , y i ) train is the i -th pair from the training set and the learnable weight vector w of all trainingexamples means probability simplex of R N .
Since the meta set D meta is balanced for all classes andclosely related with the training set, it is reasonable to assume that meta set has already achieved thebalanced data distribution that the training set aims to approximate.
For meta set, we thus can sampleeach pair from it with equal probability and present it with an empirical distribution Q :
[Equation]
[Equation]
where ( x j , y j ) meta is the j -th pair from the meta set.
To learn w , different from most automaticre-weighting methods, which minimize the classification loss on the meta set, we aim to enforce theto-be-learned distribution P ( w ) to stay close to the balanced distribution Q .
Here, we explore there-weighting method by adopting the OT distance between P ( w ) and Q :
[Equation]
(6) where cost matrix C ∈ R N × M ≥is described below and transport probability matrix T ∈ R N × M >shouldsatisfy Π( P ( w ) , Q ) := [Equation]
T | (cid:80) N
[Equation] = [Equation]
/M, (cid:80) M [Equation] =
[Equation]
= w [Equation]
3.2 Cost Function
For notation convenience, we reformulate the model as f ( x, θ ) =
f 2 ( f 1 ( x ; θ 1 ); θ 2 ) , where f 1 parameterized with θ 1 denotes the representation learning part before the classifier, and f 2 parameterizedwith θ 2 denotes the classifier.
Intuitively, the cost C ij measures the distance between pair i in training set and pair j in meta set, which can be flexibly defined in different ways.
We explore a fewconceptually intuitive options of C ij , although other reasonable choices can also be used.
Label-aware Cost
As the first option, we can define C ij with the ground-truth labels of two samples:
[Equation]
where d Lab ( · , · ) also denotes a distance measure, and y train i , y meta j are the ground-truth label vectors ofthe two samples, respectively.
Intuitively, if we use the euclidean distance, then C is a 0 − 1 matrix(we can transfer the non-zero constant to 1 ), i.e. , C ij =0 if x train i and x meta j are from the same class,and C ij =1 otherwise.
Now the OT loss is influenced by neither feature extractor θ 1 nor classifier θ 2 .
Feature-aware Cost Besides
, we can define C ij purely based on the features of samples:
[Equation]
where z train i = f 1 ( x train i ; θ 1 ) ∈ R E and z meta j = f
[Equation]
x
[Equation]
; θ 1 ) ∈ R E denote the E -dimensionalrepresentation of x train i and x meta j , respectively.
d Fea ( · , · ) denotes any commonly used distance measureand we empirically find the cosine distance is a good choice.
It is easy to see that if x train i and x meta j ’sfeatures are close, their cost is small.
Here the OT loss is influenced by the feature extractor θ 1 .
Combined Cost
Finally, we can use both features and labels to define C ij , denoted as
[Equation]
Intuitively, C ij will be small if two samples have the same label and similar features.
Empirically,we find that using the d Fea =1 − cosine( · , · ) and euclidean distance for d Lab gives better performance.
Interestingly, given the feature-aware cost (8) or label-aware cost (7), the learned weight vector can beinterpreted as the instance-level or class-level re-weighting method, respectively.
The weight vectorlearned from the combined cost can be interpreted as the combination of class-level and instance-levelweights, although no specialized design for two-component weights like previous [4]; see Fig.
3.3 Learn the Weight Vector
Given the defined cost function, we adopt the entropy regularized OT loss [21] to learn the weightvector.
We thus rewrite (6) as the following optimization problem:
[Equation]
where λ > 0 is a hyper-parameter for the entropic constraint H ( T )
= − (cid:80) ij T ij ln T ij .
Note that 151 (10) provides us a new perspective to interpret the relationship between w and T , where w is theparameter of the leader problem and T is the parameter of the follower problem, which is of the lowerpriority.
Accordingly, when we minimize (10) with respect to w using gradient descent, we shoulddifferentiate through T .
Below we investigate the following two ways to optimize the weight vector.
Optimizing w directly Specifically, at each training iteration, we define P ( w ) with current w , usethe Sinkhorn algorithm [21] to compute OT loss, then optimize w by w ∗ =
arg min w L OT .
Amortizing the learning of w We also provide an alternative method by constructing an explicitweight net to output the example weights, whose structure can be designed flexibly.
For example, wecan build the following weight net and take the sample features as input:
[Equation]
where s i is the i -th element of s ∈ R N ,
w att ∈ R 1 × A and W
vz ∈ R
A × E are the learned parameters(we omit the bias for convenience), denoted as Ω = { w att , W vz } .
Denote S ( z ; Ω ) as the weightnet parameterized by Ω , which can be optimized by Ω ∗ =
arg min Ω L OT .
Overall Algorithm and Implementations
To integrate our proposed method with deep learning frameworks, we adopt a stochastic setting,i.e. , a mini-batch setting at each iteration.
Following [4, 5], we adopt two-stage learning, wherestage 1 trains the model f ( θ ) by the standard cross-entropy loss on the imbalanced training setand stage 2 aims to learn the weight vector w and meanwhile continue to update the model f
( θ ) .
Generally, at stage 2, calculating the optimal θ and w requires two nested loops of optimization,
Require: Datasets D train , D meta , initial model parameter θ and weight vector, hyper-parameters { α,β,λ } for t = 1 , 2 ,...,t 1 do
Sample a mini-batch B from the training set D train ;
Update θ ( t +1) ← θ ( t ) − α ∇ θ L B where L B = 1 | B | (cid:80) i ∈ B ℓ (cid:16)
y i ,f (cid:16) x i ; θ ( t ) (cid:17)(cid:17) ; end for for t = t 1 + 1 ,...,t 1 + t 2 do Sample a mini-batch B from the training set D train ;
Step (a): Update ˆ θ ( t +1) ( w ( t ) )
← θ ( t ) − α ∇ θ L B where L B = 1 | B | (cid:80) i ∈
B w ( t ) i ℓ (cid:16) y
i ,f (cid:16) x i ; θ ( t ) (cid:17)(cid:17)
Use D meta to build Q in (12) and B with w t to build P ( w t ) (4);
Step (b): Compute L OT (cid:16) ˆ θ ( t +1)( w t ) , w ( t ) (cid:17) with cost (9); Optimize w ( t +1) ← w ( t ) −
β ∇ w L OT (cid:16) ˆ θ ( t +1)( w t ) , w ( t ) (cid:17)
Step (c): Update θ ( t +1) ← θ ( t ) − α ∇ θ L B where L B = 1 | B | (cid:80) i ∈
B w ( t +1)
i ℓ (cid:16)
y i ,f (cid:16) x i ; θ ( t ) (cid:17)(cid:17) end for which is cost-expensive.
Motivated by Hu et al.
[1], we optimize θ and w alternatively, correspondingto (1) and (10) respectively, where w is maintained and updated throughout the training, so thatre-estimation from scratch can be avoided in each iteration.
The implementation process of ourproposed method with w optimized directly is shown in Algorithm 1, where the key steps arehighlighted in Step (a), (b), and (c).
Specifically, at each training iteration t , in Step (a), we haveˆ θ ( t +1) ( w t ) =
{ ˆ θ ( t +1)( w t ) , ˆ θ ( t +1)( w t ) } and α is the step size for θ ; in Step (b), as the cost functionbased on features is related with ˆ θ ( t +1)( w t ) , the OT loss relies on ˆ θ ( t +1)( w t ) , and β is the step sizefor w ; in Step (c), we ameliorate model parameters θ ( t +1) .
We defer the learning of θ and Ω for theamortized learning of w to Appendix A. Discussion From Step (b), we find the gradient of w is unrelated to classifier θ 2 regardless of whichcost function we choose. If we use the label-aware cost or freeze the feature extractor parameterizedby θ 1 , which is trained in the first stage, the OT loss in Step (b) can be further reduced as L OT ( w t ) ,where we only need Steps (b)-(c) at each iteration. This is different from most of automatic reweighting methods, where the gradient of w is always related with the to-be-learned model { θ 1 , θ 2 }or classifier θ 2 (when freezing θ 1 ) for minimizing the classification loss on meta set.
Prototype-oriented OT loss (POT) Recall that we represent a balanced meta set with M samplesas distribution Q in (5), where M/K is the number of data in each class and usually larger than 1.
Computing the OT loss requires to learn a B × M -dimensional transport matrix at each iteration.
Toimprove the efficiency of algorithm, we average all samples from each class in the meta set to achieveits prototype and propose a new Q distribution over K prototypes:
[Equation]
where POT loss only needs a B × K -dimensional transport matrix.
Due to the robustness of ourmethod to Q , when dealing with a large number of classes, we can randomly sample a mini-batchfrom K prototypes at each iteration to build Q .
Related Work
Empirical Re-weighting A classic empirical re-weighting scheme is to provide the examples of eachclass with the same weight, such as inverse class frequency [11, 14].
It has been further improved bythe class-balanced loss [13], which calculates the effective number of examples as class frequency.
Focal Loss [12] uses the predicted probability to calculate higher weights for the hard examples anddynamically adjust the weights.
LDAM-DRW [17] designs a label-distribution-aware loss functionand adopts a deferred class-level re-weighting method ( i.e. , inverse class frequency).
Automatic Re-weighting The automatic re-weighting methods learn the weights with learningmechanisms. L2RW [15] adopts a meta-learning manner to learn the example weights, which areoptimized by the classification loss on the balanced meta set. Hu et al. [1] further improve L2RWby iteratively optimizing weights instead of re-estimation at each iteration.
Meta-weight-net [16]aims to learn an explicit weight net directly from data and optimize it by a meta-learning manner.
Meta-class-weight [4] defines the weight for each example as the combination of class-level weight(estimated by Cui et al.
[13]) and instance-level weight, optimized with a meta-learning approachsimilar to L2RW.
Recently, Liu et al.
[2] propose to update the weights and model under a constraint.
Our method belongs to automatic re-weighting group, and the idea of building an explicit weightnet is similar to Shu et al.
However, the major difference is that we bypass the classificationloss on the meta set and use OT to learn the weights from the view of distribution approximation,disengaging the dependence of the weight learning on the concerned classifier at each iteration.
Meta Learning and Two-stage Learning Recently, researchers have proposed to tackle the imbalance issue with meta-learning, which can be applied to learn weights (as discussed above) [4, 15, 16]or transformed semantic directions for augmenting the minority classes in MetaSAug [5].
Two-stagemethods, where the first stage and second stage mainly focus on representation learning and classifierlearning, respectively, have been proved effective for solving the imbalanced issue [5, 6, 16, 22].
BBN [23] unifies two stages with a specific cumulative learning strategy.
Optimal Transport To the best of our knowledge, the works that solve imbalanced issue with OTare still very limited.
An oversampling method via OT (OTOS) [24] aims to make synthetic samplesfollow a similar distribution to that of minority class samples.
However, ours is a novel re-weightingmethod based on OT, without augmenting samples.
Another recent work is Optimal Transport via
Linear Mapping (OTLM) [19], which performs the post-hoc correction from the OT perspective andproposes a linear mapping to replace the original exact cost matrix in OT problem.
Different from
OTLM that belongs to the post-hoc correction group and aims to learn refined prediction matrix, oursfalls into the training-aware group and aims to re-weight the training classification loss.
Experiments
We conduct extensive experiments to validate the effectiveness of our proposed method on text,image, and point cloud imbalanced classification tasks.
Notably, different from the imbalanced imageand point cloud classification, we find that optimizing the weight net is better than optimizing theweight vector directly in the text classification.
Therefore, we optimize the weight vector for theimage and point cloud cases and build a weight net for text case.
Unless specified otherwise, weadopt the combined cost and set the hyper-parameter for the entropic constraint as λ = 0 .
1 and themaximum iteration number in the Sinkhorn algorithm as 200 .
We define the imbalance factor (IF) ofa dataset as the data point amount ratio between the largest and smallest classes.
6.1 Experiments on Imbalanced Image Classification
Datasets and Baselines We evaluate our method on CIFAR-LT-10, CIFAR-LT-100, ImageNet-LTand Places-LT.
We create CIFAR-LT-10 ( CIFAR-LT-100 ) from CIFAR-10 (CIFAR-100)[25] bydownsampling samples per class with IF ∈{ 200 , 100 , 50 , 20 } [5, 13].
ImageNet-LT is built from theclassic ImageNet with 1000 classes[26] and IF =1280 / 5 [5, 22].
Places-LT is created from Places-2[27] with 365 classes and IF =4980 / 5 [4, 22].
We randomly select 10 training images per class asmeta set [5]; see more details in Appendix B. We consider the following baselines: (1) Cross-entropy(CE) , the model trained on the imbalanced training set with CE loss.
(2) Empirical re-weightingmethods , like Focal loss [12], Class-balanced (CB) loss [13] and LDAM-DRW [17].
(3) Automaticre-weighting methods , including L2RW [15], Meta-Weight-Net [16] and Meta-class-weight [4].
Meta-learning methods , including MetaSAug [5] and above methods of [4, 15, 16].
(5) Two-stagemethods , such as OLTR [22], cRT [6], LWS [6], BBN [23] and methods of [4, 5].
Experimental details and results on CIFAR-LT For a fair comparison, we use ResNet-32 [28] asthe backbone on CIFAR-LT-10 and CIFAR-LT-100.
Following Li et al.
[5], at stage 1, we use 200epochs, set the learning rate α of θ as 0.1, which is decayed by 1 e − 2 at the 160th and 180th epochs.
At stage 2, we use 40 epochs, set α as 2 e − 5 and learning rate β of weights as 1 e − 3 .
We use the SGDoptimizer with momentum 0.9, weight decay 5 e − 4 and set the batch size as 16 .
We list the recognitionresults of different methods on CIFAR-LT-10 and CIFAR-LT-100 with different imbalance factors in
Table 1.We report the average result of 5 random experiments without standard deviation which is ofsmall scale(e.g., 1e-2).
We can see that our re-weighting method outperforms CE training by a largemargin and performs better than the empirical or automatic re-weighting methods.
Remarkably, ourproposed method outperforms competing MetaSAug that conducts a meta semantic augmentationapproach to learn appropriate class-wise covariance matrices when IF is 200, 100 and 50.
Importantly,as the training data becomes more imbalanced, our method is more advantageous.
Even thoughour proposed method is inferior to MetaSAug when the dataset is less imbalanced (IF =20 ), it canstill achieve competing results and surpasses related re-weighting methods.
This suggests that ourproposed method can be used to enhance the imbalanced classification, without the requirement ofdesigning complicated models or augmenting samples on purpose.
To more comprehensively understand our method, we provide a series of ablation studies on CIFARLT-100 with IF = 200 in Table 2.
Firstly, to explore the impact of cost function, we use differentcost functions for the OT loss.
We can see that the combined cost performs better than label-awarecost and feature-aware cost, confirming the validity of combining features and labels to define cost.
Besides, using either label-aware or feature-aware cost can still achieve acceptable performance,indicating the usefulness of OT loss in the imbalanced issue.
Secondly, to explore the robustness ofthe meta distribution Q , we adopt three ways to build Q : (1) using prototypes defined in Eq.
(12) ( Ksamples) ; (2) using all samples defined in Eq.
(5) ( 10 ∗ K samples) ; (3) randomly sampling onepoint from each class ( K samples) in meta set.
We find that prototype-based meta performs best, andthe performance with random-sample meta or whole meta is still competitive, which demonstratesthe robustness of our proposed method to the distribution Q and the benefit of using the prototypes tobuild Q .
Third, we compare two ways for learning w in each iteration, where one is re-estimating wfrom scratch and another one is maintaining and updating w throughout the training ( i.e. , iterativelyoptimizing weights).
We find that iteratively optimizing performs better.
[Table] [Table]
47 BBN
[Table]
[Table]
[Table] [Table] [Table] Since cost function is essential in optimizing the OT loss, we are interested in examining the learnedweight vectors given by different cost functions.
Here, we use CIFAR-LT-10, randomly choose{ 10 , 9 , ..., 1 } training samples from class { 1 , 2 , ..., 10 } and obtain 55 samples, which are used tobuild distribution P .
Besides, the 10 prototypes from meta set are used to build the distribution Q .
Given the different cost functions, we show the learned weight vectors of 55 training samples in Fig.1, which have very different properties.
Specifically, the label-aware cost and feature-aware cost leadto class-level weights and sample-level weights, respectively.
It is reasonable that label-aware costonly decides whether the two samples (from the meta set and training set) belong to the same class,resulting in class-level measure.
However, feature-aware cost measures the distance between samplesfrom the sample-level, where each sample has its own feature.
More interestingly, the learned weightswith the combined cost own the characteristics of class-level and sample-level weights simultaneously,where example weights of different classes are far away and example weights of the same class areclose.
Coincidentally, using the combined cost to define the OT loss can reach the same goal of [4],which explicitly considers class-level and sample-level weight.
Besides, we find that the learnedexample weights of the minority class are usually more prominent than those of the majority classes.
[Table]
[Table] [Table]
To verify whether our method ameliorates the performance on minority classes we plot the confusionmatrices of CE, MetaSAug and ours on CIFAR-LT-10 with IF =200 in Fig.
As expected, although
CE training can almost perfectly classify the samples in majority classes, it suffers severe performancedegeneration in the minority classes.
MetaSAug improves the accuracies of the minority classes,where is still a big gap between the performance on the minority classes and the majority classes.
Incontrast, ours does not show a very clear preference for a certain class and outperforms the strongbaseline on the overall performance, which is the goal of on an imbalanced classification task.
Experimental details and results on Places-LT and ImageNet-LT Following [6], we employ
ResNet-152 pre-trained on the full ImageNet as the backbone on Places-LT.
For stage 1, we set theinitial learning rate as 0.01, which is decayed by 1 e − 1 every 10 epochs.
In the stage 2 of our method,we only fine-tune the last fully-connected layer for training efficiency and set α as 1 e − 4 and β as1 e − 3 within 50 epochs.
The mini-batch size is 32 and the optimizer is SGD with momentum 0 .
9and weight decay 5 e − 3 .
As shown in Table 3, our proposed method outperforms all baselines.
Itfurther suggests that our method has excellent performance in the extreme imbalance setting with IF =4980 / 5 .
For a fair comparison, we implement our method on ImageNet-LT with the same experimentalconditions of [5], from which we have taken the results of other comparison methods.
We consider ResNet-50 [28] as the backbone on ImageNet-LT.
In stage 1, we run 200 epochs and decay thelearning rate by 0.1 at the 60th and 80th epochs.
In stage 2, we implement our method for 50 epochs,set learning rate α as 2 e − 5 and β as 1 e − 2 , and only fine-tune the last fully-connected layer for trainingefficiency.
We use the SGD optimizer with momentum 0.9, weight decay 5 e − 4 and set the batch sizeas 128 .
The results on ImageNet-LT of different models reported in Table 4 indicates the effectivenessof our proposed method on ImageNet-LT when comparing with strong baseline MetaSAug.
Besides,we further consider randomly sampling a mini-batch of size 100 from all prototypes at each iterationto build Q , whose performance is comparable to the Q from all prototypes.
Thus, with a stochasticsetting for Q , our proposed method can be used to the imbalanced training set with a large number ofclasses.
We defer the time computational complexity, additional quantitative results and qualitativeresults on different image datasets to Appendix B.
6.2 Experiments on Imbalanced Text Classification
Datasets and settings Following [1, 2], we adopt the popular SST-2 for 2-class and SST-5 for 5-classsentence sentiment [29].
For a fair comparison, we use the same imbalanced datasets and settingswith [2].
Specifically, we set class 1 as the minority class and the rest as the majority classes, wherethe number of examples in the majority class is fixed as 1000 (SST-2) and 500 (SST-5) and we achieve different imbalance settings by varying the number of examples in the minority class.
Besides, thenumber of samples in the meta set is 10 for each class.
We use the BERT (base, uncased) model[30] as feature extractor and a simple 3-layer fully-connected network (FCN) with the structure in
Appendix C as classifier.
To make subsequent experiments on strong models, following [2], we usean additional balanced training set (500 samples in each class) to fine-tune the BERT model, which israndomly selected from the remaining examples in each dataset except the imbalanced training set,meta set and to-be evaluated test set.
Based on the fine-tuned BERT, we adopt the two-stage mannerfor the imbalanced text datasets, where we train the BERT + FCN in the first stage with CE loss andtrain the FCN with our proposed method by freezing the BERT in the second stage.
The settings ofthe training process are deferred to Appendix C.
Baselines We consider the following methods: (1) vanilla BERT , the vanilla pretrained languagemodel.
(2) Fine-tuned BERT , where the pretrained BERT is fine-tuned on an additional balancedtraining set.
(3) Fine-tuned BERT + CE , the fine-tuned BERT model followed by the FCN whichis further trained by the CE loss on the imbalanced training set following [1, 2].
(4) Automaticre-weighting methods , including the method of Hu et al.
[1] and constraint-based re-weighting[2].
Since few works consider imbalanced text classification, we further consider (5) Empirical reweighting methods , including re-weighting with inverse class frequency ( i.e. , Proportion) [11, 14])and LDAM-DRW [17] and (6) Logit adjustment [18] using their official codes and settings 1 2 .
Werepeat all experiments 10 times and report the mean and standard deviation.
Experimental details and results on SST-2 and SST-5 We report the text classification resultsof compared methods under different imbalance factors in Table 5.
We find that our proposedmethod outperforms all competing methods in all imbalance factor settings, which demonstrates theeffectiveness of our proposed method.
Although all methods could achieve acceptable performancein a slight imbalance, the performance of three baselines (Vanilla BERT, Fine-Tuned BERT and FineTuned BERT+CE) drop dramatically, indicating the importance of proposing specialized methods forhandling imbalanced training datasets.
Logit adjustment (post-hoc correction), is very competitive toours on SST-2, which, however, only produces similar results to the three above-mentioned baselineson SST-5.
In contrast, ours is robust to not only the imbalance factors but also the number of classes,where the results are consistent with the image case.
We provide more results in Appendix C.4.
[Table] In addition to 1D text and 2D image, we further investigate the robustness of our method on 3D pointcloud data, where we use the popular ModelNet10 [31] and defer the experiments to Appendix D.
Conclusion
This paper introduces a novel automatic re-weighting method for imbalance classification basedon optimal transport (OT).
This method presents the imbalanced training set as a to-be-learneddistribution over its training examples, each of which is associated with a probability weight.
Similarly,our method views another balanced meta set as a balanced distribution over the examples.
Byminimizing the OT distance between the two distributions in terms of the defined cost function,the learning of weight vector is formulated as a distribution approximation problem.
Our proposedre-weighting method bypasses the commonly-used classification loss on the meta set and uses OT tolearn the weights, disengaging the dependence of the weight learning on the concerned classifier ateach iteration.
This is an approach different from most of the existing re-weighting methods and mayprovide new thoughts for future work.
Experimental results on a variety of imbalanced datasets ofboth images and texts validate the effectiveness and flexibility of our proposed method
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Section 6.1.
When handlingimbalanced datasets of image classification, our proposed method is inferior to some
competitive baseline method when the dataset is less imbalanced.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] SeeAppendix D.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes] We’ve read the guidelines and make sure our submission adheres to the
ethical standards.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A](b)
Did you include complete proofs of all theoretical results?
[N/A]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] Code will be
submitted in the supplemental material.(b)
Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See Section 6.
(c) Did you report error bars (e g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] In Section 6.1, we report the average result of 5 random image experiments and omit standard deviation for the limited space, which is usually
small (e.g., 1e-2).
In Section 6.2, we repeat all text experiments 10 times and report the
mean and standard deviation.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix B.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]