Learning to Re-weight Examples with Optimal Transport for Imbalanced Classification
Abstract
Imbalanced data pose challenges for deep learning based classification models.
One of the most widely-used approaches for tackling imbalanced data is re-weighting, where training samples are associated with different weights in the loss function.
Most of existing re-weighting approaches treat the example weights as the learnable parameter and optimize the weights on the meta set, entailing expensive bilevel optimization.
In this paper, we propose a novel re-weighting method based on optimal transport (OT) from a distributional point of view.
Specifically, we view the training set as an imbalanced distribution over its samples, which is transported by OT to a balanced distribution obtained from the meta set.
The weights of the training samples are the probability mass of the imbalanced distribution and learned by minimizing the OT distance between the two distributions.
Compared with existing methods, our proposed one disengages the dependence of the weight learning on the concerned classifier at each iteration.
Experiments on image, text and point cloud datasets demonstrate that our proposed re-weighting method has excellent performance, achieving state-of-the-art results in many cases and providing a promising tool for addressing the imbalanced classification issue.
Introduction
Deep neural networks (DNNs) have achieved remarkable success in various applications, which is undoubtedly inseparable from the high-quality large-scale datasets.
Usually, the number of samples for each class in these datasets are manually selected resulting in balanced datasets.
However, most real-world datasets are imbalanced, such as a few classes (a.k.a. head or majority class) occupy most of the data while most classes (a.k.a.
tail or minority class) have a few samples.
A model trained on the imbalanced training set but without considering such class imbalance would be significantly dominated by those majority classes, and thus underperform on a balanced test dataset.
This can also be known as the long-tailed problem and exists in many domains, such as text classification [1, 2], object detection [3] and image classification [4–6].
There are rich research lines to solve the imbalance problem, including re-sampling [7–10], class-level or instance-level re-weighting [1, 2, 4, 11–18], meta-learning [4, 5, 15, 16, 19], two-stage methods [4–6, 17] and post-hoc correction [20, 21].
Inspired by [2], re-weighting strategies can be roughly grouped into empirical re-weighting and automatic re-weighting.
The former aims to design weights manually with the major insight that the minority class example will be assigned a larger weight value than that of the majority class [12–14].
However, manually setting weights can be less adaptive to different datasets [2].
The latter aims to assign adaptive weights to the examples through learning mechanisms [1, 2, 4, 15, 16].
As the representative automatic re-weighting method, L2RW [15] optimizes the weight vector as a learnable parameter with an unbiased meta set ( i.e. , validation set).
Although L2RW and its followers have received widespread attention, most of them may be limited to optimizing the weights by the classification loss on the meta set: The gradient of weights is usually coupled with the to-be-learned classifier at each training iteration.
Since classifier is the major concern in imbalanced issue [6], the dependence of weights on classifier at training stage may lead to inaccurate learning of the weights.
This paper develops a novel automatic re-weighting method for imbalanced classification based on optimal transport (OT).
As discussed by Jamal et al.
[4], the major challenge for imbalanced classification is essentially the mismatch between the imbalanced training dataset (seen by a machine learning model) and the balanced test set (used to test the learned model).
To this end, we aim to view the learning of the weight vector as the distribution approximation problem.
We adopt the two-stage learning manner motivated by [6], where stage 1 and stage 2 focus on learning the feature extractor with the standard cross-entropy loss and the classifier with our proposed method, respectively.
Specifically, we represent the imbalanced training set as a discrete empirical distribution
P over all samples within it and view the to-be-learned weight vector w as its probability measure.
Then we represent the balanced meta set as a discrete empirical distribution Q over all samples within it (in the same space with P ), which has a uniform probability measure for being balanced.
Therefore, the learning of a weight vector can be formulated as the process of learning the distribution
P to be as close to the balanced distribution Q as possible, a process facilitated by leveraging the
OT distance [22].
Notably, the cost function plays a paramount role when learning the transport plan for OT, where we use the features and ground-truth labels of samples to design it.
Due to the flexibility of our method, we can also learn an explicit weight net directly from data like [16, 23] but with a different structure, optimized by OT loss instead of the classification loss on the meta set.
Generally, at each training iteration at stage 2, we minimize the OT loss to learn the weight vector (or weight net) for the current mini-batch, which is further used to re-weight the training loss for optimizing the model.
As we can see, the gradient of weights only relies on the OT loss and thus is independent of the classifier.
More importantly, our proposed method is robust to the distribution
Q .
To save the memory consumption, we introduce the prototype-oriented OT loss by building a new distribution Q based on prototypes instead of samples (one prototype for each class).
More importantly, our proposed method can achieve a reasonably good performance even if we randomly select a mini-batch from all prototypes to build Q , making our method applicable to datasets with a large number of classes.
We summarize our main contributions as follows: (1) We formulate the learning of weight vector or weight net as the distribution approximation problem by minimizing the statistical distance between to-be-learned distribution over samples from imbalanced training set and another balanced distribution over samples from the meta set.
(2) We leverage the OT distance between the distributions to guide the learning of weight vector or weight net.
(3) We apply our method to imbalanced classification tasks including image, text and point cloud.
Experiments demonstrate that introducing the OT loss to learn the example weights can produce effective and efficient classification performance.
Related Work
Empirical Re-weighting A classic empirical re-weighting scheme is to provide the examples of each class with the same weight, such as inverse class frequency [11, 14].
It has been further improved by the class-balanced loss [13], which calculates the effective number of examples as class frequency.
Focal Loss [12] uses the predicted probability to calculate higher weights for the hard examples and dynamically adjust the weights.
LDAM-DRW [17] designs a label-distribution-aware loss function and adopts a deferred class-level re-weighting method ( i.e. , inverse class frequency).
Automatic Re-weighting The automatic re-weighting methods learn the weights with learning mechanisms.
L2RW [15] adopts a meta-learning manner to learn the example weights, which are optimized by the classification loss on the balanced meta set.
Hu et al.
[1] further improve L2RW by iteratively optimizing weights instead of re-estimation at each iteration.
Meta-weight-net [16] aims to learn an explicit weight net directly from data and optimize it by a meta-learning manner.
Meta-class-weight [4] defines the weight for each example as the combination of class-level weight (estimated by Cui et al.
[13]) and instance-level weight, optimized with a meta-learning approach similar to L2RW.
Influence-balanced loss (IB) is proposed to [18] re-weight samples by the magnitude of the gradient.
Recently, Liu et al.
[2] propose to update the weights and model under a constraint.
Our method belongs to automatic re-weighting group, and the idea of building an explicit weight net is similar to Shu et al.
However, the major difference is that we bypass the classification loss on the meta set and use OT to learn the weights from the view of distribution approximation, disengaging the dependence of the weight learning on the concerned classifier at each iteration.
Meta Learning and Two-stage Learning Recently, researchers have proposed to tackle the imbalance issue with meta-learning, which can be applied to build a Balanced Meta-Softmax (BALMS)[19], learn weights [4, 15, 16] or transformed semantic directions for augmenting the minority classes in
MetaSAug [5].
Two-stage methods, where the first stage and second stage focus on representation learning and classifier learning, respectively, have been proved effective for solving the imbalanced issue [5, 6, 16, 24].
BBN [25] unifies two stages with a specific cumulative learning strategy.
Optimal Transport Recently, OT has been used to solve the regression problem under the covariate shift [26], unsupervised domain adaption [27, 28], including sample-level, class-level or domain-level weight vector.
Although they also adopt the re-weighting strategy and OT distance, they are distinct form ours in terms of task and technical detail.
Also, the dynamic importance weighting which adopts MMD to re-weight samples for label-noise and class-prior-shift tasks [29] is also different from ours, where we provide a more flexible way for learning the weights of samples and disengage the dependence of the weight learning on the concerned classifier at each iteration.
To the best of our knowledge, the works that solve imbalanced classification problem with OT are still very limited.
An oversampling method via OT (OTOS) [30] aims to make synthetic samples follow a similar distribution to that of minority class samples.
However, ours is a novel re-weighting method based on
OT, without augmenting samples.
Another recent work is Optimal Transport via Linear Mapping (OTLM) [21], which performs the post-hoc correction from the OT perspective and proposes a linear mapping to replace the original exact cost matrix in OT problem.
Different from OTLM that belongs to the post-hoc correction group and aims to learn refined prediction matrix, ours falls into the training-aware group and aims to re-weight the training classification loss.
Background
Imbalanced Classification Consider a training set D train =
{ ( x i , y i ) } Ni =1 , where ( x, y ) is the input and target pair, x i the i -th sample, y i ∈ (0 , 1) K the one-hot associated label vector over K classes, and N the number of the entire training data.
Besides, consider a small balanced meta set D meta = { ( x j , y j ) } Mj =1 , where M is the amount of total samples and M ≪ N .
Denote the model parameterized with θ as f ( x, θ ) , where θ is usually optimized by empirical risk minimization over the training set, i.e. , θ ∗ =
arg min θ 1 N (cid:80) Ni =1 ℓ ( y i , f ( x i ; θ )) .
For notational convenience, we denote l train i ( θ ) =
ℓ ( y i , f ( x i ; θ )) to represent the training loss function of pair ( x i , y i ) .
However, the model trained by this method will prefer the majority class if the training dataset is imbalanced.
Learning to Re-Weight Examples To solve the imbalanced issue, a kind of re-weighting methods is to treat the weights as the learnable parameter and learn a fair model to the minority and the majority classes by optimizing the weighted training loss.
At each training iteration, the model is updated by
[Equation]
where w = ( w 1 , . . .
, w N ) T is the weight vector (usually with a simplex constraint) of all training examples.
Then the optimal w is obtained by making the model parameter θ ∗ ( w ) from Eq.
(1) minimize the classification loss on a balanced meta set, formulated as
[Equation]
where l meta j is the loss function of pair ( x j , y j ) from meta set and the updated w ∗ is used to ameliorate the model.
Generally, model θ consists of two key components, feature extractor and classifier, where the classifier has been proved to be the major concerning part in imbalanced issue [6].
However, the gradient of weights in Eq.
(2) always depends on the to-be-concerned classifier at each training iteration, which may result in inaccurate learning of the weights.
Most automatic re-weighting methods learn the weight vectors or weight-related parameters ( e.g. , weight net) following this line; see more details from the previous works [4, 15, 16].
Optimal Transport Theory OT has been widely used to calculate the cost of transporting one probability measure to another in various machine learning problems, such as generative models [31], text analysis [32, 33], adversarial robustness [34], and meta learning [35, 36].
Among the rich theory of OT, this work presents a brief introduction to OT for discrete distributions; see Peyré and Cuturi [22] for more details.
Consider p = (cid:80) ni
=1 a i δ x i and q = (cid:80) mj
=1 b j δ y
j
as two probability distributions, where x i and y j live in the arbitrary same space and δ is the Dirac function.
Then, we can denote a ∈ ∆ n and b ∈ ∆ m as the probability simplex of R n and R m , respectively.
The OT distance between p and q can be expressed as:
[Equation]
where ⟨· , ·⟩ is the Frobenius dot-product and C ∈ R n × m ≥is the transport cost matrix constructed by C ij =
C ( x i , y j ) .
The transport probability matrix T ∈ R n × m > 0 , which satisfies Π( p, q ) :
= { T | (cid:80) ni
=1 T ij =
b j , (cid:80) mj
=1 T ij = a i } , is learned by minimizing OT ( p, q ) .
Directly optimizing Eq.
(3) often comes at the cost of heavy computational demands, and OT with entropic regularization is introduced to allow the optimization at small computational cost in sufficient smoothness [37].
Re-weighting Method with Optimal Transport
This work views a training set as a to-be-learned distribution, whose probability measure is set as learnable weight vector w .
We use OT distance to optimize w for re-weighting the training loss.
4.1 Main Objective
Given the imbalanced training set D train , we can represent it as an empirical distribution over N pairs, where each pair ( x i , y i ) train has the sample probability w i ( i.e. , the weight), defined as:
[Equation]
[Equation]
where ( x i , y i ) train is the i -th pair from the training set and the learnable weight vector w of all training examples means probability simplex of R N .
Since the meta set D meta is balanced for all classes and closely related with the training set, it is reasonable to assume that meta set has already achieved the balanced data distribution that the training set aims to approximate.
For meta set, we thus can sample each pair from it with equal probability and present it with an empirical distribution Q :
[Equation]
where ( x j , y j ) meta is the j -th pair from the meta set.
To learn w , different from most automatic re-weighting methods, which minimize the classification loss on the meta set, we aim to enforce the to-be-learned distribution P ( w ) to stay close to the balanced distribution Q .
Here, we explore the re-weighting method by adopting the OT distance between P ( w ) and Q :
[Equation]
where cost matrix C ∈ R N × M ≥is described below and transport probability matrix T ∈ R N × M >should satisfy Π( P ( w ) , Q ) :
= { T | [Equation]
[Equation]
4.2 Cost Function
For notation convenience, we reformulate the model as f ( x, θ ) =
f 2 ( f 1 ( x ; θ 1 ); θ 2 ) , where f 1 parameterized with θ 1 denotes the representation learning part before the classifier, and f 2 parameterized with θ 2 denotes the classifier.
Intuitively, the cost C ij measures the distance between pair i in training set and pair j in meta set, which can be flexibly defined in different ways.
We explore a few conceptually intuitive options of C ij , although other reasonable choices can also be used.
Label-aware Cost
As the first option, we can define C ij with the ground-truth labels of two samples:
[Equation]
where d Lab ( · , · ) also denotes a distance measure, and y train i , y meta j are the ground-truth label vectors of the two samples, respectively.
Intuitively, if we use the euclidean distance, then C is a 0 − 1 matrix (we can transfer the non-zero constant to 1 ), i.e. , C ij {+=
+} if x train i and x meta j are from the same class, and C ij =1 otherwise.
Now the OT loss is influenced by neither feature extractor θ 1 nor classifier θ 2 .
Feature-aware Cost Besides
, we can define C ij purely based on the features of samples:
[Equation]
where z train i =
[Equation]
; θ [Equation] ) ∈ R E and z
[Equation] [Equation]
; θ 1 ) ∈ R E denote the E -dimensional representation of x train i and x meta j , respectively.
d Fea ( · , · ) denotes any commonly used distance measure and we empirically find the cosine distance is a good choice.
It is easy to see that if x train i and x meta j ’s features are close, their cost is small.
Here the OT loss is influenced by the feature extractor θ 1 .
Combined Cost
Finally, we can use both features and labels to define C ij , denoted as
[Equation]
Intuitively, C ij will be small if two samples have the same label and similar features.
Empirically, we find that using the d Fea =1 − cosine( · , · ) and euclidean distance for d Lab gives better performance.
Interestingly, given the feature-aware cost (8) or label-aware cost (7), the learned weight vector can be interpreted as the instance-level or class-level re-weighting method, respectively.
The weight vector learned from the combined cost can be interpreted as the combination of class-level and instance-level weights, although no specialized design for two-component weights like previous [4]; see Fig. 1.
4.3 Learn the Weight Vector
Given the defined cost function, we adopt the entropy regularized OT loss [37] to learn the weight vector.
We thus rewrite (6) as the following optimization problem:
[Equation]
where λ > 0 is a hyper-parameter for the entropic constraint H ( T )
= − (cid:80) ij T ij ln T ij .
Note that (10) provides us a new perspective to interpret the relationship between w and T , where w is the parameter of the leader problem and T is the parameter of the follower problem, which is of the lower priority.
Accordingly, when we minimize (10) with respect to w using gradient descent, we should differentiate through T .
Below we investigate the following two ways to optimize the weight vector.
Optimizing w directly Specifically, at each training iteration, we define P ( w ) with current w , use the Sinkhorn algorithm [37] to compute OT loss, then optimize w by w ∗ =
arg min w L OT .
Amortizing the learning of w We also provide an alternative method by constructing an explicit weight net to output the example weights, whose structure can be designed flexibly.
For example, we can build the following weight net and take the sample features as input:
[Equation]
where s i is the i -th element of s ∈ R N ,
w att ∈ R 1 × A and W
vz ∈ R
A × E are the learned parameters (we omit the bias for convenience), denoted as Ω = { w att , W vz } .
Denote S ( z ; Ω ) as the weight net parameterized by Ω , which can be optimized by Ω ∗ =
arg min Ω L OT .
Overall Algorithm and Implementations
To integrate our proposed method with deep learning frameworks, we adopt a stochastic setting, i.e. , a mini-batch setting at each iteration.
Following [4, 5], we adopt two-stage learning, where stage 1 trains the model f ( θ ) by the standard cross-entropy loss on the imbalanced training set and stage 2 aims to learn the weight vector w and meanwhile continue to update the model f
( θ ) .
Generally, at stage 2, calculating the optimal θ and w requires two nested loops of optimization, which is cost-expensive.
Motivated by Hu et al.
[1], we optimize θ and w alternatively, corresponding to (1) and (10) respectively, where w is maintained and updated throughout the training, so that re-estimation from scratch can be avoided in each iteration.
The implementation process of our proposed method with w optimized directly is shown in Algorithm 1, where the key steps are highlighted in Step (a), (b), and (c).
Specifically, at each training iteration t , in Step (a), we have ˆ θ ( t +1) ( w t ) =
{ ˆ θ ( t +1)( w t ) , ˆ θ ( t +1)( w t ) } and α is the step size for θ ; in Step (b), as the cost function based on features is related with ˆ θ ( t +1)( w t ) , the OT loss relies on ˆ θ ( t +1)( w t ) , and β is the step size for w ; in Step (c), we ameliorate model parameters θ ( t +1) .
We defer the learning of θ and Ω for the amortized learning of w .
Discussion From Step (b), we find the gradient of w is unrelated to classifier θ 2 regardless of which cost function we choose.
If we use the label-aware cost or freeze the feature extractor parameterized by θ 1 , which is trained in the first stage, the OT loss in Step (b) can be further reduced as L OT ( w t ) ,
Require: Datasets D train , D meta , initial model parameter θ and weight vector, hyper-parameters { α,β,λ } for t = 1 , 2 ,...,t 1 do
Sample a mini-batch B from the training set D train ;
Update θ ( t +1) ← θ ( t ) − α ∇ θ L B where L B = 1 | B | (cid:80) i ∈ B ℓ (cid:16)
y i ,f (cid:16) x i ; θ ( t ) (cid:17)(cid:17) ; end for for t = t 1 + 1 ,...,t 1 + t 2 do Sample a mini-batch B from the training set D train ;
Step (a): Update ˆ θ ( t +1) ( w ( t ) )
← θ ( t ) − α ∇ θ L B where L B = 1 | B | (cid:80) i ∈
B w ( t ) i ℓ (cid:16) y
i ,f (cid:16) x i ; θ ( t ) (cid:17)(cid:17)
Use D meta to build Q in (12) and B with w t to build P ( w t ) (4);
Step (b): Compute L OT (cid:16) ˆ θ ( t +1)( w t ) , w ( t ) (cid:17) with cost (9); Optimize w ( t +1) ← w ( t ) −
β ∇ w L OT (cid:16) ˆ θ ( t +1)( w t ) , w ( t ) (cid:17)
Step (c): Update θ ( t +1) ← θ ( t ) − α ∇ θ L B where L B = 1 | B | (cid:80) i ∈
B w ( t +1)
i ℓ (cid:16)
y i ,f (cid:16) x i ; θ ( t ) (cid:17)(cid:17) end for where we only need Steps (b)-(c) at each iteration.
This is different from most of automatic reweighting methods, where the gradient of w is always related with the to-be-learned model { θ 1 , θ 2 } or classifier θ 2 (when freezing θ 1 ) for minimizing the classification loss on meta set.
Prototype-oriented OT loss (POT) Recall that we represent a balanced meta set with M samples as distribution Q in (5), where M/K is the number of data in each class and usually larger than 1.
Computing the OT loss requires to learn a B × M -dimensional transport matrix at each iteration.
To improve the efficiency of algorithm, we average all samples from each class in the meta set to achieve its prototype and propose a new Q distribution over K prototypes:
[Equation]
where POT loss only needs a B × K -dimensional transport matrix.
Due to the robustness of our method to Q , when dealing with a large number of classes, we can randomly sample a mini-batch from K prototypes at each iteration to build Q .
Experiments
We conduct extensive experiments to validate the effectiveness of our proposed method on text, image, and point cloud imbalanced classification tasks.
Notably, different from the imbalanced image and point cloud classification, we find that optimizing the weight net is better than optimizing the weight vector directly in the text classification.
Therefore, we optimize the weight vector for the image and point cloud cases and build a weight net for text case.
Unless specified otherwise, we adopt the combined cost and set the hyper-parameter for the entropic constraint as λ = 0 .
1 and the maximum iteration number in the Sinkhorn algorithm as 200 .
We define the imbalance factor (IF) of a dataset as the data point amount ratio between the largest and smallest classes.
6.1 Experiments on Imbalanced Image Classification
Datasets and Baselines We evaluate our method on CIFAR-LT-10, CIFAR-LT-100, ImageNet-LT and Places-LT.
We create CIFAR-LT-10 ( CIFAR-LT-100 ) from CIFAR-10 (CIFAR-100)[38] by downsampling samples per class with IF ∈{ 200 , 100 , 50 , 20 } [5, 13].
ImageNet-LT is built from the classic ImageNet with 1000 classes[39] and IF =1280 / 5 [5, 24].
Places-LT is created from Places- [40] with 365 classes and IF =4980 / 5 [4, 24].
We randomly select 10 training images per class as meta set [5]; see more details in Appendix B. We consider the following baselines: (1) Cross-entropy (CE) , the model trained on the imbalanced training set with CE loss.
(2) Empirical re-weighting methods , like Focal loss [12], Class-balanced (CB) loss [13] and LDAM-DRW [17].
(3) Automatic re-weighting methods , including L2RW [15], IB [18], Meta-Weight-Net [16] and Meta-class-weight [4].
(4) Meta-learning methods , including MetaSAug [5] and above methods of [4, 15, 16, 19].
Two-stage methods , such as OLTR [24], cRT [6], LWS [6], BBN [25] and methods of [4, 5].
Experimental details and results on CIFAR-LT For a fair comparison, we use ResNet-32 [41] as the backbone on CIFAR-LT-10 and CIFAR-LT-100.
Following Li et al.
[5], at stage 1, we useepochs, set the learning rate α of θ as 0.1, which is decayed by 1 e − 2 at the 160th and 180th epochs.
At stage 2, we use 40 epochs, set α as 2 e − 5 and learning rate β of weights as 1 e − 3 .
We use the SGD optimizer with momentum 0.9, weight decay 5 e − 4 and set the batch size as 16 .
We list the recognition results of different methods on CIFAR-LT-10 and CIFAR-LT-100 with different imbalance factors in
Table 1.We report the average result of 5 random experiments without standard deviation which is of small scale(e.g., 1e-2).
We can see that our re-weighting method outperforms CE training by a large margin and performs better than the empirical or automatic re-weighting methods.
Remarkably, our proposed method outperforms competing MetaSAug that conducts a meta semantic augmentation approach to learn appropriate class-wise covariance matrices when IF is 200, 100 and 50.
Importantly, as the training data becomes more imbalanced, our method is more advantageous.
Even though our proposed method is inferior to MetaSAug when the dataset is less imbalanced (IF =20 ), it can still achieve competing results and surpasses related re-weighting methods.
This suggests that our proposed method can be used to enhance the imbalanced classification, without the requirement of designing complicated models or augmenting samples on purpose.
To more comprehensively understand our method, we provide a series of ablation studies on CIFARLT-100 with IF = 200 in Table 2.
Firstly, to explore the impact of cost function, we use different cost functions for the OT loss.
We can see that the combined cost performs better than label-aware cost and feature-aware cost, confirming the validity of combining features and labels to define cost.
Besides, using either label-aware or feature-aware cost can still achieve acceptable performance, indicating the usefulness of OT loss in the imbalanced issue.
Secondly, to explore the robustness of the meta distribution Q , we adopt three ways to build Q : (1) using prototypes defined in Eq.
(12) ( K samples) ; (2) using all samples defined in Eq.
(5) ( 10 ∗ K samples) ; (3) randomly sampling one point from each class ( K samples) in meta set.
We find that prototype-based meta performs best, and the performance with random-sample meta or whole meta is still competitive, which demonstrates the robustness of our proposed method to the distribution Q and the benefit of using the prototypes to build Q .
Third, we compare two ways for learning w in each iteration, where one is re-estimating w from scratch and another one is maintaining and updating w throughout the training ( i.e. , iteratively optimizing weights).
We find that iteratively optimizing performs better.
[Table]
IB [18]
[Table]
Since cost function is essential in optimizing the OT loss, we are interested in examining the learned weight vectors given by different cost functions.
Here, we use CIFAR-LT-10, randomly choose { 10 , 9 , ..., 1 } training samples from class { 1 , 2 , ..., 10 } and obtain 55 samples, which are used to build distribution P .
Besides, the 10 prototypes from meta set are used to build the distribution Q .
Given the different cost functions, we show the learned weight vectors of 55 training samples in Fig. 1, which have very different properties.
Specifically, the label-aware cost and feature-aware cost lead to class-level weights and sample-level weights, respectively.
It is reasonable that label-aware cost only decides whether the two samples (from the meta set and training set) belong to the same class, resulting in class-level measure.
However, feature-aware cost measures the distance between samples from the sample-level, where each sample has its own feature.
More interestingly, the learned weights with the combined cost own the characteristics of class-level and sample-level weights simultaneously, where example weights of different classes are far away and example weights of the same class are close.
Coincidentally, using the combined cost to define the OT loss can reach the same goal of [4], which explicitly considers class-level and sample-level weight.
Besides, we find that the learned example weights of the minority class are usually more prominent than those of the majority classes.
[Table]
[Table] [Table]
To verify whether our method ameliorates the performance on minority classes, we plot the confusion matrices of CE, MetaSAug, and ours on CIFAR-LT-10 with IF =200 in Fig.
As expected, although
CE training can almost perfectly classify the samples in majority classes, it suffers severe performance degeneration in the minority classes.
MetaSAug improves the accuracies of the minority classes, where is still a big gap between the performance on the minority classes and the majority classes.
In contrast, ours does not show a very clear preference for a certain class and outperforms the strong baseline on the overall performance, which is the goal of on an imbalanced classification task.
Experimental details and results on Places-LT and ImageNet-LT Following [6], we employ
ResNet-152 pre-trained on the full ImageNet as the backbone on Places-LT.
For stage 1, we set the initial learning rate as 0.01, which is decayed by 1 e − 1 every 10 epochs.
In the stage 2 of our method, we only fine-tune the last fully-connected layer for training efficiency and set α as 1 e − 4 and β as 1 e − 3 within 50 epochs.
The mini-batch size is 32 and the optimizer is SGD with momentum 0 .
9 and weight decay 5 e − 3 .
As shown in Table 3, our method outperforms all baselines.
It further suggests that our method has excellent performance in the extreme imbalance setting with IF =4980 / 5 .
For a fair comparison, we implement our method on ImageNet-LT with the same experimental conditions of [5], from which we have taken the results of other comparison methods.
We consider ResNet- [41] as the backbone on ImageNet-LT.
In stage 1, we run 200 epochs and decay the learning rate by 0.1 at the 60th and 80th epochs.
In stage 2, we implement our method for 50 epochs, set learning rate α as 2 e − 5 and β as 1 e − 2 , and only fine-tune the last fully-connected layer for training efficiency.
We use the SGD optimizer with momentum 0.9, weight decay 5 e − 4 and set the batch size as 128 .
The results on ImageNet-LT of different models reported in Table 4 indicate the effectiveness of our proposed method on ImageNet-LT when comparing with strong baseline MetaSAug.
Besides, we further consider randomly sampling a mini-batch of size 100 from all prototypes at each iteration to build Q , whose performance is comparable to the Q from all prototypes.
Thus, with a stochastic setting for Q , our proposed method can be used to the imbalanced training set with a large number of classes.
We defer the time computational complexity, additional quantitative results and qualitative results on different image datasets to Appendix B.
6.2 Experiments on Imbalanced Text Classification
Datasets and settings Following [1, 2], we adopt the popular SST-2 for 2-class and SST-5 for 5-class sentence sentiment [43].
For a fair comparison, we use the same imbalanced datasets and settings with [2].
Specifically, we set class 1 as the minority class and the rest as the majority classes, where the number of examples in the majority class is fixed as 1000 (SST-2) and 500 (SST-5) and we achieve different imbalance settings by varying the number of examples in the minority class.
Besides, the number of samples in the meta set is 10 for each class.
We use the BERT (base, uncased) model [44] as feature extractor and a simple 3-layer fully-connected network (FCN) with the structure in
Appendix C as classifier.
To make subsequent experiments on strong models, following [2], we use an additional balanced training set (500 samples in each class) to fine-tune the BERT model, which is randomly selected from the remaining examples in each dataset except the imbalanced training set, meta set and to-be evaluated test set.
Based on the fine-tuned BERT, we adopt the two-stage manner for the imbalanced text datasets, where we train the BERT + FCN in the first stage with CE loss and train the FCN with our proposed method by freezing the BERT in the second stage.
The settings of the training process are deferred to Appendix C.
Baselines We consider the following methods: (1) vanilla BERT , the vanilla pretrained language model.
(2) Fine-tuned BERT , where the pretrained BERT is fine-tuned on an additional balanced training set.
(3) Fine-tuned BERT + CE , the fine-tuned BERT model followed by the FCN which is further trained by the CE loss on the imbalanced training set following [1, 2].
(4) Automatic re-weighting methods , including the method of Hu et al.
[1] and constraint-based re-weighting [2].
Since few works consider imbalanced text classification, we further consider (5) Empirical reweighting methods , including re-weighting with inverse class frequency ( i.e. , Proportion) [11, 14]) and LDAM-DRW [17] and (6) Logit adjustment [20] using their official codes and settings 1 2 .
We repeat all experiments 10 times and report the mean and standard deviation.
Experimental details and results on SST-2 and SST-5 We report the text classification results of compared methods under different imbalance factors in Table 5.
We find that our proposed method outperforms all competing methods in all imbalance factor settings, which demonstrates the effectiveness of our proposed method.
Although all methods could achieve acceptable performance in a slight imbalance, the performance of three baselines (Vanilla BERT, Fine-Tuned BERT and FineTuned BERT+CE) drop dramatically, indicating the importance of proposing specialized methods for handling imbalanced training datasets.
Logit adjustment (post-hoc correction), is very competitive to ours on SST-2, which, however, only produces similar results to the three above-mentioned baselines on SST-5.
In contrast, ours is robust to not only the imbalance factors but also the number of classes, where the results are consistent with the image case.
We provide more results in Appendix C.4.
In addition to 1D text and 2D image, we further investigate the robustness of our method on 3D point cloud data, where we use the popular ModelNet10 [45] and defer the experiments to Appendix D [Table]
[Table]
Conclusion
This paper introduces a novel automatic re-weighting method for imbalance classification based on optimal transport (OT).
This method presents the imbalanced training set as a to-be-learned distribution over its training examples, each of which is associated with a probability weight.
Similarly, our method views another balanced meta set as a balanced distribution over the examples.
By minimizing the OT distance between the two distributions in terms of the defined cost function, the learning of weight vector is formulated as a distribution approximation problem.
Our proposed re-weighting method bypasses the commonly-used classification loss on the meta set and uses OT to learn the weights, disengaging the dependence of the weight learning on the concerned classifier at each iteration.
This is an approach different from most of the existing re-weighting methods and may provide new thoughts for future work.
Experimental results on a variety of imbalanced datasets of both images and texts validate the effectiveness and flexibility of our proposed method.
Acknowledgements.
This work is partially supported by a grant from the Shenzhen Science and
Technology Program (JCYJ20210324120011032) and Shenzhen Institute of Artificial Intelligence and Robotics for Society.
A Amortizing the learning of weight vector by introducing a weight net
To integrate our proposed method with deep learning frameworks, we adopt a stochastic setting, i.e. , a mini-batch setting at each iteration.
We adopt two-stage learning, where stage 1 trains the model f ( θ ) by the standard cross-entropy loss on the imbalanced training set and stage 2 aims to learn the weight vector w and meanwhile continue to update the model f
( θ ) .
Generally, at stage 2, calculating the optimal θ and w requires two nested loops of optimization, which is cost-expensive.
We optimize θ and w alternatively, corresponding to (1) and (10) respectively, where w is maintained and updated throughout the training, so that re-estimation from scratch can be avoided in each iteration.
We summarize the amortized learning of w in Algorithm 2, where the key steps are highlighted in Step (a), (b), and (c).
Specifically, at each training iteration t , in Step (a), we have ˆ θ ( t +1) ( Ω t ) =
{ ˆ θ ( t +1)
( Ω t ) , ˆ θ ( t +1)( Ω t ) } and α is the step size for θ ; in Step (b), as the cost function based on features is related with ˆ θ ( t +1)
( Ω t ) , the OT loss relies on ˆ θ ( t +1)
( Ω t ) , and β is the step size for Ω ; in Step (c), we ameliorate model parameters θ ( t +1) .
Require: Datasets D , D , initial and
} for
, t do
Sample a mini where
=
; θ
; end do β end
B More details and results about the imbalanced image classification B.1 Details about imbalanced training datasets
CIFAR-LT is the long-tailed version of CIFAR dataset, where the original CIFAR-10 (CIFAR-100) dataset [38] has 5000 (500) images per class and falls into 10 (100) classes.
Following [5], we create long-tailed training sets from CIFAR-10 and CIFAR-100 by discarding some training samples, where we vary the imbalance factor IF ∈ { 200 , 100 , 50 , 20 } .
We do not change the balanced test sets and we randomly select ten training images per class as our meta set.
We build ImageNet-LT based on the classic ImageNet [39] following [24].
After discarding some training examples, ImageNet-LT has 115.8K training examples in 1,000 classes, with an imbalance factor IF = 1280 / 5 .
The authors also provided a small balanced validation set with 20 images per class, from which we sample 10 images to construct our meta set as [5].
Besides, the original balanced ImageNet validation set is adopted as the test set (50 images per class).
Places-LT is created from Places-2 [40] by [24] with the same strategy as above.
The Places-LT contains 62.5K images from 365 categories with = 4980 / 5 .
This factor means that Places-LT is more changeling than ImageNet-LT dataset.
Similarly, there is an official balanced validation dataset with 20 images per class, where we random select 10 images to build our meta set following [4].
iNaturalist 2018 3 is collected from real world which contains 435,713 training samples incategories with the imbalance factor IF of 1000 / 2 .
Following [4, 5], we use the original validation dataset to evaluate our method and random select 2 samples from every category in training dataset to construct our meta set.
B.2 Experimental details and results on iNaturalist
Following [4], we use ResNet-50 pre-trained on ImageNet plus iNaturalist 2017 as the backbone network on iNaturalist 2018.
For stage 1, we set the initial learning rate as 0.01, which is decay by 1 e − 1 every 20 epochs.
In the stage 2 of our method, we only fine-tune the last fully connected layer and set α as 1 e − 4 and β as 1 e − 3 within 200 epochs.
The mini-batch size is 64, the optimizer is SGD with momentum 0 .
9 and weight decay is 5 e − 3 .
Considering the label vector from the iNaturalist 2018 dataset is high-dimensional and sparse, we only use the features of samples to define the cost function, i.e. , Feature-aware cost .
For training efficiency, we use Prototype to construct our meta set.
As summarized in Table 6, our proposed method outperforms all the baselines.
This observation demonstrates that our proposed re-weighting method can effectively deal with the extremely imbalanced large-scale training set with a large number of classes.
B.3 Learned weights for different classes with varying imbalanced settings
To explore the learned example weights, we average them within each class and visualize the averaged weight for each class with varying imbalanced factors in Fig.
We see that the learned weights of the class 10 (tail class) are more prominent than those of the majority classes.
This phenomenon becomes increasingly evident as the training set becomes more imbalanced.
B.4 Convergence and time complexity
In this section, we investigate the convergence and time complexity.
The optimization of the
OT distance between discrete distributions of size n , is a linear program and can be solved with combinatorial algorithms such as the simplex methods, which dampens the utility of the method when handling large datasets.
Recently, the regularization scheme proposed by Cuturi in [37] allows a very fast computation of a transportation plan, denoted as Sinkhorn algorithm.
The Sinkhorn algorithm requires the computational cost of n 2 log( n ) /ε 2 to reach ε -accuracy, which is adopted by us to solve the OT problem.
As shown in Fig.
4, we plot the variation of training loss and test classification performance as the increase of training time at the second stage.
The training loss of our proposed method can converges at around 10th epoch, where 20 epochs are usually enough for our method to yield satisfactory performance.
This means our method can reach fast convergence.
[Table]
To investigate the time complexity, we conduct experiments on CIFAR 10, CIFAR100, ImageNet-LT and Places-LT with different imbalanced factor for ten epoches and average over the time.
Models on CIFAR are evaluated with batch size of 16 on a single RTX 3080 GPU.
Models on ImageNet-LT are evaluated with batch size of 128 on 4 RTX 2080Ti GPU.
Models on Places-LT and iNaturalist 2018 are evaluated with batch size of 32 and 64 on 4 Tesla V100 GPU, respectively.
The results are summarized in Table 7.
We observe that our proposed method has an acceptable time complexity compared with the cross entropy.
It means that our proposed method achieves a better performance with an acceptable cost by introducing the OT loss for re-weighting.
B.5 Learned Transport plan matrix and weight vector
To explain the effectiveness of learning the weight vector with OT, we give a toy example.
The imbalanced training mini-batch consists of { 10 , 9 , ..., 1 } training samples from class { 1 , 2 , ..., 10 } and the balanced meta set consists of { 10 , 10 , ..., 10 } validation samples from class { 1 , 2 , ..., 10 } , which are used to compute the prototype for each class.
The process of learning the weight vector is shown in Fig.
5, where we use feature-aware cost, label-aware cost and combined cost in (a)-(c), respectively.
For all experiments, we fix the probability measure of meta set as uniform distribution, initialize the tobe-learned probability measure ( i.e. , w ) of training mini-batch with a uniform measure, and we then optimize w by minimizing the OT loss with given cost function.
Given different cost functions, we find that the learned weight vectors and transport probability matrices show different characteristics.
Given the feature-aware cost function, we can assign weight to examples instance-wisely.
However, the inaccurate discrimination of features may mislead the learning of weight.
Under the label-aware cost function, the learned weights are class-level and inversely related to the class frequency.
Given the combined cost function, the results combine the benefits of label-aware and feature-aware cost functions.
Interestingly, the weights assigned to examples of tail classes are more prominent than those of the head classes, where the weights of head classes are below the initialization 1 / 55 and those of tail classes are above 1 / 55 .
Such results demonstrate our re-weighting method can pay more attention to the tail classes.
[Table]
[Figure]
[Table] [Table]
[Table]
C More details about the text classification C.1 FCN structure
We give the FCN structure in Table 9.
C.2 Weight net structure on text classification task
We give the weight net structure used in text classification task in Fig. 6.
Different that used in image and point cloud classification tasks, we build a self attention-similar weight net and take the sample features z train i as input:
[Equation]
Specifically, the output of dimension d f from feature extractor will be encoded to the same dimension d l by two separate linear layers W 1 and W 2 .
We compute their dot products, divide by a scaled factor √ d l and apply a softmax operator to obtain expected instance weights.
The hidden shape in our experiments is shown in Table [Table]
C.3 Experimental settings
For fining tune BERT, we set the initial learning rate as 5 e − 6 , where epochs is 5 and batch size is 8.
For the first stage, we use SGD optimizer with learning rate 1 e − 4 .
For the second stage, we use SGD optimizer with α learning rate 1 e − 4 and β learning rate 4 e − 2 .
For both two stages, our minibatch size is 50 and epochs are 15 and 10, respectively.
The settings of the training process on the STTand SST-5 are listed in Table 10.
C.4 Ablation Study
To prove the robustness of our method and its variants on text data, we execute an ablation study on
SST-2 dataset, which is shown in Table 11.
We can find that combined cost achieves best performance and using label or feature cost can still obtain an acceptable result, which indicates the OT cost is flexible and robustness.
When comparing the influence of different ways in constructing meta set, we can see that both whole meta set and prototype-based meta set achieve a close performance, while random sample meta cause a noticeable performance drop.
D More details about the point cloud classification
D.1 Experiments on Imbalanced Point Cloud Classification
Dataset and Baselines In addition to 1D text and 2D image, we further investigate the robustness of our method on 3D point cloud data, where we use the popular ModelNet10 [45].
We select
[Table]
PointNet++ [46] as our backbone without other additional layers.
We use 5 examples in each class for meta set.
Specifically, we down-sample the original points to 512 followed by a random point dropout, scale and shift to process the input data, which is a common setting for point cloud classification.
We conduct the same process on all the experiments to evaluate our method fairly.
Since we are the first to conduct imbalance classification task on point cloud data, we consider following methods: (1)
Pointnet++ baseline 4 , a classic network used in point cloud classification [46].
(2) Focal loss [12]. (3) LDAM-DRW [17].
(4) Logit Adjustment [20].
Experimental details and results on ModelNet10 The classification results of different methods are summarized in Table 12, where we evaluate our method from the aspects of instance accuracy (IA) and class accuracy (CA).
We can see that ours outperforms all baselines on IA and CA.
These experimental results further demonstrate the effectiveness of integrating OT loss into imbalanced point cloud classification, showing the generalization ability of our proposed method to data modalities.
D.2 Experimental settings
For point cloud classification, we select bathtub as minority class and the rest as majority, where the number of training sample is 100 in every class.
We train the baseline model from scratch where we use Adam optimizer with learning rate 1 e − 4 .
Besides, our minibatch size is 128 and epochs are 40.
For the first and second stage, we still use Adam optimizer, where learning rate is 1 e − 4 in the 1-st stage, α learning rate 1 e − 4 and β learning rate 5 e − 3 .
For both two stages, we keep the same number of epochs and minibatch size as them in training the baseline.
For training and meta dataset, we choice 100 samples from every majority class in original training set, where we then random choice 5 samples from the rest training set to build our meta set.
For test dataset, we random sample 40 instances in every class from the original test dataset to construct our balanced test dataset
[Table]
[Table]
[Table]
D.3 Ablation study
Similarly
[Table] we execute an ablation on point cloud to investigate the influence of OT loss [Table] which is shown Table 14.
We can see that the best performance is obtained when we use the setting of weight
[Table]
vector, combined cost and whole meta set.
Besides, performance are also acceptable when we use label cost and feature cost.
The performance is also competitive when we use the setting of weight vector, combined cost and prototype-based meta set.
E Effect of Different Meta-sets
To explore whether our proposed method is effective even without an additional meta set from the validation set, we build a dynamic meta set from the given imbalanced training set.
Notably, “dynamic” means that we randomly choose the samples from the training set to build a meta set at each training stage.
We report experiment results in SST-2 dataset in Table 15.
The results show our method could still reach a comparable performance even with the meta set built from the imbalance training dataset dynamically.
[Table]
with [Table] Our
[Table]
with
[Table]
F Label-awareness Experiments
In this section, we report label-aware performance of our method in image, text and point cloud classification task.
In this setting, label-aware represents that we only use label information in meta dataset to constrain the OT loss and then obtain the updated sample weights.
Our best performance represents that we use both feature and label information in meta dataset.
As summarized in 16,and 18, all results show that our method could obtain a comparable performance under the labelaware setting, where this label information is free to be obtained and we could reach the goal by only requiring the labels in training dataset obeying a uniform distribution.
[Table]
[Table]
[Table]
This paper introduces a re-weighting method based on optimal transport to solve the imbalanced problem. The proposed re-weighting method is very different from existing re-weighting methods and may have the potential to provide new and better thoughts for imbalanced problems in the future. Notably, once a malicious imbalanced task is selected, our re-weighting method may produce a negative societal impact, similar to the progress in other machine learning methods.