Learning to Re-weight Examples with Optimal Transport for Imbalanced Classification
Abstract
Imbalanced data pose challenges for deep learning based classification models.
One of the most widely-used approaches for tackling imbalanced data is re-weighting, where training samples are associated with different weights in the loss function.
Most of existing re-weighting approaches treat the example weights as the learnable parameter and optimize the weights on the meta set, entailing expensive bilevel optimization.
In this paper, we propose a novel reweighting method based on optimal transport (OT) from a distributional point of view.
Specifically, we view the training set as an imbalanced distribution over its samples, which is transported by OT to a balanced distribution obtained from the meta set.
The weights of the training samples are the probability mass of the imbalanced distribution and learned by minimizing the OT distance between the two distributions.
Compared with existing methods, our proposed one disengages the dependence of the weight learning on the concerned classifier at each iteration.
Experiments on image, text and point cloud datasets demonstrate that our proposed re-weighting method has excellent performance, achieving state-of-the-art results in many cases and providing a promising tool for addressing the imbalanced classification issue. The code has been made available at https:// github.com/DandanGuo /reweight-imbalance-classification with OT
Introduction
Deep neural networks (DNNs) have achieved remarkable success in various applications, which is undoubtedly inseparable from the high-quality large-scale datasets.
Usually, the number of samples for each class in these datasets are manually selected resulting in balanced datasets.
However, most real-world datasets are imbalanced, such as a few classes (a.k.a. head or majority class) occupy most of the data while most classes (a.k.a.
tail or minority class) have a few samples.
A model trained on the imbalanced training set but without considering such class imbalance would be significantly dominated by those majority classes, and thus underperform on a balanced test dataset.
This can also be known as the long-tailed problem and exists in many domains, such as text classification [1, 2], object detection [3] and image classification [4–6].
There are rich research lines to solve the imbalance problem, including re-sampling [7–10], class-level or instance-level re-weighting [1, 2, 4, 11–18], meta-learning [4, 5, 15, 16, 19], two-stage methods [4–6, 17] and post-hoc correction [20, 21].
Inspired by [2], re-weighting strategies can be roughly grouped into empirical re-weighting and automatic re-weighting.
The former aims to design weights manually with the major insight that the minority class example will be assigned a larger weight value than that of the majority class [12–14].
However, manually setting weights can be less adaptive to different datasets [2].
The latter aims to assign adaptive weights to the examples through learning mechanisms [1, 2, 4, 15, 16].
As the representative automatic re-weighting method, L2RW [15] optimizes the weight vector as a learnable parameter with an unbiased meta set ( i.e. , validation set).
Although L2RW and its followers have received widespread attention, most of them may be limited to optimizing the weights by the classification loss on the meta set: The gradient of weights is usually coupled with the to-be-learned classifier at each training iteration.
Since classifier is the major concern in imbalanced issue [6], the dependence of weights on classifier at training stage may lead to inaccurate learning of the weights.
This paper develops a novel automatic re-weighting method for imbalanced classification based on optimal transport (OT).
As discussed by Jamal et al.
[4], the major challenge for imbalanced classification is essentially the mismatch between the imbalanced training dataset (seen by a machine learning model) and the balanced test set (used to test the learned model).
To this end, we aim to view the learning of the weight vector as the distribution approximation problem.
We adopt the two-stage learning manner motivated by [6], where stage 1 and stage 2 focus on learning the feature extractor with the standard cross-entropy loss and the classifier with our proposed method, respectively.
Specifically, we represent the imbalanced training set as a discrete empirical distribution
P over all samples within it and view the to-be-learned weight vector w as its probability measure.
Then we represent the balanced meta set as a discrete empirical distribution Q over all samples within it (in the same space with P ), which has a uniform probability measure for being balanced.
Therefore, the learning of a weight vector can be formulated as the process of learning the distribution
P to be as close to the balanced distribution Q as possible, a process facilitated by leveraging the
OT distance [22].
Notably, the cost function plays a paramount role when learning the transport plan for OT, where we use the features and ground-truth labels of samples to design it.
Due to the flexibility of our method, we can also learn an explicit weight net directly from data like [16, 23] but with a different structure, optimized by OT loss instead of the classification loss on the meta set.
Generally, at each training iteration at stage 2, we minimize the OT loss to learn the weight vector (or weight net) for the current mini-batch, which is further used to re-weight the training loss for optimizing the model.
As we can see, the gradient of weights only relies on the OT loss and thus is independent of the classifier.
More importantly, our proposed method is robust to the distribution
Q .
To save the memory consumption, we introduce the prototype-oriented OT loss by building a new distribution Q based on prototypes instead of samples (one prototype for each class).
More importantly, our proposed method can achieve a reasonably good performance even if we randomly select a mini-batch from all prototypes to build Q , making our method applicable to datasets with a large number of classes.
We summarize our main contributions as follows: (1) We formulate the learning of weight vector or weight net as the distribution approximation problem by minimizing the statistical distance between to-be-learned distribution over samples from imbalanced training set and another balanced distribution over samples from the meta set.
(2) We leverage the OT distance between the distributions to guide the learning of weight vector or weight net.
(3) We apply our method to imbalanced classification tasks including image, text and point cloud.
Experiments demonstrate that introducing the OT loss to learn the example weights can produce effective and efficient classification performance.
Related Work
Empirical Re-weighting A classic empirical re-weighting scheme is to provide the examples of each class with the same weight, such as inverse class frequency [11, 14].
It has been further improved by the class-balanced loss [13], which calculates the effective number of examples as class frequency.
Focal Loss [12] uses the predicted probability to calculate higher weights for the hard examples and dynamically adjust the weights.
LDAM-DRW [17] designs a label-distribution-aware loss function and adopts a deferred class-level re-weighting method ( i.e. , inverse class frequency).
Automatic Re-weighting The automatic re-weighting methods learn the weights with learning mechanisms.
L2RW [15] adopts a meta-learning manner to learn the example weights, which are optimized by the classification loss on the balanced meta set.
Hu et al.
[1] further improve L2RW by iteratively optimizing weights instead of re-estimation at each iteration.
Meta-weight-net [16] aims to learn an explicit weight net directly from data and optimize it by a meta-learning manner.
Meta-class-weight [4] defines the weight for each example as the combination of class-level weight (estimated by Cui et al.
[13]) and instance-level weight, optimized with a meta-learning approach similar to L2RW.
Influence-balanced loss (IB) is proposed to [18] re-weight samples by the magnitude of the gradient.
Recently, Liu et al.
[2] propose to update the weights and model under a constraint.
Our method belongs to automatic re-weighting group, and the idea of building an explicit weight net is similar to Shu et al.
However, the major difference is that we bypass the classification loss on the meta set and use OT to learn the weights from the view of distribution approximation, disengaging the dependence of the weight learning on the concerned classifier at each iteration.
Meta Learning and Two-stage Learning Recently, researchers have proposed to tackle the imbalance issue with meta-learning, which can be applied to build a Balanced Meta-Softmax (BALMS)[19], learn weights [4, 15, 16] or transformed semantic directions for augmenting the minority classes in
MetaSAug [5].
Two-stage methods, where the first stage and second stage focus on representation learning and classifier learning, respectively, have been proved effective for solving the imbalanced issue [5, 6, 16, 24].
BBN [25] unifies two stages with a specific cumulative learning strategy.
Optimal Transport Recently, OT has been used to solve the regression problem under the covariate shift [26], unsupervised domain adaption [27, 28], including sample-level, class-level or domain-level weight vector.
Although they also adopt the re-weighting strategy and OT distance, they are distinct form ours in terms of task and technical detail.
Also, the dynamic importance weighting which adopts MMD to re-weight samples for label-noise and class-prior-shift tasks [29] is also different from ours, where we provide a more flexible way for learning the weights of samples and disengage the dependence of the weight learning on the concerned classifier at each iteration.
To the best of our knowledge, the works that solve imbalanced classification problem with OT are still very limited.
An oversampling method via OT (OTOS) [30] aims to make synthetic samples follow a similar distribution to that of minority class samples.
However, ours is a novel re-weighting method based on
OT, without augmenting samples.
Another recent work is Optimal Transport via Linear Mapping (OTLM) [21], which performs the post-hoc correction from the OT perspective and proposes a linear mapping to replace the original exact cost matrix in OT problem.
Different from OTLM that belongs to the post-hoc correction group and aims to learn refined prediction matrix, ours falls into the training-aware group and aims to re-weight the training classification loss.
Background
Imbalanced Classification Consider a training set D train =
{ ( x i , y i ) } Ni =1 , where ( x, y ) is the input and target pair, x i the i -th sample, y i ∈ (0 , 1) K the one-hot associated label vector over K classes, and N the number of the entire training data.
Besides, consider a small balanced meta set D meta = { ( x j , y j ) } Mj =1 , where M is the amount of total samples and M ≪ N .
Denote the model parameterized with θ as f ( x, θ ) , where θ is usually optimized by empirical risk minimization over the training set, i.e. , θ ∗ =
arg min θ 1 N (cid:80) N i =1 ℓ ( y i , f ( x i ; θ )) .
For notational convenience, we denote l train i ( θ ) =
ℓ ( y i , f ( x i ; θ )) to represent the training loss function of pair ( x i , y i ) .
However, the model trained by this method will prefer the majority class if the training dataset is imbalanced.
Learning to Re-Weight Examples To solve the imbalanced issue, a kind of re-weighting methods is to treat the weights as the learnable parameter and learn a fair model to the minority and the majority classes by optimizing the weighted training loss.
At each training iteration, the model is updated by
[Equation]
where w = ( w 1 , . . .
, w N ) T is the weight vector (usually with a simplex constraint) of all training examples.
Then the optimal w is obtained by making the model parameter θ ∗ ( w ) from Eq.
(1) minimize the classification loss on a balanced meta set, formulated as
[Equation]
where l meta j is the loss function of pair ( x j , y j ) from meta set and the updated w ∗ is used to ameliorate the model.
Generally, model θ consists of two key components, feature extractor and classifier, where the classifier has been proved to be the major concerning part in imbalanced issue [6].
However, the gradient of weights in Eq.
(2) always depends on the to-be-concerned classifier at each training iteration, which may result in inaccurate learning of the weights.
Most automatic re-weighting methods learn the weight vectors or weight-related parameters ( e.g. , weight net) following this line; see more details from the previous works [4, 15, 16].
Optimal Transport Theory OT has been widely used to calculate the cost of transporting one probability measure to another in various machine learning problems, such as generative models [31], text analysis [32, 33], adversarial robustness [34], and meta learning [35, 36].
Among the rich theory of OT, this work presents a brief introduction to OT for discrete distributions; see Peyré and Cuturi [22] for more details.
Consider p = (cid:80) ni
=1 a i δ x i and q = (cid:80) mj
=1 b j δ y
j
as two probability distributions, where x i and y j live in the arbitrary same space and δ is the Dirac function.
Then, we can denote a ∈ ∆ n and b ∈ ∆ m as the probability simplex of R n and R m , respectively.
The OT distance between p and q can be expressed as:
[Equation]
where ⟨· , ·⟩ is the Frobenius dot-product and C ∈ R n × m ≥is the transport cost matrix constructed by C ij =
C ( x i , y j ) .
The transport probability matrix T ∈ R n × m > 0 , which satisfies Π( p, q ) :
= { T | (cid:80) n i =1 T ij =
b j , (cid:80) mj
=1 T ij = a i } , is learned by minimizing OT ( p, q ) .
Directly optimizing Eq.
(3) often comes at the cost of heavy computational demands, and OT with entropic regularization is introduced to allow the optimization at small computational cost in sufficient smoothness [37].
Re-weighting Method with Optimal Transport
This work views a training set as a to-be-learned distribution, whose probability measure is set as learnable weight vector w .
We use OT distance to optimize w for re-weighting the training loss.
4.1 Main Objective
Given the imbalanced training set D train , we can represent it as an empirical distribution over N pairs, where each pair ( x i , y i ) train has the sample probability w i ( i.e. , the weight), defined as:
[Equation]
where ( x i , y i ) train is the i -th pair from the training set and the learnable weight vector w of all training examples means probability simplex of R N .
Since the meta set D meta is balanced for all classes and closely related with the training set, it is reasonable to assume that meta set has already achieved the balanced data distribution that the training set aims to approximate.
For meta set, we thus can sample each pair from it with equal probability and present it with an empirical distribution Q :
[Equation]
where ( x j , y j ) meta is the j -th pair from the meta set.
To learn w , different from most automatic re-weighting methods, which minimize the classification loss on the meta set, we aim to enforce the to-be-learned distribution P ( w ) to stay close to the balanced distribution Q .
Here, we explore the re-weighting method by adopting the OT distance between P ( w ) and Q :
[Equation]
[Equation] where cost matrix C ∈ R N × M ≥is described below and transport probability matrix T ∈ R N × M >should satisfy Π( P ( w ) , Q ) :
= { T | (cid:80) N i =1 T ij =1 /M, (cid:80) M j =1 T ij = w i } .
4.2 Cost Function
For notation convenience, we reformulate the model as f ( x, θ ) =
f 2 ( f 1 ( x ; θ 1 ); θ 2 ) , where f 1 parameterized with θ 1 denotes the representation learning part before the classifier, and f 2 parameterized with θ 2 denotes the classifier.
Intuitively, the cost C ij measures the distance between pair i in training set and pair j in meta set, which can be flexibly defined in different ways.
We explore a few conceptually intuitive options of C ij , although other reasonable choices can also be used.
Label-aware Cost
As the first option, we can define C ij with the ground-truth labels of two samples:
[Equation]
where d Lab ( · , · ) also denotes a distance measure, and y train i , y meta j are the ground-truth label vectors of the two samples, respectively.
Intuitively, if we use the euclidean distance, then C is a 0 − 1 matrix (we can transfer the non-zero constant to 1 ), i.e. , C ij =
if x train i and x meta j are from the same class, and C ij =1 otherwise.
Now the OT loss is influenced by neither feature extractor θ 1 nor classifier θ 2 .
Feature-aware Cost Besides
, we can define C ij purely based on the features of samples:
[Equation]
where z train i = f 1 ( x train i ; θ
[Equation]
) ∈ R E and z meta j
[Equation]
; θ 1 ) ∈ R E denote the E -dimensional representation of x train i and x meta j , respectively.
d Fea ( · , · ) denotes any commonly used distance measure and we empirically find the cosine distance is a good choice.
It is easy to see that if x train i and x meta j ’s features are close, their cost is small.
Here the OT loss is influenced by the feature extractor θ 1 .
Combined Cost
Finally, we can use both features and labels to define C ij , denoted as
[Equation]
Intuitively, C ij will be small if two samples have the same label and similar features.
Empirically, we find that using the d Fea =1 − cosine( · , · ) and euclidean distance for d Lab gives better performance.
Interestingly, given the feature-aware cost (8) or label-aware cost (7), the learned weight vector can be interpreted as the instance-level or class-level re-weighting method, respectively.
The weight vector learned from the combined cost can be interpreted as the combination of class-level and instance-level weights, although no specialized design for two-component weights like previous [4]; see Fig. 1.
4.3 Learn the Weight Vector
Given the defined cost function, we adopt the entropy regularized OT loss [37] to learn the weight vector.
We thus rewrite (6) as the following optimization problem:
[Equation]
where λ > 0 is a hyper-parameter for the entropic constraint H ( T )
= − (cid:80) ij T ij ln T ij .
Note that (10) provides us a new perspective to interpret the relationship between w and T , where w is the parameter of the leader problem and T is the parameter of the follower problem, which is of the lower priority.
Accordingly, when we minimize (10) with respect to w using gradient descent, we should differentiate through T .
Below we investigate the following two ways to optimize the weight vector.
Optimizing w directly Specifically, at each training iteration, we define P ( w ) with current w , use the Sinkhorn algorithm [37] to compute OT loss, then optimize w by w ∗ =
arg min w L OT .
Amortizing the learning of w We also provide an alternative method by constructing an explicit weight net to output the example weights, whose structure can be designed flexibly.
For example, we can build the following weight net and take the sample features as input:
[Equation]
where s i is the i -th element of s ∈ R N ,
w att ∈ R 1 × A and W
vz ∈ R
A × E are the learned parameters (we omit the bias for convenience), denoted as Ω = { w att , W vz } .
Denote S ( z ; Ω ) as the weight net parameterized by Ω , which can be optimized by Ω ∗ =
arg min Ω L OT .
Overall Algorithm and Implementations
To integrate our proposed method with deep learning frameworks, we adopt a stochastic setting, i.e. , a mini-batch setting at each iteration.
Following [4, 5], we adopt two-stage learning, where
Require: Datasets D train , D meta , initial model parameter θ and weight vector, hyper-parameters { α, β, λ } for t = 1 , 2 , ..., t 1 do θ α f
; end for do
← θ − where L B = Use end stage 1 trains the model f ( θ ) by the standard cross-entropy loss on the imbalanced training set and stage 2 aims to learn the weight vector w and meanwhile continue to update the model f
( θ ) .
Generally, at stage 2, calculating the optimal θ and w requires two nested loops of optimization, which is cost-expensive.
Motivated by Hu et al.
[1], we optimize θ and w alternatively, corresponding to (1) and (10) respectively, where w is maintained and updated throughout the training, so that re-estimation from scratch can be avoided in each iteration.
The implementation process of our proposed method with w optimized directly is shown in Algorithm 1, where the key steps are highlighted in Step (a), (b), and (c).
Specifically, at each training iteration t , in Step (a), we have ˆ θ ( t +1) ( w t ) =
{ ˆ θ ( t +1)( w t ) , ˆ θ ( t +1)( w t ) } and α is the step size for θ ; in Step (b), as the cost function based on features is related with ˆ θ ( t +1)( w t ) , the OT loss relies on ˆ θ ( t +1)( w t ) , and β is the step size for w ; in Step (c), we ameliorate model parameters θ ( t +1) .
We defer the learning of θ and Ω for the amortized learning of w .
Discussion From Step (b), we find the gradient of w is unrelated to classifier θ 2 regardless of which cost function we choose.
If we use the label-aware cost or freeze the feature extractor parameterized by θ 1 , which is trained in the first stage, the OT loss in Step (b) can be further reduced as L OT ( w t ) , where we only need Steps (b)-(c) at each iteration.
This is different from most of automatic reweighting methods, where the gradient of w is always related with the to-be-learned model { θ 1 , θ 2 } or classifier θ 2 (when freezing θ 1 ) for minimizing the classification loss on meta set.
Prototype-oriented OT loss (POT) Recall that we represent a balanced meta set with M samples as distribution Q in (5), where M/K is the number of data in each class and usually larger than 1.
Computing the OT loss requires to learn a B × M -dimensional transport matrix at each iteration.
To improve the efficiency of algorithm, we average all samples from each class in the meta set to achieve its prototype and propose a new Q distribution over K prototypes:
[Equation]
where POT loss only needs a B × K -dimensional transport matrix.
Due to the robustness of our method to Q , when dealing with a large number of classes, we can randomly sample a mini-batch from K prototypes at each iteration to build Q .
Experiments
We conduct extensive experiments to validate the effectiveness of our proposed method on text, image, and point cloud imbalanced classification tasks.
Notably, different from the imbalanced image and point cloud classification, we find that optimizing the weight net is better than optimizing the weight vector directly in the text classification.
Therefore, we optimize the weight vector for the image and point cloud cases and build a weight net for text case.
Unless specified otherwise, we adopt the combined cost and set the hyper-parameter for the entropic constraint as λ = 0 .
1 and the maximum iteration number in the Sinkhorn algorithm as 200 .
We define the imbalance factor (IF) of a dataset as the data point amount ratio between the largest and smallest classes.
6.1 Experiments on Imbalanced Image Classification
Datasets and Baselines We evaluate our method on CIFAR-LT-10, CIFAR-LT-100, ImageNet-LT and Places-LT.
We create CIFAR-LT-10 ( CIFAR-LT-100 ) from CIFAR-10 (CIFAR-100)[38] by downsampling samples per class with IF ∈{ 200 , 100 , 50 , 20 } [5, 13].
ImageNet-LT is built from the classic ImageNet with 1000 classes[39] and IF =1280 / 5 [5, 24].
Places-LT is created from Places- [40] with 365 classes and IF =4980 / 5 [4, 24].
We randomly select 10 training images per class as meta set [5]; see more details in Appendix B. We consider the following baselines: (1) Cross-entropy (CE) , the model trained on the imbalanced training set with CE loss.
(2) Empirical re-weighting methods , like Focal loss [12], Class-balanced (CB) loss [13] and LDAM-DRW [17].
(3) Automatic re-weighting methods , including L2RW [15], IB [18], Meta-Weight-Net [16] and Meta-class-weight [4].
(4) Meta-learning methods , including MetaSAug [5] and above methods of [4, 15, 16, 19].
Two-stage methods , such as OLTR [24], cRT [6], LWS [6], BBN [25] and methods of [4, 5].
Experimental details and results on CIFAR-LT For a fair comparison, we use ResNet-32 [41] as the backbone on CIFAR-LT-10 and CIFAR-LT-100.
Following Li et al.
[5], at stage 1, we useepochs, set the learning rate α of θ as 0.1, which is decayed by 1 e − 2 at the 160th and 180th epochs.
At stage 2, we use 40 epochs, set α as 2 e − 5 and learning rate β of weights as 1 e − 3 .
We use the SGD optimizer with momentum 0.9, weight decay 5 e − 4 and set the batch size as 16 .
We list the recognition results of different methods on CIFAR-LT-10 and CIFAR-LT-100 with different imbalance factors in
Table 1.We report the average result of 5 random experiments without standard deviation which is of small scale(e.g., 1e-2).
We can see that our re-weighting method outperforms CE training by a large margin and performs better than the empirical or automatic re-weighting methods.
Remarkably, our proposed method outperforms competing MetaSAug that conducts a meta semantic augmentation approach to learn appropriate class-wise covariance matrices when IF is 200, 100 and 50.
Importantly, as the training data becomes more imbalanced, our method is more advantageous.
Even though our proposed method is inferior to MetaSAug when the dataset is less imbalanced (IF =20 ), it can still achieve competing results and surpasses related re-weighting methods.
This suggests that our proposed method can be used to enhance the imbalanced classification, without the requirement of designing complicated models or augmenting samples on purpose.
To more comprehensively understand our method, we provide a series of ablation studies on CIFARLT-100 with IF = 200 in Table 2.
Firstly, to explore the impact of cost function, we use different cost functions for the OT loss.
We can see that the combined cost performs better than label-aware cost and feature-aware cost, confirming the validity of combining features and labels to define cost.
Besides, using either label-aware or feature-aware cost can still achieve acceptable performance, indicating the usefulness of OT loss in the imbalanced issue.
Secondly, to explore the robustness of the meta distribution Q , we adopt three ways to build Q : (1) using prototypes defined in Eq.
(12) ( K samples) ; (2) using all samples defined in Eq.
(5) ( 10 ∗ K samples) ; (3) randomly sampling one point from each class ( K samples) in meta set.
We find that prototype-based meta performs best, and the performance with random-sample meta or whole meta is still competitive, which demonstrates the robustness of our proposed method to the distribution Q and the benefit of using the prototypes to build Q .
Third, we compare two ways for learning w in each iteration, where one is re-estimating w from scratch and another one is maintaining and updating w throughout the training ( i.e. , iteratively optimizing weights).
We find that iteratively optimizing performs better.
Since cost function is essential in optimizing the OT loss, we are interested in examining the learned weight vectors given by different cost functions.
Here, we use CIFAR-LT-10, randomly choose { 10 , 9 , ..., 1 } training samples from class { 1 , 2 , ..., 10 } and obtain 55 samples, which are used to build distribution P .
Besides, the 10 prototypes from meta set are used to build the distribution Q .
Given the different cost functions, we show the learned weight vectors of 55 training samples in Fig.
1, which have very different properties.
Specifically, the label-aware cost and feature-aware cost lead to class-level weights and sample-level weights, respectively.
It is reasonable that label-aware cost [Table] only decides whether the two samples (from the meta set and training set) belong to the same class, resulting in class-level measure.
However, feature-aware cost measures the distance between samples from the sample-level, where each sample has its own feature.
More interestingly, the learned weights with the combined cost own the characteristics of class-level and sample-level weights simultaneously, where example weights of different classes are far away and example weights of the same class are close.
Coincidentally, using the combined cost to define the OT loss can reach the same goal of [4], which explicitly considers class-level and sample-level weight.
Besides, we find that the learned example weights of the minority class are usually more prominent than those of the majority classes.
[Figure] [Table] [Table]
[Table]
To verify whether our method ameliorates the performance on minority classes
[Table]
we plot the confusion matrices of CE, MetaSAug
[Table]
and ours on CIFAR-LT-10 with IF =200 in Fig. 2.
As expected, although
CE training can almost perfectly classify the samples in majority classes [Table] it suffers severe performance degeneration in the minority classes.
MetaSAug improves the accuracies of the minority classes, where is still a big gap between the performance on the minority classes and the majority classes.
In contrast, ours does not show a very clear preference for a certain class and outperforms the strong baseline on the overall performance, which is the goal of on an imbalanced classification task.
Experimental details and results on Places-LT and ImageNet-LT Following [6], we employ
ResNet-152 pre-trained on the full ImageNet as the backbone on Places-LT.
For stage 1, we set the initial learning rate as 0.01, which is decayed by 1 e − 1 every 10 epochs.
In the stage 2 of our method, we only fine-tune the last fully-connected layer for training efficiency and set α as 1 e − 4 and β as 1 e − 3 within 50 epochs.
The mini-batch size is 32 and the optimizer is SGD with momentum 0 .
9 and weight decay 5 e − 3 .
As shown in Table 3, our method outperforms all baselines.
It further suggests that our method has excellent performance in the extreme imbalance setting with IF =4980 / 5 .
For a fair comparison, we implement our method on ImageNet-LT with the same experimental conditions of [5], from which we have taken the results of other comparison methods.
We consider ResNet- [41] as the backbone on ImageNet-LT.
In stage 1, we run 200 epochs and decay the learning rate by 0.1 at the 60th and 80th epochs.
In stage 2, we implement our method for 50 epochs, set learning rate α as 2 e − 5 and β as 1 e − 2 , and only fine-tune the last fully-connected layer for training efficiency.
We use the SGD optimizer with momentum 0.9, weight decay 5 e − 4 and set the batch size as 128 .
The results on ImageNet-LT of different models reported in Table 4 indicate the effectiveness of our proposed method on ImageNet-LT when comparing with strong baseline MetaSAug.
Besides, we further consider randomly sampling a mini-batch of size 100 from all prototypes at each iteration to build Q , whose performance is comparable to the Q from all prototypes.
Thus, with a stochastic setting for Q , our proposed method can be used to the imbalanced training set with a large number of classes.
We defer the time computational complexity, additional quantitative results and qualitative results on different image datasets to Appendix B.
6.2 Experiments on Imbalanced Text Classification
Datasets and settings Following [1, 2], we adopt the popular SST-2 for 2-class and SST-5 for 5-class sentence sentiment [43].
For a fair comparison, we use the same imbalanced datasets and settings with [2].
Specifically, we set class 1 as the minority class and the rest as the majority classes, where the number of examples in the majority class is fixed as 1000 (SST-2) and 500 (SST-5) and we achieve different imbalance settings by varying the number of examples in the minority class.
Besides, the number of samples in the meta set is 10 for each class.
We use the BERT (base, uncased) model [44] as feature extractor and a simple 3-layer fully-connected network (FCN) with the structure in
Appendix C as classifier.
To make subsequent experiments on strong models, following [2], we use an additional balanced training set (500 samples in each class) to fine-tune the BERT model, which is randomly selected from the remaining examples in each dataset except the imbalanced training set, meta set and to-be evaluated test set.
Based on the fine-tuned BERT, we adopt the two-stage manner for the imbalanced text datasets, where we train the BERT + FCN in the first stage with CE loss and train the FCN with our proposed method by freezing the BERT in the second stage.
The settings of the training process are deferred to Appendix C.
Baselines We consider the following methods: (1) vanilla BERT , the vanilla pretrained language model.
(2) Fine-tuned BERT , where the pretrained BERT is fine-tuned on an additional balanced training set.
(3) Fine-tuned BERT + CE , the fine-tuned BERT model followed by the FCN which is further trained by the CE loss on the imbalanced training set following [1, 2].
(4) Automatic re-weighting methods , including the method of Hu et al.
[1] and constraint-based re-weighting [2].
Since few works consider imbalanced text classification, we further consider (5) Empirical reweighting methods , including re-weighting with inverse class frequency ( i.e. , Proportion) [11, 14]) and LDAM-DRW [17] and (6) Logit adjustment [20] using their official codes and settings 1 2 .
We repeat all experiments 10 times and report the mean and standard deviation.
Experimental details and results on SST-2 and SST-5 We report the text classification results of compared methods under different imbalance factors in Table 5.
We find that our proposed method outperforms all competing methods in all imbalance factor settings, which demonstrates the effectiveness of our proposed method.
Although all methods could achieve acceptable performance in a slight imbalance, the performance of three baselines (Vanilla BERT, Fine-Tuned BERT and FineTuned BERT+CE) drop dramatically, indicating the importance of proposing specialized methods for handling imbalanced training datasets.
Logit adjustment (post-hoc correction), is very competitive to ours on SST-2, which, however, only produces similar results to the three above-mentioned baselines on SST-5.
In contrast, ours is robust to not only the imbalance factors but also the number of classes, where the results are consistent with the image case.
We provide more results in Appendix C.4.
In addition to 1D text and 2D image, we further investigate the robustness of our method on 3D point cloud data, where we use the popular ModelNet10 [45] and defer the experiments to Appendix D.
[Table]
Conclusion
This paper introduces a novel automatic re-weighting method for imbalance classification based on optimal transport (OT).
This method presents the imbalanced training set as a to-be-learned distribution over its training examples, each of which is associated with a probability weight.
Similarly, our method views another balanced meta set as a balanced distribution over the examples.
By minimizing the OT distance between the two distributions in terms of the defined cost function, the learning of weight vector is formulated as a distribution approximation problem.
Our proposed re-weighting method bypasses the commonly-used classification loss on the meta set and uses OT to learn the weights, disengaging the dependence of the weight learning on the concerned classifier at each iteration.
This is an approach different from most of the existing re-weighting methods and may provide new thoughts for future work.
Experimental results on a variety of imbalanced datasets of both images and texts validate the effectiveness and flexibility of our proposed method.
Acknowledgements.
This work is partially supported by a grant from the Shenzhen Science and
Technology Program (JCYJ20210324120011032) and Shenzhen Institute of Artificial Intelligence and Robotics for Society.