L ONG - TAILED R ECOGNITION BY R OUTING D IVERSE D ISTRIBUTION -A WARE E XPERTS
A BSTRACT
Natural data are often long-tail distributed over semantic classes.
Existing recognition methods tend to focus on gaining performance on tail classes, often at the expense of losing performance on head classes and with increased classiﬁer variance.
The low tail performance manifests itself in large inter-class confusion and high classiﬁer variance.
We aim to reduce both the bias and the variance of a long-tailed classiﬁer by RoutIng Diverse Experts (RIDE), consisting of three components: 1) a shared architecture for multiple classiﬁers (experts); 2) a distribution-aware diversity loss that encourages more diverse decisions for classes with fewer training instances; and 3) an expert routing module that dynamically assigns more ambiguous instances to additional experts.
With on-par computational complexity, RIDE signiﬁcantly outperforms the state-of-the-art methods by 5% to 7% on all the benchmarks including CIFAR100-LT, ImageNet-LT and iNaturalist 2018.
RIDE is also a universal framework that can be applied to different backbone networks and integrated into various long-tailed algorithms and training mechanisms for consistent performance gains.
Our code is publicly available at: https://github.com/frank-xwang/RIDE-LongTailRecognition.
I NTRODUCTION
Real-world recognition often has a long-tail distribution: A few classes contain many instances, whereas most classes contain only a few instances.
For extreme imbalanced classiﬁcation, learning to discriminate among them is challenging, as the classiﬁer training objective on the few tail instances can be easily overwhelmed by that for many head instances, whereas for the multitude of tail-class few-shot recognition tasks.
Long-tailed recognition is usually handled either by class re-balancing/re-weighting strategies which give more importance to tail instances (Cao et al., 2019; Kang et al., 2020; Liu et al., 2019) or by grouping methods, where long-tailed data are separated into groups by their class frequencies and models focusing on each individual group are combined to form a multi-expert framework (Zhou et al., 2020; Xiang et al., 2020).
However, all these methods generally gain on tail classes at the cost of performance loss on head classes.
Here, we quantitatively analyze the long-tail classiﬁer’s performance change in terms of bias and variance analysis with respect to ﬂuctuations in the training set: We randomly sample CIFAR (Krizhevsky, 2009) according to a long-tailed distribution a few times, train a model each time, and then estimate the per-class bias and variance of the classiﬁers.
The prediction error of model h on instance x with output Y varies with the training data D .
The expected variance with respect to D has a well-known bias-variance decomposition:
[Equation]
The model bias measures the accuracy of the prediction with respect to the true value, the variance of the method measures the stability of the prediction, and the irreducible error measures the precision of the prediction and is irrelevant to the model h .
This same concept can be extended to classiﬁcation
[Table]
L ONG - TAILED R ECOGNITION BY R OUTING D IVERSE D ISTRIBUTION -A WARE E XPERTS
Anonymous authors Paper under double-blind review
A BSTRACT
Natural data are often long-tail distributed over semantic classes.
Existing recog- nition methods tend to focus on tail performance gain, often at the expense of head performance loss from increased classiﬁer variance.
The low tail perfor- mance manifests itself in large between-class confusion and high classiﬁer variance.
We aim to reduce both the bias and the variance of a long-tailed classiﬁer by RoutIng Diverse Experts (RIDE).
It has three components: 1) a shared architecture for multiple classiﬁers (experts); 2) a distribution-aware diversity loss that encourages more diverse decisions for classes with fewer training instances; and 3) an expert routing module that dynamically assigns more ambiguous instances to additional experts.
With on-par computational complexity, RIDE signiﬁcantly outperforms the state-of-the-art methods by 5% to 7% on all the benchmarks in-
cluding CIFAR100-LT, ImageNet-LT and iNaturalist.
RIDE is also a universal framework that can be applied to different backbone networks and integrated into various re-balancing or re-weighting methods for consistent performance gains.
I NTRODUCTION
The natural data we encounter in practice often has a long tail distribution: A few classes contain many instances, while most classes contain only a few instances.
Learning discrimination among them is challenging, as the few tail instances can be easily overwhelmed by many head instances.
Long-tailed recognition is usually handled either by class re-balancing/re-weighting strategies giving more importance to tail instances (Cao et al., 2019; Kang et al., 2020; Liu et al., 2019), or by multiexpert methods, where long-tailed data are separated into parts by their frequencies and models focusing on individual parts are combined (Zhou et al., 2020; Xiang & Ding, 2020).
However, all these methods generally gain on tail classes at the cost of performance loss on head classes.
The state-of-the-art (SOTA) methods on iNaturalist (Van Horn et al., 2018) are cRT and ⌧ -norm (Kang et al., 2020) and BBN (Zhou et al., 2020).
The former belongs to the re-balancing type with a two-stage optimization for learning a good representation and classiﬁer, whereas the latter belongs to the multi-expert type with two experts focusing on head and tail classes.
We analyze the performance of a long-tail classiﬁer in terms of bias and variance with respect to ﬂuctuations in the training set: We randomly sample CIFAR100 (Krizhevsky, 2009) according to a long-tailed distribution a few times, train a model each time, and then estimate the per-class bias and variance of the classiﬁer.
(a) Comparisons of the mean accuracy, per-class bias and variance of baseline methods and our RIDE method.
Better (worse) metrics than the distribution-unaware cross entropy (CE) reference are marked in green (red)
(b) Histograms
using 0-1 loss between the predicted class and the ground-truth label (Domingos, 2000).
Below is a quick summary.
Let D be a set of training sets, x be a sample in the dataset.
A loss function L ( t ; y ) measures the cost of predicting y ∈ Y when the true value is t .
Assume the optimal prediction y ∗ for x is the prediction that minimizes E t [ L ( t ; y ∗ )] , where the subscript t denotes that the expectation is taken with respect to all possible values of t , weighted by their probabilities given x . The main prediction y m for a loss function L over the training set D (which is treated as a random variable) is deﬁned as y m = argmin y (cid:48) E D [ L ( y, y (cid:48) )] .
That is, the main prediction is the one that “differs least” from all the predictions in Y according to L .
The bias is the loss incurred by the main prediction y m relative to the optimal prediction y ∗ , and the variance is the average loss incurred by predictions y relative to the main prediction y m .
We then evaluate bias and variance with zero one loss according to the evaluation metric in (Domingos, 2000) for classiﬁcation, i.e. bias =
L zero-one ( y ∗ , y m ) , variance = E D [ L zero-one ( y m , y )] .
Fig.
1a uses the standard cross-entropy (CE) classiﬁer as a performance reference and compares three well-known long-tail methods, cRT and τ -norm (Kang et al., 2020) and
LDAM (Cao et al., 2019), with our proposed method.
The former adopts a two-stage optimization, ﬁrst representation learning and then classiﬁcation learning, whereas the latter is trained end-to-end with a marginal loss.
On the basis of the preliminary experiments, following observations can be obtained:
On the mean accuracy: All the long-tail methods increase the overall, medium-shot, and fewshot accuracies, but these previous methods all decrease the many-shot accuracy.
Our method increases accuracies on all splits.
On the model bias: All the long-tail methods reduce the overall, medium-shot, and few-shot bias.
The reduction tends to be greater for the tail classes.
Our method decreases bias more than other methods on the tail classes.
On the model variance: All the current long-tail methods increase the overall, many-shot, medium-shot, and few-shot variance, except cRT has a slight reduction for medium-shot.
Our method reduces variances throughout the class spectrum.
𝑓 !
𝜓 ! ! 𝜓 ! "
𝜓
[Figure]
[Figure] Jointly
[Figure] 𝑓 [Figure] 𝜓 !
[Figure]
𝜓
[Figure]
𝜓
[Figure] [Figure] ℒ !" #$%
[Figure]
’($)*
[Figure]
Classification loss ℒ +,-(($)
[Figure]
trainable frozen on/off on/off Routing loss ℒ /012$ c) RIDE vs. SOTA 10.
expert could dynamically assign “ambiguous” samples to extra experts is trained in stage two
At test time we combine the predictions of assigned experts to form a robust prediction.
Since tail classes are inclined to be confused with other classes, by adding the expert assignment module, the data imbalance ratio for later experts can be automatically reduced without any distribution-aware loss which allows focusing less on conﬁdent head classes more on tail classes
SOTA methods
.e
LT BBN et iNaturalist) on all experimented benchmarks.
That is, current long-tail methods sacriﬁce the head performance to improve the tail performance, and their models also become more unstable with respect to the training data variation. Our method is the only one that improves accuracies and reduces bias/variance throughout the class spectrum. Fig. 1b provides further insight into the model bias by examining the largest softmax probability in the other classes:
The smaller this value is, the less the confusion, and the lower the bias.
It shows that the poor tail performance manifests itself at increasingly larger confusion and bias, and our method signiﬁcantly reduces the confusion and bias for the tail classes.
We propose a novel multi-expert approach, called RoutIng Diverse Experts (RIDE), not only to reduce the model variance for all the classes, but also to reduce the model bias for the tail classes, both of which existing long-tail methods all fail to accomplish.
RIDE adopts multiple experts for model variance reduction and employs an additional distributionaware diversity loss L D-Diversity for reducing the model bias.
With a dynamic expert routing module,
RIDE assigns another trained and distinctive expert for a second (or third, ...) opinion when it is called for.
Both the shared architecture for experts and the routing module effectively reduce the computational complexity of our multi-expert model to a level even lower than a baseline model with the same backbone.
RIDE delivers 5% ∼ 7% higher accuracies than the current SOTA methods on CIFAR100-LT,
ImageNet-LT (Liu et al., 2019) and iNaturalist (Van Horn et al., 2018).
RIDE is also a universal framework that can be applied to different backbone networks for improving existing long-tail algorithms such as focal loss (Lin et al., 2017), LDAM (Cao et al., 2019), τ -norm (Kang et al., 2020).
RELATED WORKS
Re-balancing/re-weighting .
The classical way to achieve balance for long-tailed datasets is to control each class’s sample frequencies based on class numbers such as the under- and over-sampling method (He & Garcia, 2009).
Data augmentation is another common way to achieve sample balance.
For example, Liu et al.
(2020), Chu et al.
(2020), and Kim et al.
(2020) generate augmented samples in the feature space to supplement tail class samples.
Besides sample-wise balancing, re-weighting the loss functions can also introduce learning balance by either putting larger weights on more challenging and sparse classes (Lin et al., 2017; Cui et al., 2019; Cao et al., 2019; Wu et al., 2020) or randomly ignoring gradients from head classes (Tan et al., 2020).
However, both sample-wise and loss-wise balancing conceptually increase the learning focus on tail classes, which increases the sensitivity to small ﬂuctuations of tail classes and greatly increases the variance as in Fig.
Knowledge transfer .
Another major direction to learning balance is transferring knowledge from head to tail classes (Liu et al., 2019; Zhu & Yang, 2020;
Kang et al., 2020; Jamal et al., 2020; Wang et al., 2017).
For example, the representative approaches in this direction, OLTR (Liu et al., 2019) and inﬂated memory (Zhu & Yang, 2020), harness memory banks to store and transfer mid- and high-level features from head classes to enhance tail classes’ feature generalization.
However, this line of work usually do not have effective controls over the transferring process, and therefore would sacriﬁce the performance of head classes.
Ensemble and grouping .
A more related direction of balanced learning to this project is ensemble and grouping, in which samples are usually separated into different groups based on the position of the class from head to tail and individual models with focus on each group are ensembled at the end to form a multi-expert framework.
BBN (Zhou et al., 2020) has a two-branch architecture with an adaptive fusion procedure at the end, where one branch focuses on the head classes (by directly learning from an imbalanced dataset), and the other focuses on the tail classes (by balancing the dataset with sampling techniques).
LFME (Xiang et al., 2020) takes a step further by distilling a uniﬁed model from multiple teacher models; each focuses on classifying a relatively balanced group of the dataset (e.g., many-shot classes, medium-shot classes, and few-shot classes).
BBN and LFME still suffer from performance loss in head classes because each expert do not have a balanced access to the whole dataset, which damages the overall generalizability, especially head classes.
In our approach, we apply a diverse distribution-aware individual loss with instance-balanced sampling to each routed expert, so that the information between experts are shared to boost generalizability.
RIDE has several appealing properties over traditional ensemble methods: 1) RIDE requires much fewer parameters and computational cost, RIDE’s model complexity can even be smaller than the baseline model with higher accuracy.
By adding the expert assignment module, not all experts are activated for all samples.
The dynamic collaboration between experts enables RIDE to be more efﬁcient.
All these experts are jointly optimized, which reduces the cost for running back-propagation and optimization steps.
We share partial convolutional layers of these experts that are considered relatively task-agnostic and reduce the channel dimensions of non-shared layers.
Since the capability of each expert is lower, each expert is less easier to be over-ﬁtted on the tail classes.
RIDE: R OUTING D IVERSE D ISTRIBUTION -A WARE E XPERTS 3. O VERALL FRAMEWORK
The multi-expert framework consists of two components as in Fig.
2a: f θ , shared part across experts, and Ψ = { ψ θ 1 , ..., ψ θ n } , which is expert-independent and evaluated with the output of f θ .
Although our method is applicable to multiple backbone networks, which is also indicated in our experiments, we take ResNet as an example to illustrate our design for its commonness.
We exploit a property of convolutional neural networks in the design of the multi-expert framework: Since early CNN layers typically contain ﬁlters that generally process only task-agnostic information of an image, as indicated by the practice of freezing early layers in transfer learning, we share the ﬁrst two stages of ResNet across all experts, i.e. f θ .
Later stages, including the fully connected layer, are independent between experts, i.e. ψ
θ
i , where i ∈ [1 , n ] .
In addition, in order to further cut down the overall model complexity, the number of ﬁlters in ψ θ
i is reduced by 1/4.
Then, to obtain logits of each experts, we ﬁrst get the intermediate representation from f θ , and then pass it into each ψ θ i independently to get logits.
All these n experts are co-trained on long-tailed data with the proposed distribution aware diversity loss L D-Diversify and a classiﬁcation loss L Classify , such as cross-entropy and LDAM.
Expert assignment module is added to dynamically assign additional cascaded experts to hard samples.
Since a weak expert is sufﬁcient to provide a high-quality representation for easy samples, not all samples require extra knowledge from additional experts.
The expert assignment module is trained with a separate stage on the top of frozen feature extractor as in Fig. 2b.
Making joint decisions with geometric mean.
During testing time, assume m , where m ≤ n , is the number of experts assigned to the tested sample, we handle the outputs of these models by taking arithmetic mean on logits of m experts, which produces logits with the same ranking as taking the geometric mean on the probabilities, followed by a softmax layer to calculate the probability of the j th class: p j = softmax( 1 m (cid:80) mi =1 ψ θ i ( f θ ( x ))) .
In our setting, this approach performs better than taking the arithmetic mean on the probabilities, i.e. p j = 1 m (cid:80) mi =1 softmax( ψ θ i ( f θ ( (cid:126)x ))) .
3. D IVERSE D ISTRIBUTION -A WARE I NDIVIDUAL L OSS
Previous multi-branch methods such as BBN apply a classiﬁcation loss (e.g. cross-entropy) on aggregated output of models.
While this is intuitive due to same formula in training and evaluation, in our empirical experiments, we found that applying this method is not sufﬁcient for the best performance.
We hypothesize that taking the loss over the geometric mean in Section 3.1 leads to correlated experts, which prevents the experts to have different knowledge. In order to make the experts model decoupled, we present a loss function with two speciﬁcally-designed components. Individual loss: decoupling the training of multiple experts. We deﬁne the loss function above as “collaborative loss”, with expression: L collaborative = 1 N (cid:80) Ni
=1 L ( 1 n (cid:80) nj =1 ψ θ j ( f θ ( (cid:126)x i )) , y i ) , where
N is the mini-batch size, L is the classiﬁcation loss.
This loss is shown to behave well in several recently proposed multi-expert models.
However, in our multi-experts setting, this method only achieves accuracy comparable to a single-expert model with roughly the same number of parameters, which indicates that the collaborative loss does not train experts to have diverse knowledge to complement other experts in decision making, i.e. the collaborative loss trains correlated experts.
This means the best performance of multi-experts overall is not shown.
To reduce the effect of correlated experts, we propose a new loss function, “individual loss”, which applies the loss in the middle of the module as soon as each expert’s logits are computed.
Since the loss calculation process does not depend on post-aggregation output, it results in almost no interexpert relationship expect the shared parameters. The formal expression of it, in training time, is: L individual = 1 nN (cid:80) Ni =1 (cid:80) nj =1 L i ( ψ θ j ( f θ ( (cid:126)x i )) , y i ) .
The test time formulation is the same as the one mentioned in Section 3.1. Although it requires the model to have more than one output, which might add a little implementation difﬁculty, this loss function achieves a better accuracy and contributes a large amount of improvements in most of our experiments. A diversiﬁed distribution-aware loss is proposed to penalize the inter-expert correlation further since we discover from experiments that although the individual loss, as well as the random initialization, already leads to branches that are diverse, adding more diversity with a external loss beneﬁts an additional small amount most of the time. Since a complex model needs to be ﬁtted on a few tail class samples, it will capture the noise along with the underlying pattern in data and thus we encourage diversity in tail classes to alleviate the inﬂuence of noise, we reference the idea of temperature in contrastive loss (Hadsell et al., 2006; Wu et al., 2018) to enable the diversity loss to be distribution aware, formulated as:
[Equation]
where φ i ( (cid:126)x, (cid:126)T ) = softmax( ψ θ i ( f θ ( (cid:126)x )) /(cid:126)T ) , i is the expert index, D KL ( · , · ) is the KL-Divergence ( D KL ( P, Q ) =
(cid:80) x ∈X P ( x ) log P ( x ) Q ( x ) ), λ is the balancing factor between diversity and classiﬁcation loss between ground truth label (cid:126)y and model predictions, (cid:126)T is temperatures of each sample and division is performed as element-wise division, and X is the probability space of P ( x ) and Q ( x ) .
The optimization objective of L i D-Diversify is to encourage the diversity between expert i and other experts.
Choosing a small temperature would assign large negative gradients (i.e. smaller overall gradients) to samples with “divergent” predictions and very small negative gradients (i.e. larger overall gradients) to samples with “homogeneous” predictions.
L i D-Diversify could also be used as a regularization term.
The temperature T i for samples in class i can be calculated by:
[Equation]
where C is the number of classes, γ is the balancing factor, n i is the number of samples of class i , and η is the base temperature.
This way, we give lower temperature to tail classes, which generates higher probability for the tail classes in distributions that we apply KL Divergence on, encouraging more diversity in tail classes.
Similar to the practice of deferred reweight (Cao et al., 2019), we enable temperatures only after the network has been trained for several epochs to allow the feature extractor to learn stabilized features.
The total loss presented above can be formulated as:
[Equation]
where j and k are the expert indices, L j Classify ( ., . ) can be LDAM loss, focal loss, etc., depending on the training mechanisms we choose.
3. R OUTING D IVERSIFIED E XPERTS
An expert assignment module that dynamically routes cascaded experts is proposed to further reduce the overall computational complexity.
The expert assignment decision is based on the information from the logits and the input feature.
The input feature is normalized to stabilize training process and is transformed by a fully-connected layer to reduce dimension and aggregate information.
We only take the top z scores of logits, since other logits are generally too small to provide much information about the “ambiguity” of the sample.
z is commonly set to a number ranging from 10 to 50 .
The module has negligible computation cost and negligible additional parameters.
Let (cid:126)l i and (cid:126)v i be the top z logits and the normalized feature of expert i , respectively.
The binary output of expert assignment module is: y ea = W 2 ( (cid:126)l i ⊕ σ ( W 1 (cid:126)v i )) , where ⊕ denotes concatenation, σ ( · ) denotes ReLU function, W 1 ∈ R d (cid:48)
×
d , W 2 ∈ R 1 × ( d (cid:48)
+ z ) , and d is the dimension of input feature.
d (cid:48) is set to 16, as a too large d (cid:48) may cause overﬁtting and unstable optimization.
The expert assignment module is optimized with the routing loss , a weighted variant of binary cross entropy loss:
[Equation]
where the ground truth y is constructed as: if the current expert does not predict the sample correctly but one of the next experts gives correct prediction, the ground truth is set to 1 (considered as a positive sample), otherwise it is 0.
ω p and ω n are the re-weighting factor (set to 100 for all positive samples and 1 for all negative samples).
The expert assignment module is trained on a separate stage, with all other parameters frozen.
Each expert, except the last one, has an independent expert assignment module with shared W 2 .
Therefore, the number of expert assignment modules is n − 1 .
Since we use a binary cross entropy loss with logits input in training, the output of this module is the logits that decide whether we need to activate more experts, and the predicted probability can be obtained by appending a sigmoid function at test time.
If the predicted probability is greater or equal to 0.5, which means that our predictor gives larger probability to having additional experts, we decide to proceed to the next expert.
In case that one needs to manually adjust the proportion of samples passing through each expert, i.e. the “easiness for each expert assignment module to be satisﬁed”, one could adjust the positive weight of the binary cross entropy loss, ω p , at training time.
As observed in experiments, using a ﬁxed positive weight ( ω p = 100 ) across all our experiments leads to a good trade-off of our EA module in accuracy and computational complexity.
Self-distillation step is optional but recommended if further improvements (about 0.4% ∼ 0.8% for most experiments) are desired.
Unlike previous methods with ﬁxed number of experts (e.g. BBN,
LFME), the total number of experts can be arbitrarily adjusted to balance accuracy and computation cost.
This enables self-distillation from a model with more (6 in our setting) experts to the same model with fewer experts to obtain further improvements.
We choose knowledge distillation (Hinton et al., 2015) by default.
The implementation details and comparison on various distillation algorithms such as CRD (Tian et al., 2019) are investigated in Appendix Section A.2. [Table] [Table]
E XPERIMENTS 4. D ATASETS AND I MPLEMENTATIONS
We conduct experiments on three major long-tailed recognition benchmarks and different backbone networks to prove the effectiveness and universality of RIDE:
CIFAR100-LT (Cao et al., 2019):
The long-tailed version of CIFAR100 follows an exponential decay in sample sizes across different classes.
We conduct experiments on CIFAR100-LT with an imbalance factor of 100, with the ResNet-32 (He et al., 2016) as a backbone network.
ImageNet-LT (Liu et al., 2019): RIDE with ResNet-50 and ResNeXt-50 (Xie et al., 2017) are experimented on ImageNet-LT.
More details and experiments on other backbones are listed in appendix.
iNaturalist (Van Horn et al., 2018):
We use ResNet-50 as the backbone network and apply the same training recipe as ImageNet-LT except with batch size 512 as Kang et al.
(2020) does.
4. E XPERIMENTAL R ESULTS
CIFAR100-LT .
Table 1 shows that RIDE outperforms state-of-the-art methods by a large margin on
CIFAR100-LT.
The average computational cost is even about 10% less than baseline models when we only apply 2 cascaded experts as in BBN.
Compared with LFME (Xiang et al., 2020) and BBN (Zhou et al., 2020), which also apply multiple experts, RIDE signiﬁcantly surpasses both by more than 5.3% and 6.5%, respectively.
Since LDAM is end-to-end optimized, we choose it as the default training method for simplicity, unless otherwise noticed. F o c a l l o s s C E L D A M c R T - n o r m
C o s i n e Mechanisms 30. A cc u r a c y ( % ) 38.
[Figure] [Figure] [Figure] [Table]
Integrating RIDE with various methods .
Fig.
3 indicates that our method will likely be able to beneﬁt from the future advancements to the loss function and training process.
Consistent improvements can be observed.
Cosine classiﬁer is a baseline that we constructed in which we normalize the weights of the classiﬁer and apply resample in the classiﬁer retrain stage, similar to cRT.
Since two-stage methods require an additional training stage, we use LDAM as our default choice.
ImageNet-LT .
We further evaluate RIDE on ImageNet-LT with various backbones as in Table 2.
Compared with current SOTA methods, LWS and cRT, RIDE achieves new state-of-the-art results and outperforms SOTA by more than 7.7% with ResNet-50.
ResNeXt-50 is based on group convolution (Xie et al., 2017), which divides all ﬁlters into several groups and aggregates information from multiple groups.
ResNeXt-50 generally performs better than ResNet-50 on multiple tasks.
Using ResNeXt-50, we can see a performance improvement of 6.9%. [Table]
iNaturalist .
iNaturalist 2018 is a naturally imbalanced ﬁne grained dataset with 8,142 categories.
Table 3 shows that RIDE outperforms current SOTA by 6.3%.
Surprisingly, RIDE obtains very similar results in many-shots, medium-shots and few-shots, which is a very ideal result for the long tailed recognition task.
Current SOTA method BBN also uses multiple experts.
Compared with
BBN, which signiﬁcantly decreases the performance on many-shot classes by about 23%, RIDE increases the accuracy on few-shot without sacriﬁcing the performance on many-shot categories.
Contribution of each component of RIDE.
RIDE is jointly trained with L D-Diversify and L Classify , we use LDAM as L Classify by default.
Table 4 shows that the architectural change from the original ResNet-32 to the RIDE variant with 2 ∼ 4 experts contributes 2.7% ∼ 4.3% accuracy improvements.
The change from applying our classiﬁcation loss naively to model output (collaborative loss) to applying to each expert as an individual loss brings 1.5% improvements. Adding diversity further [Table]
[Figure] [Figure]
For RIDE with 3 or 4 experts, more than half of many-shot instances only require one expert.
On the contrary, more than 76% samples of few-shot classes require opinions from additional experts improves about 0.9%.
The computation cost is greatly reduced by adding the expert assignment module.
Knowledge distillation from RIDE with 6 experts obtains another 0.6% increase.
All these coherent components of RIDE enable it to get 7.1% increase compared with baseline LDAM.
Inﬂuence of expert number.
Fig.
4 shows the inﬂuence of expert number for each split.
Compared with many-shot, which only obtains 3.8% relative improvements by using 8 experts, the accuracy of few-shot classes is relative increased by more than 16%, which indicates that few-shot classes can enjoy more beneﬁts from using more experts.
No distillation is applied in this comparison
Number of experts allocated to each split.
Fig.
5 shows that: Most instances in the many-shot classes are assigned only 1 expert, whereas those in few-shot classes are often assigned more experts.
The low conﬁdence in low-shot instances requires the model to seek a second (or a third, ...) opinion.
S UMMARY
We propose a novel multi-expert approach for long-tailed recognition.
It trains partially shared diverse distribution-aware experts and routes an instance to additional experts when necessary, with computational complexity comparable to a single expert.
RIDE signiﬁcantly outperforms SOTA methods by a large margin on all the benchmarks including CIFAR100-LT, ImageNet-LT, and iNaturalist.
It can also be applied to various backbones and methods with consistent performance gains.
A A PPENDIX A. D ATASETS AND I MPLEMENTATIONS
We conduct experiments on three major long-tailed recognition benchmarks and different backbone networks to prove the effectiveness and universality of RIDE:
CIFAR100-LT (Krizhevsky, 2009; Cao et al., 2019):
The original version of CIFAR-100 contains 50,000 images on training set and 10,000 images on validation set with 100 categories.
The long-tailed version of CIFAR-100 follows an exponential decay in sample sizes across different categories.
We conduct experiment on CIFAR100-LT with an imbalance factor of 100,
i
e. the ratio between the most frequent class and the least frequent class.
To make fair comparison with previous works, we follow the training recipe of (Cao et al., 2019) on
CIFAR100-LT.
We train the ResNet-32 (He et al., 2016) backbone network by SGD optimizer with a momentum of 0.9.
CIFAR100-LT is trained for 200 epochs with standard data augmentations (He et al., 2016)
and a
batch size of 128 on one RTX
2080Ti
GPU
The learning rate is initialized as
and decayed by 0.01 at epoch 120 and 160 respectively.
ImageNet-LT (Deng et al., 2009; Liu et al., 2019): ImageNet-LT is constructed by sampling a subset of ImageNet-2012 following the Pareto distribution with the power value α = 6 (Liu et al., 2019).
ImageNet-LT consists of 115.8k images from 1,000 categories, with the largest and smallest categories containing 1,280 and 5 images, respectively.
Multiple backbone networks are experimented on ImageNet-LT, including ResNet-10, ResNetand ResNeXt-50 (Xie et al., 2017).
All backbone networks are trained with a batch size of 256 on
RTX 2080Ti GPUs for 100 epochs using SGD with an initial learning rate of 0.1 decayed by 0.1 at 60 epochs and 80 epochs.
We utilize standard data augmentations as in (He et al., 2016).
iNaturalist (Van Horn et al., 2018):
The iNaturalist-2018 dataset is an imbalanced datasets with 437,513 training images from 8,142 classes with a balanced test set of 24,426 images.
We use
ResNet-50 as the backbone network and apply the same training recipe as ImageNet-LT, except that we use a batch size of 512.
A. A DDITIONAL E XPERIMENTS
[Figure]
Ablation study on distillation methods.
Self-distillation step is optional but recommended if further improvements (0.4% ∼ 0.8% for most experiments) are desired.
We apply distillation from a more powerful model with more experts into a model with fewer experts.
A simple way to transfer knowledge is knowledge distillation (KD)
(Hinton et al., 2015), which applies KD loss ( L KD =
T 2 D KL ( (cid:126) l teacher /T, (cid:126) l expert i /T ) ) to match the distribution of logits of a teacher and a student.
We found that for teacher model with more experts using smaller distillation loss factor gives better performance.
We hypothesize that since we distill from the same teachers, giving large distillation factor prevents the branches from becoming as diversiﬁed as it is able to.
We also explored other distillation methods, such as CRD (Tian et al., 2019), PKT (Passalis & Tefas, 2018), and SP (Tung
[Table]
Mori, 2019), and compared the differences in Table 5.
Although adding other methods along with
KD may boost performance, the difference is small.
Therefore, we opt for simplicity and use KD only unless otherwise noticed.
Detailed results for ImageNet-LT experiments .
We list details of our ResNet-50 experiments in
ImageNet-LT on Table 7.
With 2 experts, we are able to achieve about 7% gain in accuracy with computational cost about 10% less than baseline.
In contrast to previous methods that sacriﬁce many-shot accuracy to get few-shot accuracy, we improve on all three splits on ImageNet-LT.
From 3 experts to 4 experts, we keep the same many-shot accuracy while increasing the few-shot accuracy, indicating that we are using the additional computational power to improve on the hardest part of the data rather than uniformly applying to all samples.
We also list our ResNet-10 and ResNeXt-50 experiments on Table 6 and 8, respectively, to compare against other works evaluated on these backbones.
Our method also achieves lower computational cost and higher performance when compared to other methods.
As illustrated in Fig.
6, our approach provides a comprehensive treatment to all the many-shot, medium-shot and few-shot classes, achieving substantial improvements to current state-of-the-art on all aspects.
Compared with cRT which reduces the performance on the many-shot classes, RIDE can achieves signiﬁcantly better performance on the few-shot classes without impairing the manyshot classes.
Similar observations can be obtained in the comparison with the state-of-the-art method
BBN (Zhou et al., 2020) on iNaturalist.
Comparison with ensemble method.
Since our method requires the joint decision from several experts, which raw ensembles also do, we also compare against ensembles of LDAM in Fig.7 on
CIFAR100-LT.
In the ﬁgure, even our method with 4 experts has less computational cost than the minimum computational cost for the ensemble of 2 LDAM models.
This indicates that our model is much more efﬁcient and powerful in terms of computational cost and accuracy than ensemble on long-tailed datasets.
t-SNE visualization .
We also provide the t-SNE visualization of embedding space on CIFAR100LT as in Fig. 8.
Compared with the baseline method LDAM, the feature embedding of RIDE is more compact for both the head and tail classes and better separated from the neighboring classes.
This greatly reduces the difﬁculty for the classiﬁer to distinguish the tail category.
[Figure]
A cc u r a c y 47.
RIDE and Ensembles of LDAM
[Figure] [Figure]
What if we apply RIDE to balanced datasets?
We also conducted experiments on CIFAR to check if our method can achieve similar performance gains on balanced datasets.
However, we only obtained an improvement of about 1%, which is much smaller than the improvements observed on the CIFAR100-LT.
Compared with balanced datasets, long-tailed datasets can get more beneﬁts from RIDE.
[Table]
CE) †
[Table]
[Table]
[Table]