Self-Supervised Aggregation of Diverse Experts for Test-Agnostic Long-Tailed Recognition
Abstract
Existing long-tailed recognition methods, aiming to train class-balanced models from long-tailed data, generally assume the models would be evaluated on the uniform test class distribution.
However, practical test class distributions often violate this assumption ( e.g., being either long-tailed or even inversely long-tailed), which may lead existing methods to fail in real applications.
In this paper, we study a more practical yet challenging task, called test-agnostic long-tailed recognition , where the training class distribution is long-tailed while the test class distribution is agnostic and not necessarily uniform .
In addition to the issue of class imbalance, this task poses another challenge: the class distribution shift between the training and test data is unknown.
To tackle this task, we propose a novel approach, called Self-supervised Aggregation of Diverse Experts , which consists of two strategies: (i) a new skill-diverse expert learning strategy that trains multiple experts from a single and stationary long-tailed dataset to separately handle different class distributions; (ii) a novel test-time expert aggregation strategy that leverages self-supervision to aggregate the learned multiple experts for handling unknown test class distributions.
We theoretically show that our self-supervised strategy has a provable ability to simulate test-agnostic class distributions.
Promising empirical results demonstrate the effectiveness of our method on both vanilla and test-agnostic long-tailed recognition.
Code is available at https://github. com/Vanint/SADE-AgnosticLT .
Introduction
Real-world visual recognition datasets typically exhibit a long-tailed distribution, where a few classes contain numerous samples (called head classes), but the others are associated with only a few instances (called tail classes)
Due to the class imbalance, the trained model is easily biased towards head classes and perform poorly on tail classes [2, 58].
To tackle this issue, numerous studies have explored long-tailed recognition for learning well-performing models from imbalanced data [20, 56].
Most existing long-tailed studies [3, 9, 10, 48, 52] assume the test class distribution is uniform, i.e., each class has an equal amount of test data.
Therefore, they develop various techniques, e.g., class resampling [13, 18, 25, 55], cost-sensitive learning [11, 36, 41, 47] or ensemble learning [2, 13, 27, 53], to re-balance the model performance on different classes for fitting the uniform class distribution.
However, this assumption does not always hold in real applications, where actual test data may follow any kind of class distribution, being either uniform, long-tailed, or even inversely long-tailed to the training data (cf.
Figure 1(a)).
For example, one may train a recognition model for autonomous cars based on the training data collected from city areas, where pedestrians are majority classes and stone obstacles are minority classes.
However, when the model is deployed to mountain areas, the pedestrians become the minority while the stones become the majority.
In this case, the test class distribution is inverse to the training one, and existing methods may perform poorly.
[Figure]
However, the resulting models may fail to handle practical test class distributions that skew arbitrarily
Our method seeks to learn a multi-expert model from a single long-tailed training set, where different experts are skilled in handling different class distributions, respectively.
By reasonably aggregating these experts at test time, our method is able to handle unknown test class distributions.
To address the issue of varying class distributions, as the first research attempt, LADE [17] assumes the test class distribution to be known and uses the knowledge to post-adjust model predictions.
However, the actual test class distribution is usually unknown a priori, making LADE not applicable in practice.
Therefore, we study a more realistic yet challenging problem, namely test-agnostic long-tailed recognition , where the training class distribution is long-tailed while the test distribution is agnostic .
To tackle this problem, motivated by the idea of "divide and conquer", we propose to learn multiple experts with diverse skills that excel at handling different class distributions (cf.
Figure 1(b)).
As long as these skill-diverse experts can be aggregated suitably at test time, the multi-expert model would manage to handle the unknown test class distribution.
Following this idea, we develop a novel approach, namely Self-supervised Aggregation of Diverse Experts (SADE).
The first challenge for SADE is how to learn multiple diverse experts from a single and stationary long-tailed training dataset.
To handle this challenge, we empirically evaluate existing long-tailed methods in this task, and find that the models trained by existing methods have a simulation correlation between the learned class distribution and the training loss function .
That is, the models learned by various losses are skilled in handling class distributions with different skewness.
For example, the model trained with the conventional softmax loss simulates the long-tailed training class distribution, while the models obtained from existing long-tailed methods are good at the uniform class distribution.
Inspired by this finding, SADE presents a simple but effective skill-diverse expert learning strategy to generate experts with different distribution preferences from a single long-tailed training distribution.
Here, various experts are trained with different expertise-guided objective functions to deal with different class distributions, respectively.
As a result, the learned experts are more diverse than previous multi-expert long-tailed methods [49, 63], leading to better ensemble performance, and in aggregate simulate a wide spectrum of possible class distributions.
The other challenge is how to aggregate these skill-diverse experts for handling test-agnostic class distributions based on only unlabeled test data .
To tackle this challenge, we empirically investigate the property of different experts, and observe that there is a positive correlation between expertise and prediction stability , i.e., stronger experts have higher prediction consistency between different perturbed views of samples from their favorable classes.
Motivated by this finding, we develop a novel self-supervised strategy, namely prediction stability maximization, to adaptively aggregate experts based on only unlabeled test data.
We theoretically show that maximizing the prediction stability enables SADE to learn an aggregation weight that maximizes the mutual information between the predicted label distribution and the true class distribution.
In this way, the resulting model is able to simulate unknown test class distributions.
We empirically verify the superiority of SADE on both vanilla and test-agnostic long-tailed recognition.
Specifically, SADE achieves promising performance on vanilla long-tailed recognition under all benchmark datasets.
For instance, SADE achieves 58.8 % accuracy on ImageNet-LT with more than 2 % accuracy gain over previous state-of-the-art ensemble long-tailed methods, i.e., RIDE [49] and ACE [2].
More importantly, SADE is the first long-tailed approach that is able to handle various test-agnostic class distributions without knowing the true class distribution of test data in advance.
Note that SADE even outperforms LADE [17] that uses knowledge of the test class distribution.
Compared to previous long-tailed methods ( e.g., LADE [17] and RIDE [49]), our method offers the following advantages: (i) SADE does not assume the test class distribution to be known, and provides the first practical approach to handling test-agnostic long-tailed recognition; (ii) SADE develops a simple diversity-promoting strategy to learn skill-diverse experts from a single and stationary long-tailed dataset; (iii) SADE presents a novel self-supervised strategy to aggregate skill-diverse experts at test time, by maximizing prediction consistency between unlabeled test samples’ perturbed views; (iv) the presented self-supervised strategy has a provable ability to simulate test-agnostic class distributions, which opens the opportunity for tackling unknown class distribution shifts at test time.
Related Work Long-tailed recognition
Existing long-tailed recognition methods, related to our study, can be categorized into three types: class re-balancing, logit adjustment and ensemble learning.
Specifically, class re-balancing resorts to re-sampling [4, 13, 18, 25] or cost-sensitive learning [3, 10, 16, 61] to balance different classes during model training.
Logit adjustment [17, 33, 37, 43] adjusts models’ output logits via the label frequencies of training data at inference time, for obtaining a large relative margin between head and tail classes.
Ensemble-based methods [2, 13, 53, 63], e.g., RIDE [49], are based on multiple experts, which seek to capture heterogeneous knowledge, followed by ensemble aggregation.
More discussions on the difference between our method and RIDE [49] are provided in
Appendix D.3.
Regarding test-agnostic long-tailed recognition, LADE [17] assumes the test class distribution is available and uses it to post-adjust model predictions.
However, the true test class distribution is usually unknown a priori, making LADE inapplicable.
In contrast, our method does not rely on the true test distribution for handling this problem, but presents a novel self-supervised strategy to aggregate skill-diverse experts at test time for test-agnostic class distributions.
Moreover, some ensemble-based long-tailed methods [39] aggregate experts based on a labeled uniform validation set.
However, as the test class distribution could be different from the validation one, simply aggregating experts on the validation set is unable to handle test-agnostic long-tailed recognition.
Test-time training Test-time training [23, 26, 30, 40, 46] is a transductive learning paradigm for handling distribution shifts [28, 32, 34, 38, 45, 59] between training and test data, and has been applied with success to out-of-domain generalization [19, 35] and dynamic scene deblurring [6].
In this study, we explore this paradigm to handle test-agnostic long-tailed recognition, where the issue of class distribution shifts is the main challenge.
However, most existing test-time training methods seek to handle covariate distribution shifts instead of class distribution shifts, so simply leveraging them cannot resolve test-agnostic long-tailed recognition, as shown in our experiment (cf.
Table 9).
Problem Formulation
Long-tailed recognition aims to learn a well-performing classification model from a training dataset with long-tailed class distribution.
Let D s =
{ x i , y i } n s i =1 denote the long-tailed training set, where y i is the class label of the sample x i .
The total number of training data over C classes is n s = (cid:80) Ck =1 n k , where n k denotes the number of samples in class k .
Without loss of generality, we follow a common assumption [17, 25] that the classes are sorted by cardinality in decreasing order ( i.e., if i 1 <
i 2 , then n i 1 ≥ n i 2 ), and n 1 ≫ n C .
The imbalance ratio is defined as max( n k ) / min( n k ) = n 1 / n C .
The test data D t = { x j , y j } n t j =1 is defined in a similar way.
Most existing long-tailed recognition methods assume the test class distribution is uniform ( i.e., p t ( y )
= 1 /C ), and seek to train models from the long-tailed training distribution p s ( y ) to perform well on the uniform test distribution.
However, such an assumption does not always hold in practice.
The actual test class distribution in real-world applications may also be long-tailed ( i.e., p t ( y )
=
p s ( y ) ), or even inversely long-tailed to the training data ( i.e., p t ( y )
=
inv ( p s ( y )) ).
Here, inv ( · ) indicates that the order of the long tail on classes is flipped.
As a result, the models learned by existing methods may fail when the actual test class distribution is different from the assumed one.
To address this issue, we propose to study a more practical yet challenging long-tailed recognition problem, i.e.,
Test-agnostic Long-tailed Recognition .
This task aims to learn a recognition model from long-tailed training data, where the resulting model would be evaluated on multiple test sets that follow different class distributions.
This task is challenging due to the integration of two challenges: (1) the severe class imbalance in the training data makes it difficult to train models;
(2) unknown class distribution shifts between training and test data ( i.e., p t ( y ) ̸
= p s ( y ) ) makes models hard to generalize.
[Table]
Method
To tackle the above problem
[Table]
inspired by the idea of "divide and conquer"
[Table]
we propose to learn multiple skill-diverse experts that excel at handling different class distributions.
By reasonably fusing these experts at test time
[Table]
the multi-expert model would manage to handle unknown class distribution shifts and resolve test-agnostic long-tailed recognition.
Following this idea
[Table]
we develop a novel
Self-supervised Aggregation of Diverse Experts (SADE) approach.
Specifically, SADE consists of two innovative strategies: (1) learning skill-diverse experts from a single long-tailed training dataset; (2) test-time aggregating experts with self-supervision to handle test-agnostic class distributions.
4.1 Skill-diverse Expert Learning
As shown in Figure
[Table]
SADE builds a three-expert model that comprises two components: (1) an expert-shared backbone f θ ; (2) independent expert networks E 1 , E 2 and E 3 .
When training the model, the key challenge is how to learn skill-diverse experts from a single and stationary long-tailed training dataset.
Existing ensemble-based long-tailed methods [13, 49] seek to train experts for the uniform test class distribution, and hence the trained experts are not differentiated sufficiently for handling various class distributions (refer to Table 6 for an example).
To tackle this challenge, we first empirically investigate existing long-tailed methods in this task.
From Table 1, we find that there is a simulation correlation between the learned class distribution and the training loss function .
That is, the models learned by different losses are good at dealing with class distributions with different skewness.
For instance, the model trained with the softmax loss is good at the long-tailed distribution, while the models obtained from long-tailed methods are skilled in the uniform distribution.
[Figure]
Motivated by this finding, we develop a simple skill-diverse expert learning strategy to generate experts with different distribution preferences.
To be specific, the forward expert E 1 seeks to be good at the long-tailed class distribution and performs well on many-shot classes.
The uniform expert E 2 strives to be skilled in the uniform distribution.
The backward expert E 3 aims at the inversely long-tailed distribution and performs well on few-shot classes.
Here, the forward and backward experts are necessary since they span a wide spectrum of possible class distributions, while the uniform expert ensures retaining high accuracy on the uniform distribution.
To this end, we use three different expertise-guided losses to train the three experts, respectively.
The forward expert E
We use the softmax cross-entropy loss to train this expert, so that it directly simulates the original long-tailed training class distribution:
[Equation]
where v 1 ( · ) is the output logits of the forward expert E 1 , and σ ( · ) is the softmax function.
The uniform expert E
We aim to train this expert to simulate the uniform class distribution.
Inspired by the effectiveness of logit adjusted losses for long-tailed recognition [33], we resort to the balanced softmax loss [21].
Specifically, let ˆ y k
=
[Equation]
be the prediction probability.
The balanced softmax adjusts the prediction probability by compensating for the long-tailed class distribution with the prior of training label frequencies: ˆ y k
=
[Equation]
where π k = n k n denotes the training label frequency of class k .
Then, given v 2 ( · ) as the output logits of the expert E 2 , the balanced softmax loss for the expert E 2 is defined as:
[Equation]
Intuitively, by adjusting logits to compensate for the long-tailed distribution with the prior π , this loss enables E 2 to output class-balanced predictions that simulate the uniform distribution.
The backward expert E
We seek to train this expert to simulate the inversely long-tailed class distribution.
To this end, we propose a new inverse softmax loss , based on the same rationale of logit adjusted losses [21, 33].
Specifically, we adjust the prediction probability by: ˆ y k =
[Equation]
) , where the inverse training prior ¯ π is obtained by inverting the order of training label frequencies π .
Then, the new inverse softmax loss for the expert E 3 is defined as:
[Equation]
where v 3 ( · ) denotes the output logits of E 3 and λ is a hyper-parameter.
Intuitively, this loss adjusts logits to compensate for the long-tailed distribution with π , and further applies reverse adjustment with ¯ π .
This enables E 3 to simulate the inversely long-tailed distribution (cf.
Table 6 for verification).
4.2 Test-time Self-supervised Aggregation
Based on the skill-diverse learning strategy, the three experts in SADE are skilled in different class distributions.
The remaining challenge is how to fuse them to deal with unknown test class distributions.
A basic principle for expert aggregation is that the experts should play a bigger role in situations where they have expertise.
Nevertheless, how to detect strong experts for unknown test class distribution remains unknown.
Our key insight is that strong experts should be more stable in predicting the samples from their skilled classes, even though these samples are perturbed.
Empirical observation To verify this hypothesis, we estimate the prediction stability of experts by comparing the cosine similarity between their predictions for a sample’s two augmented views.
Here, the data views are generated by the data augmentation techniques in MoCo v2 [5].
From Table 2, we find that there is a positive correlation between expertise and prediction stability , i.e., stronger experts have higher prediction similarity between different views of samples from their favorable classes.
Following this finding, we propose to explore the relative prediction stability to detect strong experts and weight experts for the unknown test class distribution.
Consequently, we develop a novel self-supervised strategy, namely prediction stability maximization.
Prediction stability maximization
This strategy learns aggregation weights for experts (with frozen parameters) by maximizing model prediction stability for unlabeled test samples.
As shown in
Figure 3, the method comprises three major components as follows.
Data view generation
For a given sample x , we conduct two stochastic data augmentations to generate the sample’s two views, i.e., x 1 and x 2 .
Here, we use the same augmentation techniques as the advanced contrastive learning method, i.e., MoCo v2 [5], which has been shown effective in self-supervised learning.
Learnable aggregation weight
Given the output logits of three experts ( v 1 , v 2 , v 3 ) ∈ R 3 × C , we aggregate experts with a learnable aggregation weight w =
[ w 1 , w 2 , w 3 ] ∈ R 3 and obtain the final softmax prediction by ˆ y = σ ( w 1 · v 1 + w 2 · v 2 + w 3 · v 3 ) , where w is normalized before aggregation, i.e., w 1 + w 2 + w 3 =1 .
[Table]
Maximizepredictionstability
Logits
[Table]
𝑦
[Table]
Objective function Given the view predictions of unlabeled test data, we maximize the prediction stability based on the cosine similarity between the view predictions:
[Equation]
Here, ˆ y 1 and ˆ y 2 are normalized by the softmax function.
In test-time training, only the aggregation weight w is updated.
Since stronger experts have higher prediction similarity for their skilled classes, maximizing the prediction stability S would learn higher weights for stronger experts regarding the unknown test class distribution.
Moreover, the self-supervised aggregation strategy can be conducted in an online manner for streaming test data.
The pseudo-code of SADE is provided in Appendix B.
Theoretical Analysis We then theoretically analyze the prediction stability maximization strategy to conceptually understand why it works.
To this end, we first define the random variables of predictions and labels as ˆ Y ∼ p (ˆ y ) and Y ∼ p t ( y ) .
We have the following result:
Theorem 1.
The prediction stability S is positive proportional to the mutual information between the predicted label distribution and the test class distribution
I ( ˆ Y ; Y ) , and negative proportional to the prediction entropy H ( ˆ Y ) :
[Equation]
Please refer to Appendix A for proofs.
According to Theorem 1, maximizing the prediction stability
S enables SADE to learn an aggregation weight that maximizes the mutual information between the predicted label distribution p (ˆ y ) and the test class distribution p t ( y ) , as well as minimizing the prediction entropy.
Since minimizing entropy helps to improve the confidence of the classifier output [12], the aggregation weight is learned to simulate the test class distribution p t ( y ) and increase the prediction confidence.
This property intuitively explains why our method has the potential to tackle the challenging task of test-agnostic long-tailed recognition at test time.
Experiments
In this section, we first evaluate the superiority of SADE on both vanilla and test-agnostic long-tailed recognition.
We then verify the effectiveness of SADE in terms of its two strategies, i.e., skill-diverse expert learning and test-time self-supervised aggregation.
More ablation studies are reported in appendices.
Here, we begin with the experimental settings.
5.1 Experimental Setups Datasets
We use four benchmark datasets ( i.e., ImageNet-LT [31], CIFAR100-LT [3], PlacesLT [31], and iNaturalist 2018 [44]) to simulate real-world long-tailed class distributions.
Their data statistics and imbalance ratios are summarized in Appendix C.1.
The imbalance ratio is defined as max n j / min n j , where n j denotes the data number of class j .
Note that CIFAR100-LT has three variants with different imbalance ratios.
[Table]
Baselines
We compare SADE with state-of-the-art long-tailed methods, including two-stage methods (Decouple [25], MiSLAS [62]), logit-adjusted training (Balanced Softmax [21], LADE [17]), ensemble learning (BBN [63], ACE [2], RIDE [49]), classifier design (Causal [42])
[Table]
and representation learning (PaCo [8]).
Note that LADE uses the prior of test class distribution for post-adjustment (although it is unavailable in practice), while all other methods do not use this prior
[Table]
Evaluation protocols
In test-agnostic long-tailed recognition, following LADE [17], the models are evaluated on multiple sets of test data that follow different class distributions, in terms of micro accuracy.
Same as LADE [17], we construct three kinds of test class distributions, i.e., the uniform distribution, forward long-tailed distributions as training data, and backward long-tailed distributions.
In the backward ones, the order of the long tail on classes is flipped.
More details of test data construction are provided in Appendix C.2.
Besides, we also evaluate methods on vanilla long-tailed recognition [25, 31], where the models are evaluated on the uniform test class distribution.
Here, the accuracy on three class sub-groups is also reported, i.e., many-shot classes (more than 100 training images), medium-shot classes (20 ∼ 100 images) and few-shot classes (less than 20 images).
Implementation details We use the same setup for all the baselines and our method.
Specifically, following [17, 49], we use ResNeXt-50 for ImageNet-LT, ResNet-32 for CIFAR100-LT, ResNetfor Places-LT and ResNet-50 for iNaturalist 2018 as backbones, respectively.
Moreover, we adopt the cosine classifier for prediction on all datasets.
If not specified, we use the SGD optimizer with the momentum of 0.9 for training 200 epochs and set the initial learning rate as 0.1 with linear decay.
We set λ =2 for ImageNet-LT and CIFAR100-LT, and λ =1 for the remaining datasets.
During test-time training, we train the aggregation weights for 5 epochs with the batch size 128, where we use the same optimizer and learning rate as the training phase.
More implementation details and the hyper-parameter statistics are reported in Appendix C.3.
5.2 Superiority on Vanilla Long-tailed Recognition
[Table]
This subsection compares SADE with state-of-the-art long-tailed methods on vanilla long-tailed recognition.
Specifically, as shown in Tables 3-4, Softmax trains the model with only cross-entropy, so it simulates the longtailed training distribution and performs well on manyshot classes.
However, it performs poorly on mediumshot and few-shot classes, leading to worse overall performance.
In contrast, existing long-tailed methods ( e.g.,
Decouple, Causal) seek to simulate the uniform class distribution, so their performance is more class-balanced, leading to better overall performance.
However, as these methods mainly seek balanced performance, they inevitably sacrifice the performance on many-shot classes.
To address this, RIDE and ACE explore ensemble learning for long-tailed recognition and achieve better performance on tail classes without sacrificing the head-class performance.
In comparison, based on the increasing expert diversity derived from skill-diverse expert learning, our method performs the best on all datasets, e.g., with more than 2 % accuracy gain on ImageNet-LT compared to RIDE and ACE.
These results demonstrate the superiority of SADE over the compared methods that are particularly designed for the uniform test class distribution.
Note that SADE also outperforms baselines in experiments with stronger data augmentation ( i.e., RandAugment [7]) and other architectures, as reported in Appendix D.1.
[Table] [Table]
5.3 Superiority on Test-agnostic Long-tailed Recognition
In this subsection
[Table]
we evaluate SADE on test-agnostic long-tailed recognition.
The results on various test class distributions are reported in Table 5.
Specifically, since Softmax seeks to simulate the longtailed training distribution, it performs well on forward long-tailed test distributions.
However, its performance on the uniform and backward long-tailed distributions is poor.
In contrast, existing longtailed methods show more balanced performance among classes, leading to better overall accuracy.
However, the resulting models by these methods suffer from a simulation bias, i.e., performing similarly among classes on various class distributions (c.f. Table 1).
As a result, they cannot adapt to diverse test class distributions well.
To handle this task, LADE assumes the test class distribution to be known and uses this information to adjust its predictions, leading to better performance on various test class distributions.
However, since obtaining the actual test class distribution is difficult in real applications, the methods requiring such knowledge may be not applicable in practice.
Moreover, in some specific cases like Forward-LT-3 and Backward-LT-3 distributions of iNaturalist 2018, the number of test samples on some classes becomes zero.
In such cases, the test prior cannot be used in
LADE, since adjusting logits with log 0 results in biased predictions.
In contrast, without relying on the knowledge of test class distributions, our SADE presents an innovative self-supervised strategy to deal with unknown class distributions, and obtains even better performance than LADE that uses the test class prior
(c.f. Table 5).
The promising results demonstrate the effectiveness and practicality of our method on test-agnostic long-tailed recognition.
Note that the performance advantages of SADE become larger as the test data get more imbalanced.
Due to the page limitation, the results on more datasets are reported in Appendix D.2.
5.4 Effectiveness of Skill-diverse Expert Learning
We next examine our skill-diverse expert learning strategy.
The results are reported in Table 6, where
RIDE [49] is a state-of-the-art ensemble-based method.
RIDE trains each expert with cross-entropy independently and uses KL-Divergence to improve expert diversity.
However, simply maximizing the divergence of expert predictions cannot learn visibly diverse experts (cf.
Table 6).
In contrast, the three experts learned by our strategy have significantly diverse expertise, excelling at many-shot classes, the uniform distribution (with higher overall performance), and few-shot classes, respectively.
As a result, the increasing expert diversity leads to a non-trivial gain for the ensemble performance of SADE compared to RIDE.
Moreover, consistent results on more datasets are reported in Appendix D.3, while the ablation studies of the expert learning strategy are provided in Appendix E.
[Table]
[Table] [Table]
5.5 Effectiveness of Test-time Self-supervised Aggregation
This subsection evaluates our test-time self-supervised aggregation strategy.
Effectiveness in expert aggregation.
As shown in Table 7, our self-supervised strategy learns suitable expert weights for various unknown test class distributions.
For forward long-tailed distributions, the weight of the forward expert E 1 is higher; while for backward long-tailed ones, the weight of the backward expert E 3 is relatively high.
This enables our multi-expert model to boost the performance on dominant classes for unknown test distributions, leading to better ensemble performance (cf.
Table 8), particularly as test data get more skewed.
The results on more datasets are reported in Appendix D.4, while more ablation studies of our strategy are shown in Appendix F.
Superiority over test-time training methods.
We then verify the superiority of our self-supervised strategy over existing test-time training approaches on various test class distributions.
Specifically, we adopt three non-trivial baselines: (i) Test-time pseudo-labeling uses the multi-expert model to iteratively generate pseudo labels for unlabeled test data and uses them to fine-tune the model; (ii)
Test class distribution estimation leverages BBSE [29] to estimate the test class distribution and uses it to pose-adjust model predictions; (iii) Tent [46] fine-tunes the batch normalization layers of models through entropy minimization on unlabeled test data.
The results in Table 9 show that directly applying existing test-time training methods cannot handle well the class distribution shifts, particularly on the inversely long-tailed class distribution.
In comparison, our self-supervised strategy is able to aggregate multiple experts appropriately for the unknown test class distribution (cf.
Table 7), leading to promising performance gains on various test class distributions (cf.
Table 9).
[Table]
Effectiveness on partial class distributions.
Real-world test data may follow any type of class distribution, including partial class distributions ( i.e., not all of the classes appear in the test data).
Motivated by this, we further evaluate
SADE on three partial class distributions: only many-shot classes, only medium-shot classes, and only few-shot classes.
The results in Table 10 demonstrate the effectiveness of SADE in tackling more complex test class distributions.
Conclusion
In this paper, we have explored a practical yet challenging task of test-agnostic long-tailed recognition , where the test class distribution is unknown and not necessarily uniform.
To tackle this task, we present a novel approach, namely Self-supervised Aggregation of Diverse Experts (SADE), which consists of two innovative strategies, i.e., skill-diverse expert learning and test-time self-supervised aggregation.
We theoretically analyze our proposed method and also empirically show that SADE achieves new state-of-the-art performance on both vanilla and test-agnostic long-tailed recognition.
Acknowledgments
This work was partially supported by NUS ODPRT
Grant R252-000-A81-133 and NUS Advanced
Research and Technology Innovation Centre (ARTIC) Project Reference (ECT-RP2).
We also gratefully appreciate the support of MindSpore, CANN (Compute Architecture for Neural Networks) and Ascend AI Processor used for this research.
A Proofs for Theorem
Proof.
We first recall several key notations and define some new notations.
The random variables of model predictions and ground-truth labels are defined as ˆ Y ∼ p (ˆ y ) and Y ∼ p ( y ) , respectively.
The number of classes is denoted by C .
Moreover, we further denote the test sample set of the class k by
Z k , in which the total number of samples in this class is denoted by
|Z k | .
Let c k = 1 |Z k | (cid:80) ˆ y ∈Z
k ˆ y represent the hard mean of all predictions of samples from the class k , and let c = indicate equality up to a multiplicative and/or additive constant.
As shown in Eq.(4), the optimization objective of our test-time self-supervised aggregation method is to maximize S
= (cid:80) n t j =1 ˆ y 1 j · ˆ y 2 j , where n t denotes the number of test samples.
For convenience, we simplify the first data view to be the original data, so the objective function becomes (cid:80) n t j =1 ˆ y j · ˆ y 1 j .
Maximizing such an objective is equivalent to minimizing (cid:80) n t j =1 − ˆ y j
· ˆ y 1 j .
Here, we assume the data augmentations are strong enough to generate representative data views that can simulate the test data from the same class.
In this sense, the new data view can be regarded as an independent sample from the same class.
Following this, we analyze our method by connecting − ˆ y j
· ˆ y 1 j to (cid:80) ˆ y j ∈Z
k
[Equation]
ˆ
[Equation]
j −
[Equation]
k ∥ 2 , which is similar to the tightness term in the center loss [50]:
[Equation]
where we use the property of the normalized predictions, i.e., ∥ ˆ y j ∥ 2 = ∥ ˆ y 1 j ∥ 2 = 1 , and the definition of the class hard mean c k = 1 |Z k | (cid:80) ˆ y ∈Z k ˆ y .
By summing over all classes k , we obtain:
[Equation]
Based on this equation, following [1, 57], we can interpret (cid:80) n t j =1 − ˆ y j · ˆ y 1 j as a conditional crossentropy between ˆ Y and another random variable ¯ Y , whose conditional distribution given Y is a standard Gaussian centered around c Y : ¯ Y | Y ∼N ( c y , i ) :
[Equation]
Hence, we know that (cid:80) n t j =1 − ˆ y j
· ˆ y 1 j is an upper bound on the conditional entropy of predictions ˆ Y given labels Y :
[Equation]
where the symbol c ≥ represents “larger than" up to a multiplicative and/or an additive constant.
Moreover, when ˆ Y | Y ∼N ( c y , i ) , the bound is tight.
As a result, minimizing (cid:80) n t j =1 − ˆ y j · ˆ y 1 j is equivalent to minimizing H ( ˆ Y | Y ) :
[Equation]
Meanwhile, the mutual information between predictions ˆ Y and labels Y can be represented by:
[Equation] [Equation]
Combining Eqs.(5-6), we have:
[Equation]
Since S = (cid:80) n t j =1 ˆ y j · ˆ y 1 j , we obtain:
[Equation]
which concludes the proof for Theorem 1.
B Pseudo-code
This appendix provides the pseudo-code 1 of SADE, which consists of skill-diverse expert learning and test-time self-supervised aggregation.
Here, the skill-diverse expert learning strategy is summarized in Algorithm 1.
For simplicity, we depict the pseudo-code based on batch size 1, but we conduct batch gradient descent in practice.
Require:
Epochs T ; Hyper-parameters λ for L inv Initialize: Network backbone f θ ; Experts E 1 , E 2 , E1: for e=1,..., T do 2: for x ∈ D s do // batch sampling in practice 3:
Obtain logits v 1 based on f θ and E 1 ; 4:
Obtain logits v 2 based on f θ and E 2 ; 5:
Obtain logits v 3 based on f θ and E 3 ; 6:
Compute loss L ce with v 1 for Expert E 1 ; // Eq.(1) 7:
Compute loss L bal with v 2 for Expert E 2 ; // Eq.(2) 8:
Compute loss L inv with v 3 for Expert E 3 ; // Eq.(3) 9:
Train the model with L ce + L bal + L inv .
10: end for 11: end for Output: The trained model { f θ , E 1 , E 2 , E 3 }
After training the multiple skill-diverse experts with Algorithm 1, the final prediction of the multiexpert model for vanilla long-tailed recognition is the arithmetic mean of the prediction logits of these experts, followed by a softmax function.
When it comes to test-agnostic long-tailed recognition, we need to aggregate these skill-diverse experts to handle the unknown test class distribution based on Algorithm 2.
Here, to avoid the learned weights of some weak experts becoming zero, we give a stopping condition in Algorithm 2: if the weight for one expert is less than 0.05, we stop test-time training.
Retaining a small amount of weight for each expert is sufficient to ensure the effect of ensemble learning.
Require: Epochs T ′ ;
The trained backbone f θ ; The trained experts E 1 , E 2 , E 3 Initialize: Expert aggregation weights w // uniform initialization 1: for e=1,..., T ′ do 2: for x ∈ D t do // batch sampling in practice 3:
Draw two data augmentation functions t ∼T , t ′ ∼T ; 4:
Generate data views x 1 = t ( x ) , x 2 = t ′ ( x ) ; 5:
Obtain logits v 11 , v 12 , v 13 for the view x 1 ; 6:
Obtain logits v 21 , v 22 , v 23 for the view x 2 ; 7:
Normalize expert weights w via softmax function; 8:
Conduct predictions ˆ y 1 , ˆ y 2 based on ˆ y =
wv ; 9:
Compute prediction stability S ; // Eq.
Maximize S to update w ; 11: end for 12:
If w i ≤ 0 .
for any w i ∈ w , then stop training.
13: end for
Output: Expert aggregation weights w
Note that, in test-agnostic long-tailed recognition, each model is only trained once on long-tailed training data and then directly evaluated on multiple test sets.
Our test-time self-supervised strategy adapts the trained multi-expert model using only unlabeled test data during testing.
C More Experimental Settings
In this appendix, we provide more details on experimental settings.
C.1 Benchmark Datasets
We use four benchmark datasets ( i.e., ImageNet-LT [31], CIFAR100-LT [3], Places-LT [31], and iNaturalist 2018 [44]) to simulate real-world long-tailed class distributions.
These datasets suffer from severe class imbalance [22, 60].Their data statistics are summarized in Table 11, where CIFAR100-LT has three variants with different imbalance ratios.
The imbalance ratio is defined as max n j / min n j , where n j denotes the data number of class j .
[Table]
C.2 Construction of Test-agnostic Long-tailed Datasets
Following LADE [17], we construct three kinds of test class distributions, i.e., the uniform distribution, forward long-tailed distributions and backward long-tailed distributions.
In the backward ones, the long-tailed class order is flipped.
Here, the forward and backward long-tailed test distributions contain multiple different imbalance ratios, i.e., ρ ∈ { 2 , 5 , 10 , 25 , 50 } .
Note that LADE [17] only constructed multiple distribution-agnostic test datasets for ImageNet-LT; while in this study, we use the same way to construct distribution-agnostic test datasets for the remaining benchmark datasets, i.e., CIFAR100-LT, Places-LT and iNaturalist 2018, as illustrated below.
Considering the long-tailed training classes are sorted in a decreasing order, the various test datasets are constructed as follows: (1) Forward long-tailed distribution: the number of the j -th class is n j = N · ρ ( j − 1) /C , where N indicates the sample number per class in the original uniform test dataset and C is the number of classes.
(2) Backward long-tailed distribution: the number of the j -th class is n j = N · ρ ( C − j ) /C .
In the backward long-tailed distributions, the order of the long tail on classes is flipped, so the distribution shift between training and test data is large, especially when the imbalance ratio gets higher.
For ImageNet-LT, CIFAR100-LT and Places-LT, since there are enough test samples per class, we follow the setting in LADE [17] and construct the imbalance ratio set by ρ ∈ { 2 , 5 , 10 , 25 , 50 } .
For iNaturalist 2018, since each class only contains three test samples, we adjust the imbalance ratio set to ρ ∈ { 2 , 3 } .
Note that when we set ρ = 3 , there are some classes in iNaturalist 2018 containing no test sample.
All these constructed distribution-agnostic long-tailed datasets will be publicly available along with our code.
C.3 More Implementation Details of Our Method
We implement our method in PyTorch.
Following [17, 49], we use ResNeXt-50 for ImageNet-LT,
ResNet-32 for CIFAR100-LT, ResNet-152 for Places-LT and ResNet-50 for iNaturalist 2018 as backbones, respectively.
Moreover, we adopt the cosine classifier for prediction on all datasets.
Although we have depicted the skill-diverse multi-expert framework in Section 4.1, we give more details about it here.
Without loss of generality, we take ResNet [15] as an example to illustrate the multi-expert model.
Since the shallow layers extract more general features and deeper layers extract more task-specific features [54], the three-expert model uses the first two stages of ResNet as the expert-shared backbone, while the later stages of ResNet and the fully-connected layer constitute independent components of each expert.
To be more specific, the number of convolutional filters in each expert is reduced by 1/4, since by sharing the backbone and using fewer filters in each expert [49, 63], the computational complexity of the model is reduced compared to the model with independent experts.
The final the arithmetic mean the prediction logits of these experts, by a softmax function.
In the training phase, the data augmentations are the same as previous long-tailed studies [17, 25].
If not specified, we use the SGD optimizer with the momentum of 0.9 and set the initial learning rate as 0.1 with linear decay.
More specifically, for ImageNet-LT, we train models for 180 epochs with batch size 64 and a learning rate of 0.025 (cosine decay).
For CIFAR100-LT, the training epoch is 200 and the batch size is 128.
For Places-LT, following [31], we use ImageNet pre-trained ResNet-152 as the backbone, while the batch size is set to 128 and the training epoch is 30.
Besides, the learning rate is 0.01 for the classifier and 0.001 for all other layers.
For iNaturalist 2018, we set the training epoch to 200, the batch size to 512 and the learning rate to 0.2.
In our inverse softmax loss, we set λ =2 for
ImageNet-LT and CIFAR100-LT, and λ =1 for the remaining datasets.
In the test-time training, we use the same augmentations as MoCo v2 [5] to generate different data views, i.e., random resized crop, color jitter, gray scale, Gaussian blur and horizontal flip.
If not specified, we train the aggregation weights for 5 epochs with the batch size 128, where we adopt the same optimizer and learning rate as the training phase.
More detailed statistics of network architectures and hyper-parameters are reported in Table 12.
Based on these hyper-parameters, we conduct experiments on 1 TITAN RTX 2080 GPU for CIFAR100-LT, 4 GPUs for iNaturalist18, and 2 GPUs for ImageNet-LT and Places-LT, respectively.
The source code of our method is available in the supplementary material.
[Table]
C.4 Discussions on Evaluation Metric
As mentioned in Section 5.1, we follow LADE [17] and use micro accuracy to evaluate model performance on test-agnostic long-tailed recognition.
In this appendix, we explain why micro accuracy is a better metric than macro accuracy when the test dataset exhibits a non-uniform class distribution.
For instance, in the test scenario with a backward long-tailed class distribution, the tail classes are more frequently encountered than the head classes, and thus should have larger weights in evaluation.
However, simply using macro accuracy treats all the categories equally and cannot differentiate classes of different frequencies.
For example, one may train a recognition model for autonomous cars based on the training data collected from city areas, where pedestrians are majority classes and stone obstacles are minority classes.
Assume the model accuracy is 60% on pedestrians and 40% on stones.
If deploying the model to city areas, where pedestrians/stones are assumed to have 500/50 test data, then the macro accuracy is 50% and the micro accuracy is 500 × 0 .
6+50 × 0 .
4 500+ ≈ 58% .
In contrast, when deploying the model to mountain areas, the pedestrians become the minority, while stones become the majority.
Assuming the test data numbers are changed to 50/500 on pedestrians/stones, the micro accuracy is adjusted to 50 × 0 .
6+500 × 0 .
4 50+ ≈ 42% ,
but the macro accuracy is still 50%.
In this case, macro accuracy is less informative than micro accuracy for measuring model performance.
Therefore, micro accuracy is a better metric to evaluate the performance of test-agnostic long-tailed recognition.
D More Empirical Results D.1 More Results on Vanilla Long-tailed Recognition Accuracy on class subsets
In the main paper, we have provided the average performance over all classes on the uniform test class distribution.
In this appendix, we further report the accuracy regarding various class subsets (c.f.
Table 13), making the results more complete.
[Table]
Results on stonger data augmentations Inspired by PaCo [8], we further evaluate SADE training with stronger data augmentation ( i
[Table]
e., RandAugment [7]) for 400 epochs.
The results in Tablefurther demonstrate the state-of-the-art performance of SADE
[Table] [Table]
Results on more neural architectures In addition to using the common practice of backbones as previous long-tailed studies [17, 49]
[Table]
we further evaluate SADE on more neural architectures.
The results in Table 15 demonstrate that SADE is able to train different network backbones well
[Table] [Table]
Results on more datasets We also conduct experiments on CIFAR10-LT with imbalance ratios of 10 and 100.
Promising results in Table 16 further demonstrate the effectiveness and superiority of our proposed method.
[Table]
D.2 More Results on Test-agnostic Long-tailed Recognition
In the main paper [Table] we have provided the overall performance on four benchmark datasets with various test class distributions (cf.
Table 5).
In this appendix, we further verify the effectiveness of our method on more dataset settings ( i
[Table]
e., CIFAR100-IR10 and CIFAR100-IR50), as shown in Table 17.
[Table]
Furthermore, we plot the results of all methods under these benchmark datasets with various test class distributions in Figure 4.
To be specific, Softmax only performs well on highly-imbalanced forward long-tailed class distributions.
Existing long-tailed baselines outperform Softmax, but they cannot handle backward test class distributions well.
In contrast, our method consistently outperforms baselines on all benchmark datasets, particularly on the backward long-tailed test distributions with a relatively large imbalance ratio.
[Figure] [Figure]
D.3 More Results on Skill-diverse Expert Learning
This appendix further evaluates the skill-diverse expert learning strategy on CIFAR100-LT, Places-LT and iNaturalist 2018 datasets.
We report the results in Table 18, from which we draw the following observations.
RIDE [49] is one of the state-of-the-art ensemble-based long-tailed methods, which tries to learn diverse distribution-aware experts by maximizing the divergence among expert predictions.
However, such a method cannot learn sufficiently diverse experts.
As shown in Table 18, the three experts in RIDE perform very similarly on various groups of classes under all benchmark datasets, and each expert has similar overall performance on each dataset.
Such results demonstrate that simply maximizing the KL divergence of different experts’ predictions is not sufficient to learn visibly diverse distribution-aware experts.
In contrast, our proposed method learns the skill-diverse experts by directly training each expert with their customized expertise-guided objective functions, respectively.
To be specific, the forward expert E 1 seeks to learn the long-tailed training distribution, so we directly train it with the crossentropy loss.
For the uniform expert E 2 , we use the balanced softmax loss to simulate the uniform test distribution.
For the backward expert E 3 , we design a novel inverse softmax loss to train the expert, so that it simulates the inversely long-tailed class distribution.
Table 18 shows that the three experts trained by our method are visibly diverse and skilled in handling different class distributions.
Specifically, the forward expert is skilled in many-shot classes, the uniform expert is more balanced with higher overall performance, and the backward expert is good at few-shot classes.
Because of such a novel design that enhances expert diversity, our method achieves more promising ensemble performance compared to RIDE.
[Table]
D.4 More Results on Test-time Self-supervised Aggregation
This appendix provides more results to examine the effectiveness of our test-time self-supervised aggregation strategy.
We report results in Table 19, from which we draw several observations.
First of all, our method is able to learn suitable expert aggregation weights for test-agnostic class distributions, without relying on the true test class distribution.
For the forward long-tailed test distribution, where the test data number of many-shot classes is more than that of medium-shot and few-shot classes, our method learns a higher weight for the forward expert E 1 who is skilled in many-shot classes, and learns relatively low weights for the expert E 2 and expert E 3 who are good at medium-shot and few-shot classes.
Meanwhile, for the uniform test class distribution where all classes have the same number of test samples, our test-time expert aggregation strategy learns relatively balanced weights for the three experts.
For example, on the uniform ImageNet-LT test data, the learned weights by our strategy are 0.33, 0.33 and 0.34 for the three experts, respectively.
In addition, for the backward long-tailed test distributions, our method learns a higher weight for the backward expert E 3 and a relatively low weight for the forward expert E 1 .
Note that when the class imbalance ratio becomes larger, our method is able to learn more diverse expert weights adaptively for fitting the actual test class distributions.
Such results not only demonstrate the effectiveness of our proposed strategy, but also verify the theoretical analysis that our method can simulate the unknown test class distribution.
To our best knowledge, such an ability is quite promising, since it is difficult to know the true test class distributions in real-world application.
Therefore, our method opens the opportunity for tackling unknown class distribution shifts at test time, and can serve as a better candidate to handle real-world long-tailed learning applications.
The results show that our self-supervised strategy is able to learn suitable expert weights for various unknown test class distributions.
[Table]
Relying on the learned expert weights
[Table]
our method aggregates the three experts appropriately and achieves better performance on the dominant test classes, thus obtaining promising performance gains on various test distributions, as shown in Table 20.
Note that the performance gain compared to existing methods gets larger as the test dataset gets more imbalanced.
For example, on CIFAR100-LT with the imbalance ratio of 50, our test-time self-supervised strategy obtains a 7.7 % performance gain on the Forward-LT-50 distribution and obtains a 9.2 % performance gain on the Backward-LTdistribution, both of which are non-trivial.
Such an observation is also supported by the visualization result of Figure 5, which plots the results of existing methods on ImageNet-LT with different test class distributions regarding the three class subsets.
In addition, since the imbalance degrees of the test datasets are relatively low on iNaturalist 2018, the simulated test class distributions are thus relatively balanced.
As a result, the obtained performance improvement is not that significant, compared to other datasets.
However, if there are more iNaturalist test samples following highly imbalanced test class distributions in real applications, our method would obtain more promising results.
[Table]
) - - - - - - - - Forward-LT43.0 44.0 33.1 42.
45.6 40.6 27.3 42.6(+0.2) - - - - - - - - Forward-LT43.4 42.4 32.6 41.
44.9 41.2 29.5 41.3(+0.0) 73.9 72.4 72.0 72.
[Table]
70.7 72.5(+0.1) Uniform 43.1 42.4 33.2 40.
[Table] [Table]
36.8 40.9(+
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
9(+
[Table]
) Backward-LT42.
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
5(+
[Table]
4) Backward-LT-
[Table] [Table]
[Table] [Table]
[Table] [Table]
(+2.
[Table]
) - - - - - - - - Backward-LT-10 43.
[Table] [Table]
35.2 43.2 41.3 41.4(+2.5) - - - - - - - - Backward-LT-25 44.6 42.
[Table] [Table]
43.5 41.1 42.0(+4.2) - - - - - - - - Backward-LT-50 42.2 43.
[Table] [Table]
[Table]
43.5 40.5 41.6(+4.7) - - - - - - - -
[Figure]
data, with the imbalance ratio N , respectively.
The results show that existing methods perform very similarly on various test class distributions in terms of their performance on many-shot, medium-shot and few-shot classes.
In contrast, our proposed method is capable of adaptingbi to various test class distributions in terms of many-shot, medium-shot and few-shot performance, thus leading to better overall performance on each test class distribution.
E Ablation Studies on Skill-diverse Expert Learning E.1 Discussion on Expert Number
In SADE, we consider three experts, where the “forward" and “backward" experts are necessary since they span a wide spectrum of possible test class distributions, while the “uniform" expert ensures that we retain high accuracy on the uniform test class distributions.
Nevertheless, our approach can be straightforwardly extended to more than three experts.
For the models with more experts, we adjust the hyper-parameter λ in Eq.
(3) for the new experts and keep the hyper-parameters of the original three experts unchanged, so that different experts are skilled in different types of class distributions.
Following this, we further evaluate the influence of the expert number on our method based on
ImageNet.
To be specific, when there are four experts, we set λ
= 1 for the new expert; while when there are five experts, we set λ = 0 .
5 and λ = 1 for the two newly-added experts, respectively.
As shown in Table 21, with the increasing number of experts, the ensemble performance of our method is improved on vanilla long-tailed recognition, e.g., four experts obtain a 1.2 % performance gain compared to three experts on ImageNet-LT.
As a result, our method with more experts obtains consistent performance improvement in test-agnostic long-tailed recognition on various test class distributions compared to three experts, as shown in Table 22.
Even so, only three experts are sufficient to handle varied test class distributions, and provide a good trade-off between performance and efficiency.
[Table] [Table]
E.2 Hyper-parameters in Inverse Softmax Loss
This appendix evaluates the influence of the hyper-parameter λ in the inverse softmax loss for the backward expert, where we fix all other hyper-parameters and only adjust the value of λ .
As shown in
Table 23, with the increase of λ , the backward expert simulates more inversely long-tailed distribution (to the training data), and thus the ensemble performance on few-shot classes is better.
Moreover, when λ ∈ { 2 , 3 } , our method achieves a better trade-off between head classes and tail classes, leading to relatively better overall performance on ImageNet-LT.
[Table]
F Ablation Studies on Test-time Self-supervised Aggregation F.1 Influences of Training Epoch
As illustrated in Section 5.
[Table]
we set the training epoch of our test-time self-supervised aggregation strategy to 5 on all datasets.
Here
[Table]
we further evaluate the influence of the epoch number, where we adjust the epoch number from 1 to 100.
As shown in Table
[Table]
when the training epoch number is larger than 5, the learned expert weights by our method are converged on ImageNet-LT, which verifies that our method is robust enough.
The corresponding performance on various test class distributions is reported in Table 25.
[Table] [Table]
F.2 Influences of Batch Size
In previous results
[Table]
we set the batch size of test-time self-supervised aggregation to 128 on all datasets.
In this appendix
[Table]
we further evaluate the influence of the batch size on our strategy, where we adjust the batch size from 64 to 256.
As shown in Table 26, with different batch sizes, the learned expert weights by our method keep nearly the same, which shows that our method is insensitive to the batch size.
The corresponding performance on various test class distributions is reported in Table 27, where the performance is also nearly the same when using different batch sizes.
[Table] [Table]
43.3 58.8 Backward-LT64.
44.8 57.1 Backward-LT63.
47.5 55.5 Backward-LT60.
48.7 54.4 Backward-LT60.
49.6 53.6 Backward-LT61.
F.3 Influences of Learning Rate
In this appendix, we evaluate the influence of the learning rate on our self-supervised strategy, where we adjust the learning rate from 0.001 to 0.5.
As shown in Table 28, with the increase of the learning rate, the learned expert weights by our method are sharper and fit the unknown test class distributions better.
For example, when the learning rate is 0.001, the weight for expert E 1 is 0.
on the Forward-LT-50 test distribution, while when the learning rate increases to 0.5, the weight for expert E 1 becomes 0.57 on the Forward-LT-50 test distribution.
Similar phenomenons are also observed on backward long-tailed test class distributions.
By observing the corresponding model performance on various test class distributions in Table
[Table]
we find that when the learning rate is too small ( e.g., 0.001), our test-time self-supervised aggregation strategy is unable to converge, given a fixed training epoch number of 5.
In contrast, given the same training epoch, our method can obtain better performance by reasonably increasing the learning rate.
[Table] [Table]
F.4 Results of Prediction Confidence
In our theoretical analysis ( i.e., Theorem 1), we find that our test-time self-supervised aggregation strategy not only simulates the test class distribution, but also makes the model predictions more confident.
In this appendix, we evaluate whether our strategy can really improve the prediction confidence of models on various unknown test class distributions of ImageNet-LT.
To this end, we compare the prediction confidence of our method without and with test-time self-supervised aggregation in terms of the hard mean of the highest prediction probability on all test samples.
As shown in Table
[Table]
our test-time self-supervised aggregation strategy enables the deep model to have higher prediction confidence.
For example, on the Forward-LT-50 test distribution, our strategy obtains 0.015 confidence improvement, which is non-trivial since it is an average value for a large number of samples (more than 10,000 samples).
In addition, when the class imbalance ratio becomes larger, our method is able to obtain more apparent confidence improvement
[Table]
[Table]
F.5 Run-time Cost of Test-time Aggregation
One may be interested in the run-time cost of our test-time self-supervised aggregation strategy, so we further report its running time on Forward-LT-50 and Forward-LT-25 test class distributions for illustration.
As shown in Table 31, our test-time self-supervised aggregation strategy is fast in terms of per-epoch time.
The actual average additional time is only 0.009 seconds per sample at test time on
V100 GPUs.
The result is easy to interpret since we freeze the model parameters and only learn the aggregation weights, which is much more efficient than training the whole model.
More importantly, the goal of this paper is to handle a practical yet challenging test-agnostic long-tailed recognition task.
For solving this challenging problem, we believe it is acceptable to allow models to be trained more, while the promising results in previous experiments have demonstrated the effectiveness of our proposed test-time self-supervised learning strategy in handling this problem.
In the future, we will further extend the proposed method for better computational efficiency, e.g., exploring dynamic network routing.
[Table]
F.6 Test-time Self-supervised Aggregation on Streaming Test Data
In the previous experiments
[Table]
we conduct the test-time self-supervised aggregation strategy in an offline manner.
However, as mentioned in Section 4.2, our test-time strategy can also be conducted in an online manner and does not require access to all the test data in advance.
To verify this
[Table]
we further conduct our test-time strategy on steaming test data of ImageNet-LT.
As shown in Table 32, our test-time strategy performs well on the streaming test data.
Even when the test data come in one by one, our test-time self-supervised strategy still outperforms the state-of-the-art baseline (i
[Table]
e., offline Tent [46]) by a large margin.
[Table]
G More Discussions on Model Complexity
In this appendix, we discuss the model complexity of our method in terms of the number of parameters, multiply-accumulate operations (MACs) and top-1 accuracy on test-agnostic long-tailed recognition.
As shown in Table 33, both SADE and RIDE belong to ensemble-based long-tailed learning methods, so they have more parameters (about 1.5x) and MACs (about 1.4x) than the original backbone model, where we do not use the efficient expert assignment trick in [49] for both methods.
Because of the ensemble effectiveness of the multi-expert scheme, both methods perform much better than non-ensemble methods ( e.g., Softmax and other long-tailed methods).
In addition, since our method and RIDE use the same multi-expert framework, both methods have the same number of parameters and MACs.
Nevertheless, by using our proposed skill-diverse expert learning and test-time selfsupervised aggregation strategies, our method performs much better than RIDE with no increase in model parameters and computational costs.
One may concern the multi-expert scheme leads to more model parameters and higher computational costs than the original backbone.
However, note that the main focus of this paper is to solve the challenging test-agnostic long-tailed recognition, while promising results have shown that our method addresses this problem well.
In this sense, slightly increasing the model complexity is acceptable for solving this practical yet challenging problem.
Moreover, since there have already been many studies [14, 51] showing effectiveness in improving the efficiency of the multi-expert scheme, we think the computation increment is not a severe issue and we leave it to the future.
Here, we not use the
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
H Potential Limitations
One concern is that this work only focuses on long-tailed classification problems.
However, we believe this is enough for a new challenging task of test-agnostic long-tailed recognition
[Table]
while how to extending to object detection and instance segmentation will be explored in the future.
Another potential concern is the model complexity of our method.
However, as discussed in Appendix G, the computation increment is not a very severe issue, while how to further accelerate our method will be explored in future.
In addition, one may also expect to evaluate the proposed method on more test class distributions.
However, as shown in Section 5.
[Table]
we have demonstrated the effectiveness of our method on the uniform class distribution, the forward and backward long-tailed class distributions with various imbalance ratios
[Table]
and even partial class distributions.
Therefore, we believe the empirical verification is sufficient for verifying our method
[Table]
and the extension to more complex test class distributions is left to the future
[Table]