Self-Supervised Aggregation of Diverse Experts for Test-Agnostic Long-Tailed Recognition
Anonymous Author s)
AffiliationAddress email
Abstract
Existing long-tailed recognition methods, aiming to train class-balanced modelsfrom long-tailed data, generally assume the models would be evaluated on theuniform test class distribution.
However, practical test class distributions oftenviolate this assumption ( e.g., being either long-tailed or even inversely long-tailed),which may lead existing methods to fail in real applications.
In this paper, we studya more practical yet challenging task, called test-agnostic long-tailed recognition ,where the training class distribution is long-tailed while the test class distributionis agnostic and not necessarily uniform .
In addition to the issue of class imbalance,this task poses another challenge: the class distribution shift between the trainingand test data is unknown.
To tackle this task, we propose a novel approach, called
Self-supervised Aggregation of Diverse Experts , which consists of two strategies:(i) a new skill-diverse expert learning strategy that trains multiple experts froma single and stationary long-tailed dataset to separately handle different classdistributions; (ii) a novel test-time expert aggregation strategy that leverages selfsupervision to aggregate the learned multiple experts for handling unknown testclass distributions.
We theoretically show that our self-supervised strategy has aprovable ability to simulate test-agnostic class distributions.
Promising empiricalresults demonstrate the effectiveness of our method on both vanilla and test-agnosticlong-tailed recognition.
Source code is available in the supplementary material.
Introduction
Real-world visual recognition datasets typically exhibit a long-tailed distribution, where a few classescontain numerous samples (called head classes), but the others are associated with only a few data(called tail classes)
Due to the class imbalance, the trained model is easily biased towardshead classes and perform poorly on tail classes [2, 55].
To tackle this issue, numerous studies haveexplored long-tailed recognition for learning well-performing models from imbalanced data [20, 53].
Most existing long-tailed studies [3, 9, 10, 45, 49] assume the test class distribution is uniform, i.e.,each class has an equal amount of test data.
Therefore, they develop various techniques, e.g., class resampling [13, 18, 24, 52], cost-sensitive learning [11, 34, 38, 44] or ensemble learning [2, 13, 26, 50],to re-balance the model performance on different classes for fitting the uniform class distribution.
However, this assumption does not always hold in real applications, where actual test data may followany kind of class distribution, being either uniform, long-tailed, or even inversely long-tailed to thetraining data (cf.
Figure 1(a)).
For example, one may train a recognition model for autonomouscars based on the training data collected from city areas, where pedestrians are majority classes andstone obstacles are minority classes.
However, when the model is deployed to mountain areas, thepedestrians become the minority while the stones become the majority.
In this case, the test classdistribution is inverse to the training one, and existing methods may perform poorly.
[Figure]
However, the resulting models may fail to handle practical test class distributions that skew arbitrarily
Our method seeks to learn a multi-expert model from a single long-tailed training set, where different experts are skilled in handling different class distributions, respectively.
By reasonably aggregating these experts at test time, our method is able to handle unknown test class distributions.
To address the issue of varying class distributions, as the first research attempt, LADE [17] assumesthe test class distribution to be known and uses the knowledge to post-adjust model predictions.
However, the actual test class distribution is usually unknown a priori, making LADE not applicablein practice.
Therefore, we study a more realistic yet challenging problem, namely test-agnosticlong-tailed recognition , where the training class distribution is long-tailed while the test distribution isagnostic .
To tackle this problem, motivated by the idea of "divide and conquer", we propose to learnmultiple experts with diverse skills that excel at handling different class distributions (cf.
Figure 1(b)).
As long as these skill-diverse experts can be aggregated suitably at test time, the multi-expert modelwould manage to handle the unknown test class distribution.
Following this idea, we develop a novelapproach, namely Self-supervised Aggregation of Diverse Experts (SADE).
The first challenge for SADE is how to learn multiple diverse experts from a single and stationarylong-tailed training dataset.
To handle this challenge, we empirically evaluate existing long-tailedmethods in this task, and find that the models trained by existing methods have a simulation correlationbetween the learned class distribution and the training loss function .
That is, the models learned byvarious losses are skilled in handling class distributions with different skewness.
For example, themodel trained with the conventional softmax loss simulates the long-tailed training class distribution,while the models obtained from existing long-tailed methods are good at the uniform class distribution.
Inspired by this finding, SADE presents a simple but effective skill-diverse expert learning strategy togenerate experts with different distribution preferences from a single long-tailed training distribution.
Here, various experts are trained with different expertise-guided objective functions to deal withdifferent class distributions, respectively.
As a result, the learned experts are more diverse thanprevious multi-expert long-tailed methods [46, 57], leading to better ensemble performance, and inaggregate simulate a wide spectrum of possible class distributions.
The other challenge is how to aggregate these skill-diverse experts for handling test-agnostic classdistributions based on only unlabeled test data.
To tackle this challenge, we empirically investigatethe property of different experts, and observe that there is a positive correlation between expertiseand prediction stability , i.e., stronger experts have higher prediction consistency between differentperturbed views of samples from their favorable classes.
Motivated by this finding, we develop a novelself-supervised strategy, namely prediction stability maximization, to adaptively aggregate expertsbased on only unlabeled test data.
We theoretically show that maximizing the prediction stabilityenables SADE to learn an aggregation weight that maximizes the mutual information between thepredicted label distribution and the true class distribution.
In this way, the resulting model is able tosimulate unknown test class distributions.
We empirically verify the superiority of SADE on both vanilla and test-agnostic long-tailed recognition.
Specifically, SADE achieves promising performance on vanilla long-tailed recognition underall benchmark datasets.
For instance, SADE achieves 58.8 % accuracy on ImageNet-LT with morethan 2 % accuracy gain over previous state-of-the-art ensemble long-tailed methods, i.e., RIDE [46]and ACE [2].
More importantly, SADE is the first long-tailed approach that is able to handle varioustest-agnostic class distributions without knowing the true class distribution of test data in advance.
Note that SADE even outperforms LADE [17] that uses knowledge of the test class distribution.
Compared to previous long-tailed methods ( e.g., LADE [17] and RIDE [46]), our method offers thefollowing advantages: (i) SADE does not assume the test class distribution to be known, and providesthe first practical approach to handling test-agnostic long-tailed recognition; (ii) SADE developsa simple diversity-promoting strategy to learn skill-diverse experts from a single and stationarylong-tailed dataset; (iii) SADE presents a novel self-supervised strategy to aggregate skill-diverseexperts at test time, by maximizing prediction consistency between unlabeled test samples’ perturbedviews; (iv) the presented self-supervised strategy has a provable ability to simulate test-agnostic classdistributions, which opens the opportunity for tackling unknown class distribution shifts at test time.
Related WorkLong-tailed recognition
Existing long-tailed recognition methods, related to our study, can becategorized into three types: class re-balancing, logit adjustment and ensemble learning.
Specifically,class re-balancing resorts to re-sampling [4, 13, 18, 24] or cost-sensitive learning [3, 10, 16] tobalance different classes during model training.
Logit adjustment [17, 31, 35, 40] adjusts models’output logits via the label frequencies of training data at inference time, for obtaining a large relativemargin between head and tail classes.
Ensemble-based methods [2, 13, 50, 57], e.g., RIDE [46], arebased on multiple experts, which seek to capture heterogeneous knowledge, followed by ensembleaggregation.
More discussions on the difference between our method and RIDE [46] are provided in
Appendix D.3.
Regarding test-agnostic long-tailed recognition, LADE [17] assumes the test classdistribution is available and uses it to post-adjust model predictions.
However, the true test classdistribution is usually unknown a priori, making LADE inapplicable.
In contrast, our method does notrely on the true test distribution for handling this problem, but presents a novel self-supervised strategyto aggregate skill-diverse experts at test time for test-agnostic class distributions.
Moreover, someensemble-based long-tailed methods [36] aggregate experts based on a labeled uniform validation set.
However, as the test class distribution could be different from the validation one, simply aggregatingexperts on the validation set is unable to handle test-agnostic long-tailed recognition.
Test-time training Test-time training [22, 25, 28, 37, 43] is a transductive learning paradigm forhandling distribution shifts between training and test data [30, 32, 42], and has been applied withsuccess to out-of-domain generalization [19, 33] and dynamic scene deblurring [6].
In this study,we explore this paradigm to handle test-agnostic long-tailed recognition, where the issue of classdistribution shifts is the main challenge.
However, most existing test-time training methods seekto handle covariate distribution shifts instead of class distribution shifts, so simply leveraging themcannot resolve test-agnostic long-tailed recognition, as shown in our experiment (cf.
Table 9).
Problem Formulation
Long-tailed recognition aims to learn a well-performing classification model from a training datasetwith long-tailed class distribution.
Let D s =
{ x i , y i } n s i =1 denote the long-tailed training set, where y iis the class label of the sample x i .
The total number of training data over C classes is n s = (cid:80) Ck =1 n k ,where n k denotes the number of samples in class k .
Without loss of generality, we follow a commonassumption [17, 24] that the classes are sorted by cardinality in decreasing order ( i.e., if i 1 < i 2 , thenn i 1 ≥ n i 2 ), and n 1 ≫ n C .
The imbalance ratio is defined as max( n k ) / min( n k ) = n 1 / n C .
The testdata D t = { x j , y j } n t j =1 is defined in a similar way.
Most existing long-tailed recognition methods assume the test class distribution is uniform ( i.e.,p t ( y )
= 1 /C ), and seek to train models from the long-tailed training distribution p s ( y ) to performwell on the uniform test distribution.
However, such an assumption does not always hold in practice.
The actual test class distribution in real-world applications may also be long-tailed ( i.e., p t ( y )
=
p s ( y ) ), or even inversely long-tailed to the training data ( i.e., p t ( y )
=
inv ( p s ( y )) ).
Here, inv ( · )indicates that the order of the long tail on classes is flipped.
As a result, the models learned by existingmethods may fail when the actual test class distribution is different from the assumed one.
To addressthis issue, we propose to study a more practical yet challenging long-tailed recognition problem, i.e.,
Test-agnostic Long-tailed Recognition .
This task aims to learn a recognition model from long-tailedtraining data, where the resulting model would be evaluated on multiple test sets that follow differentclass distributions.
This task is challenging due to the integration of two challenges: (1) the severeclass imbalance in the training data makes it difficult to train models;
(2) unknown class distributionshifts between training and test data ( i.e., p t ( y ) ̸
= p s ( y ) ) makes models hard to generalize.
[Table]
Method
To tackle the above problem, inspired by the idea of "divide and conquer", we propose to learnmultiple skill-diverse experts that excel at handling different class distributions.
By reasonably fusingthese experts at test time, the multi-expert model would manage to handle unknown class distributionshifts and resolve test-agnostic long-tailed recognition.
Following this idea, we develop a novel
Self-supervised Aggregation of Diverse Experts (SADE) approach.
Specifically, SADE consists oftwo innovative strategies: (1) learning skill-diverse experts from a single long-tailed training dataset;(2) test-time aggregating experts with self-supervision to handle test-agnostic class distributions.
4.1 Skill-diverse Expert Learning
As shown in Figure 2, SADE builds a three-expert model 1 that comprises two components: (1) anexpert-shared backbone f θ ; (2) independent expert networks E 1 , E 2 and E 3 .
When training themodel, the key challenge is how to learn skill-diverse experts from a single and stationary long-tailedtraining dataset.
Existing ensemble-based long-tailed methods [13, 46] seek to train experts for theuniform test class distribution, and hence the trained experts are not differentiated sufficiently forhandling various class distributions (refer to Table 6 for an example).
To tackle this challenge, wefirst empirically investigate existing long-tailed methods in this task.
From Table 1, we find that thereis a simulation correlation between the learned class distribution and the training loss function .
Thatis, the models learned by different losses are good at dealing with class distributions with differentskewness.
For instance, the model trained with the softmax loss is good at the long-tailed distribution,while the models obtained from long-tailed methods are skilled in the uniform distribution.
[Figure]
Motivated by this finding, we develop a simpleskill-diverse expert learning strategy to generateexperts with different distribution preferences.
To be specific, the forward expert E 1 seeks to begood at the long-tailed class distribution and performs well on many-shot classes.
The uniformexpert E 2 strives to be skilled in the uniform distribution.
The backward expert E 3 aims at theinversely long-tailed distribution and performswell on few-shot classes.
Here, the forwardand backward experts are necessary since theyspan a wide spectrum of possible class distributions, while the uniform expert ensures retaininghigh accuracy on the uniform distribution.
Tothis end, we use three different expertise-guidedlosses to train the three experts, respectively.
The forward expert E
We use the softmax cross-entropy loss to train this expert, so that it directlysimulates the original long-tailed training class distribution:
[Equation]
where v 1 ( · ) is the output logits of the forward expert E 1 , and σ ( · ) is the softmax function.
The uniform expert E
We aim to train this expert to simulate the uniform class distribution.
Inspired by the effectiveness of logit adjusted losses for long-tailed recognition [31], we resort tothe balanced softmax loss [21].
Specifically, let ˆ y k
=
[Equation]
be the prediction probability.
The balanced softmax adjusts the prediction probability by compensating for the long-tailed classdistribution with the prior of training label frequencies:
[Equation]
where π k = n k n denotes the training label frequency of class k .
Then, given v 2 ( · ) as the output logitsof the expert E 2 , the balanced softmax loss for the expert E 2 is defined as:
[Equation]
Intuitively, by adjusting logits to compensate for the long-tailed distribution with the prior π , this lossenables E 2 to output class-balanced predictions that simulate the uniform distribution.
The backward expert E
We seek to train this expert to simulate the inversely long-tailedclass distribution.
To this end, we propose a new inverse softmax loss , based on the same rationale of logit adjusted losses [21, 31].
Specifically, we adjust the prediction probability by:ˆ
[Equation]
) , where the inverse training prior ¯ π is obtained by inverting the orderof training label frequencies π .
Then, the new inverse softmax loss for the expert E 3 is defined as:
[Equation]
log ¯ π ) , (3) where v 3 ( · ) denotes the output logits of E 3 and λ is a hyper-parameter.
Intuitively, this loss adjustslogits to compensate for the long-tailed distribution with π , and further applies reverse adjustmentwith ¯ π .
This enables E 3 to simulate the inversely long-tailed distribution (cf.
Table 6 for verification).
4.2 Test-time Self-supervised Aggregation
Based on the skill-diverse learning strategy, the three experts in SADE are skilled in differentclass distributions.
The remaining challenge is how to fuse them to deal with unknown test classdistributions.
A basic principle for expert aggregation is that the experts should play a bigger role insituations where they have expertise.
Nevertheless, how to detect strong experts for unknown testclass distribution remains unknown.
Our key insight is that strong experts should be more stable inpredicting the samples from their skilled classes, even though these samples are perturbed.
Empirical observation To verify this hypothesis, we estimate the prediction stability of experts bycomparing the cosine similarity between their predictions for a sample’s two augmented views.
Here,the data views are generated by the data augmentation techniques in MoCo v2 [5].
From Table 2,we find that there is a positive correlation between expertise and prediction stability , i.e., strongerexperts have higher prediction similarity between different views of samples from their favorableclasses.
Following this finding, we propose to explore the relative prediction stability to detect strongexperts and weight experts for the unknown test class distribution.
Consequently, we develop a novelself-supervised strategy, namely prediction stability maximization.
Prediction stability maximization
This strategy learns aggregation weights for experts (withfrozen parameters) by maximizing model prediction stability for unlabeled test samples.
As shown in
Figure 3, the method comprises three major components as follows.
Data view generation
For a given sample x , we conduct two stochastic data augmentations togenerate the sample’s two views, i.e., x 1 and x 2 .
Here, we use the same augmentation techniquesas the advanced contrastive learning method, i.e., MoCo v2 [5], which has been shown effective inself-supervised learning.
Learnable aggregation weight
Given the output logits of three experts ( v 1 , v 2 , v 3 ) ∈ R 3 × C , weaggregate experts with a learnable aggregation weight w =
[ w 1 , w 2 , w 3 ] ∈ R 3 and obtain the finalsoftmax prediction by ˆ y = σ ( w 1 · v 1 + w 2 · v 2 + w 3 · v 3 ) , where w is normalized before aggregation,i.e., w 1 + w 2 + w 3 =1 . expert E is good at many skilled in experts tend to have better prediction consistency for samples from their skilled classes. Here, the ratio is
[Table]
Maximizepredictionstability Logits 𝑥
[Table]
𝑦-
[Table]
Objective function Given the view predictions of unlabeled test data, we maximize the predictionstability based on the cosine similarity between the view predictions:
[Equation]
Here, ˆ y 1 and ˆ y 2 are normalized by the softmax function.
In test-time training, only the aggregationweight w is updated.
Since stronger experts have higher prediction similarity for their skilled classes,maximizing the prediction stability S would learn higher weights for stronger experts regarding theunknown test class distribution.
Moreover, the self-supervised aggregation strategy can be conductedin an online manner for streaming test data.
The pseudo-code of SADE is provided in Appendix B.
Theoretical Analysis We then theoretically analyze the prediction stability maximization strategy toconceptually understand why it works.
To this end, we first define the random variables of predictionsand labels as ˆ Y ∼ p (ˆ y ) and Y ∼ p t ( y ) .
We have the following result:
Theorem 1.
The prediction stability S is positive proportional to the mutual information betweenthe predicted label distribution and the test class distribution
I ( ˆ Y ; Y ) , and negative proportional tothe prediction entropy H ( ˆ Y ) :
[Equation]
Please refer to Appendix A for proofs.
According to Theorem 1, maximizing the prediction stability
S enables SADE to learn an aggregation weight that maximizes the mutual information
betweenthe predicted label distribution p (ˆ y ) and the test class distribution p t ( y ) , as well as minimizingthe prediction entropy.
Since minimizing entropy helps to improve the confidence of the classifieroutput [12], the aggregation weight is learned to simulate the test class distribution p t ( y ) and increasethe prediction confidence.
This property intuitively explains why our method has the potential totackle the challenging task of test-agnostic long-tailed recognition at test time.
Experiments
In this section, we first evaluate the superiority of SADE on both vanilla and test-agnostic long-tailedrecognition.
We then verify the effectiveness of SADE in terms of its two strategies, i.e., skill-diverseexpert learning and test-time self-supervised aggregation.
More ablation studies are reported inappendices.
Here, we begin with the experimental settings.
5.1 Experimental SetupsDatasets
We use four benchmark datasets ( i.e., ImageNet-LT [29], CIFAR100-LT [3], PlacesLT [29], and iNaturalist 2018 [41]) to simulate real-world long-tailed class distributions.
Their datastatistics and imbalance ratios are summarized in Appendix C.1.
The imbalance ratio is defined asmax n j / min n j , where n j denotes the data number of class j .
Note that CIFAR100-LT has threevariants with different imbalance ratios.
[Table]
Baselines
We compare SADE with state-of-the-art long-tailed methods, including two-stagemethods (Decouple [24], MiSLAS [56]), logit-adjusted training (Balanced Softmax [21], LADE [17]),ensemble learning (BBN [57], ACE [2], RIDE [46]), classifier design (Causal [39])
[Table]
and representationlearning (PaCo [8]).
Note that LADE uses the prior of test class distribution for post-adjustment(although it is unavailable in practice), while all other methods do not use this prior.
Evaluation protocols
In test-agnostic long-tailed recognition, following LADE [17], the modelsare evaluated on multiple sets of test data that follow different class distributions, in terms of microaccuracy.
Same as LADE [17], we construct three kinds of test class distributions, i.e., the uniformdistribution, forward long-tailed distributions as training data, and backward long-tailed distributions.
In the backward ones, the order of the long tail on classes is flipped.
More details of test dataconstruction are provided in Appendix C.2.
Besides, we also evaluate methods on vanilla long-tailedrecognition [24, 29], where the models are evaluated on the uniform test class distribution.
Here, theaccuracy on three class sub-groups is also reported, i.e., many-shot classes (more than 100 trainingimages), medium-shot classes (20 ∼ 100 images) and few-shot classes (less than 20 images).
Implementation details We use the same setup for all the baselines and our method.
Specifically,following [17, 46], we use ResNeXt-50 for ImageNet-LT, ResNet-32 for CIFAR100-LT, ResNet-152for Places-LT and ResNet-50 for iNaturalist 2018 as backbones, respectively.
Moreover, we adoptthe cosine classifier for prediction on all datasets.
If not specified, we use the SGD optimizer withthe momentum of 0.9 for training 200 epochs and set the initial learning rate as 0.1 with lineardecay.
We set λ =2 for ImageNet-LT and CIFAR100-LT, and λ =1 for the remaining datasets.
Duringtest-time training, we train the aggregation weights for 5 epochs with the batch size 128, where weuse the same optimizer and learning rate as the training phase.
More implementation details and thehyper-parameter statistics are reported in Appendix C.3.
5.2 Superiority on Vanilla Long-tailed Recognition
[Table]
This subsection compares SADE with state-of-the-artlong-tailed methods on vanilla long-tailed recognition.
Specifically, as shown in Tables 3-4, Softmax trains themodel with only cross-entropy, so it simulates the longtailed training distribution and performs well on manyshot classes.
However, it performs poorly on mediumshot and few-shot classes, leading to worse overall performance.
In contrast, existing long-tailed methods ( e.g.,
Decouple, Causal) seek to simulate the uniform class distribution, so their performance is more class-balanced,leading to better overall performance.
However, as thesemethods mainly seek balanced performance, they inevitably sacrifice the performance on many-shot classes.
To address this, RIDE and ACE exploreensemble learning for long-tailed recognition and achieve better performance on tail classes withoutsacrificing the head-class performance.
In comparison, based on the increasing expert diversityderived from skill-diverse expert learning, our method performs the best on all datasets, e.g., withmore than 2 % accuracy gain on ImageNet-LT compared to RIDE and ACE.
These results demonstratethe superiority of SADE over the compared methods that are particularly designed for the uniformtest class distribution.
Note that SADE also outperforms baselines in experiments with stronger dataaugmentation ( i.e., RandAugment [7]) and other architectures, as reported in Appendix D.1.
[Table] [Table]
5.3 Superiority on Test-agnostic Long-tailed Recognition
In this subsection
[Table]
we evaluate SADE on test-agnostic long-tailed recognition.
The results on varioustest class distributions are reported in Table 5.
Specifically, since Softmax seeks to simulate the longtailed training distribution, it performs well on forward long-tailed test distributions.
However, itsperformance on the uniform and backward long-tailed distributions is poor.
In contrast, existing longtailed methods show more balanced performance among classes, leading to better overall accuracy.
However, the resulting models by these methods suffer from a simulation bias, i.e., performingsimilarly among classes on various class distributions (c.f. Table 1).
As a result, they cannot adapt todiverse test class distributions well.
To handle this task, LADE assumes the test class distribution tobe known and uses this information to adjust its predictions, leading to better performance on varioustest class distributions.
However, since obtaining the actual test class distribution is difficult in realapplications, the methods requiring such knowledge may be not applicable in practice.
Moreover,in some specific cases like Forward-LT-3 and Backward-LT-3 distributions of iNaturalist 2018, thenumber of test samples on some classes becomes zero.
In such cases, the test prior cannot be used in
LADE, since adjusting logits with log 0 results in biased predictions.
In contrast, without relying onthe knowledge of test class distributions, our SADE presents an innovative self-supervised strategy todeal with unknown class distributions, and obtains even better performance than LADE that uses thetest class prior
(c.f. Table 5).
The promising results demonstrate the effectiveness and practicality ofour method on test-agnostic long-tailed recognition.
Note that the performance advantages of SADEbecome larger as the test data get more imbalanced.
Due to the page limitation, the results on moredatasets are reported in Appendix D.2.
5.4 Effectiveness of Skill-diverse Expert Learning
We next examine our skill-diverse expert learning strategy.
The results are reported in Table 6, where
RIDE [46] is a state-of-the-art ensemble-based method.
RIDE trains each expert with cross-entropyindependently and uses KL-Divergence to improve expert diversity.
However, simply maximizing thedivergence of expert predictions cannot learn visibly diverse experts (cf.
Table 6).
In contrast, the threeexperts learned by our strategy have significantly diverse expertise, excelling at many-shot classes, theuniform distribution (with higher overall performance), and few-shot classes, respectively.
As a result,the increasing expert diversity leads to a non-trivial gain for the ensemble performance of SADEcompared to RIDE.
Moreover, consistent results on more datasets are reported in Appendix D.3,while the ablation studies of the expert learning strategy are provided in Appendix E.
[Table]
[Table] [Table]
5.5 Effectiveness of Test-time Self-supervised Aggregation
[Table]
This subsection evaluates our test-time self-supervised aggregation strategy.
Effectiveness in expert aggregation.
As shown in Table 7, our self-supervised strategy learnssuitable expert weights for various unknown test class distributions.
For forward long-tailed distributions, the weight of the forward expert E 1 is higher; while for backward long-tailed ones, theweight of the backward expert E 3 is relatively high.
This enables our multi-expert model to boostthe performance on dominant classes for unknown test distributions, leading to better ensembleperformance (cf.
Table 8), particularly as test data get more skewed.
The results on more datasets arereported in Appendix D.4, while more ablation studies of our strategy are shown in Appendix F.
Superiority over test-time training methods.
We then verify the superiority of our self-supervisedstrategy over existing test-time training approaches on various test class distributions.
Specifically,we adopt three non-trivial baselines: (i) Test-time pseudo-labeling uses the multi-expert model toiteratively generate pseudo labels for unlabeled test data and uses them to fine-tune the model; (ii)
Test class distribution estimation leverages BBSE [27] to estimate the test class distribution anduses it to pose-adjust model predictions; (iii) Tent [43] fine-tunes the batch normalization layersof models through entropy minimization on unlabeled test data.
The results in Table 9 show thatdirectly applying existing test-time training methods cannot handle well the class distribution shifts,particularly on the inversely long-tailed class distribution.
In comparison, our self-supervised strategyis able to aggregate multiple experts appropriately for the unknown test class distribution (cf.
Table 7),leading to promising performance gains on various test class distributions (cf.
Table 9).
[Table]
Effectiveness on partial class distributions.
Real-world test data may follow any type ofclass distribution, including partial class distributions ( i.e., not all of the classes appear in thetest data).
Motivated by this, we further evaluate
SADE on three partial class distributions: onlymany-shot classes, only medium-shot classes,and only few-shot classes.
The results in Table 10 demonstrate the effectiveness of SADEin tackling more complex test class distributions.
Conclusion
In this paper, we have explored a practical yet challenging task of test-agnostic long-tailed recognition ,where the test class distribution is unknown and not necessarily uniform.
To tackle this task, wepresent a novel approach, namely Self-supervised Aggregation of Diverse Experts (SADE), whichconsists of two innovative strategies, i.e., skill-diverse expert learning and test-time self-supervisedaggregation.
We theoretically analyze our proposed method and also empirically show that SADEachieves new state-of-the-art performance on both vanilla and test-agnostic long-tailed recognition.
Checklist
For all authors...
[Table] (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
[Table]
(b) Did you describe the limitations of your work?
[Yes] Please refer to Appendix H.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
This is a
fundamental research that does not have particular negative social impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes] Please refer to
Appendix A.
If you ran experiments...
[Table]
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] Please refer tothe supplemental material. [Table]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] Please refer to Section 5.1 and Appendix C.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[N/A]
The common practice in long-tailed recognition does notreport error bars, so we follow the previous papers and do not report them.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] Please refer to Appendix C.3 fordetails on different datasets.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[N/A] All the used benchmark datasets are
publicly available.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
We submitted the source codes of our method as an anonymized zip file.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A] These datasets are open-source benchmark datasets.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A] These datasets are open-source benchmark
datasets.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
Appendices: Self-Supervised Aggregation of Diverse Experts for Test-Agnostic Long-Tailed Recognition
We organize the appendices as follows:
• Appendix A: the proofs for Theorem 1.
• Appendix B: the pseudo-code of the proposed method.
• Appendix C: more details of experimental settings.
• Appendix D: more empirical results on vanilla long-tailed recognition, test-agnostic long-tailedrecognition, skill-diverse expert learning, and test-time self-supervised aggregation.
• Appendix E: more ablation studies on expert learning and the proposed inverse softmax loss.
• Appendix F: more ablation studies on test-time self-supervised aggregation.
• Appendix G: more discussion on model complexity.
• Appendix H: discussion on potential limitations.
A Proofs for Theorem 1
Proof.
We first recall several key notations and define some new notations.
The random variables ofmodel predictions and ground-truth labels are defined as ˆ Y ∼ p (ˆ y ) and Y ∼ p ( y ) , respectively.
Thenumber of classes is denoted by C .
Moreover, we further denote the test sample set of the class k by
Z k , in which the total number of samples in this class is denoted by
|Z k | .
Let c k = 1 |Z k | (cid:80) ˆ y ∈Z k ˆ yrepresent the hard mean of all predictions of samples from the class k , and let c = indicate equality upto a multiplicative and/or additive constant.
As shown in Eq.(4), the optimization objective of our test-time self-supervised aggregation method isto maximize S
= (cid:80) n t j =1 ˆ y 1 j · ˆ y 2 j , where n t denotes the number of test samples.
For convenience, wesimplify the first data view to be the original data, so the objective function becomes (cid:80) n t j =1 ˆ y j · ˆ y 1 j .
Maximizing such an objective is equivalent to minimizing (cid:80) n t j =1 − ˆ y j
· ˆ y 1 j .
Here, we assume the dataaugmentations are strong enough to generate representative data views that can simulate the test datafrom the same class.
In this sense, the new data view can be regarded as an independent sample fromthe same class.
Following this, we analyze our method by connecting − ˆ y j
· ˆ y 1 j to (cid:80) ˆ y j ∈Z k
∥ ˆ y j − c k ∥ 2 ,which is similar to the tightness term in the center loss [47]:
[Equation]
where we use the property of the normalized predictions, i.e., ∥ ˆ y j ∥ 2 = ∥ ˆ y 1 j ∥ 2 = 1 , and the definitionof the class hard mean c k = 1 |Z k | (cid:80) ˆ y ∈Z k ˆ y .
By summing over all classes k , we obtain:
[Equation]
Based on this equation, following [1, 54], we can interpret (cid:80) n t j =1 − ˆ y j · ˆ y 1 j as a conditional crossentropy between ˆ Y and another random variable ¯ Y , whose conditional distribution given Y is astandard Gaussian centered around c Y : ¯ Y | Y ∼N ( c y , i ) :
[Equation]
Hence, we know that (cid:80) n t j =1 − ˆ y j
· ˆ y 1 j is an upper bound on the conditional entropy of predictions ˆ Ygiven labels Y :
[Equation]
where the symbol c ≥ represents “larger than" up to a multiplicative and/or an additive constant.
Moreover, when ˆ Y | Y ∼N ( c y , i ) , the bound is tight.
As a result, minimizing (cid:80) n t j =1 − ˆ y j · ˆ y 1 j isequivalent to minimizing H ( ˆ Y | Y ) :
[Equation]
Meanwhile, the mutual information between predictions ˆ Y and labels Y can be represented by:
[Equation]
Combining Eqs.(5-6), we have:
[Equation]
Since S = (cid:80) n t j =1 ˆ y j · ˆ y 1 j , we obtain:
[Equation]
which concludes the proof for Theorem 1.
B Pseudo-code
This appendix provides the pseudo-code 2 of SADE, which consists of skill-diverse expert learning andtest-time self-supervised aggregation.
Here, the skill-diverse expert learning strategy is summarizedin Algorithm 1.
For simplicity, we depict the pseudo-code based on batch size 1, but we conductbatch gradient descent in practice.
Require:
Epochs T ; Hyper-parameters λ for L inv Initialize: Network backbone f θ ; Experts E 1 , E 2 , E1: for e=1,..., T do 2: for x ∈ D s do // batch sampling in practice 3:
Obtain logits v 1 based on f θ and E 1 ; 4:
Obtain logits v 2 based on f θ and E 2 ; 5:
Obtain logits v 3 based on f θ and E 3 ; 6:
Compute loss L ce with v 1 for Expert E 1 ; // Eq.(1) 7:
Compute loss L bal with v 2 for Expert E 2 ; // Eq.(2) 8:
Compute loss L inv with v 3 for Expert E 3 ; // Eq.(3) 9:
Train the model with L ce + L bal + L inv .
10: end for 11: end for Output: The trained model { f
[Table]
, E 1 , E 2 , E 3 }
After training the multiple skill-diverse experts with Algorithm 1, the final prediction of the multiexpert model for vanilla long-tailed recognition is the arithmetic mean of the prediction logits ofthese experts, followed by a softmax function.
When it comes to test-agnostic long-tailed recognition, we need to aggregate these skill-diverseexperts to handle the unknown test class distribution based on Algorithm 2.
Here, to avoid the learnedweights of some weak experts becoming zero, we give a stopping condition in Algorithm 2: if theweight for one expert is less than 0.05, we stop test-time training.
Retaining a small amount of weightfor each expert is sufficient to ensure the effect of ensemble learning.
Require: Epochs T ′ ;
The trained backbone f θ ; The trained experts E 1 , E 2 , E 3 Initialize: Expert aggregation weights w // uniform initialization 1: for e=1,..., T ′ do 2: for x ∈ D t do // batch sampling in practice 3:
Draw two data augmentation functions t ∼T t ′ ∼T ; 4:
Generate data views x t x x t ′ ( x ) ; 5:
Obtain logits v 11 , v 12 , v 13 for the view x 1 ; 6:
Obtain logits v 21 , v 22 , v 23 for the view x 2 ; 7:
Normalize expert weights w via softmax function; 8:
Conduct predictions ˆ y 1 , ˆ y 2 based on ˆ y =
wv ; 9:
Compute prediction stability S ; // Eq.
Maximize S to update w ; 11: end for 12:
If w i ≤ 0 .
for any w i ∈ w , then stop training.
13: end for
Output: Expert aggregation weights w
Note that, in test-agnostic long-tailed recognition, each model is only trained once on long-tailedtraining data and then directly evaluated on multiple test sets.
Our test-time self-supervised strategyadapts the trained multi-expert model using only unlabeled test data during testing.
C More Experimental Settings
In this appendix, we provide more details on experimental settings.
C.1 Benchmark Datasets
We use four benchmark datasets ( i.e., ImageNet-LT [29], CIFAR100-LT [3], Places-LT [29], andiNaturalist 2018 [41]) to simulate real-world long-tailed class distributions.
Their data statistics aresummarized in Table 11, where CIFAR100-LT has three variants with different imbalance ratios.
Theimbalance ratio is defined as max n j / min n j , where n j denotes the data number of class j .
[Table]
C.2 Construction of Test-agnostic Long-tailed Datasets
Following LADE [17], we construct three kinds of test class distributions, i.e., the uniform distribution,forward long-tailed distributions and backward long-tailed distributions.
In the backward ones, thelong-tailed class order is flipped.
Here, the forward and backward long-tailed test distributionscontain multiple different imbalance ratios, i.e., ρ ∈ { 2 , 5 , 10 , 25 , 50 } .
Note that LADE [17] onlyconstructed multiple distribution-agnostic test datasets for ImageNet-LT; while in this study, we usethe same way to construct distribution-agnostic test datasets for the remaining benchmark datasets,i.e., CIFAR100-LT, Places-LT and iNaturalist 2018, as illustrated below.
Considering the long-tailed training classes are sorted in a decreasing order, the various test datasetsare constructed as follows:
(1) Forward long-tailed distribution
: the number of the j -th class isn j = N · ρ ( j − 1) /C , where N indicates the sample number per class in the original uniform testdataset and C is the number of classes.
(2) Backward long-tailed distribution: the number of the j -thclass is n j = N · ρ ( C − j ) /C .
In the backward long-tailed distributions, the order of the long tail onclasses is flipped, so the distribution shift between training and test data is large, especially when theimbalance ratio gets higher.
For ImageNet-LT, CIFAR100-LT and Places-LT, since there are enough test samples per class, wefollow the setting in LADE [17] and construct the imbalance ratio set by ρ ∈ { 2 , 5 , 10 , 25 , 50 } .
ForiNaturalist 2018, since each class only contains three test samples, we adjust the imbalance ratio setto ρ ∈ { 2 , 3 } .
Note that when we set ρ = 3 , there are some classes in iNaturalist 2018 containing notest sample.
All these constructed distribution-agnostic long-tailed datasets will be publicly availablealong with our code.
C.3 More Implementation Details of Our Method
We implement our method in PyTorch.
Following [17, 46], we use ResNeXt-50 for ImageNet-LT,
ResNet-32 for CIFAR100-LT, ResNet-152 for Places-LT and ResNet-50 for iNaturalist 2018 asbackbones, respectively.
Moreover, we adopt the cosine classifier for prediction on all datasets.
Although we have depicted the skill-diverse multi-expert framework in Section 4.1, we give moredetails about it here.
Without loss of generality, we take ResNet [15] as an example to illustrate themulti-expert model.
Since the shallow layers extract more general features and deeper layers extractmore task-specific features [51], the three-expert model uses the first two stages of ResNet as theexpert-shared backbone, while the later stages of ResNet and the fully-connected layer constituteindependent components of each expert.
To be more specific, the number of convolutional filtersin each expert is reduced by 1/4, since by sharing the backbone and using fewer filters in eachexpert [46, 57], the computational complexity of the model is reduced compared to the model withindependent experts.
The final prediction is the arithmetic mean of the prediction logits of theseexperts, followed by a softmax function.
In the training phase, the data augmentations are the same as previous long-tailed studies [17, 24].
Ifnot specified, we use the SGD optimizer with the momentum of 0.9 and set the initial learning rate as0.1 with linear decay.
More specifically, for ImageNet-LT, we train models for 180 epochs with batchsize 64 and a learning rate of 0.025 (cosine decay).
For CIFAR100-LT, the training epoch is 200 andthe batch size is 128.
For Places-LT, following [29], we use ImageNet pre-trained ResNet-152 as thebackbone, while the batch size is set to 128 and the training epoch is 30.
Besides, the learning rate is0.01 for the classifier and 0.001 for all other layers.
For iNaturalist 2018, we set the training epoch to200, the batch size to 512 and the learning rate to 0.2.
In our inverse softmax loss, we set λ =2 for
ImageNet-LT and CIFAR100-LT, and λ =1 for the remaining datasets.
In the test-time training, we use the same augmentations as MoCo v2 [5] to generate different dataviews, i.e., random resized crop, color jitter, gray scale, Gaussian blur and horizontal flip.
If notspecified, we train the aggregation weights for 5 epochs with the batch size 128, where we adopt thesame optimizer and learning rate as the training phase.
More detailed statistics of network architectures and hyper-parameters are reported in Table 12.
Basedon these hyper-parameters, we conduct experiments on 1 TITAN RTX 2080 GPU for CIFAR100-LT,4 GPUs for iNaturalist18, and 2 GPUs for ImageNet-LT and Places-LT, respectively.
The source codeof our method is available in the supplementary material.
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
C.4 Discussions on Evaluation Metric
As mentioned in Section 5.1, we follow LADE [17] and use micro accuracy to evaluate modelperformance on test-agnostic long-tailed recognition.
In this appendix, we explain why microaccuracy is a better metric than macro accuracy when the test dataset exhibits a non-uniform classdistribution.
For instance, in the test scenario with a backward long-tailed class distribution, the tailclasses are more frequently encountered than the head classes, and thus should have larger weightsin evaluation.
However, simply using macro accuracy treats all the categories equally and cannotdifferentiate classes of different frequencies.
For example, one may train a recognition model for autonomous cars based on the training datacollected from city areas, where pedestrians are majority classes and stone obstacles are minorityclasses.
Assume the model accuracy is 60% on pedestrians and 40% on stones.
If deploying themodel to city areas, where pedestrians/stones are assumed to have 500/50 test data, then the macroaccuracy is 50% and the micro accuracy is 500 × 0 .
6+50 × 0 .
4 500+ ≈ 58% .
In contrast, when deploying themodel to mountain areas, the pedestrians become the minority, while stones become the majority.
Assuming the test data numbers are changed to 50/500 on pedestrians/stones, the micro accuracy isadjusted to 50 × 0 .
6+500 × 0 .
4 50+ ≈ 42% ,
but the macro accuracy is still 50%.
In this case, macro accuracyis less informative than micro accuracy for measuring model performance.
Therefore, micro accuracyis a better metric to evaluate the performance of test-agnostic long-tailed recognition.
D More Empirical ResultsD.1 More Results on Vanilla Long-tailed RecognitionAccuracy on class subsets
In the main paper, we have provided the average performance overall classes on the uniform test class distribution.
In this appendix, we further report the accuracyregarding various class subsets (c.f.
Table 13), making the results more complete
[Table]
[Table]
Results on stonger data augmentations Inspired by PaCo [8], we further evaluate SADE trainingwith stronger data augmentation ( i
[Table]
e., RandAugment [7]) for 400 epochs.
The results in Table 14further demonstrate the state-of-the-art performance of SADE
[Table]
work [8]
[Table]
Results on more neural architectures In addition to using the common practice of backbones as
[Table]
previous long-tailed studies [17, 46]
[Table]
we further evaluate SADE on more neural architectures.
Theresults in Table 15 demonstrate that SADE is able to train different network backbones well
[Table] [Table]
Results on more datasets We also conduct experiments on CIFAR10-LT with imbalance ratios of10 and 100.
Promising results in Table 16 further demonstrate the effectiveness and superiority of ourproposed method.
[Table]
D.2 More Results on Test-agnostic Long-tailed Recognition
In the main paper, we have provided the overall performance on four benchmark datasets with varioustest class distributions (cf.
Table 5).
In this appendix, we further verify the effectiveness of ourmethod on more dataset settings ( i
[Table]
e., CIFAR100-IR10 and CIFAR100-IR50), as shown in Table
[Table] [Table]
Furthermore, we plot the results of all methods under these benchmark datasets with various testclass distributions in Figure 4.
To be specific, Softmax only performs well on highly-imbalancedforward long-tailed class distributions.
Existing long-tailed baselines outperform Softmax, but theycannot handle backward test class distributions well.
In contrast, our method consistently outperformsbaselines on all benchmark datasets, particularly on the backward long-tailed test distributions with arelatively large imbalance ratio.
[Figure] [Figure]
D.3 More Results on Skill-diverse Expert Learning
This appendix further evaluates the skill-diverse expert learning strategy on CIFAR100-LT, Places-LTand iNaturalist 2018 datasets.
We report the results in Table 18, from which we draw the followingobservations.
RIDE [46] is one of the state-of-the-art ensemble-based long-tailed methods, which triesto learn diverse distribution-aware experts by maximizing the divergence among expert predictions.
However, such a method cannot learn sufficiently diverse experts.
As shown in Table 18, the threeexperts in RIDE perform very similarly on various groups of classes under all benchmark datasets,and each expert has similar overall performance on each dataset.
Such results demonstrate that simplymaximizing the KL divergence of different experts’ predictions is not sufficient to learn visiblydiverse distribution-aware experts.
In contrast, our proposed method learns the skill-diverse experts by directly training each expertwith their customized expertise-guided objective functions, respectively.
To be specific, the forwardexpert E 1 seeks to learn the long-tailed training distribution, so we directly train it with the crossentropy loss.
For the uniform expert E 2 , we use the balanced softmax loss to simulate the uniformtest distribution.
For the backward expert E 3 , we design a novel inverse softmax loss to train theexpert, so that it simulates the inversely long-tailed class distribution.
Table 18 shows that the threeexperts trained by our method are visibly diverse and skilled in handling different class distributions.
Specifically, the forward expert is skilled in many-shot classes, the uniform expert is more balancedwith higher overall performance, and the backward expert is good at few-shot classes.
Because ofsuch a novel design that enhances expert diversity, our method achieves more promising ensembleperformance compared to RIDE.
[Table]
D.4 More Results on Test-time Self-supervised Aggregation
This appendix provides more results to examine the effectiveness of our test-time self-supervisedaggregation strategy.
We report results in Table 19, from which we draw several observations.
First of all, our method is able to learn suitable expert aggregation weights for test-agnostic classdistributions, without relying on the true test class distribution.
For the forward long-tailed testdistribution, where the test data number of many-shot classes is more than that of medium-shotand few-shot classes, our method learns a higher weight for the forward expert E 1 who is skilledin many-shot classes, and learns relatively low weights for the expert E 2 and expert E 3 who aregood at medium-shot and few-shot classes.
Meanwhile, for the uniform test class distribution whereall classes have the same number of test samples, our test-time expert aggregation strategy learnsrelatively balanced weights for the three experts.
For example, on the uniform ImageNet-LT test data,the learned weights by our strategy are 0.33, 0.33 and 0.34 for the three experts, respectively.
Inaddition, for the backward long-tailed test distributions, our method learns a higher weight for thebackward expert E 3 and a relatively low weight for the forward expert E 1 .
Note that when the classimbalance ratio becomes larger, our method is able to learn more diverse expert weights adaptivelyfor fitting the actual test class distributions.
Such results not only demonstrate the effectiveness of our proposed strategy, but also verify thetheoretical analysis that our method can simulate the unknown test class distribution.
To our bestknowledge, such an ability is quite promising, since it is difficult to know the true test class distributions in real-world application.
Therefore, our method opens the opportunity for tackling unknownclass distribution shifts at test time, and can serve as a better candidate to handle real-world long-tailedlearning applications.
The results show that our self-supervised strategy is able to learn suitable expert weights for various unknown test class distributions
[Table]
Relying on the learned expert weights, our method aggregates the three experts appropriately andachieves better performance on the dominant test classes, thus obtaining promising performancegains on various test distributions, as shown in Table 20.
Note that the performance gain compared toexisting methods gets larger as the test dataset gets more imbalanced.
For example, on CIFAR100-LTwith the imbalance ratio of 50, our test-time self-supervised strategy obtains a 7.7 % performancegain on the Forward-LT-50 distribution and obtains a 9.2 % performance gain on the Backward-LT-50distribution, both of which are non-trivial.
Such an observation is also supported by the visualizationresult of Figure 5, which plots the results of existing methods on ImageNet-LT with different testclass distributions regarding the three class subsets.
In addition, since the imbalance degrees of the test datasets are relatively low on iNaturalist 2018, thesimulated test class distributions are thus relatively balanced.
As a result, the obtained performanceimprovement is not that significant, compared to other datasets.
However, if there are more iNaturalisttest samples following highly imbalanced test class distributions in real applications, our methodwould obtain more promising results
[Table] [Table]
.0 43.3(+1.0) - - - - - - - - Forward-LT43.0 44.0 33.1 42.
45.6 40.6 27.3 42.6(+0.2) - - - - - - - - Forward-LT43.4 42.4 32.6 41.
44.9 41.2 29.5 41.3(+0.0) 73.
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
72.5(+0.1) Uniform 43.
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
) Backward-LT42.
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
4) Backward-LT-
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
) - - - - - - - - Backward-LT-10 43.5 42.
[Table] [Table]
[Table]
41.4(+2.5) - - - - - - - - Backward-LT-25 44.
[Table] [Table]
[Table] [Table]
[Table] [Table]
42.0(+4.2) - - - - - - - - Backward-LT-50 42.
[Table]
[Table] [Table]
[Table] [Table]
6(+4.7) - - - - - - - -
[Figure]
data, with the imbalance ratio N , respectively.
The results show that existing methods perform very similarly on various test class distributions in terms of their performance on many-shot, medium-shot and few-shot classes.
In contrast, our proposed method is capable of adaptingbi to various test class distributions in terms of many-shot, medium-shot and few-shot performance, thus leading to better overall performance on each test class distribution.
E Ablation Studies on
Skill diverse Expert Learning
[Table]
E.1 Discussion on Expert Number
In SADE, we consider three experts, where the “forward" and “backward" experts are necessary sincethey span a wide spectrum of possible test class distributions, while the “uniform" expert ensures thatwe retain high accuracy on the uniform test class distributions.
Nevertheless, our approach can bestraightforwardly extended to more than three experts.
For the models with more experts, we adjustthe hyper-parameter λ in Eq. (3) for the new experts and keep the hyper-parameters of the originalthree experts unchanged, so that different experts are skilled in different types of class distributions.
Following this, we further evaluate the influence of the expert number on our method based on
ImageNet.
To be specific, when there are four experts, we set λ = 1 for the new expert; while whenthere are five experts, we set λ = 0 . 5 and λ = 1 for the two newly-added experts, respectively.
As shown in Table 21, with the increasing number of experts, the ensemble performance of ourmethod is improved on vanilla long-tailed recognition, e.g., four experts obtain a 1.2 % performancegain compared to three experts on ImageNet-LT.
As a result, our method with more experts obtainsconsistent performance improvement in test-agnostic long-tailed recognition on various test classdistributions compared to three experts, as shown in Table 22.
Even so, only three experts aresufficient to handle varied test class distributions, and provide a good trade-off between performanceand efficiency.
[Table] [Table]
E.2 Hyper-parameters in Inverse Softmax Loss
This appendix evaluates the influence of the hyper-parameter λ in the inverse softmax loss for thebackward expert, where we fix all other hyper-parameters and only adjust the value of λ .
As shown in
Table 23, with the increase of λ , the backward expert simulates more inversely long-tailed distribution(to the training data), and thus the ensemble performance on few-shot classes is better.
Moreover,when λ ∈ { 2 , 3 } , our method achieves a better trade-off between head classes and tail classes, leadingto relatively better overall performance on ImageNet-LT.
[Table]
[Table]
F Ablation Studies on Test-time Self-supervised AggregationF.1 Influences of Training Epoch
As illustrated in Section 5.
[Table]
we set the training epoch of our test-time self-supervised aggregationstrategy to 5 on all datasets.
Here
[Table]
we further evaluate the influence of the epoch number
[Table]
where weadjust the epoch number from 1 to 100.
As shown in Table
[Table]
when the training epoch number islarger than 5, the learned expert weights by our method are converged on ImageNet-LT, which verifiesthat our method is robust enough.
The corresponding performance on various test class distributionsis reported in Table 25.
[Table] [Table]
F.2 Influences of Batch Size
In previous results
[Table]
we set the batch size of test-time self-supervised aggregation to 128 on all datasets.
In this appendix
[Table]
we further evaluate the influence of the batch size on our strategy, where we adjustthe batch size from 64 to 256.
As shown in Table 26, with different batch sizes, the learned expertweights by our method keep nearly the same, which shows that our method is insensitive to the batchsize.
The corresponding performance on various test class distributions is reported in Table
[Table]
wherethe performance is also nearly the same when using different batch sizes.
[Table]
[Table]
[Table]
43.3 58.8 Backward-LT64.
44.8 57.1 Backward-LT63.
47.5 55.5 Backward-LT60.
48.7 54.4 Backward-LT60.
49.6 53.6 Backward-LT61.
F.3 Influences of Learning Rate
In this appendix, we evaluate the influence of the learning rate on our self-supervised strategy,where we adjust the learning rate from 0.001 to 0.5.
As shown in Table 28, with the increase of thelearning rate, the learned expert weights by our method are sharper and fit the unknown test classdistributions better.
For example, when the learning rate is 0.001, the weight for expert E 1 is 0.36on the Forward-LT-50 test distribution, while when the learning rate increases to 0.5, the weightfor expert E 1 becomes 0.57 on the Forward-LT-50 test distribution.
Similar phenomenons are alsoobserved on backward long-tailed test class distributions.
[Table]
By observing the corresponding model performance on various test class distributions in Table
[Table]
we find that when the learning rate is too small ( e.g., 0.001), our test-time self-supervised aggregationstrategy is unable to converge, given a fixed training epoch number of 5.
In contrast, given the sametraining epoch, our method can obtain better performance by reasonably increasing the learning rate.
[Table] [Table]
[Table]
F.4 Results of Prediction Confidence
In our theoretical analysis ( i.e., Theorem 1), we find that our test-time self-supervised aggregationstrategy not only simulates the test class distribution, but also makes the model predictions moreconfident.
In this appendix, we evaluate whether our strategy can really improve the predictionconfidence of models on various unknown test class distributions of ImageNet-LT.
To this end,we compare the prediction confidence of our method without and with test-time self-supervisedaggregation in terms of the hard mean of the highest prediction probability on all test samples.
As shown in Table 30, our test-time self-supervised aggregation strategy enables the deep model tohave higher prediction confidence.
For example, on the Forward-LT-50 test distribution, our strategyobtains 0.015 confidence improvement, which is non-trivial since it is an average value for a largenumber of samples (more than 10,000 samples).
In addition, when the class imbalance ratio becomeslarger, our method is able to obtain more apparent confidence improvement.
[Table]
F.5 Run-time Cost of Test-time Aggregation
One may be interested in the run-time cost of our test-time self-supervised aggregation strategy, sowe further report its running time on Forward-LT-50 and Forward-LT-25 test class distributions forillustration.
As shown in Table 31, our test-time self-supervised aggregation strategy is fast in termsof per-epoch time.
The actual average additional time is only 0.009 seconds per sample at test time on
V100 GPUs.
The result is easy to interpret since we freeze the model parameters and only learn theaggregation weights, which is much more efficient than training the whole model.
More importantly,the goal of this paper is to handle a practical yet challenging test-agnostic long-tailed recognitiontask.
For solving this challenging problem, we believe it is acceptable to allow models to be trainedmore, while the promising results in previous experiments have demonstrated the effectiveness ofour proposed test-time self-supervised learning strategy in handling this problem. In the future, wewill further extend the proposed method for better computational efficiency, e.g., exploring dynamicnetwork routing. [Table] F.6 Test-time Self-supervised Aggregation on Streaming Test Data
In the previous experiments
[Table]
we conduct the test-time self-supervised aggregation strategy in anoffline manner.
However, as mentioned in Section 4.2, our test-time strategy can also be conductedin an online manner and does not require access to all the test data in advance.
To verify this
[Table]
wefurther conduct our test-time strategy on steaming test data of ImageNet-LT.
As shown in Table 32,our test-time strategy performs well on the streaming test data.
Even when the test data come inone by one, our test-time self-supervised strategy still outperforms the state-of-the-art baseline (i.e.,offline Tent [43]) by a large margin.
[Table]
G More Discussions on Model Complexity
In this appendix, we discuss the model complexity of our method in terms of the number of parameters,multiply-accumulate operations (MACs) and top-1 accuracy on test-agnostic long-tailed recognition.
As shown in Table 33, both SADE and RIDE belong to ensemble-based long-tailed learning methods,so they have more parameters (about 1.5x) and MACs (about 1.4x) than the original backbone model,where we do not use the efficient expert assignment trick in [46] for both methods.
Because ofthe ensemble effectiveness of the multi-expert scheme, both methods perform much better thannon-ensemble methods ( e.g., Softmax and other long-tailed methods).
In addition, since our methodand RIDE use the same multi-expert framework, both methods have the same number of parametersand MACs.
Nevertheless, by using our proposed skill-diverse expert learning and test-time selfsupervised aggregation strategies, our method performs much better than RIDE with no increase inmodel parameters and computational costs.
One may concern the multi-expert scheme leads to more model parameters and higher computationalcosts than the original backbone.
However, note that the main focus of this paper is to solve thechallenging test-agnostic long-tailed recognition, while promising results have shown that our methodaddresses this problem well.
In this sense, slightly increasing the model complexity is acceptablefor solving this practical yet challenging problem.
Moreover, since there have already been manystudies [14, 48] showing effectiveness in improving the efficiency of the multi-expert scheme, wethink the computation increment is not a severe issue and we leave it to the future.
Here, we do not use the
[Figure]
[Table]
H Potential Limitations
One concern is that this work only focuses on long-tailed classification problems.
However, webelieve this is enough for a new challenging task of test-agnostic long-tailed recognition
[Table]
while howto extending to object detection and instance segmentation will be explored in the future.
Anotherpotential concern is the model complexity of our method.
However, as discussed in Appendix G, thecomputation increment is not a very severe issue
[Table]
while how to further accelerate our method will beexplored in future.
In addition, one may also expect to evaluate the proposed method on more testclass distributions.
However, as shown in Section 5.3, we have demonstrated the effectiveness of ourmethod on the uniform class distribution, the forward and backward long-tailed class distributionswith various imbalance ratios, and even partial class distributions.
Therefore, we believe the empiricalverification is sufficient for verifying our method, and the extension to more complex test classdistributions is left to the future.