L EARNING
B INARY
N ETWORKS
ON
L ONG -T
AILED
D ISTRIBUTIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
In deploying deep models to real world scenarios, there are a number of issues including computational resource constraints and long-tailed data distributions.
For the ﬁrst time in the literature, we address the combined challenge of learning long-tailed distributions under the extreme resource constraints of using binary networks as backbones.
Speciﬁcally, we propose a framework of calibrating off-the-shelf pretrained full precision weights that are learned on non-long-tailed distributions when training binary networks on long-tailed datasets.
In the framework, we additionally propose a novel adversarial balancing and a multi-resolution learning method for better generalization to diverse semantic domains and input resolutions.
We conduct extensive empirical evaluations on 15 datasets including newly derived long-tailed datasets from existing balanced datasets, which is the largest benchmark in the literature.
Our empirical studies show that our proposed method outperforms prior arts by large margins, e.g ., at least +14 .
33% on average.
I NTRODUCTION
In recent years, there grows emphasis on resource constraints, especially for edge devices, in learning deep models, resulting in breakthroughs such as MobileNet (Howard et al., 2017) and YOLOV7 (Wang et al., 2022) that concern not only accuracy but also computational costs.
This has attracted attention for deep learning both in the research and the industrial communities.
Besides, long-tailed (LT) training data are frequently encountered in the wild (He & Garcia, 2009).
Thus, many deep learning methods have been developed to combat it (Cui et al., 2021;
He et al., 2021;
Zhong et al., 2021).
Yet, the devices of daily usages, of many which aim to utilize these deep models, suffer from lack of sufﬁcient computing power.
Thus, for real world deployment of such deep models, methodological advances in LT recognition should also consider the resource constraints.
Unfortunately, current LT recognition methods largely assume sufﬁcient computing resources and are designed to work with a large number of full precision parameters, i.e ., ﬂoating point (FP) weights.
While recognition on LT distributions by FP models may have signiﬁcantly improved (Cui et al., 2021;
He et al., 2021; Zhong et al., 2021), it is not clear whether these improvements would immediately translate to real world scenarios or not where resource constraints limit the model selection to those with lower capacity.
To this end, we argue that it is necessary to benchmark and improve the performance of long-tailed recognition with capacity-limited models.
As binary networks are at the extreme end of capacity-limited models (Rastegari et al., 2016), the long-tailed recognition performance using the 1-bit networks would roughly correspond to the ‘worst case scenario’ for resource constrained LT.
If we can show sufﬁcient LT performance with binary networks, we can reasonably expect, at the very least, matching or better LT performance with N -bit models, where N > 1 .
Here, we take an initiative to benchmark and develop long-tailed recognition methods using binary networks as a challenging reference.
In the LT scenario, the data scarcity in the tail classes is one of the major issues that may cause problems such as the class weights having varying magnitudes for head to tail classes (Kang et al., 2019;
Alshammari et al., 2022), leading to disappointing performance.
Prior methods (Liu et al., 2019;
Kozerawski et al., 2020; Park et al., 2022) use cosine classiﬁers which eliminate the effect of class weight norms as a discriminative statistic and improve accuracy.
However, since binary networks lack learning capacity and exhibit worse generalization performance than FP networks (Rastegari et al., 2016; Courbariaux et al., 2016), we want to reduce the adverse effect of the uneven class
[Figure]
i
[Figure]
r yF P
[Figure]
[Figure] [Figure]
# o f
S a m p l e s
One can reduce the high variance in binary networks by providing additional supervisory signals to the binary networks during training.
The additional supervision would supplement the insufﬁcient information due to the scarce data in the tail classes.
In a different line of work, binary network learning methods (Martinez et al., 2020) employ additional supervision by a pretrained FP network used as a teacher in knowledge distillation when training binary networks.
Furthermore, works in
LT recognition with FP backbones also use distillation (He et al., 2021) to improve the accuracy.
However, these methods are not straightforwardly applicable to our scenario for the following reason.
As most of pretrained FP networks are learned on non LT datasets, using them as teachers may not be helpful for training binary networks on LT datasets due to the domain gap.
One can train the
FP teacher on the target LT datasets from scratch (He et al., 2021) before distillation but that is not scalable.
Recent works in various domains such as prompt engineering (Radford et al., 2021; Liu et al., 2021a) and self-supervised learning (Bachman et al., 2019; Chen et al., 2020) suggest utilizing pretrained weights is desirable for scalability (He et al., 2020).
Here, we propose a way to utilize pretrained teacher networks on non LT data ( e.g . from open source projects) for LT distillation without training the teacher from scratch.
Speciﬁcally, we propose a ‘Calibrate and Distill’ framework where we calibrate the pretrained teacher on non LT data by attaching and training the classiﬁer on the target LT dataset.
We investigate how the proposed method affects the variance and visualize the results in Fig.
As shown in the ﬁgure, the calibrate and distill method signiﬁcantly decreases the ‘Binary-FP’ gap, suggesting a reduction of the high variance in binary networks.
Note that while the magnitudes are within similar magnitude ranges, they are not equal for all classes, which could be useful for discriminating different classes.
In addition, semantic domains and image resolutions vary across multiple real-world data.
In addressing the variety, we found that balancing the amount of knowledge transferred from the distillation loss to the feature extractor and classiﬁer part of the binary network largely affects the performance, depending on how varied the target data is to the pretraining data.
However, manual tuning may result in good performance only for a limited number of benchmark datasets and may not generalize to various scenarios.
For better generalization to different data distributions, we propose a novel adversarially learned balancing scheme for long-tailed recognition that learns the balance dependent on the input data without requiring ﬁnicky hyper-parameter tuning.
Furthermore, we incorporate various input resolutions for better generalization to different data distributions (Jacobs et al., 1995; Rosenfeld, 2013).
This would ease the burden for the model to learn various optimal receptive ﬁelds that can help generalize to the differently sized data.
But using multi-resolution inputs (Rosenfeld, 2013) may incur increasing computational costs depending on what and how many resolutions are used.
For negligible training time increase in multi-scale learning, we propose to selectively use multi-resolution inputs to mitigate the extra computation costs within our calibrate and distill framework.
Combining all the proposals, we name our method as
Ca librate a n d D istill: L ong-Tailed Recognition on the E dge or CANDLE for short.
We empirically evaluate on a wide variety of datasets totalling to 15, where some are derived by under-sampling existing non LT datasets and some are from existing LT benchmarks, that far outnumbers the previously used datasets in the LT literature.
On our empirical validations in the various datasets, our method outperforms prior arts by large margin of at least +14 .
33% on average.
We summarize our contributions as follows:
Addressing resource constrained long-tailed recognition with binary networks for the ﬁrst time.
• Proposing a calibrate and distill framework for long-tailed recognition with binary networks utilizing the off-the-shelf pretrained teachers on non-LT distributions.
• Proposing a adversarially learned balancing scheme and efﬁcient usage of multi-resolution inputs for the calibrate and distill framework.
• Evaluating the methods in a widely varied set of 15 long-tailed datasets for a comprehensive empirical study (the largest empirical study in the LT recognition literature).
R ELATED W ORK
Long-tailed recognition.
As training deep learning models on LT distributions is challenging, numerous works have been proposed.
We categorize them into class rebalancing (He & Garcia, 2009; Buda et al., 2018;
Shen et al., 2016), logit adjustment (Cao et al., 2019; Cui et al., 2019;
Tan et al., 2020), two stage methods – sequential learning of representation and classiﬁer (Kang et al., 2019; Xiang et al., 2020; Ren et al., 2020; Tang et al., 2020;
Li et al., 2020), and knowledge distillation (He et al., 2021; Wang et al., 2020a; Zhang et al., 2021a; Xiang et al., 2020).
Prior arts usually train the model from the scratch on the target LT dataset, which may be less efﬁcient than using pretrained weights.
For instance, two stage methods (Kang et al., 2019; Xiang et al., 2020; Ren et al., 2020; Tang et al., 2020; Li et al., 2020) perform representation learning without utilizing off-the-shelf pretrained weights.
Granted, the pretrained weights are usually trained on a non-LT dataset with a considerable domain gap.
However, we aim to utilize those pretrained weights on non-LT distributions for more efﬁcient and effective long-tailed recognition.
Furthermore, some works tend to increase the model capacity for the long-tailed problem (Wang et al., 2020a;
Zhou et al., 2020; Cui et al., 2021; 2022) which incurs additional costs.
For the realworld scenario, one has to consider the resource constraints, thus resource efﬁcient methods should be preferred.
Additionally, most works are benchmarked only on a few datasets with little regards to actual data variety in the wild.
To more accurately benchmark LT methods that consider the realistic variety, we use a total of 15 different LT datasets whereas previous works mostly showed results on 3-9 (Cao et al., 2019;
Kang et al., 2019; Cui et al., 2021; 2022) datasets.
Binary networks.
As one of the ways of bringing extreme efﬁciency to deep learning models, many prior arts have been proposed that either improves upon the underlying architecture of binary networks (Rastegari et al., 2016; Lin et al., 2017;
Liu et al., 2018; 2020; Bulat et al., 2021), use neural architecture search to ﬁnd better binary networks (Kim et al., 2020; Kim & Choi, 2021;
Bulat et al., 2020), elaborate training schemes for effective optimization of binary networks (Martinez et al., 2020; Han et al., 2020; Meng et al., 2020;
Liu et al., 2021b;
Le et al., 2022), or explore using binary networks as backbones for previously unused scenarios such as unsupervised learning (Kim & Choi, 2022; Shen et al., 2021).
Recent successful attempts in training binary networks utilize a FP network to provide additional supervisory signals (Martinez et al., 2020; Liu et al., 2020;
Bulat et al., 2021; 2020; Kim et al., 2020; Kim & Choi, 2021; 2022).
However, these methods use teacher networks that are trained on data with similar distributions to that of the target data which amounts to little domain gap in the teacher network.
As this process is not scalable, we propose to utilize teacher networks trained on non-LT data for many different target LT data distributions, which is scalable but challenging.
Additionally, most existing works focus on the supervised classiﬁcation problem and methods that consider other scenarios such as unsupervised representation learning (Kim & Choi, 2022; Shen et al., 2021) and object detection (Wang et al., 2020b) are relatively scarce.
As such, there is hardly any previous work regarding long-tailed recognition with binary networks which brings resource efﬁciency to learning on long-tailed distributions.
Here, we aim to train resource efﬁcient binary
[Figure]
Binary
[Figure]
Tail
[Figure] [Figure]
[Figure]
W e i g h t
N o r m # of Samples
Binary
FP
[Figure]
# o
[Figure]
S a m p l e s
[Figure]
A PPROACH
To develop long-tailed recognition methods for real world deployment that considers realistic resource constraints, we use binary networks as backbone networks, which are extremely efﬁcient neural networks with single bit weights and activations (Rastegari et al., 2016)
(see Sec.
A.1 for a brief preliminary).
We ﬁrst explain how the class weight norms (Kang et al., 2019; Alshammari et al., 2022) can be used as an approximate to visualize the variance of a model and analyze its trends in LT (Sec. 3.1).
With the analysis, we describe the proposed ‘Calibrate and Distill’ framework (Sec. 3.2) and plot the classiﬁer weight norms to investigate how the proposed framework reduces the high variance in binary networks (Fig. 4).
Additionally, to make the model more generalizable to the various semantic domains and image resolutions present in the wild that may differ from benchmark to benchmark, we further propose adversarially learned balancing (Sec. 3.3) and an efﬁcient usage of multi-resolution inputs with negligible computational cost increase (Sec. 3.4).
3. H IGH V ARIANCE FOR B INARY N ETWORKS IN L ONG -T AILED D ISTRIBUTION
The scarcity of data for tail classes impacts the bias and variance of a model when it is trained on LT distributions (Wang et al., 2020a).
Speciﬁcally, we believe the model with linear classiﬁers is likely to exhibit high variance as it ﬁts to the few training samples in the tail classes.
We hypothesize that the above will be more severe for binary networks as they have limited capacity, making the models more susceptible to overﬁtting to the scarce training data in the tail.
Previous works (Kang et al., 2019; Alshammari et al., 2022) analyzed the imbalance in the classiﬁer weight norms for classes ranging from head to tail.
We carefully re-visit how the classiﬁer weight norm analysis could be used in our scenario as there are a number of differences that might affect the analysis as follows.
First, we found that previous works (Kang et al., 2019; Zhou et al., 2020; Alshammari et al., 2022) used the SGD optimizer with weight-decay when analyzing the classiﬁer weight norm.
However, using the Adam optimizer signiﬁcantly outperforms using the SGD optimizer when training binary networks (Han et al., 2020).
Thus, using SGD will lead to inaccurate observations as the analysis will be done for models that have not sufﬁciently ﬁt to the training data yet.
Second, weight decay is a regularization technique that pushes the weights to be closer to zero, which is not desirable for binary networks (Han et al., 2020) which quantize the weights to +1 / − 1 .
Furthermore, as weight decay regularizes the training, it may result in the binary network underﬁtting to the training data, similar to using the SGD optimizer (Han et al., 2020; Kim & Choi, 2021).
Perhaps even more problematic, weight decay may artiﬁcially reduce the weight magnitudes, making accurate analysis of the variance challenging.
Thus, we use the Adam optimizer with zero weight decay in observing the classiﬁer weight norms for our experiments Please see Sec.
A.10 for more discussion on the difference in weight norm trend with Adam and SGD.
We visualize the classiﬁer weight norms in Fig.
2-(a).
When the model is trained from scratch, we observe that the classiﬁer weight norms get bigger as the classes become more tail.
A similar phenomenon can be observed in machine learning literature where the model is trained to have high variance ( e.g ., high weight magnitudes) when it ﬁts excessively to the training data (Kohavi et al., 1996; Von Luxburg & Sch ¨ olkopf, 2011).
Thus, one can interpret the above observation as a signal of overﬁtting to the scarce training samples in the tail, leading to the weights for the tail classes having high magnitudes and large variance (see Sec.
A.2 for more details).
In line with our hypothesis, this trend is intensiﬁed for binary networks.
As shown in Fig.
2-(a), the binary networks exhibit consistently larger weights with widening gap between the binary and ﬂoating point models (see
Fig. 1) towards the tail classes.
In contrast, we apply the proposed CANDLE as depicted in Fig.
3 and conduct the same analysis using classiﬁer weight norms in Fig. 2-(b).
The magnitudes of the binary weights are largely reduced with the gap between the binary and FP being reduced signiﬁcantly (as also shown in Fig 1).
We attribute the reduction in variance to the ‘Calibrate and Distill’ component of CANDLE that supplies additional supervisory signals in the tail classes from the FP teacher.
3. C ALIBRATE AND D ISTILL
To overcome the high variance issue, we propose a ‘Calibrate and Distill’ scheme that provides additional information to the binary network during training over the given scarce training data in the tail classes.
Inspired by recent methods in training binary networks that use a FP teacher for distillation (Martinez et al., 2020), we use pretrained FP networks as distillation teacher networks in training the binary network on LT data.
Note that most existing pretrained weights are not necessarily trained on LT distributions and it is also not scalable to manually acquire pretrained weights on each target LT datasets.
Here, we propose the ‘Calibrate and Distill’ scheme to utilize non-LT pretrained weights for use on LT datasets in an efﬁcient manner.
Speciﬁcally, we propose to ﬁrst calibrate the teacher network with non-LT pretrained weights and then transfer the knowledge from the calibrated teacher.
In particular, we attach a randomly initialized classiﬁer to the non-LT pretrained FP network.
During calibration, the classiﬁer is trained using the cross entropy loss L CE on the target LT datasets using Zhong et al.
(2021), which is more efﬁcient than training the entire teacher.
We use the KL divergence and feature similarity losses between the calibrated teacher and the binary networks for distillation following (Romero et al., 2014;
Heo et al., 2019).
The optimization objective is
[Equation]
where D LT is the target LT dataset and f T ( · ) , f Bθ ( · ) are the calibrated teacher and binary networks, respectively.
e T ( · ) , e Bθ ( · ) are the teacher and binary encoders, i.e ., feature extractors, and
L KL ( · , · ) , L FS ( · , · ) are the KL divergence and cosine distance, inspired by Chen et al.
; He et al
[Figure]
W e i g h t
N o r m # of Samples
Baseline
Calibrate and Distill
# o f
S a m p l e s
When we empirically compare our ‘Calibrate and Distill’ to the baseline, i.e ., using the pretrained weights without calibrating with feature distillation (Romero et al., 2014; Heo et al., 2019), we see an accuracy gain of +8 .
75% on average (see Tab. 3).
More interestingly, observing the classiﬁer weight norms we found that the proposed ‘Calibrate and Distill’ method is highly effective in mitigating the high variance compared to the baseline as shown in Fig.
4-(a).
Note that while the high variance in binary networks is reduced through ‘Calibrate and Distill’, varying semantic domains with diverse image resolutions in the real world data are not sufﬁciently addressed yet.
3. A DVERSARIALLY L EARNED B ALANCING
The encoder of our teacher network is pretrained on non-LT data but the classiﬁer is trained on the target LT datasets.
Optimizing the binary encoder and the classiﬁer altogether with Eq.
1 will disregard this difference and may disrupt the training of the binary encoder by using the gradients from L KL in the same way that we use it to train the binary classiﬁer.
Thus, depending on how much the LT target data differs from the non-LT pretraining data, we may employ different strategies for balancing L FS and L KL via λ to train the binary encoder and the classiﬁer.
However, manually tuning λ for L KL and L FS for each different target data is not desirable for generalization.
To address the semantic diversity without manual tuning, we propose to parameterize and learn λ during training as well.
To that end, we ﬁrst split the binary network’s parameters θ into the binary encoder parameters ( θ e ) and classiﬁer parameters ( θ c ), and optimize them separately as
[Equation]
To learn λ , we use L KL and L FS as inputs and use an attention module with learnable parameters φ to output a single scalar value λ φ such that λ φ is dependent on the two loss functions as
[Equation]
where Atten φ ( · ) is a MLP that takes in L KL ( x ) and L FS ( x ) , and outputs λ φ .
Note that minimizing with respect to φ results in the learned λ
φ outputting zero as the balancing coefﬁcient for whichever loss function is larger i.e ., 0 for L FS in Eq.
2 if L FS > L KL .
While this does minimize the loss, it also prevents the gradient ﬂow from L FS when optimizing with respect to θ .
Thus, we employ adversarial learning where λ φ is learned to maximize the loss which is then minimized by optimizing θ .
We can rewrite our objective as
[Equation]
3. L EARNING WITH M ULTI -R ESOLUTION I NPUTS
To make our model more robust to various image resolutions frequent in the wild, we argue that using multi-resolution inputs during training would be a plausible solution (Rosenfeld, 2013).
Speciﬁcally, we propose to use 128 × 128 , 224 × 224 and 480 × 480 resolution images as multi-resolution inputs, following Tan & Le (2019).
However, directly using multi-resolution inputs dramatically increases the training time as shown in Fig.
5 (Single-Res vs .
Direct Multi-Res).
Thus, we propose a novel way to efﬁciently use multi-resolution inputs for our ‘Calibrate and Distill’ framework such that there is negligible
As we harness supervisory signals from both the input data and the teacher network, we can select which supervisory signal source the multi-resolution inputs are used.
Instead of using multiresolution inputs in both sources, we only use them when obtaining supervisory signals from the teacher.
To do so, we ﬁrst calibrate the teacher using multi-resolution inputs as shown in Fig.
As the calibration only trains the classiﬁer (see Sec. 3.2), the usage of multi-resolution inputs is relatively inexpensive.
During distillation, we feed the multi-resolution inputs to the teacher only and use single-resolution inputs for the binary network.
Because the teacher is not part of the backward computational graph, the increase in training time is negligible (see Fig. 5).
The ‘VRAM peak’ indicates the peak consumption of GPU memory during training.
Our scheme does incur some
VRAM peak increase but is much more efﬁcient than training schemes that directly use the multiresolution inputs to both teacher and binary networks.
Note that with the multi-resolution inputs, the feature similarity loss is calculated using only the channels that correspond to the input with the same resolution as that of the input given to the binary network as depicted in Fig.
We use calibrate and distill with the adversarially learned balancing and an efﬁcient way of using multi-resolution inputs as our full method named as CANDLE .
E XPERIMENTS
Setup.
For empirical evaluations, we use 12 small-scale and 3 large-scale LT datasets as detailed in Sec.
A.4.
We use pretrained FP networks from public open source projects that are either trained on ImageNet-1K for the small-scale and on ImageNet-21K for the large-scale LT datasets.
Following (Liu et al., 2021b), we use the Adam optimizer with zero weight decay for training binary networks.
Further implementation details are in Sec.
A.5.
Note that we use the authors’ implementation of prior arts when available or re-implement by reproducing the reported results.
More details on the compared prior arts are in Sec.
A.3.
We will publicly release the code for CANDLE along with the newly derived LT datasets.
Extended benchmarks for long-tail recognition.
In addition to developing a method for LT recognition using binary networks, we also want to improve how the performance of LT methods are measured to better reﬂect the real-world scenario (Yang et al., 2022).
To that end, we compile a large number of derived LT datasets from commonly used computer vision datasets along with the datasets often used in the LT literature to gather 12 small-scale and 3 large-scale datasets for evaluation.
We denote the imbalance ratio in a parenthesis following the dataset names for the small-scale datasets and add a postﬁx of ‘ − LT’ to each of the large-scale dataset names.
More details regarding on the newly added datasets along with dataset statistics for all 15 datasets are given in Sec.
A.4.
4. L ONG -T AILED R ECOGNITION A CCURACY IN V ARIOUS D ATASETS
We present comparative results of our method with various state of the arts LT methods tuned for binary networks.
We show the average accuracy over multiple datasets.
Please see Sec.
A.11 for head, medium, and tail class accuracy and per-class accuracy plots for CANDLE.
All results of
CANDLE are with linear classiﬁers (see Sec.
A.12 for results with cosine classiﬁers on small-scale
We compare with
(Kang et al.
BALMS (Ren et al
PaCo (Cui et al
MiSLAS (Zhong et al
DiVE (He et al superior performance large margins at least
[Table] [Table]
datasets).
The best accuracy is in bold and the second best is underline.
Surprisingly, DiVE (He et al., 2021) does not perform well on the small-scale datasets as the teacher network used in DiVE’s distillation process, e.g ., BALMS
(Ren et al., 2020) exhibits disappointing accuracy ( ∼ 26 .
65% mean acc.)
in the ﬁrst place.
Hence, we also present results with DiVE ∗ which uses the same teacher network as our method.
More discussion regarding some of the other low performing methods on the small-scale datasets such as τ -norm (Kang et al., 2019) and BALMS is in Sec.
A.6.
In small-scale datasets.
We summarize the comparative results on the 12 small-scale datasets in Tab. 1 along with the mean accuracy over all 12 datasets.
The mean accuracy over thedatasets suggests that CANDLE performs better than the prior arts by noticeable margins of at least +15 .
More precisely, the proposed method outperforms all other existing works by large margins for the popular CIFAR-10 (10,100) and CIFAR-100 (10,100) datasets.
In addition, our method exhibits superior performance in some of the newly added LT datasets with limited data such as CUB-200-2011 (10).
Interestingly, comparing DiVE and DiVE ∗ , we can see that using our calibrated teacher boosts the performance signiﬁcantly.
However, the performance difference between DiVE ∗ and CANDLE ( +20 . 27% ) also indicates that our adversarially learned balancing (Sec. 3.3) and multi-resolution teacher is effective (Sec. 3.4).
Additional results on changing the teacher network to the larger one used for large-scale experiments (Sec. 4) is in Sec.
A.7, which show that the mean accuracy improves when using the larger teacher.
Further comparison against the ensemble-based method, TADE (Zhang et al., 2021b), is also in Sec.
A.8.
In large-scale datasets.
Similar to Tab. 1, we compare our method to prior arts on the large-scale datasets in Tab. 2.
While our multi-resolution teacher is far more efﬁcient than directly using multiresolution (Fig. 5), the VRAM peak still increases.
Thus, we could not run the multi-resolution experiments with the large-scale datasets with our resources and only use single-resolution, indicated using † in Tab. 2.
Still, even without using multi-resolution, the proposed method outperforms prior arts with binary networks on all three large-scale datasets.
Looking at the mean accuracy, our method shows a margin of at least +7 .
93% when compared to the previous LT works.
4. L EARNED λ φ IN D IFFERENT D ATASETS
The learned values of λ φ (Eq. 4) for each dataset vs .
training iteration is visualized in Fig.
Note that we do not schedule λ φ in any speciﬁc way.
Interestingly, we observe smooth transitions from the initial values to 1 for all the shown datasets.
The initial values are different as they depend on the different initial loss values for each dataset.
In addition, while the learned λ φ converges to 1 (using only the L FS ) in Eq. 4, the way in which the λ φ changes is different depending on the dataset.
Combined with the empirical gain ( +5 .
40% mean average accuracy in Tab. 3) of using adversarially learned balancing, the various different ways in which λ φ varies in Fig.
6 imply that adversarially learned balancing (Eq. 4) is effective in balancing L KL and L FS on a wide variety of LT datasets.
[Table]
[Figure]
[Table]
[Figure]
the mean accuracy substantially.
[Table]
4. A BLATION S TUDIES
We ﬁnally conduct ablation studies of our 1 (cid:13) ‘Calibrate & Distill’, 2 (cid:13) ‘Adv.
Learned Bal.’
[Table]
and 3 (cid:13) ‘Multi-Res.’
on the 12 small-scale datasets and report the mean average, head, medium, and tail class accuracy in Tab. 3.
The baseline is set to using the uncalibrated teacher with feature distillation (Romero et al., 2014; Heo et al., 2019) (refer to Sec. 3.2 for detail).
Note that the baseline achieves 45 .
27% mean average accuracy, only behind MiSLAS.
All proposed components of CANDLE improve the mean average accuracy, e.g ., +10 .
32% for ‘Calibrate & Distill’, +5 .
40% for ‘Adv.
Learned Bal.’, and +1 .
08% for ‘Multi-Res.’.
Furthermore, the mean tail class accuracy also improves consistently with each component, indicating the effectiveness of the respective components for LT.
The average accuracy for each dataset for the ablated models can be found in Sec.
A.9.
C ONCLUSION
To develop efﬁcient deep learning models for LT, we use binary networks as backbones for learning long-tailed recognition.
In doing so, we analyze a high variance issue and propose to mitigate it using our ‘Calibrate and Distill’ framework where pretrained weights on non-LT data are utilized as distillation teachers for learning binary networks on LT target data distributions.
We further propose adversarially learned balancing and efﬁcient usage of multi-resolution inputs to better prepare the model for the wide variety of semantic domains and input resolutions present in the wild.
We empirically evaluate the proposed CANDLE and other existing works on a total of 15 datasets, which is the largest benchmark in the literature.
The proposed method achieves superior performance in all empirical validations with ablations showing clear beneﬁts of each component of CANDLE.
E THICS S TATEMENT
This work aims to equip edge-compatible binary networks with long-tailed recognition capabilities.
Thus, though there is no intent on the authors, application of deep models to real-world scenarios with long-tail data distributions might become prevalent.
This may expose the public to discrimination by the deployed deep models due to unsolved issues in deep learning such as model bias.
We will take all available measures to prevent such outcomes as that is not the intention of this work.
R EPRODUCIBILITY S TATEMENT
We take the reproducibility in deep learning very seriously and highlight some of the contents in the manuscript that might help in reproducing our work.
First, we will release the code and newly derived datasets used in our experiments as mentioned in Sec. 4.
Second, we include additional information regarding the extended long-tailed benchmark in Sec.
A.4 that may help with future work looking to use the datasets that were used in this work.
Third, we include relevant implementation details in both Sec. 4 and Sec.
A.5.
Last, we present our ﬁnal optimization objective (Eq. 4) and the overall view of our method (Fig. 3) with necessary details to reproduce the methodology
A A PPENDIX A. P RELIMINARY ON L EARNING B INARY N ETWORKS (S EC . 3)
Let W F ∈ R C
i × C
o × k × k be the 32-bit ﬂoating point weights of a convolution layer where C i , C o and k represent the number of input channels, the number of output channels and the kernel size, respectively.
The corresponding binarized weights are given by W B = sign( W F ) , where the sign( · ) outputs +1 / − 1 depending on the sign of the input.
The ﬂoating point input activation
A F ∈ R C i × w × h , where w, h are the width and height of the input activation, can be similarly binarized to the binary input activation by A B = sign( A F ) .
A ﬂoating point scaling factor is also used, which can either be the L1-norm of W F (Rastegari et al., 2016) or learned by backpropagation (Bulat & Tzimiropoulos, 2019).
The forward pass using convolutions is approximated as following
[Equation]
where ∗ , (cid:12) , ⊕ denote the ﬂoating point convolution operation, Hadamard product, and the binary
XNOR convolution with popcount, respectively.
STE (Courbariaux et al., 2016) is used for the weight update in the backward pass.
A. D ETAILED A NALYSIS ON T HE H IGH V ARIANCE (S EC . 3.1)
We give a more detailed explanation on the high variance analysis in Sec. 3.1.
As we are more interested in the high weight magnitudes which signals high variance, we focused on showing the classiﬁer weight norms in the main part of the paper.
Here, we also show the bias portion of the classiﬁer and see how it is affected for binary networks in LT conditions.
As seen in Fig.
7-(a), we see the classiﬁer weight norm ( e.g . variance) become larger and larger for the tail classes when the binary network is trained from scratch.
Looking at the bias in Fig.
7-(b), we see the classiﬁer bias getting smaller and smaller for the tail classes accordingly when the binary network is trained from scratch.
In contrast, when CANDLE is applied instead, the large increase in the classiﬁer weight norm for the tail classes is substantially mitigated, resulting in the weight norm barely changing from head to tail classes.
Similarly, the bias is also relatively unchanging compared to training from scratch and shows that CANDLE stabilizes the binary network in terms of both variance and bias.
[Figure]
A. D ETAILS ON C OMPARED M ETHODS (S EC . 4)
We give brief summary of the compared methods.
τ -norm (Kang et al., 2019) is a two-stage decoupling method, which consists of classiﬁer re-training(cRT) and learnable weight scaling(LWS).
BALMS(Ren et al., 2020) is also a two-stage method, where the balanced softmax loss and meta sampler is used in classiﬁer training stage.
PaCo(Cui et al., 2021) uses supervised contrastive learning to learn the class centers in long-tailed data distributions.
MiSLAS(Zhong et al., 2021) focuses on calibrating the model properly on LT data with a two-stage method that uses CutMix(Yun et al., 2019) to enhance the representation learning stage.
The classiﬁer learning stage consists of labelaware smoothing and shift learning on batch normalization layers.
DiVE(He
et al., 2021) is a distil- lation method where the teacher network is trained by existing well perfroming LT-methods such as
BALMS and then trains student model with KL divergence loss and balanced softmax loss.
A. D ETAILS ON THE E XTENDED LT B ENCHMARK (S EC . 4)
[Table]
the newly added LT datasets from existing computer vision datasets e.g ., Caltech101 (Li et al., 2022),
CUB-200-2011 (Wah et al., 2011), Stanford Dogs
(Khosla et al., 2011), Stanford Cars (Krause et al., 2013), DTD (Cimpoi et al., 2014), FGVC-Aircraft (Maji et al., 2013), Flowers-102 (Nilsback &
Zisserman, 2008)
[Table]
and Fruits-360 (Mures¸an & Oltean, 2017).
We ﬁrst sort the class indices with the number of samples per class to create a class order.
We then set the imbalance ratio and sub-sample the data in each of the classes.
In sub-sampling the data, we use an exponential decaying curve to determine the class sample frequency from head to tail classes.
As the number of data in each of the classes is limited, we often can only use a certain imbalance ratio such as 10 or 100.
We then detail the dataset statistics for all the LT datasets used in our experiments in Tab. 4, including the newly added ones and the previously used datasets in the LT literature.
The added datasets have a wide variety of semantic domains which may have not been covered by the existing LT datasets in the literature.
A. F URTHER I MPLEMENTATION D ETAILS (S EC . 4)
In the calibration stage, we used the class-balanced sampling and label-aware smoothing following
MiSLAS (Zhong et al., 2021).
We use a pretrained ResNet-152 (He et al., 2015) backbone for the
ImageNet1K teacher and a pretrained EfﬁcientNet (Tan & Le, 2019) backbone for the ImageNet21K teacher.
In distillation stage, we used ReActNet (Liu et al., 2020) as our binary backbone network.
Atten φ is a MLP with 3 hidden layers with the number of channels expanding by 16 per layer starting at 2 channels in the ﬁrst layer.
The MLP takes two scalar values L KL ( x ) and L FS ( x ) turned into a 2-dimensional vector and outputs a single scalar value which is passed through a sigmoid function to be used as λ φ .
To process the multi resolution inputs to the teacher network, we use the size interpolation & concat module.
The size interpolation & concat module is comprised of 1x1 convolutions that match the number of channels to that of the binary network and a nearest interpolation step where the different sized feature maps from the multi-resolution inputs are spatialy resized.
Once the feature maps from the different resolution inputs have the same spatial size and number of channels, they are channel-wise concatenated.
To optimize the min-max objective of
Eq. 4, we perform 1 maximization step per every 1 minimization step.
The binary networks are trained for 400 epochs with a batch size of 256.
We use the cosine annealing scheduling for our
[Table]
[Table]
learning rate scheduler, where the initial learning rate is set as 0 .
01 for most of the datasets.
Although we did try different learning rates, we found the difference to be negligible.
We used one GPU for training on the 12 small-scale datasets and four GPUs for training on the 3 large-scale datasets.
A. M ORE D ISCUSSION R EGARDING τ - NORM AND BALMS (S EC . 4.1)
On comparison with existing LT methods on the small-scale datasets in Tab. 1, τ -norm and BALMS seems to show relatively low accuracy.
We hypothesize the following reasons for the low accuracy.
For τ -norm, it is based on how the weight norms distributed when the model is trained with the SGD optimizer.
Since we use the Adam optimizer to train the models due to having binary backbones (Sec. 3.1), the core assumption for τ -norm may not hold and hence the resulting low accuracy.
For
BALMS, it may have lacked the sufﬁcient number of epochs to properly converge as we use binary backbone networks which usually take longer to train than ﬂoating point models (Courbariaux et al., 2015).
A. E FFECT OF U SING D IFFERENT T EACHER N ETWORKS FOR CANDLE (S EC . 4.1)
We used the ImageNet1k pretrained teacher network for small-scale experiments in Tab. 1 as we felt that the teacher network could still provide sufﬁcient supervisory signals for the small-scale datasets.
Nonetheless, we also conduct experiments on the small-scale datasets with the larger teacher network pretrained on ImageNet21k which was originally used only for the large-scale datasets.
As shown in Tab. 5, not all datasets beneﬁt from the larger teacher network such as Caltech (10), CIFAR-10 (10), CIFAR-100 (10), and Stanford Cars (10).
However, there are noticeable gains of +6 .
51% on CUB-200-2011-10 and +4 .
72% on Stanford Dogs-10 and the resulting mean accuracy is improved by +1 .
For the large-scale datasets in Tab. 2, we could not use the
ImageNet1K teacher as that is a direct superset of ImageNet-LT.
The above reason is why we used the ImageNet21K teacher for the large-scale experiments.
However, the gains in Tab. 5 suggest that one could use the ImageNet21K teacher for the small-scale datasets as well.
A. A DDITIONAL C OMPARISON TO TADE (Z HANG ET AL ., 2021 B ) (S EC . 4.1)
For a more comprehensive comparison, we also present results for TADE (Zhang et al., 2021b), which is an ensemble-based method.
As shown in Tab. 6, we show results for TADE on thesmall-scale datasets along with CANDLE.
Our method still outperforms TADE by large margins of +24 .
94% in terms of mean accuracy.
Not only that, the proposed method seems to show higher accuracy both on the frequently used CIFAR-10 (10, 100) and CIFAR-100 (10, 100) datasets but also on the newly added datasets such as CUB-200-2011 (10), Stanford Dogs (10), and Stanford
Cars (10).
A. M ORE D ETAILED A BLATION S TUDY R ESULTS (S EC . 4.3)
We presented the mean accuracy over the 12 small-scale datasets for our ablation studies in Tab.to more clearly present the performance of the ablated models.
We also report the accuracy on each of the 12 datasets in Tab.
7 to provide more details regarding our ablation studies.
However, we also emphasize that looking at the accuracy of a particular dataset only may not give the most accurate description.
Going from 1 (cid:13) to 1 (cid:13) + 2 (cid:13) + 3 (cid:13) (=CANDLE), 9 out of 12 datasets show an accuracy increase.
On the 3 datasets where an accuracy decrease is observed, the accuracy drops by − 5 .
17% at most, whereas on 9 datasets where the accuracy increases, it increases by up to +41 .
Ultimately, the mean accuracy consistently goes up when add more and more parts of CANDLE, as also shown in Tab. 3.
[Table]
CUB-200-2011 (10) 12.
Stanford Dogs (10) 25. Stanford Cars (10) 25.
DTD (10) 21.
FGVC-Aircraft (10) 28.
Flowers-102 (10) 51.
Fruits-360 (100) 99.
Mean Acc.
[Figure]
[Figure]
W e i g h t
N o r m # of Samples
SGD with WD (Acc: 13.24%)
SGD without WD (Acc: 15.46%)
# o f
S a m p l e s
A. M ORE DISCUSSION ON DIFFERENCE IN WEIGHT NORM TREND WITH A DAM OR SGD
As discussed in Sec. 3.1, we use the Adam optimizer for training the binary networks.
This results in a different trend in the classiﬁer weight norms than some prior works (Kang et al., 2019; Alshammari et al., 2022).
To give a more information regarding this matter, we also plot the classiﬁer weight norm trend of training binary networks from scratch on CIFAR100 (100) using the SGD optimizer in Fig.
We see that using the SGD shows similar trends in terms of the classiﬁer weight norms as the prior arts (Kang et al., 2019; Alshammari et al., 2022).
However, the accuracy of the learned binary networks with SGD is very low for both with and without weight decay.
We believe that further developing our analysis on such an under-ﬁtted model will not be meaningful and use Adam optimizer instead.
Note that combined with results in Fig.
2, we see opposite trends in weight norms depending on whether we use Adam or SGD for both binary and FP networks.
With Adam, we have weight norms increasing as the classes become tail, and with SGD, we have weight norms decreasing as the classes become tail instead.
We conjecture that Adam adjusts the learning rates such that parameters corresponding to frequently occurring head class data use relatively small learning rates and parameters corresponding to infrequent tail class data use relatively large learning rates. As a result, the optimization may result in the tail class weights having large weight norms due to the relatively larger learning rate used in the update. Note that more thorough studies on the above phenomenon is warranted to gain a deeper understanding and we leave that as an interesting future direction, respecting the original scope of this work. A. A DDITIONAL I NFORMATION R EGARDING H EAD , M EDIAN , AND T AIL C LASS A CCURACY
To provide more information regarding the experimental results of CANDLE, we provide average, head, medium, and tail class accuracy of CANDLE for the 15 datasets in Tab. 8.
For most of the datasets, CANDLE shows high accuracy at the tail classes compared to the average, head, or the medium classes.
This is especially true for large-scale datasets such as Places-LT, ImageNet-LT, and iNat-LT where the tail class accuracy is either on par or higher than the average accuracy.
Additionally, in Fig.
9, 10, 11, we plot a per-class accuracy improvement of CANDLE over the baseline used in Tab.
3 following Kozerawski et al.
(2020) on Places-LT, ImageNet-LT, and iNat-LT, respectively.
The ﬁgures show that CANDLE improves the accuracy at the tail and medium classes by large margins for all three datasets.
Additionally, in the case of ImagetNet-LT, even the head accuracy shows noticeable improvements.
The ﬁgures empirically show that CANDLE shows good [Table] average accuracy not because it is only good at head classes but mostly because it improves the accuracy of tail and medium classes by large margins.
CANDLE w/ Cosine Cls. CANDLE w/ Linear Cls. Average Head
Medium
Tail
Average
Head
Medium
Tail
[Table]
[Table]
[Table]
CIFAR
[Table]
[Table]
[Table]
CUB
[Table]
[Table]
[Table]
[Table]
DTD
[Table]
FGVC-
[Table]
[Table]
A. R ESULTS WITH C OSINE C LASSIFIERS FOR CANDLE ON S MALL -S CALE D ATASETS
We present results of the proposed method with the cosine classiﬁers instead of the linear classiﬁer in Tab. 9.
As shown in the table, using linear classiﬁers results in better average accuracy than cosine classiﬁers for all datasets except Caltech101 (10).
Not only is the average accuracy higher for linear classiﬁers, but all head, medium, and tail class accuracy show clear advantages of linear classiﬁers over cosine classiﬁers on all 12 datasets.
Taking the mean over the 12 datasets shows a +4 .
25% gain in average accuracy as well as a +10 .
21% gain in tail class accuracy.
This is in-line with our hypothesis in Sec.
1 that cosine classiﬁers may not work as well for binary networks due to the reduced usefulness of weight norms magnitudes in supplementing the limited capacity of binary networks
A. L IMITATIONS .
We mainly focus on the algorithmic aspects of learning binary networks with accurate LT recognition capabilities.
Actual deployment of LT capable binary networks to the real-world requires more application-oriented development on top of the proposed algorithmic improvements and is out of scope of this work.
Additionally, while our usage of multi-resolution inputs incurs little increase in training time, an increase in the VRAM peak is observed.
Reducing the VRAM peak increase would be an interesting future direction for making the algorithm more efﬁcient.