D ELVING INTO S EMANTIC S CALE I MBALANCE
Anonymous authors
Paper under double-blind review
A BSTRACT
Model bias triggered by long-tailed data has been widely studied.
However, measure based on the number of samples cannot explicate three phenomena simultaneously: (1) Given enough data, the classification performance gain is marginal with additional samples.
(2) Classification performance decays precipitously as the number of training samples decreases when there is insufficient data.
(3) Model trained on sample-balanced datasets still has different biases for different classes.
In this work, we define and quantify the semantic scale of classes, which is used to measure the feature diversity of classes.
It is exciting to find experimentally that there is a marginal effect of semantic scale, which perfectly describes the first two phenomena.
Further, the quantitative measurement of semantic scale imbalance is proposed, which can accurately reflect model bias on multiple datasets, even on sample-balanced data, revealing a novel perspective for the study of class imbalance.
Due to the prevalence of semantic scale imbalance, we propose semantic-scalebalanced learning, including a general loss improvement scheme and a dynamic re-weighting training framework that overcomes the challenge of calculating semantic scales in real-time during iterations.
Comprehensive experiments show that dynamic semantic-scale-balanced learning consistently enables the model to perform superiorly on large-scale long-tailed and non-long-tailed natural and medical datasets, which is a good starting point for mitigating the prevalent but unnoticed model bias.
I NTRODUCE
In practical tasks, long-tailed class imbalance is a common problem, and the imbalance in number makes the trained model easily biased towards the dominant head classes and perform poorly on the tail classes (22; 69).
However, what is overlooked is that, in addition to long-tailed data, our study finds that the model trained on sample-balanced data still shows different biases for different classes.
This model bias is not taken into account by the study for class imbalance problem, and it cannot be ameliorated by the current methods proposed for long-tailed data (4; 23; 30; 61).
For natural datasets, classes artificially divided by different semantic concepts correspond to different semantic scales, which can lead to different degrees of optimization when the deep metric is a single scale (47).
In this study, we attempt to uncover more information from the data itself and introduce and quantify the semantic scale imbalance for representing more general model bias.
The semantic-scale-balanced learning is further proposed, which is used to improve loss to mitigate model bias.
The classes corresponding to different semantic concepts have different feature diversity, and we equate the diversity to the semantic scale.
Usually, the finer the semantic concept of a class label, the less rich the feature diversity, the less information a model can extract, and the worse the model performs on that class (10; 47; 6; 66; 8; 64).
The manifold distribution hypothesis (50) states that a specific class of natural data is concentrated on a low-dimensional manifold.
The larger the range of value variation along a specific dimension of the manifold, such as illumination and angle, the richer the feature and the larger the volume of the manifold.
For example, in Figure 1, since "Swan" is a subclass of "Bird", its semantic concept is finer, so the feature diversity of "Bird" is richer than that of "Swan", and the corresponding volume of manifold is larger.
Obviously, the semantic scale can be measured by the volume of manifold.
We present a reliable and numerically stable quantitative measurement of the semantic scale and further define the semantic scale imbalance.
To avoid confusion, we refer to the volume of the manifold calculated on the sample space as the sample volume and the feature space as the feature volume.
In addition, our innovative study for semantic scale imbalance can simultaneously and naturally explain the following two phenomena that cannot be explicated by existing studies of class imbalance: (1) As the number of samples increases linearly, the model performance shows a trend of rapid improvement in the early stages and leveling off in the later stages (53).
(2) Even models trained on the dataset with balanced sample numbers still suffer from class bias.
The classification experiments on samplebalanced datasets show that the models are biased towards the classes with larger semantic scales, such as the decision surface shown by the green line.
In this case, the re-weighting strategy based on the number of samples does NOT work, while our proposed re-weighting approach based on the semantic scale biases the decision surface toward the class with a larger feature volume (red line).
The experiments demonstrate that semantic scale imbalance is more widely present in natural datasets than sample number imbalance, allowing the study scope for class imbalance to be extended to arbitrary datasets.
Then, how to mitigate the adverse effects of semantic scale imbalance?
We are inspired by classification methods for long-tailed data, and current solutions for class imbalance problem usually adopt re-sampling strategies (51; 71; 5) and cost-sensitive learning (21; 65; 32).
However, re-sampling may either introduce a large number of duplicate samples, making the model susceptible to overfitting when oversampling, or discard valuable samples when undersampling.
Therefore by drawing on the classical re-weighting strategy (49), we propose the dynamic semantic-scale-balanced learning.
Its core idea is to dynamically rather than invariably measure the degree of imbalance between semantic scales in the feature space, to achieve a dynamic evaluation of
the weaker classes and assign greater weights to their corresponding losses
In this work, our key contributions are summarized as:
(1) We propose the novel idea of leveraging the volume of manifold to measure the semantic scale (Sec 3.2).
It is also innovative to find that the semantic scale has the marginal effect (Sec 3.3), and that the semantic scale of the dataset is highly consistent with model performance in terms of trends.
(2) We introduce and define semantic scale imbalance, aiming to measure the degree of class imbalance by semantic scale rather than sample number, which reveals a new perspective for the study of class imbalance problem.
Experiments show that semantic scale imbalance is prevalent in the dataset and can more accurately reflect model bias that affects model performance (Sec 3.
(3) Semantic
scale- balanced learning is proposed to mitigate model bias, which includes a general loss improvement scheme (Sec 4.1) and a dynamic re-weighting training framework (Sec 4.2) that overcomes the challenge of calculating semantic scales in real-time during iterations.
Comprehensive experiments demonstrate that semantic-scale-balanced learning is applicable to a variety of datasets and achieves significant performance gains on multiple vision tasks (Sec 5).
S LOW D RIFT P HENOMENON OF F EATURES AND M ARGINAL E FFECT
Since the model parameters are changing during training, (58) studies the drifting speed of embeddings by measuring the difference in features of the same instance across training iterations.
Experiments on the Stanford Online Products (SOP) dataset (37) show that the features change drastically in the early stage of training, become relatively stable after traversing the dataset twice, and drift gets extremely slowly when the learning rate decreases.
This ensures that it is reasonable to leverage historical features to calculate semantic scales.
Marginal effect (10) describes that in the early stages of model training, the network is able to learn features quickly, but since there is information overlap among samples, as the number of samples increases, the information of data will gradually saturate and the improvement in model performance from newly added samples will diminish.
The effective number of samples is proposed to represent the information of data, but its limitation is that it does not work when the number of samples for each class is balanced.
Assume that the volume of each sample is unit volume 1 and define the set of all samples for a class as Ω with volume N and N ≥ 1 .
A new sample may overlap with a previous sample such that the probability of overlap is P and that of non-overlap is 1 − P .
As the information of data increases, the probability P will be higher.
Define the effective number of samples as E n , and E n = 1 + β 1 − β n − 1 1 − β = 1 − β n 1 − β (10), where n denotes the number of samples, hyperparameter β = N − 1 N ∈ [0 , 1) controls how fast E n grows as n increases.
When N
= 1 , β = 0 and E n = 1 , meaning that all samples can be represented by a single prototype via data augmentation.
When N → ∞ , β → 1 , implying that there is no overlapping, then
[Equation]
= n , which indicates that the effective number of samples does not increase faster than the number of samples, but this is not the case in our experimental results.
The effective number of samples E n is an exponential function of the number of samples n .
The hyperparameters β corresponding to classes of different grain should be different.
However, the selection of β requires more information from the data itself, but this problem is not addressed by (10), which is forced to assume that β is the same for all classes.
In this case, compared to the number of samples, E n simply uses the exponential function to obtain smoother weights.
Furthermore, when the number of samples is balanced, E n is the same for each class and cannot be used to mitigate model bias, so we attempt to mine the data for information (or feature diversity) of each class to facilitate the study of imbalance problem.
S EMANTIC S CALE I MBALANCE
In this section, first, sample volume, feature volume, and semantic scale imbalance are defined.
Next, we derive a quantitative measurement of feature volume to measure the semantic scale from the perspective of singular value decomposition of the data matrix and information theory.
Then, the marginal effect of semantic scale is investigated.
Finally, we discuss the relationship between semantic scale imbalance and model bias.
3. D EFINITIONS
Different semantic concepts correspond to different semantic scales, for example, the scale of "Bird" is larger than that of "Swan".
For each class, we equate its feature diversity to its semantic scale and measure the semantic scale by the volume of subspace spanned by samples or features.
Deep neural networks can be viewed as a combination of a feature mapping function f ( x, θ ) and a trained downstream classifier g ( z ) , i.e., x → z ( θ ) →
y .
Let the samples of a class be X
=
[ x 1 , x 2 , . . .
, x m ] , and the embeddings learned by deep neural networks are represented as Z = (cid:8)
z i | z i = f
( x i , θ ) ∈
R
d , i = 1 , 2 , . . .
, m (cid:9) .
Definition 3.1.
(Sample volume)
The volume of the subspace spanned by sample set X .
Definition 3.2.
(Feature volume)
The volume of the subspace spanned by feature vectors Z .
Definition 3.3.
(Semantic scale imbalance)
A phenomenon of imbalance in the size of semantic scales measured by sample volume or feature volume.
3. Q UANTIFICATION OF S EMANTIC S CALE
Given the data X
=
[ x 1 , x 2 , . . .
, x m ] and the learned embeddings Z = [ z 1 , z 2 , . . .
z m ] ∈ R d × m , z i = f
( x i , θ ) ∈
R
d , i = 1 , 2 , . . .
, m , the volume of subspace spanned by the random vector z i (i.e., the feature volume) is derived below, and the sample volume can be calculated in the same way (Appendix E).
The covariance matrix of random vector z i is estimated as Σ = E
[Equation]
are the eigenvalues of real symmetric matrix.
The singular value decomposition (SVD) of Z yields Z = U Σ V T and the singular values are σ j = (cid:112) λ j , j = 1 , 2 , . . .
, d .
The volume of the space spanned by the vector z i is proportional to the product of all singular values of the feature matrix
Z , i.e.,
Vol ( Z ) ∝ d (cid:81) j
=1 σ j = (cid:115) d (cid:81) j
=1 λ j .
After the determinant expansion, the characteristic polynomial of the matrix Σ is Φ( λ ) = det( λI − Σ)
=
λ d − ( a 11 + a 22 + · · · a dd ) λ
d − 1 + · · · + ( − 1) d detΣ , and λ 1 λ 2 · · · λ d = detΣ .
Therefore, the volume of the space spanned by the vector z i is proportional to the square root of the determinant of the covariance matrix of Z :
[Equation]
The same result can be derived from the volume of a parallel hexahedron defined by vectors (Appendix
G).
Considering that real-world metric tools typically have a dynamic range, for example, a ruler always has multiple scales (1mm, 1cm, or even 10cm) to measure objects of different scales.
We expect the quantitative measurement of feature volume to have a multi-scale metric and therefore use the sphere packing method (7; 33), which is normally adopted in information theory, to implement it.
The feature volume is positively correlated with the number of spheres, and since the sphere packing has error at the subspace boundary, the error of the finite feature vectors is assumed to be independent additive Gaussian noise:
z i ′ =
z i + w i , where w i ∼ N (0 , ε 2 n I ) .
The estimate of the number of spheres of radius ε needed to pack the space spanned by all vectors is N ε =
Vol ( Z ′ )/
Vol ( w ) .
The adjustment of the metric scale can be achieved by tuning the radius ε of the spheres, thus controlling the measurement result of feature volume.
Then the covariance matrix of the vector z i is Σ ′ =
ε 2 n I + 1 m ZZ T ∈
R
d × d , such that Vol ( Z ′ )
∝ (cid:113) det( ε 2 n I + 1 m ZZ T ) and Vol ( w ) ∝ (cid:113) det( ε 2 n I ) .
The feature volume is proportional to N ε :
[Equation]
The dimension of feature z i is d , so let n = d .
In order to increase the numerical stability, we perform a logarithmic transformation of the above equation, which does not affect the monotonicity of the function, and we can obtain Vol ( Z ) ∝ log 2 (cid:113) det (cid:0) I + nmε 2 ZZ T (cid:1)
= 12 log det (cid:0)
I + dmε 2 ZZ T (cid:1) .
In practical training, it is essential to normalize the feature vectors so that their mean value is 0.
In this work, we set ε = 1000 , and the value of ε does not affect the relative size of the space spanned by feature vectors of each class.
The volume of the space spanned by Z can be can be written as:
[Equation]
where Z mean is the mean value of Z , Vol ( Z ) > 0 when the number of samples m > 1 .
We measure the semantic scale S ′ by the feature volume, i.e., S ′ =
Vol ( Z ) , and the larger S ′ , the richer the feature diversity, which we verify in Appendix C using multiple Stanford point cloud manifolds.
3. M ARGINAL E FFECT OF S EMANTIC S CALE
The marginal effect describes that the feature richness will gradually saturate as the number of samples increases, so the change of semantic scale should also conform to the marginal effect.
Figure 2 illustrates that as the number of samples increases, the semantic scale S ′ measured by sample volume gradually saturates, which indicates that the quantitative measurement of semantic scale is reasonable.
In addition, the growth rate of the semantic scale varies across classes, which is determined by the grain size of the class itself.
It leads to different semantic scales even if all classes have the same number of samples.
To investigate why adding samples has a marginal improvement in model performance when the training samples are sufficient, we use the following method to generate new training datasets for classification experiments: assume that the total number of classes in the original dataset is C , and m samples are randomly selected from each class to form a sub-dataset with a total number of samples of C × m .
The sub-datasets generated based on CIFAR-10, CIFAR-100 (26) and Mini-ImageNet (54) are shown in Appendix D.1.
We train ResNet-18 and ResNet-34 (14) on each of the 31 training sets in Table 6, and the sum of the semantic scales for all classes and the corresponding top-1 accuracy are shown in Figure 2.
We are pleasantly surprised to find that when the semantic scale increases rapidly, the model performance improves swiftly with it, and when the semantic scale becomes saturated, the improvement is small.
3. S EMANTIC S CALE I MBALANCE AND M ODEL B IAS 3.4. Q UANTIFICATION OF S EMANTIC S CALE I MBALANCE
The previous subsection shows that the sum of semantic scales for all classes in the dataset is highly correlated with the model performance we further investigate the relationship between semantic scale and model bias for different classes.
When a class is closer to other classes, the model performs worse on that class (36; 1).
Therefore, inter-class interference is additionally considered when quantifying the degree of imbalance between semantic scales of classes.
When class i is closer to other classes, a smaller weight w i is applied to the semantic scale of class i .
Specifically, the semantic scale of m classes after maximum normalization is assumed to be
S ′ =
[ S ′ 1 , S ′ 2 , . . .
, S ′ m ] T , and the centers of all classes are C
=
[ c 1 , c 2 , . .
, c m ] T .
Define the distance between the centers of class i and class j as d i,j = ∥
c i − c j ∥ 2 , the weight w i = 1 m − 1 (cid:80) mj
=1 ∥ c i − c j ∥ 2 .
The weights of m classes are written as W ′ =
[ w 1 , w 2 , . . .
, w m ] T .
After the maximum normalization and logarithmic transformation of W ′ , we can obtain
W =log 2 ( α + W ′ ) , α ≥ 1 , where α is used to control the smoothing degree of W .
After considering the inter-class distance, the semantic scale
S = S
′ ⊙ W , and the role of S ′ in dominating the degree of imbalance is greater when α is larger.
To obtain the most appropriate α , we calculate the Pearson correlation coefficients between the semantic scale and the accuracy of ResNet-18 and ResNet-34 trained on CIFAR-10-LT and CIFAR10, as shown in Table 1.
The experimental settings are in Appendix D.2.
It can be found that S ′ is more dominant on long-tailed data than on non-long-tailed data, and the improved S is better than S ′ and far better than the number of samples in reflecting model bias.
In the following experiments, we let α be 2 on the long-tailed data and 1 on the non-long-tailed data.
[Table]
3.4. S EMANTIC S CALE I MBALANCE ON L ONG -T AILED D ATA
Previous studies have roughly attributed model bias to the imbalance in the number of samples.
The experimental results in the first row of Figure 3 show that even though the number of MNIST-LT-1 is similar to that of MNIST-LT-2, the class-wise accuracy on MNIST-LT-1 is closer to that on MNIST, just as their semantic scales are also more similar.
In addition, Figure 3 indicate that models on certain classes with fewer samples outperform those on classes with more samples and that the semantic scale S reflects model bias more accurately.
Further, we also observe that the accuracy of the CIFAR-100LT does not show a significant decreasing trend, which can be explained by the marginal effect of the semantic scale (Sec 3.3).
3.4. S EMANTIC S CALE I MBALANCE ON N ON -L ONG -T AILED D ATA
Figure 4 demonstrates the model bias not only on long-tailed data but also on sample-balanced data
Usually, the classes with smaller semantic scales have lower accuracies.
Depending on the size of the semantic scale, it can make it possible for the weaker and dominant classes to be well differentiated.
It should be noted that the weaker classes are not random, and experiments in Figure 4 show that the models always perform worse on the same classes.
More semantic scale imbalance of the datasets is shown in Appendix D.4.
In summary, semantic scale imbalance can represent model bias more generally and appropriately, and further, we expect to improve the overall performance of the model when facing the semantic scale imbalance problem.
Therefore, we propose the dynamic semantic-scale-balanced learning by drawing on the re-weighting strategy.
D YNAMIC S EMANTIC -S CALE -B ALANCED L EARNING
Deep neural networks can be viewed as a combination of the feature mapping function and the classifier, and several studies have shown that model bias is mainly caused by classifier bias (71; 59; 2), so we are more concerned with semantic scale imbalance in the feature space, i.e., semantic scale measured by the feature volume .
In this section, we propose a general semantic-scale-based loss improvement scheme and design a training framework for the successful application of the scheme.
4. D YNAMIC S EMANTIC -S CALE -B ALANCED L OSS
During training, the feature vectors corresponding to the samples vary with the model parameters, and thus the semantic scale per class is constantly changing.
Compared with the traditional re-weighting strategy, we propose to calculate the degree of imbalance between semantic scales in real-time at each iteration in order to dynamically evaluate the weaker classes and assign greater weights to their corresponding losses.
Specifically, for class i at each iteration, normalized re-weighting terms α i ∝ 1 S i , C (cid:80) i =
1 α i = 1 , inversely proportional to the semantic scales that take into account inter-class interference, are introduced, and C is the total number of classes.
Given the embedding z of a sample and label y
i , the dynamic semantic-scale-balanced ( DSB ) loss can be expressed as
DSB ( z, y i ) =
1 S i L ( z, y i ) , i = 1 , 2 , . . .
, C, where y i is the label of the sample from class i .
How to combine general loss to generate DSB loss is described in Appendix F.1.
It should be noted that all methods that re-balance the loss function or adjust the sampling rate based on the sample number can be improved with the semantic scale, since both the semantic scale and the sample number are natural measures and they are not model-dependent.
However, the number of samples used at each iteration is limited, and it is not possible to obtain the features of all samples for calculating the semantic scales.
Therefore, we propose a dynamic re-weighting training framework that enables DSB loss to be successfully applied.
4. D YNAMIC R E -W EIGHTING T RAINING F RAMEWORK
Inspired by the slow drift phenomenon of features (58; 31), we design a storage pool Q to store and update historical features and propose a three-stage training framework.
A mini-batch of features can be dynamically updated at each iteration, and the semantic scale of each class is calculated using all the features in the storage pool.
The three-stage training framework is shown in Figure 12 and
Algorithm 2 (More details are in Appendix F.2), with the following textual description
In the first stage , all the features and labels generated by the 1st epoch are stored in Q , but they cannot be used directly to calculate semantic scales due to the large drift of historical features from current features in the early stage.
(2) The second stage corresponds to epoch 2 to epoch n .
At each iteration, the oldest mini-batch features and labels in Q are removed and those generated by the current iteration are stored.
The goal is to continuously update the features in Q until the feature drift is small enough.
We set n to 5 in our experiments, and the original loss function is used in the first two stages.
Figure 5 shows the effect of n on the model performance.
A larger n does not hurt the model performance, but only takes a little more time.
Experience suggests that setting n to 5 is sufficient.
(3) The third stage corresponds to epoch > n .
At each iteration, the semantic scales are calculated using the features in Q after updating Q , and the original loss is re-weighted.
The comparison and analysis of the video memory and training speed are in Appendix F.2.
We answer possible questions about the methods section in detail in Appendix B.
E XPERIMENTS
To validate the superiority and generality of the proposed dynamic semantic-scale-balanced learning, we design three experiments.
The first experiment is conducted on large-scale long-tailed datasets,
ImageNet-LT and iNaturalist2018 (52), to confirm the superior performance of our approach on long-tailed data.
The second experiment uses large-scale ImageNet (41) and sample-balanced
CIFAR-100, and the third experiment selects CIFAR-100-LT and benchmark datasets commonly used in deep metric learning (CUB-2011 (55) and Cars196 (25)).
The latter two experiments demonstrate the generality of our approach on non-long-tailed data.
More experimental results are provided in Appendix D.5 and Appendix D.6 (Results on the fundus dataset OIA-ODIR (27)).
5. R ESULTS ON I MAGE N ET -LT AND I N ATURALIST
ImageNet-LT is a long-tailed version of ImageNet containing 1,000 classes with between 1,280 andsamples per class.
iNaturalist2018 is a real-world, extremely unbalanced dataset containing 437, images from 8,142 classes.
For fair comparisons, we adopt the official training and validation splits.
Table 2 shows that when CE, Focal (29) and RIDE are combined with our approach (Appendix
D.1)
, the model overall performance is significantly improved.
For example, the overall accuracy of
[Table]
on ImageNet-LT and iNaturalist2018, respectively.
We also report the performance on three subsets (Head: more than 100 images, Middle: 20-100 images,
Tail: less than 20 images) of these two datasets.
It can be observed that our proposed method has the largest improvement for the tail subset without compromising the performance of the head subset, where DSB-CE and DSB-Focal improve 13.7% and 10.6% , respectively, over the original method in the tail subset of ImageNet-LT, effectively alleviating the model bias.
5. R ESULTS ON I MAGE N ET AND CIFAR-
[Table]
We use the ILSVRC2012 split contains 1,281,167 training and 50,000 validation images.
Each class of CIFAR-100 contains 500 images for training and 100 images for testing.
The results in Table 3 indicate that our approach is able to achieve performance gains greater than 1% for a variety of networks on both datasets.
In particular, it enables VGG16 to improve 1.3% and 1.5% on ImageNet and CIFAR-100, respectively, compared to the original method.
This implies that there is a semantic scale imbalance in non-long-tailed datasets and it affects the model performance.
5. R ESULTS ON CUB-2011, C ARS 196 AND CIFAR-100-LT
Since we also improve on the classical losses (NormSoftmax and SoftTriple (38)) in the field of deep metric learning
[Table]
we abide by the widely adopted backbone network, experimental parameters
[Table]
and the division of datasets in this field (Appendix D.5).
The two improved loss functions are denoted as
DSB-NSM and DSB-ST, respectively, and their formulas are given in Appendix F.1.
[Table]
Results on CUB-2011 and Cars196 .
Table 4 summarizes the performance of our method with 64 and 512 embeddings, respectively.
The experiments show that DSB loss is able to consistently improve by more than 1% on R1 and NMI.
DSB-ST with 512 embeddings performs superiorly on
Cars196, where R@1 and R@2 exceed the Circle loss (46) by 2.2% and 1.5% , respectively
[Table] [Table]
Results on CIFAR-100-LT .
Class-balanced loss (CB loss) that performs well on long-tailed data and is also based on the re-weighting strategy is selected for comparison with DSB loss.
Analyzing the results in Table 5, the DSB loss outperforms the CB loss overall.
Among them, when the imbalance factor of long-tailed CIFAR-100 is 200, DSB-ST performs significantly better than CB-ST, with higher performance than SoftTriple on R@1, R@2 and NMI by 2.7% , 2.8% and 1.9% .
Experiment Summary .
Extensive experiments confirm that dynamic semantic-scale-balanced learning has superior performance not only on long-tailed datasets, but also on non-long-tailed datasets, and even on sample-balanced datasets.
This also means that the semantic scale imbalance needs to be paid extensive attention.
D ISCUSSION
In this work, we pioneer the concept and quantitative measurement of semantic scale imbalance, and make two important discoveries: (1) semantic scale has marginal effects, and (2) semantic scale imbalance can accurately describe model bias.
It is important to note that our proposed semantic scale, like the number of samples, is a natural measure of class imbalance and does not depend on the model’s predictions (See Related Work in Appendix A).
Semantic scale can guide data augmentation, e.g., semantic scale imbalance can evaluate which classes are the weaker classes that need to be augmented, and marginal effects can assist us to select a more appropriate number of samples.
We expect that our work will bring more attention to the more prevalent model bias, improve the robustness of models and promote the development of fairer AI.
A R ELATED W ORK
Real-world datasets tend to long-tailed.
The extreme imbalance in the number of samples for longtailed data prevents the classification model from learning the distribution of the tail classes adequately, which leads to poor performance on the tail classes.
Therefore, the methods of re-balancing the number of samples (23; 13; 12; 70) and balancing the loss incurred per class (11; 72; 45), i.e., re-sampling and cost-sensitive learning, are proposed.
Among them cost-sensitive learning is most relevant to our work.
(40) proposes to use the frequency of labels to adjust the loss during training to mitigate class bias.
(29) assigns weights to the loss for each class, and the hard samples are given higher weights.
Recent studies have shown that re-weighting the loss strictly by the inverse of the number of samples is moderate (34; 35).
Some methods that generate weights for re-weighting in a more "smooth" manner perform better, such as taking the square root of the number of samples (34) as weights.
CB loss (10) attributes the better performance of this more "smooth" approach to the presence of marginal effects, while other studies (49; 17; 56) attribute it to the negative gradient over-suppression.
Distributionbalanced loss (60) proposes negative-tolerant regularization to mitigate gradient differences, and recalculates the loss by calculating the ratio of the expected to the actual sampling frequency for each class.
Seesaw Loss (56) leverages mitigation and compensation factors to dynamically suppress excessive negative sample gradients on the tail classes while complementing the penalty for misclassified samples.
Furthermore, with the proposal of decoupled training (71), (23) adopts decoupled training, using cross-entropy loss to learn features in the first stage and re-balancing loss to learn the classifier in the second stage.
Unlike the above studies of re-balancing loss, which all re-balance loss based on the number of samples or the ratio of positive and negative gradients, our work proposes a novel measure, called semantic scale.
Compared to the number of samples, the semantic scale also considers the sample distribution scope.
In contrast to gradient-based measures, the semantic scale does not depend on the model output and gradient back-propagation, and is a natural measure similar to the number of samples.
Work on model robustness has focused on the out-of-domain generalization performance of models, an area known as "out-of-distribution generalization of models".
For example, (44) aims to maintain the good performance of the model when the test distribution deviates from the training distribution.
Similarly, (28) aims to allow the model to learn more information outside the domain.
Unlike them, we are concerned with the problem that model bias introduced by unbalanced data makes the model perform poorly in certain classes.
B E XPLANATION OF A FEW KEY POINTS B. H OW DOES THE SECTION "S LOW DRIFT PHENOMENON AND MARGINAL EFFECTS OF CHARACTERISTICS " RELATE TO THE REST OF THE PAPER ? (1) Association with effective number of samples
The effective number of samples discusses the relationship between the sample number and feature diversity in a class, and it argues that feature diversity has marginal effects.
However, the effective number of samples has many major drawbacks (introduced in Section 2), such as it does not work on sample-balanced datasets and does not give a quantitative measure of feature diversity.
We therefore propose the semantic scale to measure feature diversity, which perfectly fits the marginal effects, and is the first quantitative measure of feature diversity.
In addition, our approach simultaneously explains three phenomena that cannot be explicated by other methods, which indicates the reliability of our proposed method.
(2) Association with feature slow drift
Since experiments show a very high correlation between semantic scale and model bias, we propose to re-weight the loss function with the inverse of the semantic scale.
The features are dynamically changing during training, and all the feature vectors are needed to calculate the semantic scale of each class.
Obviously, the data in one batch is not enough, so we propose to dynamically update and store the historical features to calculate the semantic scale, and the feature slow drift phenomenon ensures the feasibility of this operation.
Section 2 is indispensable for the whole paper, it ensures the coherence of the paper.
B. W HY FOCUS ON THE RELATIONSHIP BETWEEN SEMANTIC SCALE AND ACCURACY ?
It is important to note that we are concerned with the model bias introduced by unbalanced data, which causes models to perform poorly on some classes.
In the past, researchers believed that models performed poorly on classes with fewer samples and therefore defined classes with fewer samples as tail classes and classes with more samples as head classes, proposing a long-tailed identification task.
However, we observe that the model does not necessarily perform poorly on classes with fewer samples, which explains why some of the tail classes are "overbalanced" in many long-tailed identification methods.
The higher similarity between our proposed semantic scale and model performance allows us to redefine the imbalance problem by replacing the number of samples with semantic scale.
The superior performance achieved on the sample-balanced datasets shows that our proposed semantic scale imbalance is reliable.
The semantic scale measure does not depend on the model and is calculated directly from the data.
Even more surprising is that the semantic scale of the class can predict the performance of the class, which can lead to further understanding of what the model learns from the data.
It can facilitate the development of data-driven artificial intelligence.
B. W HY SHOULD THE LOSS FUNCTION BE DYNAMICALLY WEIGHTED ?
The working process of DSB loss: in each iteration, the semantic scale of each class is calculated in the feature space, and the loss function is re-weighted by the inverse of the semantic scale.
Why is the loss "dynamically" weighted?
Because the semantic scales change continuously as the features change during training, and we need to update the features in each iteration and re-calculate the semantic scales to re-weight the loss function.
The term "dynamic" refers to the dynamic update of the semantic scale in each iteration.
However, there is difficulty in implementing dynamic weighting, i.e., all features are needed to calculate the semantic scale, and we cannot extract the features of all samples in each iteration, which would be time consuming.
Therefore, we analyze the "feature slow drift" phenomenon in Section 2 and propose to calculate the semantic scale by dynamically storing and updating the historical features (i.e., dynamic re-weighting training framework).
Comparative experiments on the training speed and memory consumption of the above training framework are presented in Appendix F.2, and the results show that our approach is efficient.
C E XPERIMENTS ON S TANFORD POINT CLOUD MANIFOLDS
Since ( Z − Z mean )( Z − Z mean ) T is a real symmetric matrix, it is semi-positive definite.
Further,
I + pm
( Z − Z mean )
( Z − Z mean ) T is a positive definite matrix and therefore det ( I + pm
( Z −
Z mean )( Z − Z mean ) T ) > 0 .
The semantic scale measure is derived from the singular value decomposition of the data matrix, which is jointly determined by most of the samples.
Therefore, our method is insensitive to noisy samples, i.e., the semantic scale measure is numerically stable.
The semantic scales of multiple Stanford point cloud manifolds with different sizes are calculated and plotted in Figure 6.
Let the center point of bunny be C bunny .
We increase the volume of bunny by performing w ∗ ( bunny − C bunny ) , and the other point clouds are scaled up in this manner.
As the object manifold is scaled up, the calculated volume then increases slowly and monotonically, indicating that our method can accurately measure the relative size of the manifold volume and is numerically stable, an advantage that will help mitigate the effects of noisy samples.
D E XPERIMENTAL D ETAILS D. M ARGINAL E FFECT OF S EMANTIC S CALE
We use the following method to generate new training datasets for classification experiments: assume that the total number of classes in the original dataset is C , and m samples are randomly selected from each class to form a sub-dataset with a total number of samples of C × m .
The details of the sub-datasets generated based on CIFAR-10, CIFAR-100 and Mini-ImageNet are in Table 6.
[Figure]
D. Q UANTIFICATION OF S EMANTIC S CALE I MBALANCE
We train ResNet-18 and ResNet-34 on CIFAR-10-LT and CIFAR-10 with an imbalance factor of 200, respectively
[Figure]
and the test set of CIFAR-10-LT is consistent with CIFAR-10.
During training, the batch size is fixed to
[Figure]
and the optimizer adopts Adam.
The learning rate is initially 0.01 and becomes 0.98 × the previous learning rate after each epoch.
We do not employ other additional tricks and data augmentation strategies
[Figure]
D. S EMANTIC S CALE I MBALANCE ON L ONG -T AILED D ATA
We artificially produce two long-tailed versions of the MNIST dataset, called MNIST-LT-1 and
MNIST-LT-2.
The number of samples per class is listed in Table
[Table] [Table]
Figure 3 shows the class-wise accuracies of ResNet-18 and ResNet-34 trained on CIFAR-10-LT and
CIFAR-100-LT with the same training settings as in Appendix D.2.
Taking CIFAR-10 as an example, labels 1 to 10 correspond to: airplane , automobile , bird , cat , deer , dog , frog , horse , ship , truck .
The prediction scores of classification experiments on CIFAR-10-LT and CIFAR-10 find that cat (label 4) is most easily confused with dog (label 6), as shown by their lowest accuracy on CIFAR-10 (Figure 4).
However, the accuracy of cat and dog on CIFAR-10-LT is higher than that of deer (label 5), which is due to the dominant role of semantic scale S ′ in S for long-tailed data.
D. S EMANTIC S CALE I MBALANCE FOR M ORE D ATASETS
We have demonstrated the semantic scale imbalance on MNIST, MNIST-LT, CIFAR-10, CIFAR10-LT, CIFAR-100 and CIFAR-100-LT in Section 3.4.
Figure 7 additionally shows the degree of semantic scale imbalance on CUB-2011, Cars196, and Mini-ImageNet, indicating that the semantic scale imbalance is indeed prevalent in all kinds of datasets.
[Figure]
D
E XPERIMENTAL S ETTINGS FOR S ECTION
.3 AND
ORE
XPERIMENTS
E XPERIMENTAL S ETTINGS FOR S ECTION
Backbone Network and Experimental Parameters .
Since we improve on the classical loss in the field of deep metric learning, we abide by the widely adopted backbone network, experimental parameters, and the division of datasets in this field.
The BN-Inception (20; 48) pre-trained on
ImageNet is adopted as the backbone network, and the training set is augmented by using random horizontal flipping and random cropping.
All images are cropped to 224 × 224 as the input of the network.
The output of the network after global average pooling is fed into a single fully connected layer to obtain 64- or 512-dimensional feature embeddings, and then all embeddings are clustered by
K-means.
The model is optimized by Adam with the batch size as 32 and the number of epochs as 50.
We evaluate the performance of the learned embeddings with Recall@K and Normalized Mutual
Information (NMI).
The remaining experimental parameters used in the training are consistent with those reported in NormSoftmax and SoftTriple (38).
Dataset Introduction (CUB-2011, Cars196 and CIFAR-100-LT) .
The CUB-2011 dataset has 5, images in the first 100 classes for training and 5,924 images in the second 100 classes for testing.
The
Cars196 dataset consists of 196 classes totaling 16,185 images, with the first 98 classes for training and the remaining classes for testing.
The CIFAR-100 has 100 classes, each containing 600 images.
We create three long-tailed CIFAR-100 with the first 60 classes for training (See Figure 8a) and test on the remaining classes.
D.5. M ORE E XPERIMENTS
The long-tailed Cars196 is created using the first 98 classes for training (See Figure 8b) and the test set is the remaining classes.
Both Mini-ImageNet and CIFAR-100 datasets contain 100 classes, each containing 600 samples.
For the Mini-ImageNet dataset, the first 64 classes are the training set and the last 36 classes are the test set, and for the CIFAR-100 dataset, the first 60 classes are the training set and the remaining classes are the test set.
Note that the experiments on CIFAR-100 in this section are different from the classification experiments on CIFAR-100 in Sec 5.3.
The purpose of this experiments is to complement the effectiveness of our proposed method on both long-tailed and sample-balanced datasets for the field of deep metric learning.
[Table]
Table 8 shows the performance comparison of DSB-NSM and DSB-ST on the long-tailed Car196.
When the imbalance factor is 10, DSB-NSM outperforms NSM (NormSoftmax) by 3.1% on R@ and DSB-ST outperforms ST (SoftTriple) by 2.1% .
When the imbalance factor is 50, DSB-NSM and DSB-ST improve 2.8% and 2.5% , respectively, on R@1 compared to the original method.
In addition, DSB loss performs better than CB loss on all metrics.
Table 9 shows that compared with the original losses, DSB-NSM and DSB-ST are able to consistently improve R@1 by 1.3% on average and NMI by 1-2% for both sample-balanced datasets, with all other metrics outperforming the original.
The results in Tables 8 and 9 further confirm that our proposed dynamic semantic-scale-balanced learning is applicable to long-tailed and sample-balanced datasets in the field of deep metric learning, and has broad application prospects.
[Table]
D. R ESULTS ON THE FUNDUS DATASETS OIA-ODIR AND OIA-ODIR-B
Dataset Introduction .
The OIA-ODIR dataset (27) was made public in 2019, and it contains a total of 10,000 fundus images in 8 classes.
The eight classes are: Normal(N), hypertensive retinopathy(D), glaucoma(G), cataract(C), agerelated macular degeneration(A) , hypertension complication (H), pathologic myopia (M), other disease / abnormality(O).
Considering that O usually appears together with other diseases, to reduce ambiguity, we adopt the data splitting scheme of (63), using only the data of the first 7 classes, and the number of training samples and test samples for each class is shown in Figure 9.
The OIA-ODIR dataset suffers from an unbalanced number of samples.
To fully validate our method, we produced a balanced version of the OIA-ODIR dataset, OIA-ODIR-B, by using the class with the least number of samples as the benchmark.
As shown in Figure 9, each class of OIA-ODIR- contains 103 training samples and 46 test samples.
In addition to the number of samples, we plot the degree of semantic scale imbalance for the training sets of OIA-ODIR and OIA-ODIR-BS in Figure 9.
Backbone Network and Experimental Parameters .
We used ResNet-50, pre-trained on ImageNet, as the backbone network.
An adam optimizer with a learning rate of 0.1 (linear decay), a momentum of 0.9, and a weight decay factor of 0.005 was adopted to train all networks.
In keeping with (63), average precision (AP) was used as the performance metric of the model.
Results on OIA-ODIR .
We improved the advanced class rebalancing method (BS (39), Focal loss (10), LDAM (3)) and the classification results are plotted in Figure 10.
The experimental findings are summarized as follows.
• Although the sample from class H is the smallest, all methods outperform on class H than on class C, class M, and class A.
This again shows that the number of samples is not the best measure of class imbalance.
• Methods based on sample numbers usually result in larger boosts for the classes with the smallest sample numbers and thus fail to give more attention to class C, class M, and
class A.
Our method has the most significant boosts for these three classes, indicating that semantic scale imbalance can more accurately reflect the difficulty of the classes.
Results on OIA-ODIR-B .
Since the class rebalancing method based on the number of samples cannot be applied to the dataset with a balanced number of samples, we additionally adopted
VGG16, ResNet18 and SE-ResNet50 as the backbone network to test the effect of DSB-CE on CE enhancement, and the experimental results are shown in Figure 11.
The experimental findings are summarized as follows.
• With a balanced number of samples, the model still performs poorly on class C, class M and class A. Figure 9 shows that the semantic scales of these three classes are significantly smaller than the other classes.
• Our approach results in significant performance gains for all models on class C, class M, and class A, and promotes more balanced model performance on all classes, which is important in medical AI.
E P SEUDO C ODE FOR S AMPLE V OLUME
We provide the pseudo code for the calculation of the sample volume in Algorithm 1.
In this work, we resize the image to (16, 16, 3) and then calculate the sample volume after flattening.
Algorithm Calculation
Input:
Onput: Sample volumes for all classes for j = 1 to C do Select the sample set D j = { ( x i , y i ) m j for class j from D , m is the number of samples for j
Resize the image to ( imagesize , imagesize , 3)
Flatten the image into a vector of length d = imagesize × imagesize × 3 and store it in
Z j = (cid:2) z 1 , z 2 , . .
z m j (cid:3) ∈ R d
× m j
Z j = Z j − NumPy.mean
( Z j , 1)
Calculate the covariance matrix Σ j = 1 m j Z j Z Tj
Calculate the sample volume Vol (Σ j ) = 12 log 2 det ( I + d Σ j ) for class j end for
F D YNAMIC S EMANTIC -S CALE -B ALANCED L EARNING F. DSB-NSM, DSB-ST AND DSB-F OCAL L OSS
Given the embedding z of a sample and label y
i , the dynamic semantic-scale-balanced (DSB) loss can be expressed as:
[Equation]
) where y i is the label of the sample from class i .
To show how to combine the general loss to generate the dynamic semantic-scale-balanced loss, we improve the NormSoftmax (NSM) crossentropy loss and SoftTriple (ST) loss.
NormSoftmax removes the bias term in the last linear layer and an L2 normalization module is added to the inputs and weights before the SoftMax loss.
[ w 1 , w 2 , · · · , w C ] ∈ R d × C is the last fully connected layer, then the DSB-NSM with temperature σ generated by embedding z can be written as:
[Equation]
The SoftTriple loss combined with the semantic-scale-balanced term is expressed as
[Equation]
where λ is a scaling factor and δ is a hyperparameter.
The relaxed similarity between embedding z and class c is defined as D
[Equation]
, where k is the number of centers for each class.
The purpose of Focal loss is to apply small loss weights to samples with high classification confidence, thus increasing the proportion of loss of hard samples with low classification confidence to the overall loss.
The α -balanced variant of Focal loss regulates the proportion of loss among samples while assigning different weights to each class, which is denoted as FL ( p t )
=
− α t (1 − p t ) γ log ( p t ) , where p t is the probability that the sample belongs to the true class.
When α t = 1 S
i , Focal loss is transformed into DSB-Focal loss.
F. D YNAMIC R E -W EIGHTING T RAINING F RAMEWORK
Given the training samples X
= [ x 1 , x 2 , . . .
, x N ] containing C classes and corresponding labels Y =
[ y 1 , y 2 , . . .
, y N ] , the number of samples per class is N i
( i = 1 , 2 . . .
, C ) , and the total number of samples is N .
The d-dimensional features extracted by the CNNs are denoted as
Z =
[ z 1 , z 2 , . . .
, z N ] ∈ R d × N .
In this work, we conduct experiments for two types of tasks, image classification and deep metric learning.
In the field of deep metric learning, 64-dimensional features are generally adopted, while the features extracted by the network in image classification tasks tend to be of high dimensionality.
For example, the feature dimension extracted by ResNet-50 is 2048, which will occupy more video memory.
Therefore, when saving historical features in the classification task, one-dimensional average pooling is performed on all features to reduce the feature dimension to 64, which is consistent with the common feature dimension in the field of deep metric learning while preserving the geometry of the distribution (because the pooling operation is translation invariant, rotation invariant, and scale invariant).
In the following, we describe the three-stage training framework in detail.
The three-stage training framework is shown in Figure 12:
(1) In the first stage , all the features and labels generated by the 1st epoch are stored in Q , which is denoted as:
[Equation]
Q contains the features and labels of all samples, but in the early stage of training, the historical features have a large drift from the current features and cannot be used directly to calculate the semantic scale.
(2) The second stage corresponds to epoch 2 to epoch n .
At each iteration, the oldest mini-batch features and labels in Q are removed and those generated by the current iteration are stored.
The goal is to continuously update the features in Q until the feature drift is small enough.
We set n to 5 in our experiments, and the original loss function is used in the first two stages.
Figure 5 shows the effect of n on the model performance.
A larger n does not hurt the model performance, but only takes a little more time.
Experience suggests that setting n to 5 is sufficient.
(3) The third stage corresponds to epoch > n .
At each iteration, the semantic scales are calculated using the features in Q after updating Q , and the original loss is re-weighted.
Algorithm 2 shows how to apply the dynamic re-weighting training framework by taking DSB-ST loss as an example.
Algorithm
Input: Training set D
= { ( x i , y i ) }
Mi =1 , total training epochs N , defined encoder is model()
Initialize the queue Q for epoch = 1 to N do for iteration = 0 to
M batchsize do
Sample a mini-batch { ( x i , y i ) } batchsizei = from D
Calculate features F =
[ f 1 , . .
, f i , . . .
, f batchsize ] , f i = model (x i ) , i = 1 , . .
. , batchsize
Store F and label y into Q : enqueue
The proposed three-stage training framework overcomes the difficulty of not being able to calculate class-wise semantic scales during training due to the limited number of samples per batch.
In fact, there is a simple and brute-force method to achieve the goal of calculating class-wise semantic scales in real time, which is to extract features of all samples using the current model after each iteration.
However, this would take a lot of time, for example, when training ImageNet with batch size of 512, one epoch contains about 2,500 iterations.
This also means that all features need to be extracted 2,500 times, which is unacceptable.
Our method causes almost no reduction in training speed.
In terms of video memory consumption, even the extracted features of a million-level dataset like ImageNet can all be placed on one graphics card (about 6,000 MB of video memory is needed).
In this work, we use 4 NVIDIA 2080Ti GPUs to train all the models.
A comparison of the video memory consumption and training speed for some experiments is shown in Table 10.
It can be noticed that the consumption of our method is negligible for the video memory, and the training speed is about 90% of the original method. [Table]
G V OLUME F ORMULA FOR THE L OW -D IMENSIONAL P ARALLEL H EXAHEDRON IN THE H IGH -D IMENSIONAL S PACE
In Section 3.2 of the main content [Table] we deduce from the singular value decomposition of the matrix
Z =
[ z 1 , z 2 , . . .
z m ] ∈ R d × m composed of features that the volume Vol ( Z ) of the subspace spanned by z i is proportional to (cid:113) det( 1 m ZZ T ) .
Here [Table] we assume that in R d , given m d-dimensional vectors, these vectors will define a parallel hexahedron in R n .
The problem is how to calculate the parallel hexahedron.
For example, consider two vectors
[Equation]
[Equation] [Equation] The parallel hexahedron defined by these two vectors is a parallelogram in R 3 .
We want to find a formula to calculate the area of the parallelogram.
(Note that the true three-dimensional volume of the planar parallelogram is 0, just as the length of the point is 0 and the area of the line is 0.
Here, we are trying to measure the two-dimensional “volume" of the parallelogram.)
Next we will introduce two special cases of parallel hexahedral volume, for a single vector
[Equation]
whose parallel hexahedral is itself.
Here “volume" means the length of the vector, and according to the Pythagorean theorem its volume is
[Equation]
Another case is to give n vectors in R n .
Suppose that these n vectors are
[Equation]
we can know that the volume of the resulting parallel hexahedron is
[Equation]
In the non-special case, the formula for the volume of a low-dimensional parallel hexahedron in a high-dimensional space will contain Results (7) and (8).
Here, we first present the final formula and then discuss why it is reasonable.
Write the k vectors z 1 , . .
z k in R n as column vectors.
Let
Z = [ z 1 , . . .
, z k ] ∈ R n × k , and the volume of the parallel hexahedron derived from the vectors z 1 , . .
z k is
[Equation]
det[
[Equation]
We now discuss why (cid:112) det[ Z T Z ] must be the volume in the general case.
Lemma E.1.
For a matrix Z =
[ z 1 , . . .
, z k ] , we can get
[Equation]
where z i · z j denotes the dot product of the vectors
z i and z j and |
z i | =
√ z i · z i denotes the length of the vector.
The proof of Lemma 1 needs to focus only on
[Equation]
If we apply any linear transformation that preserves angularity and length in R n (in other words, if we perform a rotation operation on R n ), the numbers | z i | and z i · z j do not change.
The multiple sets of all linear transformations that preserve angle and length in R n form a group, called the orthogonal group and denoted as O( n ) .
This allows us to reduce the problem to that of finding the volume of a parallel hexahedron in R k .
Proof.
It is known that
[Equation]
To prove that the above equation must be the formula for the volume, we first consider a set of standard basis of R n :
[Equation]
According to Lemma 1, we are able to find a rotation of R n , which is able to maintain both length and angle, and also rotate our vectors z 1 , . .
z k such that they can be fully represented linearly by the first k standard vectors e 1 , . . .
, e k (which is geometrically reasonable).
After the rotation, the latter n − k dimensions of each vector zi are 0.
Therefore we can think of our parallel hexahedron as consisting of k vectors in R k and we already know how to calculate it, which is
[Equation]