L IFTING
I MBALANCED R EGRESSION WITH S ELFS UPERVISED L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
A new inﬂuential task called imbalanced regression has received a great deal of attention.
Different from previous works, this task particularly aims to solve the imbalanced problem in the regression world.
The exploration of this task is still at a preliminary stage, so more attempts are needed.
In this paper, we ﬁrst discover that the beneﬁt of Self-supervised Learning (SSL) can help reveal the bias caused by the severe skewness in the training data.
Therefore, we work on a seamless marriage of imbalanced regression and SSL.
But with this comes the ﬁrst question of how to measure the similarity and dissimilarity between raw and augmented samples under the regression sense, for which the deﬁnition is clear in the classiﬁcation.
To tackle this issue, we provide a formal deﬁnition of similarity and dissimilarity in the regression task.
In addition, a common practice in SSL to produce augmented samples is to put noise onto the raw samples; the second problem is, it is not guaranteed that the augmented samples are similar to original samples when scaling to a deep network by directly adding random noise to the input.
To approach this problem, we speciﬁcally limit the volume of noise on the output to get a meaningful augmented sample by back propagation.
Experimental results show that our approach achieves state-of-the-art performance.
I NTRODUCTION
Regression is a fundamental task in machine learning and statistical analysis, which involves modeling the relationship, between one or more independent and dependent variables.
As distinct from the discrete output values in classiﬁcation, regression makes predictions for continuous values.
The issue of imbalanced recognition, arising from the conjunction of imbalance and classiﬁcation, has been drawing the attention of researchers recently (Shu et al., 2019;
Cao et al., 2019;
Ren et al., 2020;
Jamal et al., 2020; Kang et al., 2020; 2021).
By analogy, it is natural to question whether long-tailed regression exists (or one can also call it imbalanced regression, the equivalence of these two names in the paper).
Exactly what we expected, a fresh task, namely long-tailed regression, has recently been proposed by Yang et al.
Regardless of the newness of the task, there is no way to neglect its immense signiﬁcance in terms of application and research.
For example, age estimation from facial images; health indicators for various populations in healthcare (Soualhi et al., 2014), these quantities usually exhibit a long-tailed distribution.
Going further, in terms of performance, progress in the long-tailed regression research is also far from satisfactory.
More recently, self-supervised learning (SSL) has emerged as a new rage in the research community, owing to its straightforward learning paradigm yet excellent performance (Noroozi & Favaro, 2016;
Gidaris et al., 2018; Wu et al., 2018;
He et al., 2020).
The excellence of the approach has been admired not only on classiﬁcation with well-balanced datasets, but also surprisingly on long-tailed classiﬁcation by Kang et al.
; Wang et al.
; Cui et al.
; Samuel & Chechik (2021) who found its beneﬁts.
It prompts some thoughts on the potential and practical applications, in particular, applications to SSL for long-tailed regression (Yang et al., 2021).
Mindful that the majority of available techniques on SSL are in the context of classiﬁcation (Oord et al., 2018; Hjelm et al., 2018; Chen et al., 2020a; Henaff, 2020).
Due to the difference between classiﬁcation and regression (no hard boundary existing between outputs in regression task as shown in Figure 1a) (Razi &
Athappilly, 2005; Breiman et al., 2017; Liaw et al., 2002;
Yang et al., 2021), it is not guaranteed that SSL can help address the imbalanced regression problem well, and it may be difﬁcult to make a straightforward migration from classiﬁcation to regression task.
Upon deeper pondering, there are
[Figure]
obstacles and problems that we face: Can SSL really relieve the long-tailed regression problem ?
If so, how might we solve the imbalanced regression problem by utilizing SSL ?
Figure 1a illustrates the motivation, by adding noise to the image of the cat, it is very unlikely to be misclassiﬁed into a class that differs signiﬁcantly from it, such as a bird, since they are discrete and the categories diverge greatly from each other.
In the case of regression, however, it is not at all feasible when a similar migration was made.
We imagine that if noise is added to a face image, then the network’s estimation of the person’s age may be signiﬁcantly biased.
In this regard, it becomes essential to work out and comprehend the degree of noise that induces “similarity and dissimilarity” amongst output values from a regression standpoint.
On the foundation of this deﬁnition, questions are raised about exploiting noise sensibly and effectively.
With this paper, we try for the ﬁrst time to propose answers to these questions.
And we summarize our contributions as follows:
• To the best of our knowledge, we are the ﬁrst to broaden the concept of similarity and dissimilarity between augmented samples and original samples from classiﬁcation to regression.
And we are also the ﬁrst to reveal imbalanced regression problem with SSL.
• Having been convinced of the beneﬁcial effect of applying SSL on a simple experiment, we extend it to the neural network-based long-tailed regression task.
To ensure that the noise is manageable w.r.t.
the similarity of the regression, we propose to generate noise on regression values within a predeﬁned threshold.
• We obtain an optimization problem on noise with an efﬁcient approximate solution algorithm based on a ﬁrst-order Taylor expansion.
The augmented input samples are obtained by back-propagation.
• As a practical trial for the proposed method, the results reveal that we attained not only the best results, but also the possibility to mix it freely with other training techniques.
P RELIMINARY
In this section, we cover the description of imbalanced regression, and then, with reference to the concept of similarity and dissimilarity in classiﬁcation, we broaden this concept to regression, and provide a new deﬁnition of similarity and dissimilarity in regression.
Notations.
In this paper, lowercase typeface letters ( e.g. , x ) represent scalar, lowercase bold typeface letters ( e.g. , x ) represent vectors, uppercase typeface letters ( e.g. , X ) stand for random variable, uppercase bold typeface letters ( e.g. , X ) represent matrix.
diag ( · ) means diagonalising a vector into a matrix.
2. P ROBLEM S TATEMENT
A collection of paired training data { ( x si , y si ) } N s i =1 , validation samples { ( x vi , y vi ) } N v i =1 and testing samples { ( x ti , y ti ) } N t i =1 are presumed to be available, where x oi ∈ R d , o ∈ { s, v, t } , and y oi are a continuous variables.
We suppose that y oi is well-bounded and notate the maximum and minimum values of y
oi as y omax and y omin
accordingly.
Further, we are able to partition y oi into B bins [ y o 0 , y o 1 ) , [ y o 1 , y o 2 ) , . . .
, [ y oB − 1 , y oB ) with equal interval ∆ y
=
y omax − y omin B .
In so doing we could put every y oi in the dataset into the corresponding bin fulﬁlling the condition y ob < y oi ≤ y ob +1 , where b ∈ { 0 , 1 , . . .
, B − 1 } .
With three splits (training, validation, testing), if we count the number of data points that fall into each bin and form these into a vector m o =
( m o 1 , . . .
, m oB ) .
The imbalanced character of the normal regressions lies in the existence of m sb i , m sb j ∈ m s , allowing m sb i (cid:29) m sb j to hold.
It is the opposite of this that the values of the elements of the two vectors m v and m t , are approximately equal.
A desire for imbalanced regression enables model z oi = f
( x oi ; θ ) produced on an imbalanced training set to be potentially generalized to a balanced validation and testing set to the maximum extent possible.
Typically, f ( x oi ; θ ) may be a neural network with parameter θ because of its impressive ﬁtting ability.
In what follows, for the sake of convenience, we will drop the superscript o .
In the introduction section, we elaborate on the predicament that the lack of a clear deﬁnition of the concepts of similarity and dissimilarity in the regression, as regression involves continuous targets, where hard boundaries between classes do not exist.
As a solution to this problem, we formally propose the following deﬁnition:
Deﬁnition 1 There exists some threshold δ , regarding two data samples ( x i , ˆ y i ) , (˜ x i , ˜ y i ) , ˜ x i is obtained by adding some noise r to x i , and ˜ y
i and ˆ y i is inferred by the model f (; θ ) from ˜ x
i and x i respectively.
We remark that ( x i , ˆ y i ) , (˜ x i , ˜ y i ) are similar samples of the model f ( x i ; θ ) (no need for well-trained model) if the distance between the two samples is less than δ under some distance function D ( · , · ) , such as l 1 or l 2 distance.
Mathematically, it can be phrased in this following:
[Equation]
where I ( x i , ˜ x i ) indicates the trueness of the condition whether two samples x i and ˜ x i are similar samples.
The parameter δ here is manually speciﬁed and task-dependent.
We display a virtual example of this deﬁnition as illustrated in Figure 1b.
We select images from AgeDB-DIR and attach noises to them, afterward we estimate the ages of the original images and their corresponding noisy images with the network.
Finally, we identify a pair of original image and noisy image as not similar if their difference exceeds the threshold δ , and vice versa.
M OTIVATION
Herewith we illustrate, for simple shallow neural networks, that SSL facilitates long-tailed regression in a positive way.
Aside from this, we would characterize the difﬁculties involved in extending our approach to deep neural networks, i.e. for controlling the level of noise to guarantee the creation of similar samples under the sense of regression.
We ﬁrst describe the data generation, together with a description of the speciﬁc experimental settings followed by an analysis.
The network structure and implementation details are available in the appendix.
Data generation.
Consider two components of data following two different Gaussian distribution, in particular, x 1 ∼ N ( µ 1 · 1 d , diag ( σ 1 · 1 d )) , x 2 ∼ N ( µ 2 · 1 d , diag ( σ 2 · 1 d )) .
1 d denotes a vector with all 1’s.
The values of µ 1 ( σ 1 ) and µ 2 ( σ 2 ) are quite different.
With linear mapping to xand x 2 with parameter w ∈ R d , it yields the output value y =
w T x .
As a result, we get two collections of training data { x s 1 ,i , y s 1 ,i } N s
i
=1 , { x s 2 ,j , y s 2 ,j } N s 2 j =1 .
Suppose N s 1 > N s 2 , and we measure
[Figure]
imbalance here in terms of R =
N s
N s 2 .
We pick up the value of R in { 1 , 2 , 4 , 6 , 8 , 10 , 100 } .
Despite the training samples are imbalanced, the testing set { x t 1 ,i , y t 1 ,i } N t 1 i =1 , { x t 2 ,j , y t 2 ,j } N t 2 j =1 we generate is balanced.
Additionally, we produce similar samples by controlling the noise so that the condition min( y 1 ,i , y 2 ,j ) (cid:29) δ, ∀ i ∈ { 1 , . . .
, N 1 } , j ∈ { 1 , . .
. , N 2 } is satisﬁed.
Experimental settings.
We compare two training methods here: (i) Representation Learning : It is trained in the ﬁrst T 1 epochs using a self-supervised approach, and the resulting network is treated as an initialized network, and in the second T 2 epochs it is trained in a standard supervised learning manner.(ii)
Supervised Learning : it is entirely trained using supervised learning in the T 1 + T 2 epochs.
As per the normal process, the model is trained on the training set and tested on the testing set.
To evaluate we use Mean Square Error (MSE) as a criterion.
Here we use the vanilla mean squared loss function.
Experimental ﬁndings.
We plot the variation curve of MSE with the imbalance coefﬁcient on the testing set for the two training methods.
As illustrated in Figure 2a, what has been clearly observed is that the MSE for both methods grows with an increase in the imbalance coefﬁcient.
More importantly, it is found that SSL could alleviate this problem to some degree.
The experimental ﬁndings, thus, lead us to mitigate the imbalance regression problem with SSL.
Remark It is true that the above experiments concede that SSL can be beneﬁcial to long-tailed regression, albeit with some restrictions that remain when we adapt it to neural networks.
Well, yes, we have imposed settings for noise, namely, the noise added is controllable in terms of its impact on the regression values.
Unfortunately, scaling the technique beyond shallow to deep networks, there is a difﬁculty in keeping the difference between the output values of similar samples under regression within a threshold, a concern that arises in the case of a sample augmented with random noise, where the difference between the output values of the noisy sample and the original sample is more than the threshold, as shown in Figure 2b, hence the challenge stems from ﬁnding valid noise to ensure the generation of similar samples .
M ETHOD
Just as we showed before, it is not feasible to merely port the above method to regression tasks.
Instead, it will backﬁre if at all we do so.
Hence, it is essential to control the noise r to limit the gap between ˆ y i and ˜ y
i in the length of a bin ∆ y .
We search for the most anisotropic noise causing the generated samples to be similar or dissimilar to the original samples, which means that we have to impose constraints on the output values.
Rather than adding noise to the input, we blaze new a trail, that is, we apply noise to the output value.
Mathematically, it can be expressed by the following formula:
[Equation]
where
[Equation]
where D denotes the distance loss function, and ˜ y
i is the noisy output w.r.t.
noises z i and r i .
Here z i is randomly sampled from [ − ∆ y, ∆ y ] .
This allows us to assure that ˜ y
i and ˆ y i are similar, while simultaneously enhancing the local smoothness of the output.
Normally it is not possible to obtain r to ﬁnd a closed-form solution.
Since neural networks are highly dimensional and nonlinear, the analytical solutions can not be derived for r .
For the purpose of facilitating analysis and simultaneously providing efﬁcient training of the network, we approximate D (ˆ y i , ˜ y
i ; θ ) by the ﬁrst-order Taylor expansion:
[Equation]
Where a = D (ˆ y i , ˜ y
i ; θ ) | x = x
i is a constant term around x i .
Based on this, the close-form solution of r i can be calculated as
[Equation]
Therefore, it is obviously that r i = (cid:15) ∇ x i D
(ˆ y i , ˜ y
i ; θ ) ||∇ x i D (ˆ y i , ˜ y
i ; θ ) || .
Furthermore, as Eq. 4 presented, ˜ y
i
=
ˆ y i + z i , so r i can be further denoted as
[Equation]
Input: Mini-batch B with M samples, model f ( · ; θ ) with its parameters θ , bin length ∆ y , limitation coefﬁcient of noise (cid:15) , divergence distance function D , values range of input samples ( v min , v max ), function clip ( · , v max , v min ) to clip values that out of range of ( v min , v max )
Result:
Updated parameters of the model ˆ θ .
1 for x i in B do
Initial random noise z i , s.t.
| z i | < ∆ y ; ˆ y i =
f ( x i ; θ ) , ˜ y
i
=
ˆ y i + z i ;
Calculate r i by taking the gradient of D (ˆ y i , ˜ y
i ; θ ) w.r.t.
x i : g i = ∇ x i D (ˆ y i , ˜ y
i ; θ ) =
∇ x i D ( f ( x i ; θ ) , f ( x i ; θ ) + z i ) r
i ← (cid:15) × g i /
||
g
i || 2 ;
Update ˜
y
i by taking x i
+ r i as the input of f
: ˜ x
i = clip ( x i
+ r i , v max , v min )
; ˜ y
i = f
( ˜ x i ; θ ) ;
Update r i by by taking the gradient of D (ˆ y i , ˜ y
i ; θ ) w.r.t.
x i : g i = ∇ x i D (
˜ x
= x + r obtained in this way, however, may exceed the boundary value of the feature (For example, the range of image pixels is [0 , 255] ).
To cope with such an accident, we constrain the feature values to this range by clipping their values.
On the other hand, considering that a single back-propagation optimization may not be an accurate approximation, we therefore performed a twostep optimization of r .
In our experiments, one-step optimisation approach does not adequately yield optimal performance, and more iterations will introduce more computational cost to training without signiﬁcant performance gain.
Up to now, our proposed S elf- S upervised I mbalanced R egression (SSIR) algorithm is completely introduced.
In order to show the SSIR more clearly.
We give the algorithm workﬂow in Algorithm 1.
Our training strategy includes normal regression training and self-supervised regression.
For the regression training, we directly apply supervised training to minimize the loss function between the outputs of the model ˆ y and the corresponding ground truth y .
Besides, an additional coefﬁcient λ is introduced to the ﬁnal loss.
Our overall training function is shown below:
[Equation]
E XPERIMENTS
In this section, we report the results on three datasets to demonstrate the superiority of our proposed methods, when compared with state-of-the-art baselines.
Speciﬁcally, we compare our method with
VANILLA , SMOTER (Torgo et al., 2013), SMOGN (Branco et al., 2017), RRT (Yang et al., 2021),
INV , SQINV , FOCAL-R (Yang et al., 2021), LDS (Yang et al., 2021), FDS (Yang et al., 2021) and some of their combinations on three imbalanced regression datasets including IMDN-WIKIDIR , AgeDB-DIR and NYUD2-DIR .
Note that, in all the experiments, the LDS is combined with re-weighting methods
i
[Table]
e. SQINV or INV .
In otherwise, our proposed SSIR can be combined with various methods in training process that demonstrates the ﬂexibility of SSIR.
The evaluation metrics used on IMDB-WIKI-DIR and AgeDB-DIR datasets are MAE and GM (Yang et al., 2021), besides,
RMSE and δ 1 are also employed on NYUD2-DIR dataset.
For the details of datasets, baselines, evaluation metrics and implementation details, we put them in the appendix.
[Table]
5. M AIN R ESULTS
In the experiments on IMDB-WIKI-DIR dataset
[Table]
we combine our proposed SSIR with SQINV and
FOCAL-R respectively.
As shown in Table 1, our method achieves the best performance with 7. and 4.26 on MAE and GM accordingly.
Compared with the second-best method RRT + LDS +
FDS, there has improvements of 0.02 and 0.05 in MAE and GM with our approach.
Besides, in comparison with other FOCAL-R-based methods, greater performance can be achieved with the combination of FOCAL-R and SSIR with a better GM score of 4.39 and the MAE score of 7.88 .
For re-weighting-based methods, the combination of SQINV and SSIR outperforms the second-best re-weighting-based method SQINV + LDS + FDS and improves 0.15 and 0.11 on MAE and GM respectively.
[Table]
In the experiments on AgeDB-DIR
[Table]
with the same experimental settings on IMDB-WIKI-DIR
[Table]
we combine our proposed SSIR with SQINV and FOCAL-R respectively.
Table 2 illustrates the main results of our method and baselines on AgeDB-DIR dataset.
Clearly our method achieves the best performance on both MAE and GM with 7.35 and 4.61 respectively.
As the combination of the reweighting method and SSIR, SQINV + SSIR performs better than the other four re-weighting-based methods with 0.06 and 0.11 improvements on MAE and GM separately, compared to the second-best re-weighting-based method SQINV + LDS + FDS.
For FOCAL-R-based methods, the association with SSIR can further decrease the testing error than FOCAL-R and FOCAL-R + LDS + FDS.
For NYUD2-DIR dataset, we combine SSIR with VANILLA and re-weighting method INV respectively.
As presented in Table 3, our method INV + SSIR achieves the best performance among other methods.
Speciﬁcally, INV + SSIR outperforms the second-best method INV + LDS + FDS on
RMSE with 0.047, and our method achieves the best performance on δ 1 with 0.075.
Compared with VANILLA, the combination with SSIR can improve the performance signiﬁcantly with 0.07 on
RMSE and 0.007 on δ 1 .
[Table]
[Figure]
5. P ARAMETER AND C ASE S TUDY
Parameter Study.
Figure 3a illustrates the results of parameter study of FOCAL-R + SSIR on
AgeDB-DIR dataset.
As it shows, each line denotes the results of different λ when ﬁxing (cid:15) , and we conduct such experiments with the (cid:15) of 5e-3, 1e-3, 5e-4 and 1e-4.
The overall results show that our method achieves the best performance with (cid:15)
= 1 e − 3 and λ = 1 . 0 .
Case Study.
We also conduct case studies to visualize some results of selected examples.
As
Figure 3b shows, the gaps between the outputs and their associated noisy outputs of all the chosenexamples do not exceed a bin length, which means our proposed method can successfully control the noise to limit the noisy outputs within a reasonable range.
Visualization on Noisy Samples.
Moreover, we also show three examples and their corresponding noisy examples with (cid:15) of 0.05, 0.1, 0.3, 0.5 and 1.0 respectively in Figure 4.
Obviously, with the (cid:15) increasing, the noisy examples become unclearer that means the noise level is positively correlated with (cid:15) .
R ELATED W ORK
Presented here are the literature on imbalance problems and SSL.
Concerning imbalance problems, the introduction focuses on classiﬁcation and regression.
For SSL, we review some of the advances and applications.
6. I MBALANCED P ROBLEM
From the object of imbalanced problems, it can be divided into two categories: imbalanced classiﬁcation and imbalanced regression.
Imbalanced classiﬁcation.
In the imbalanced classiﬁcation task, existing methods mainly adopt data-based or model-based solutions (Yang et al., 2021).
The core idea of data-based methods is reducing the gap of sampling numbers between many-shot classes and few-shot classes (Chawla et al., 2002;
He et al., 2008).
For the model-based methods, several works have been proposed to tackle such challenges based on lots of aspects.
Jamal et al.
(2020) proposed to solve long-tailed classiﬁcation through domain adaptation perspective.
Kang et al.
(2020) used a two-stage learning strategy to learn representations for the classiﬁcation layer using a long-tailed training set.
Kang et al.
(2021) explored the impact of balanced feature spaces when dealing with imbalanced data, and proposed a SSL-based method to better learn the representation for imbalanced classiﬁcation.
Besides, SSL on unbalanced classiﬁcation was similarly explored in Wang et al.
; Cui et al.
(2021); Samuel & Chechik (2021).
Furthermore, there also has some methods addressing the problem by well-designed loss functions (Zhang et al., 2017;
Cao et al., 2019; Cui et al., 2019; Ren et al., 2020) and other learning paradigms, i.e. meta learning (Shu et al., 2019; Wang et al., 2020).
Imbalanced regression.
Imbalanced problem has also been explored in regression problems.
Conventional works mainly applied data-based methods to solve the imbalanced problem in regression tasks.
One of the ﬁrst works to address this problem, is published by Torgo et al.
(2013) using
SMOTE (Chawla et al., 2002).
Branco et al.
(2017) proposed a Gaussian noise-based synthetic case generation method for imbalanced regression.
After that, they further proposed a bagging-based method to aggregate various data pre-processing strategies (Branco et al., 2018) for the same task.
Recently, Yang et al.
(2021) further delved into such problem by analyzing the error distribution and the statistic distribution of features, both w.r.t.
labels, and proposed two distribution smoothing methods for labels and features.
And they also proposed the Focal-R, a new loss function for imbalanced regression, motivated by Focal loss (Lin et al., 2017).
6. S ELF -S UPERVISED L EARNING
Recently, SSL has received extensive attention in the ﬁeld of deep learning.
Different from conventional unsupervised learning and supervised learning, SSL can use the semantic or contexture supervision information to learn robust and well-generalization representations without manual annotations, include contrastive learning-based methods (Chen et al., 2020a;b;c;
He et al., 2020) and some other pretext tasks-based methods, such as image inpainting (Pathak et al., 2016; Jenni &
Favaro, 2018), context prediction (Doersch et al., 2015), rotation prediction (Gidaris et al., 2018) and so on.
Whereas we tackle the optimization problem in a very similar way, the difference concerns the fact that we apply noise to the output directly rather than to the input, as well as the fact that the problem we analyze varies from semi-supervised classiﬁcation problems to imbalanced regression problems.
C ONCLUSION
In this paper, we explore the Self-Supervised Learning (SSL) in imbalanced regression scenario.
Speciﬁcally, we propose S elf- S upervised I mbalanced R egression (SSIR) to better address the imbalanced problem in regression tasks.
In SSIR, instead of adding noise to the inputs directly to generate the noisy samples, we add noise, which is limited by the bin size, to the output of the model to avoid the excessive dissimilarity between samples and noisy samples, followed by back propagation steps to obtain the meaningful noisy samples.
This work provides a new paradigm to tackle this challenge.
Further, extensive experiments have shown the superiority of our proposed SSIR over existing state-of-the-art baselines.
A PPENDIX
In the appendix, we give the implementation details in the motivation, as well as the details of the datasets and baseline methods included in the experiments.
Besides, we also present more detailed experimental results of parameter studies, and loss curves within different settings.
8. C OMPARISON WITH R ANDOM N OISE M ETHOD
With the addition of random noise, the combined SSIR with FOCAL-R and SQINV approaches can achieve the performance on MAE with 7.57 and 7.71 respectively on AgeDB-DIR dataset.
In contrast, the combined SSIR with FOCAL-R and SQINV approaches with noise generation proposed in our paper can achieve MAE performance with 7.35 and 7.49 respectively.
The experimental results demonstrate the superiority of our noise generation approach.
8. C OMPARISON WITH O NE - STEP OPTIMIZATION M ETHOD
To quantitatively compare the beneﬁts of two-step approximation against one-step approximation, we conducted a comparison experiment on the AgeDB-DIR dataset.
The setup of the comparison experiment is following: we perform a one-step optimization of r and provide results combined with SQINV; except for this, we keep the other hyperparameters consistent with our experiments.
When performing one-step optimization, the combined SSIR with SQINV approach can achieve the performance on MAE with 7.62 .
In contrast, the combined SSIR with SQINV approach with a twostep approximation can achieve MAE performance with 7.49 .
The experimental results demonstrate the superiority of two-step approximation.
However, we have found in our experiments that there is no signiﬁcant increase in performance but an increase in computational overhead when the number of optimization steps is greater than or equal to 3.
Therefore, in order to balance accuracy and efﬁciency, we have used two-step optimization in our paper.
8. E XPERIMENTS S ETTINGS
Datasets We evaluate our proposed method on three large-scale deep imbalanced regression datasets, the tasks of which are age estimation and depth estimation.
For all the datasets used in our experiments, we follow the settings in (Yang et al., 2021) and segment the target bins into three subsets: many-shot which contains more than 100 training images, medium-shot in which the number of training samples in each bin is in the range of 20 to 100, and few-shot that contains less than 20 training samples in each bin.
The details of the datasets are as following:
• IMDB-WIKI-DIR: IMDB-WIKI-DIR is an imbalanced age estimation dataset.
This dataset is created by Yang et al.
based on the IMDB-WIKI dataset (Rothe et al., 2018).
The original IMDB-WIKI contains 523.0K face images with associated ages.
In IMDB-WIKIDIR, the training set contains 191.5K images, and both validation and testing sets include 11.0K bin-balanced images separately.
The length of a bin is set to 1 year with the range from 0 to 186, and the density of each bin is from 1 to 7,149.
• AgeDB-DIR: AgeDB-DIR is also a imbalanced age estimation dataset proposed by Yang et al.
(2021) using AgeDB dataset (Moschoglou et al., 2017).
In the dataset, the age range is from 0 to 101 and the bin length is 1 year.
The maximal and minimal numbers of images in a bin are 353 and 1 respectively.
The training set contains 12,208 images with imbalanced densities among bins, while the validation and testing sets contain 2,140 bin-balanced images respectively.
• NYUD2-DIR: The NYUD2-DIR is a dataset constructed by Yang et al.
(2021) for imbalanced indoor scenes depth estimation, which is an imbalanced version of NYU Depth Dataset V2 (Silberman et al., 2012).
The dataset contains images of different indoor scenes and their corresponding depth maps.
The bin length is set to 0.1 meters.
The maximal and minimal depth is 10 meters and 0.7 meters separately in the dataset.
Following Yang et al.
(2021), there are 50K images in the training set and 654 images in the testing set.
In each bin, the number of pixels is in the range from 1 .
13 × 10 6 to 1 .
46 × 10 8 .
Besides, the number
of testing pixels selected for each bin from testing images is set to 9,357 (the minimal number of pixels in each bin in the testing set), which aims to make the testing set
[Table] [Table]
[Table] [Table]
[Table] [Table]
Baselines We conduct experiments to compare our proposed method with the following baselines:
• Vanilla model:
The pure backbone model that does not apply any techniques to improve the performance is presented as VANILLA .
Following (Yang et al., 2021), we use ResNet50 (He et al., 2016) as VANILLA for both IMDB-WIKI-DIR and AgeDB-DIR datasets, and ResNet-50-based encoder-decoder method (Hu et al., 2019) for NYUD2-DIR dataset.
• Synthetic samples: We choose existing synthetic samples methods, which are used for imbalanced regression, as baselines, including SMOTER (Torgo et al., 2013) and SMOGN (Branco et al., 2017).
• Focal-R: FOCAL-R
(Yang et al., 2021) is the regression version of Focal loss (Lin et al., 2017), which uses a continuous function, e.g. Sigmoid function, to map the absolute error into the range of 0 to 1 instead of the scaling factor in Focal loss.
• Regressor Re-Training
(RRT): RRT (Yang et al., 2021) is a two-stage training strategy inspired by Kang et al.
(2020), in which the feature and classiﬁer are decoupled.
• Re-weighting: We also select Re-weighting methods as baselines, including inversefrequency weighting ( INV ) and its square-root weighting variant ( SQINV ) (Yang et al., 2021).
• Distribution Smoothing: Distribution Smoothing (Yang et al., 2021) includes Label Distribution Smoothing (LDS) and Feature Distribution Smoothing (FDS).
Such methods are designed to utilize the information from nearby bins by smoothing the distribution of labels or features.
Note that, LDS should be used with re-weighting methods, and we follow Yang et al. (2021) to combine LDS with INV and SQINV .
Evaluation metrics Here, we will introduce the details of all the evaluation metrics used in our experiments.
• Mean Absolute Error (MAE): MAE is a common evaluation metric in regression problems, which is used to measure the averaged absolute difference between ground truth and prediction results.
MAE can be denoted as 1 N (cid:80) Ni
=
| ˆ y
i − y
i | , where N denotes the number of total samples, ˆ y i and y i are the prediction result and ground truth of the i -th sample respectively.
• Root Mean Squared Error (RMSE): Similar to MAE, RMSE is also a widely used evaluation metric in regression tasks.
Different from MAE, RMSE is used to measure the rooted averaged squared difference between ground truth and prediction results, which can be presented as 1 N (cid:80) Ni
=
(ˆ y i − y i ) 2 , where N denotes the number of total samples, ˆ y i and y
i are the prediction result and ground truth of the i -th sample respectively.
• error Geometric Mean (GM): GM is a new evaluation metric proposed by Yang et al.
Instead of applying arithmetic mean, GM uses geometric mean over the prediction errors, which is deﬁned as ( (cid:81)
Ni =
1 e i ) 1 N , where e i (cid:44) | ˆ
y
i − y i | denotes the L 1 error of the i -th sample.
• Threshold Accuracy ( δ i ): For NYUD2-DIR dataset, we follow Yang et al.
and use standard depth estimation evaluation metric
δ i which is deﬁned as max( d i g i , g i d i )
=
δ i < 1 .
25 i , where g i and d i denote the ground truth value and the the prediction value of a pixel respectively.
Here, in our experiments on NYUD2-DIR, we set i to 1 as the settings in (Yang et al., 2021).
8. I MPLEMENTATION D ETAILS IN M AIN E XPERIMENTS
IMDB-WIKI-DIR: For IMDB-WIKI-DIR dataset, we use ResNet-50 (He et al., 2016) as backbone and train it with our proposed SSIR method for 90 epochs using Adam optimizer (Kingma & Ba, 2014) with L 1 loss.
The initial learning rate is set to 1e-3 and decayed to 1e-4 and 1e-5 at the 60-th and 80-th epoch respectively.
Besides, we set the batch size, λ and (cid:15) to 256, 0.1 and 1e-3 separately.
AgeDB-DIR: For AgeDB-DIR dataset, the settings of backbone, optimizer, learning rate, loss function and batch size are the same with that of IMDB-WIKI-DIR.
For the training of FocalR+SSIR, we set the λ and (cid:15) to 1.0 and 1e-3 respectively.
Besides, they are set to 0.1 and 1e-3 for
SQINV+SSIR.
NYUD2-DIR: For the experiments on NYUD2-DIR, we apply a ResNet-50-based encoder-decoder architecture (Hu et al., 2019).
We train our method by Adam optimizer with 20 epochs.
The initial learning rate is set to 1e-4 and decayed by 0.1 every 5 epochs.
We set the batch size and weight decay to 32 and 1e-4 respectively.
For λ and (cid:15) , we set them to 1.0 and 5e-4 for VANILLA+SSIR, and 2.5, 5e-4 for INVERSE+SSIR.
8. I MPLEMENTATION D ETAILS IN M OTIVATION E XPERIMENTS
For the settings in the data generation of our motivation, we set the mean and variance of the two
Gaussian distribution µ 1 , σ 1 and µ 2 , σ 2 to 10, 25 and -10, 25 respectively, and the dimension of the generated data d is set to 100.
We apply a two-layer Multi-Layer Perception (MLP) withhidden neurons as the prediction model in the experiments.
For the settings of the training process in the experiments, we set epochs T 1 and T 2 to 1000 and 500 respectively.
The learning rates of the two training methods: Representation Learning and Supervised Learning , are set to 1e-3 and 1erespectively.
Besides, we set the weight decay of the training process in the experiments to 5e-5.
8. P ARAMETER S TUDIES
Here, we study the different parameter settings of (cid:15) and λ on all three datasets and report all the results of the experiments that we conducted.
Table 4 illustrates the experiments results of different settings of (cid:15) and λ .
The top part of the table is the results of different parameters settings of SQINV + SSIR, and the bottom part shows the results of FOCAL-R + SSIR on different settings of (cid:15) and λ .
The results of the parameter study of our method on AgeDB-DIR dataset are presented in Table 5.
As the table shows
[Table]
we present the results of different (cid:15) and λ settings of our approach.
The top part of the table is regarding with SQINV + SSIR, and the bottom part denotes the results of different parameter settings of FOCAL-R + SSIR.
[Table]
Table 6 is the parameter study regards to (cid:15) and λ on NYUD2-DIR dataset.
The top part of the table illustrates the results of different parameters settings of VANILLA + SSIR while the bottom part shows that of INVERSE + SSIR.
8. T RAINING AND V ALIDATION L OSSES
We present the training and validation loss of our proposed approach on IMDB-WIKI-DIR and
AgeDB-DIR datasets.
Figure 5a and 5b present the training and validation losses w.r.t.
training epochs of SQINV + SSIR and FOCAL-R + SSIR respectively on IMDB-WIKI-DIR dataset.
Figure 5c and 5d illustrate the training and validation losses of SQINV + SSIR and FOCAL-R + SSIR on
AgeDB-DIR dataset.
[Table] [Table]
[Figure]