S UPERVISED C ONTRASTIVE R EGRESSION WITH S AMPLE R ANKING
Anonymous authors
Paper under double-blind review
A BSTRACT
Deep regression models typically learn in an end-to-end fashion and do not explicitly try to learn a regression-aware representation.
Their representations tend to be fragmented and fail to capture the continuous nature of regression tasks.
In this paper, we propose Supervised Contrastive Regression (SupCR) , a framework that learns a regression-aware representation by contrasting samples against each other based on their target distance.
SupCR is orthogonal to existing regression models, and can be used in combination with such models to improve performance.
Extensive experiments using five real-world regression datasets that span computer vision, human-computer interaction, and healthcare show that using SupCR achieves the state-of-the-art performance and consistently improves prior regression baselines on all datasets, tasks, and input modalities.
SupCR also improves robustness to data corruptions, resilience to reduced training data, performance on transfer learning, and generalization to unseen targets.
I NTRODUCTION
Regression problems are ubiquitous and fundamental in the real world.
They include estimating age from human appearance (Rothe et al., 2015), predicting health scores from physiological signals (Engemann et al., 2022), and detecting gaze directions from webcam images (Zhang et al., 2017b).
Since regression targets are continuous, the most widely used approach to train a regression model is to have the model predict the target value, and use the distance (e.g., L1 or L2 distance) between the prediction and the ground-truth target as the loss function (Zhang et al., 2017a;b; Schrumpf et al., 2021; Engemann et al., 2022).
There are also works that control the relationship between predictions and targets by converting the regression task into a classification task and training the model with the cross-entropy loss (Rothe et al., 2015; Niu et al., 2016; Shi et al., 2021).
However, all previous methods focus on imposing constraints on the final predictions in an end-to-end fashion, but do not explicitly consider the representations learned by the model.
Their representations tend to be fragmented and fail to capture the continuous relationships underlying regression tasks.
For example, Figure 1(a) shows the representation learned by the L1 loss in the task of predicting weather temperature from webcam outdoor images (Chu et al., 2018), where the images are captured byoutdoor webcams at different locations.
The representation learned by the L1 model does not exhibit the continuous ground-truth temperatures; rather it is grouped by different webcams in a fragmented manner.
Such unordered and fragmented representation is sub-optimal for the regression task and can even hamper performance since it contains distracting information (e.g., the capturing webcam).
While there is a rich literature on representation learning, past methods are focused on classification problems.
In particular, contrastive learning (Chen et al., 2020a; He et al., 2020; Chen et al., 2020b) and supervised contrastive learning (SupCon)
(Khosla et al., 2020) have been proven highly effective in representation learning.
However, as shown in Figure 1(b), which plots the representation learned by SupCon for the visual temperature prediction task mentioned above, such method produces a sub-optimal representation for regression problems because it ignores the continuous order between the samples in a regression task.
In this paper, we introduce Supervised Contrastive Regression (SupCR) , a new framework for deep regression learning, where we first learn a representation that ensures the distances in the embedding space are ordered according to the target values.
We then use this representation to predict the targets.
To learn such a regression-aware representation, we contrast samples against each other based on
[Figure]
their label/target distances. Our method explicitly leverages the ordered relationships between the samples to optimize the representation for the downstream regression task.
As shown in Figure 1(c),
SupCR leads to a representation that captures the intrinsic ordered relationships between the samples.
Moreover, our framework is orthogonal to existing regression methods as one can use any kind of regression method to map our learned representation to the prediction values.
Extensive experiments using five real-world regression datasets that span computer vision, humancomputer interaction, and healthcare verify the superior performance of our framework.
The results show that SupCR achieves the state-of-the-art performance and consistently improves the performance of previous regression methods by 8.7% on average across all datasets, tasks, and input modalities.
Furthermore, our experimental results also show that SupCR delivers multiple desirable properties: 1) improved robustness to data corruptions; 2) enhanced resilience to reduced training data; 3) higher performance on transfer learning; and 4) better generalization to unseen targets.
R ELATED W ORK
Regression Learning.
Deep learning has achieved a great success in addressing regression tasks (Rothe et al., 2015;
Zhang et al., 2017a;b; Schrumpf et al., 2021; Engemann et al., 2022).
The most straightforward and widely used way to train a regression model is to have the model predict the target value, and use the distance between the prediction and the ground-truth target as the loss function.
Common distance metrics used as the loss function in regression tasks are the L1 loss, the MSE loss and the Huber loss (Huber, 1992).
Past work has also proposed variants to the basic methods described above.
One branch of prior works (Rothe et al., 2015; Gao et al., 2017; 2018; Pan et al., 2018) divide the regression range into small bins to convert the problem into a classification problem, and use classification loss, e.g., the cross-entropy loss, to train the model.
Another branch of prior work (Niu et al., 2016; Fu et al., 2018; Cao et al., 2020; Shi et al., 2021) casts regression as an ordinal classification problem.
They design multiple ordered thresholds and use multiple binary classifiers, one for each threshold, to learn whether the target is larger than each threshold.
Past work however focuses on forcing the final predictions of the model to be close to the target and does so in an end-to-end manner.
In contrast, this paper focuses on ordering the samples in the embedding space in accordance with the label order, and it does so by first learning a representation that contrasts samples against each other, and then learning a predictor to map the representation to the prediction values.
Therefore, our method is orthogonal to prior methods for regression learning as those prior methods can be applied to training our predictor and learning the final predictions with the help of our ordered representation.
Contrastive Learning.
Contrastive learning, which aligns positive pairs and repulses negative pairs in the representation space, has demonstrated improved performance for self-supervised learning (Chen et al., 2020a;
He et al., 2020; Chen et al., 2020b; Chuang et al., 2020).
Its supervised version, named supervised contrastive learning (SupCon)
(Khosla et al., 2020), which defines samples from the same class as positive pairs and samples from different classes as negative pairs, was also shown to outperform the cross-entropy loss on image classification tasks (Khosla et al., 2020).
It is also beneficial for learning in challenging settings such as noisy labels (Li et al., 2022a), long-tailed classification (Kang et al., 2020; Li et al., 2022b) and out-of-domain detection (Zeng et al., 2021).
A couple of recent papers have used the concept of “contrastive” or contrastive learning in the context of continuous labels.
In particular, Yu et al.
(2021) learns a model for action quality assessment by regressing the relative scores between two input videos.
This work differs from ours in that it does not use the standard contrastive learning framework.
Wang et al.
(2022) proposes to improve domain adaptation for gaze estimation by adding a contrastive loss term to the L1 loss.
They show that their approach is beneficial for adapting a gaze estimation model from one dataset (i.e., one domain) to another, but the approach produces no benefits and even reduces performance for the source dataset.
In contrast, our approach improves performance for the source dataset as opposed to being specific to domain adaptation.
Besides, Dufumier et al.
(2021a;b) use a contrastive loss reweighted by continuous meta-data for classification.
Schneider et al.
(2022) learns low-dimensional and interpretable embeddings to encode behavioral and neural data using a generalized contrastive loss which samples positives and negatives according to continuous behavior or time labels.
However, the continuous meta-data in (Dufumier et al., 2021a;b) and the continuous labels in (Schneider et al., 2022) are not the targets in their tasks and they are not working on regression problems.
M ETHOD
For a regression task, we aim to train a neural network composed of a feature encoder f ( · ) :
X → R d e and a predictor p ( · ) : R d e → R d t to predict the target y ∈ R d t based on the input data x ∈ X .
Given an input batch of data, similar to contrastive learning methods (Chen et al., 2020a; Khosla et al., 2020), we first apply data augmentation twice to obtain two views of the batch.
The two views are fed into the encoder f ( · ) to obtain a d e -dimensional feature embedding for each augmented input data.
Our supervised contrastive regression loss, L SupCR , is computed on the feature embeddings.
To use the learned representation for regression, we freeze the encoder f ( · ) and train the predictor on top of it using a regression loss (e.g., L1 loss).
3. S UPERVISED C ONTRASTIVE R EGRESSION L OSS
We would like our loss function to ensure distances in the embedding space are ordered according to distances in the label space.
But, how can we design a contrastive loss that delivers this property?
Below, we first define our loss function, then explain it in the context of positive and negative contrastive samples, and finally provide a theoretical analysis to prove that as our loss approaches its minimum, the learned representation will be ordered according to label distances.
For a positive integer I , let [ I ] denote the set { 1 , 2 , · · · , I } .
Given a randomly sampled batch of
N input and label pairs { ( x n , y n ) } n ∈ [ N ] , we apply data augmentation to the batch and obtain a two-view batch { (˜ x ℓ , ˜ y ℓ ) } ℓ ∈ [2 N ] , where
˜ x 2 n = t ( x n ) , ˜ x 2 n − 1 = t ′ ( x n ) ( t and t ′ are independently sampled data augmentations operations) and ˜ y 2 n = ˜
y 2 n − 1 = y n , ∀ n ∈ [ N ] .
The augmented batch is fed into the encoder f ( · ) to obtain v l = f (˜ x l ) ∈ R d e , ∀ l ∈ [2 N ] .
Our supervised contrastive regression loss is then computed over the feature embeddings { v l } l ∈ [2 N ] as:
[Equation]
where sim( · , · ) is the function measuring the similarity between two feature embeddings (e.g., negative L2 norm), d ( · , · ) is the function measuring the distance between two labels (e.g., L distance), 1 [ · ] ∈ { 0 , 1 } is an indicator function evaluating to 1 iff the conditions in the square brackets are satisfied, and τ is the temperature parameter.
At a high level, our loss can be explained in the context of positive/negative pairs, commonly used in contrastive learning.
Contrastive learning and SupCon focus on classification.
Hence, positive samples are those that come from the same class or the same input image, and all other samples are negative samples.
In regression problems, there are no classes but continuous labels.
Any two [Figure] [Figure] [Figure]
[Figure] [Figure]
[Figure]
samples can be thought of as a positive pair or a negative pair, depending on context.
To exploit the inherent continuity underlying the labels, we define positive and negative samples in a relative way, based on their label distance to the anchor. In particular, we rank samples with respect to their label distance to the anchor.
We then contrast the anchor to the sample in the batch that is closest in the label space, which will be treated as the positive sample.
All other samples are farther from the anchor and should be negative samples.
We also contrast the anchor to the second closest sample in the batch.
In this case, only samples in the batch that are farther from the positive sample (i.e. samples whose ranks is three or higher) can act as negative samples.
We continue this process for higher rank samples (the third closest, fourth closest, etc.), and anchors in a batch.
Figure 2(a) shows an example batch, and Figure 2(b) shows two positive pairs and their corresponding negative pair(s).
Specifically, for any anchor sample i , any other sample j in the same batch can be used to create a positive pair with corresponding negative samples set to all samples in the batch whose labels differ from i ’s label by more than the label of j .
In other words, L SupCR aims to force the feature similarity between i and j greater than the feature similarity between i and any other sample k in the batch if the label distance between i and k is greater than i and j .
Therefore, optimizing
L SupCR will make the feature embedding ordered according to the order in the label space.
The design of L SupCR sufficiently leverages the ordered relationships between samples reflected in the continuous label space of regression problems, which is also the intrinsic difference between regression problems and classification problems.
3. T HEORETICAL A NALYSIS
In this section, we analytically show that optimizing L SupCR will make the feature embedding ordered according to the order in the label space.
To show this, we first derive a lower bound of L SupCR , and show L SupCR can be arbitrarily close to it.
Next, we formalize the concept of δ -ordering which refers to the feature embeddings being ordered according to the order in the label space, and show that as L SupCR gets sufficiently close to the lower bound, the feature embeddings will be δ -ordered.
All proofs are in Appendix A.
Notations.
Let s i,j := sim( v i , v j ) /τ , ∀ i, j ∈ [2 N ] , d i,j :=
d (˜ y
i , ˜ y j ) , ∀ i, j ∈ [2 N ] , D i, 1 <
D i, 2 < · · · < D i,M i be the sorted label distances starting from the i -th sample, i.e., sort ( { d i,j | j ∈ [2 N ] \{ i }} ) , ∀ i ∈ [2 N ] , and n i,m := |{ j | d i,j = D i,m , j ∈ [2 N ] \{ i }}| be the number of samples whose distance from the i -th sample equals D i,m , ∀ i ∈ [2 N ] , m ∈
[ M i ] .
First, we show that L
[Equation]
Theorem 1 (Lower bound of L SupCR ) .
L ⋆ is a lower bound of L SupCR , i.e., L SupCR > L ⋆ .
Next, we show that L SupCR can be arbitrarily close to its lower bound L ⋆ .
Theorem 2 (Lower bound tightness) .
For any ϵ > 0 , there exists a set of feature embeddings such that L SupCR <
L ⋆ + ϵ .
Then, we define a property called δ -ordered for the feature embeddings { v l } l ∈ [2 N ] .
Definition 1 ( δ -ordered feature embedddings) .
For any 0 < δ < 1 , the feature embeddings { v l } l ∈ [2 N ] are δ -ordered if ∀ i ∈ [2 N ] , j, k ∈ [2 N ] \{ i } ,
[Equation]
It says that a δ -ordered set of feature embeddings satisfies the following properties: First, for all j and k such that d i,j = d i,k , the gap between s i,j and s i,k should be smaller than δ ; Second, for all j and k such that d i,j < d i,k , s i,j should be larger than s i,k by at least 1 δ .
Notice that 1 δ > δ , which means the feature similarity gap between a sample pair with different label distance to the anchor is always larger than the feature similarity gap between a sample pair with equal label distance to the anchor.
Finally, we show that for any 0 < δ < 1 , when L SupCR is close enough to its lower bound L ⋆ , the feature embeddings will be δ -ordered.
Theorem 3 (Main theorem) .
For any 0 < δ < 1 , there exist ϵ > 0 , such that
if L SupCR <
L ⋆ + ϵ , then the feature embeddings are δ -ordered.
E XPERIMENTS
In this section, we evaluate our proposed method empirically.
We first benchmark our method and compare it with the state-of-the-art regression baselines.
Then, we evaluate the desirable properties of our learned representations, including the robustness to data corruptions, the resilience to reduced training data, the performance on transfer learning, and the generalization ability to unseen targets.
Finally, we analyze and evaluate the design variants for our method.
Datasets.
We benchmark our method on five regression datasets for common real-world tasks in computer vision, human-computer interaction, and healthcare.
1) AgeDB (Moschoglou et al., 2017) is used for predicting age from face images.
It contains 16, in-the-wild images of celebrities and the corresponding age labels.
The age range is between 0 and 101.
It is split into a 12,208-image training set, a 2140-image validation set and a 2140-image test set.
2) TUAB (Obeid & Picone, 2016; Engemann et al., 2022) is used for brain-age estimation from EEG resting-state signals.
The dataset comes from EEG exams at the Temple University Hospital in Philadelphia.
Following Engemann et al.
(2022), we use only the non-pathological subjects, so that we may consider their chronological age as their brain-age label.
The dataset includes 1, 21-channel EEG signals sampled at 200Hz from individuals whose age ranges from 0 to 95.
It is split into a 1,246-subject training set and a 139-subject test set.
3) MPIIFaceGaze (Zhang et al., 2017a;b) is used for gaze direction estimation from face images.
It contains 213,659 face images collected from 15 participants during their natural everyday laptop use.
We subsample and split it into a 33,000-image training set, a 6,000-image validation set and a 6000image test set with no overlapping participants.
The gaze direction is described as a 2-dimensional vector with the pitch angle in the first dimension and the yaw angle in the second dimension.
The range of the pitch angle is -40° to 10° and the range of the yaw angle is -45° to 45°.
4) SkyFinder (Mihail et al., 2016;
Chu et al., 2018) is used for temperature prediction from outdoor webcam images.
It contains 35,417 images captured by 44 cameras around 11am on each day under a wide range of weather and illumination conditions.
The temperature range is − 20 ◦ C to 49 ◦ C .
It is split into a 28,373-image training set, a 3,522-image validation set and a 3,522-image test set.
5) IMDB-WIKI (Rothe et al., 2015) is a large dataset for predicting age from face images, which contains 523,051 celebrities images and the corresponding age labels.
The age range is between 0 and 186 (some images are mislabeled).
We use this dataset to test our method’s resilience to reduced training data, performance on transfer learning, and ability to generalize to unseen targets.
We subsample the dataset to create a variable size training set, and keep the validation set and test set unchanged with 11,022 images in each.
[Table] [Table]
[Table] [Table]
Metrics.
We report two kinds of metrics: the prediction error and coefficient of determination (R 2 ).
Prediction errors have practical meaning and are convenient for interpretation, while R 2 quantifies how much the model outperform a dummy regressor that always predicts the mean value of the training labels.
For age, brain-age, and temperature, the mean absolute error (MAE) is reported as the prediction error.
For gaze direction, the angular error is reported as the prediction error.
Baselines.
We implemented seven typical regression methods as baselines.
L1, MSE and H UBER have the model directly predict the target value and train the model with an error-based loss function, where L1 uses the mean absolute error, MSE uses the mean squared error and H UBER uses an MSE term when the error is below a threshold and an L1 term otherwise.
DEX (Rothe et al., 2015) and
DLDL- V 2 (Gao et al., 2018) divide the regression range of each label dimension into several bins and learn the probability distribution over the bins.
DEX (Rothe et al., 2015) optimizes a cross-entropy loss between the predicted distribution and the one-hot ground-truth labels, while DLDL- V 2 (Gao et al., 2018) jointly optimizes a KL loss between the predicted distribution and a normal distribution centered at the ground-truth value, as well as an L1 loss between the expectation of the predicted distribution and the ground-truth value.
During inference, they output the expectation of the predicted distribution for each label dimension.
OR (Niu et al., 2016) and CORN (Shi et al., 2021) design multiple ordered threshold for each label dimension, and learn a binary classifier for each threshold.
OR (Niu et al., 2016) optimizes a binary cross-entropy loss for each binary classifier to learn whether the target value is larger than each threshold, while CORN (Shi et al., 2021) learns whether the target value is larger than each threshold conditioning on it is larger than the previous threshold.
During inference, they aggregate all binary classification results to produce the final results.
Experiment Settings.
We use ResNet-18 (He et al., 2016) as the backbone model for AgeDB ,
IMDB-WIKI , MPIIFaceGaze and SkyFinder datasets.
For TUAB , we use a 24-layer 1D ResNet (He et al., 2016) as the backbone model to process the EEG signals.
We use the linear regressor as the predictor.
We use the SGD optimizer and cosine learning rate annealing (Loshchilov & Hutter, 2016) for training.
The batch size is set to 256.
We pick the best learning rates and weight decays by grid search for both our method and the baselines.
We train all the baselines and the encoder of our method for 400 epochs, and the linear regressor of our method for 100 epochs.
The same data augmentations are adopted for all baselines and our method: random crop and resize (with random flip), color distortions for AgeDB , IMDB
4. M AIN R ESULTS
As explained earlier, our model learns a regression-suitable representation that can be used by any of the baselines.
Thus, in our comparison, we first train the encoder with the proposed L SupCR , and then freeze the encoder and train a predictor on top of it using each of the baseline methods.
We then compare the original baseline without our representation, to the same baseline with our representation.
For example, L1 means training the encoder and predictor end-to-end with L1 loss and S UP CR(L1) means training the encoder with L SupCR and then training the predictor with L1 loss.
Table 1, Table 2, Table 3 and Table 4 show the evaluation results on AgeDB , TUAB , MPIIFaceGaze and
SkyFinder , respectively.
Green numbers highlight the performance gains brought by using our representation and the best numbers are shown in bold.
As all tables indicate, S UP CR consistently achieves the best performance on both metrics across all datasets.
Moreover, incorporating S UP CR to learn the representation consistently reduces the prediction error of all baselines by 5.8%, 10.1%, 11.7% and 7.0% on average on the AgeDB , TUAB , MPIIFaceGaze and SkyFinder , respectively.
4. R OBUSTNESS TO D ATA C ORRUPTIONS
[Figure]
It is well known that deep neural networks are highly vulnerable to out of distribution data and different forms of corruptions (Hendrycks & Dietterich, 2019), such as noise, blur and color distortions.
Using the corruption generation process in ImageNet-C benchmark (Hendrycks & Dietterich, 2019), we corrupt the
AgeDB test set with 19 diverse corruption types under different severity levels.
In Figure 3, we compare
S UP CR(L1) with L1 by testing the model (trained on clean AgeDB training set) on corrupted test set and reporting the average mean absolute error over all types of corruptions for each severity level.
The results show that the representation learned by S UP CR is more robust to unforeseen data corruptions, and S UP CR suffers from lesser performance degradation as corruption severity increases.
4. R ESILIENCE TO R EDUCED T RAINING D ATA
32k 16k 8k 4k 2k
M ean
A b s o l u t e E rr o r
( M
AE )
Number of Training Samples
L
SupCR(L1)
Much of the success of modern deep learning is due to the availability of massive training datasets.
However, in many real-world settings, collecting a large training set is often infeasible due to the time and cost of labeling.
Thus, it is desirable to improve model resilience to reduced training data.
In Figure 4, we subsample IMDB-WIKI to produce varying size training sets, and compare S UP CR(L1) with L1.
The results show that S UP CR is more resilient to reduced training data, and exhibits lesser performance degradation as the number of training samples decreases.
4. T RANSFER L EARNING
[Table]
[Table]
We evaluate the transfer learning performance by first pre-training the feature encoder on a large dataset and then using either linear probing (fixed encoder) or fine-tuning to learn a predictor on a small dataset.
We investigate two scenarios: transferring from AgeDB which contains ∼ 12k samples to a subsampled IMDB-WIKI of 2k samples and transferring from another subsampled
IMDB-WIKI of 32k samples to AgeDB .
As shown in Table 5, S UP CR(L1) outperforms L1 in both the linear probing and fine-tuning settings in both two scenarios.
4. G ENERALIZATION TO U NSEEN T ARGETS
[Table]
In real-world regression tasks, it is common that some targets are unseen during training.
As in Yang et al.
(2021), we curate two subset of
IMDB-WIKI that contain unseen targets.
The test set is a uniform distribution across the whole target range.
Table 6 shows the label distributions of the two training set, where pink shading indicates regions of unseen targets, and the blue shading represents the distribution of seen targets.
The first training set consists of 3, training samples with a bi-modal Gaussian distribution over the target space, and the second training set consists of 3,662 training samples with a tri-modal Gaussian distribution over the target space.
We report the prediction error on the seen and unseen targets separately.
The results show that S UP CR (L1) outperforms L1 by a larger margin on the unseen targets without sacrificing the performance on the seen targets.
4. D ESIGN V ARIANTS
Similarity Measure and Projection Head.
We explore potential variants of our method to pick the best design.
Table 7 compares the performance of our method ( L SupCR with similarity measure using the negative L2 norm) with alternative designs that use a different similarity measure and potentially support a projection head.
The results are on the AgeDB dataset and using S UP CR(L1).
Specifically, the table shows the performance of using different feature similarity measures sim( · , · ) .
We see that among cosine similarity, i.e., sim( v i , v j ) =
[Equation]
negative
[Equation]
[Table]
L2 norm delivers the best performance.
The table also shows the performance with and without a projection head.
To train the encoder network f ( · ) : X → R d e , one can involve a projection head h ( · ) : R d e → R d p during training to calculate the loss on { h ( f (˜ x i ))
} i =1 ∈ [2 N ] , and then discard the projection head to use f ( x i ) for inference.
One can also directly calculate the loss on { f (˜ x i ) } i ∈
[2 N ] and then use f ( x i ) for inference.
S IM CLR (Chen et al., 2020a) and S UP C ON (Khosla et al., 2020)
both use the former approach.
For L SimCLR , the results show that adding the projection head can benefit the regression performance.
This is because L SimCLR aims to extract features that are invariant to data augmentations and can remove information that may be useful for the downstream regression task.
For L SupCon and L SupCR , however, it is better to train the encoder without the projection head, since both L SupCon and L SupCR leverage the label information to extract features that directly target on the downstream task.
Moreover, we see that using L SupCon or L SupCR as the loss function delivers better performance than using L SimCLR , verifying that it is helpful to utilize label information during encoder training.
We also see that L SupCR outperforms L SupCon by a large margin, highlighting the superiority of L SupCR which explicitly leverages the ordered relationships between samples for regression problems.
[Table]
Training Scheme.
There are usually three schemes to train the feature encoder:
(1) Linear probing : first trains the feature encoder using the representation learning loss, then freezes the encoder and trains a linear regressor on top of it using a regression loss.
(2) Fine-tuning : first trains the feature encoder using the representation learning loss, then fine-tunes the whole model using a regression loss.
(3) Regularization : trains the whole model while jointly optimizing the representation learning loss and the regression loss.
Table 8 shows the performance on AgeDB for the three schemes using L SupCR as the representation learning loss and L1 as the regression loss.
We see that all of the three schemes can improve performance over using the regression loss L1 alone to train a model.
Further, unlike classification problems where fine-tuning often delivers the best performance, freezing the feature encoder performs the best.
This is because in the case of regression, back-propagating the L1 loss to the representation can destroy the order in the embedding space learned by L SupCR , which leads to poorer performance.
C ONCLUSION
In this paper, we propose Supervised Contrastive Regression (SupCR), a framework that learns a regression-aware representation by contrasting samples against each other according to their target distance.
The proposed framework is orthogonal to existing regression models, and can be used in combination with such models to improve performance.
It achieves the state-of-the-art performance and consistently improves prior regression baselines across different datasets, tasks, and input modalities.
It also improves the robustness to data corruptions, resilience to reduced training data, performance on transfer learning, and generalization to unseen targets.
A P ROOFS
Theorem 1 (Lower bound of L SupCR ) .
L ⋆ is a lower bound of L SupCR , i.e., L SupCR > L ⋆ .
Proof
L SupCR =
[Equation]
∀ i ∈
[2 N ] , m ∈
[ M i ] , from Jensen’s Inequality we have
[Equation]
Thus
[Equation]
by plugging Equation 3 into Equation
[Equation]
we have
[Equation]
Theorem 2 (Lower bound tightness) .
For any ϵ > 0 , there exists a set of feature embeddings such that L SupCR <
L ⋆ + ϵ .
Proof.
We will show ∀ ϵ > 0 , there is a set of feature embeddings where
[Equation] ,k if d i,j = d i,k and γ := log 2 N min i ∈ [2 N ] ,m ∈
[ Mi ] n i,m ϵ , ∀ i ∈ [2 N ] , j, k ∈ [2 N ] \{ i } , such that L SupCR <
L ⋆ + ϵ .
For such a set of feature embeddings, ∀ i ∈ [2 N ] , m ∈ [ M i ] , j ∈ { j ∈ [2 N ] \{ i } | d i,j
[Equation]
D i,m } ,
[Equation]
i
[Equation]
since s i,k = s i,j for all k such that d i,k = D i,m = d i,j
[Equation]
and
[Equation]
since s i,k − s i,j < − γ for all k such that d i,k > D i,m = d i,j and s i,k − s i,j = 0 for all k such that d i,k
= D i,m = d i,j .
From Equation 2 we have
L SupCR = −
[Equation]
By plugging Equation 5 and 6 into Equation 7 we have
[Equation]
Theorem 3 (Main theorem) .
For any 0 < δ < 1 , there exist ϵ > 0 , such that
if L SupCR <
L ⋆ + ϵ , then the feature embeddings are δ -ordered.
Proof.
We will show ∀ 0 < δ < 1 , there is a
[Equation]
such that when L SupCR <
L ⋆ + ϵ , the feature embeddings are δ -ordered.
We first show that | s i,j − s i,k | < δ if d i,j = d i,k , ∀ i ∈ [2 N ] , j, k ∈ [2 N ] \{ i } when L SupCR <
L ⋆ + ϵ .
From Equation 2 we have
[Equation]
Let p i,m :=
[Equation]
∀ i ∈ [
[Equation]
N ] , m ∈ [ M i ] , by splitting out the maximum term and the minimum term we have
[Equation]
Let θ i,m :=
[Equation]
we have
[Equation]
and
[Equation]
Then, from Jensen’s inequality we know exp
[Equation]
thus − log [Equation] [Equation] By plugging Equation 11, 12 and 14 into Equation 10, we have
L SupCR >
[Equation]
Let h ( θ ) : = n i,m log(1+exp( η i,m )+( n i,m − 2) θ )
− η i,m − ( n i,m − 2) log( θ ) .
From derivative analysis we know h ( θ ) decreases monotonically when θ ∈ (cid:104) 1 , 1+exp( η i,m )(cid:105) and increases monotonically when θ ∈ (cid:104) 1+exp( η i,m ), exp( η i,m ) (cid:105) , thus
h
[Equation]
[Equation]
By plugging Equation 16 into Equation 15, we have
[Equation]
) − η i,
[Equation]
(cid:19) .
Then, since η i,m ≥ 0 , we have 2 log 1+exp( η i,m )− η i,m ≥ 0 .
Thus, ∀ i ∈ [2 N ] , m ∈
[ M i ] ,
L SupCR > L ⋆
[Equation]
If L SupCR < L ⋆ + ϵ ≤ L ⋆ +
[Equation]
then
[Equation]
Since y ( x ) = 2 log 1+exp( x )− x increases monotonically when x > 0 , we have η i,m < δ .
Hence, ∀ i ∈ [2 N ] , j, k ∈ [2 N ] \{ i } , if d i,j = d i,k
= D i,m , | s i,j − s i,k | ≤ η i,m < δ .
Next, we show s i,j > s i,k
+ δ if d i,j < d i,k when L SupCR < L ⋆ + ϵ .
From Equation 2 we have
L SupCR = −
[Equation]
and combining it with Equation 3 we have
L SupCR ≥ L ⋆ +
[Equation]
> L ⋆
[Equation]
[Equation]
∀ i ∈ [2 N ] , j ∈ [2 N ] \{ i } , k ∈ { k ∈ [2 N ] \{ i }
| d i,j < d i,k } .
When L SupCR < L ⋆ + ϵ , we already have | s i,l − s i,j | < δ , ∀ d i,l = d i,j , which derives s i,l − s i,j < δ and thus exp( s i,
l − s i,j ) < exp( δ ) .
By putting this into Equation 20, we have ∀ i ∈ [2 N ] , j ∈ [ [Equation] [Equation]
[Equation]
22) where r i,j ∈ [ M i ] is the index such that D i,r i,j = d i,j .
Further, given L SupCR < L ⋆ +
[Equation]
<
[Equation]
+ [Equation] +
[Equation]
) , we have
[Equation] [Equation]
which derives s [Equation] ,j < d i,k } .
Finally, ∀ i ∈ [2 N ] , j, k ∈ [2 N ] \{ i } , s i,j < s i,k − 1 δ if d i,j > d i,k directly follows from s i,j > s i,k + 1 δ if d i,j < d i,k . B V ISUALIZATION OF D ATA A UGMENTATION Figure B shows examples of original and augmented data samples on each dataset. For AgeDB and
SkyFinder , random crop and resize (with random horizontal flip), color distortions are used as data augmentation; For TUAB , random crop is used as data augmentation; For MPIIFaceGaze , random crop and resize (without random horizontal flip), color distortions are used as data augmentation.
C D ETAILED E XPERIMENT S ETTINGS
For the encoder training of our method and regression learning baselines, we selected the best learning rates and weight decays for each dataset by grid search, with a grid of learning rates from { 0 . 01 , 0 . 05 , 0 . 1 , 0 . 2 , 0 . 5 , 1 . 0 } and weight decays from { 10 − 6 , 10 − 5 , 10 − 4 , 10 − 3 } .
For the predictor training of our method, we adopted the same search setting above except for adding no weight decay to the search choices of weight decays.
For temperature parameter τ , we searched from { 0 . 1 , 0 . 2 , 0 . 5 , 1 . 0 , 2 . 0 , 5 . 0 } and selected the best, which is 2.0.
…… 21 c hanne l s …… ……
Original
Augmented …… ……
A ge D
B
T U AB
M P II F a c e G a z e
S ky F i nde r
For the classification-based baselines, we divided the regression range into small bins.
For AgeDB , the target range is 0 ∼ 101 and the bin size is set to 1; for TUAB , the target range is 0 ∼ 95 and the bin size is set to 1; for MPIIFaceGaze , the target range is -40 ∼ 10 for the pitch angle and -45 ∼for the yaw angle, and the bin size is set to 0.5 for the pitch angle and is set to 1 for the yaw angle.
For SkyFinder , the target range is -20 ∼ 49 and the bin size is set to 1.
For the implementation of
SupCon (Khosla et al., 2020) on regression datasets, the regression range is divided into small bins in the same way and each bin is regarded as a class.
Samples from the same class, i.e., sharing the same target bin as the anchor, are considered as positive samples.