L EARNING L ABEL E NCODINGS FOR D EEP R EGRESSION
A BSTRACT
Deep regression networks are widely used to tackle the problem of predicting a continuous value for a given input.
Task-specialized approaches for training regression networks have shown significant improvement over generic approaches, such as direct regression.
More recently, a generic approach based on regression by binary classification using binary-encoded labels has shown significant improvement over direct regression.
The space of label encodings for regression is large.
Lacking heretofore have been automated approaches to find a good label encoding for a given application.
This paper introduces Regularized Label Encoding Learning (RLEL) for end-to-end training of an entire network and its label encoding.
RLEL provides a generic approach for tackling regression.
Underlying RLEL is our observation that the search space of label encodings can be constrained and efficiently explored by using a continuous search space of real-valued label encodings combined with a regularization function designed to encourage encodings with certain properties.
These properties balance the probability of classification error in individual bits against error correction capability.
Label encodings found by
RLEL result in lower or comparable errors to manually designed label encodings.
Applying RLEL results in 10 .
9% and 12 .
4% improvement in Mean Absolute Error (MAE) over direct regression and multiclass classification, respectively.
Our evaluation demonstrates that RLEL can be combined with off-the-shelf feature extractors and is suitable across different architectures, datasets, and tasks.
Code is available at https://github.com/ubc-aamodt-group/RLEL_regression .
I NTRODUCTION
Deep regression is an important problem with applications in several fields, including robotics and autonomous vehicles.
Recently, neural radiance fields (NeRF) regression networks have shown promising results in novel view synthesis, 3D reconstruction, and scene representation (Liu et al., 2020; Yu et al., 2021).
However, a typical generic approach to direct regression, in which the network is trained by minimizing the mean squared or absolute error between targets and predictions, performs poorly compared to task-specialized approaches (Yang et al., 2018;
Ruiz et al., 2018;
Niu et al., 2016;
Fu et al., 2018).
Recently, generic approaches based on regression by binary classification have shown significant improvement over direct regression using custom-designed label encodings (Shah et al., 2022).
In this approach, a real-valued label is quantized and converted to an M -bit binary code, and these binary-encoded labels are used to train M binary classifiers.
In the prediction phase, the output code of classifiers is converted to real-valued prediction using a decoding function.
Moreover, binary-encoded labels have been proposed for ordinal regression (Li & Lin, 2006; Niu et al., 2016) and multiclass classification (Allwein et al., 2001; Cissé et al., 2012).
The use of binary-encoded labels for regression has multiple advantages.
Additionally, predicting a set of values (e.g., classifiers’ output) instead of one value (direct regression) introduces ensemble diversity, which improves accuracy (Song et al., 2021).
Furthermore, encoded labels introduce redundancy in the label presentation, which improves error correcting capability and accuracy (Dietterich & Bakiri, 1995).
Finding suitable label encoding for a given problem is challenging due to the vast design space.
Related work on ordinal regression has primarily leveraged unary codes (Li & Lin, 2006; Niu et al., 2016; Fu et al., 2018).
Different approaches for label encoding design, including autoencoder, random search, and simulated annealing, have been proposed to design suitable encoding for multiclass classification (Cissé et al., 2012;
Dietterich & Bakiri, 1995; Song et al., 2021).
However, these encodings
[Table]
[Figure] [Table] [Figure] perform relatively poorly for regression due to differences in task objectives (Section 2).
More recently, Shah et al.
(2022) analyzed and proposed properties of suitable encodings for regression.
They empirically demonstrated the effectiveness of manually designed encodings guided by these properties.
While establishing the benefits of exploring the space of label encodings for a given task, they did not provide an automated approach to do so.
[-L a b e l 0 2 4 6 8 10 12 14 16 18 Bit 
-]
In this work we propose Regularized Label Encoding Learning (RLEL) , an end-to-end approach to train the network and label encoding together.
Binary-encoded labels have discrete search space.
This work proposes to relax the assumption of using discrete search space for label encodings.
Label encoding design can be approached by regularized search through a continuous space of real-valued label encodings, enabling the use of continuous optimization approaches.
Such a formulation enables end-to-end learning of the network parameters and label encoding.
We propose two regularization functions to encourage certain properties in the learned label encoding during training.
Specifically, while operating on real-valued label encoding, the regularization functions employed by RLEL are designed to encourage properties previously identified as being helpful for binary-valued label encodings (Shah et al., 2022).
The first property encourages the distance between learned encoded labels to be proportional to the difference between corresponding label values, which reduces the regression error.
Further, each bit of label encoding can be considered a binary classifier.
The second property proposes to reduce the complexity of a binary classifier’s decision boundary by reducing the number of bit transitions ( 0 → 1 and 1 → 0 transitions in the classifier’s target over the range of labels) in the corresponding bit in binarized label encoding.
Figure 1 demonstrates the effect of proposed regularizers on the learned label encodings and regression errors.
Figure 1a plots the L1 distance between learned encodings for different target labels versus the difference between corresponding label values.
The L1 distance between encodings for distant targets is low without the regularizer.
In contrast, the proposed regularizer encourages the learned label encoding to follow the first design property.
Figure 1b plots learned label encoding (binarized representation for clarity).
Each row represents encoding for a target value, and each column represents a classifier’s output over the range of target labels.
The use of regularizer R2 reduces the number of bit-transitions (i.e., 1 → 0 and
→ 1 transitions in a column) to enforce the second design property and consequently reduces the regression error.
We demonstrate that the regularization approach employed by RLEL encourages the desired properties in the label encodings.
We evaluate the proposed approach on 11 benchmarks, covering diverse datasets, network architectures, and regression tasks, such as head pose estimation, facial landmark detection, age estimation, and autonomous driving.
Label encodings found by RLEL result in lower or comparable errors to manually designed codes and outperform generic encoding design approaches (Gamal et al., 1987; Cissé et al., 2012; Shah et al., 2022).
Further, RLEL results in lower error than direct regression and multiclass classification by 10 . 9% and 12 .
4% , respectively, and even outperforms several task-specialized approaches.
We make the following contributions to this work:
[Figure] [Figure] [Figure] [Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure] [Figure] [Figure] [Figure]
[Figure]
encoding
[Figure]
[Figure]
• We provide an efficient label encoding design approach by combining regularizers with continuous search space of label encodings.
[-Hadamard encoding Target Label Binary-encoded Label B 4 B 3 B 2 B 0 0 0 01 0 0 01 1 0 01 1 1 01 1 1 10 1 1 10 0 1 10 0 0(c) Johnson encoding Target B 
-]• We analyze properties of suitable encodings in the continuous search space and propose regularization functions for end-to-end learning of network parameters and label encoding.
• We evaluate the proposed approach on 11 benchmarks and show significant improvement over different encoding design methods and generic regression approaches.
B ACKGROUND AND R ELATED W ORK
This section summarizes relevant background information on regression by binary classification approach and different code design approaches.
Task-specific regression approaches are summarized in Appendix A.3.
However, a generic regression approach applicable to all tasks is desirable.
2. R EGRESSION BY B INARY C LASSIFICATION
A regression problem can be converted to a set of binary classification subproblems.
Prior works proposed to use N binary classifiers for scaled and quantized target labels ∈ { 1 , 2 , ..., N } (Niu et al., 2016; Fu et al., 2018).
Here, classifier- k ’s target output is 1 if the target label is greater than k , else 0 .
Figure 2a represents the target output of binary classifiers for this setup.
Shah et al.
(2022) proposed Binary-encoded Labels (BEL) , a generalized framework for regression by binary classification.
In the proposed approach, a real-valued target label is quantized and converted to a binary code B of length M using an encoding function E .
M binary classifiers are trained using binary-encoded target labels B ∈ { 0 , 1 } M .
During inference, the output of binary classifiers, i.e., predicted code, is converted to the real-valued prediction using a decoding function D .
Using encoded labels introduces error-correction capability, i.e., tolerance to classification error.
Hamming distance between two codes (number of differing bits) gives a measure of error-correction capability.
Error-correcting codes, such as Hadamard codes (Figure 2b), have been proposed to encode labels in multiclass classification (Dietterich & Bakiri, 1995; Verma & Swami, 2019).
BEL showed that such codes are not suitable for regression due to differences in task objectives and classifiers’ error probability distribution, and proposed properties of suitable codes for regression.
The first property suggests a trade-off between classification errors and error correction properties.
Each classifier learns a decision boundary for bit transitions from 1 → 0 and 0 → 1 in the classifier’s target bit over the numeric range of labels (green lines within a column in Figure 2).
For example, in
Johnson encoding (Libaw & Craig, 1953) (Figure 2c), the classifier for bit B 3 learns two decision boundaries for bit transitions in intervals (2 , 3) and (6 , 7) .
The number of intervals for which the classifier has to learn a separate decision boundary increases with bit transitions, increasing its complexity.
Hadamard codes have excellent error-correction properties but have several bit transitions (Figure 2b); this increases the complexity of a classifier’s decision boundary and reduces its classification accuracy compared to unary and Johnson codes (Figure 2a and Figure 2c). Rahaman et al.
(2019) introduced the term spectral bias and demonstrated that neural networks prioritize learning low-frequency functions (i.e., lower local fluctuations).
The spectral bias of neural networks provides insights into accuracy improvement with the reduction in the number of bit transitions.
Second, Hamming distance between two codes should increase with the difference between corresponding label values to reduce the probability of making large absolute errors between predicted and target labels.
The probability that erroneous predicted code for label X will be decoded as Y decreases as the hamming distance between codes for values X and Y increases.
Thus the above rule reduces the regression error.
Last, the encoding design should also consider the error probability of classifiers.
BEL shows that the error probability of classifiers is not uniform for regression and increases near bit transitions, as shown for classifier B 1 in Figure 2d.
Here, the probability of predicting 8 for target label 1 is very low, as the bit differing between corresponding codes (B 1 ) has a very low classification error probability.
BEL shows that this property can be exploited to design better codes for regression.
These three factors significantly affect the suitability of encodings.
BEL demonstrates that simple codes sampled based on these properties, such as unary or Johnson code, result in lower errors than widely used error-correcting Hadamard code.
2. E NCODING DESIGN
Encoding design is a well-studied problem with applications in several fields.
Iterative approaches, such as simulated annealing or random walk, have been proposed for code design (Dietterich &
Bakiri, 1995; Song et al., 2021).
However, iterative approaches are computationally expensive as each iteration requires full/partial training of the network to measure the error for sample encodings.
Works on multiclass classification using binary classifiers have demonstrated the effectiveness of error-correcting codes such as Hadamard or random codes (Verma & Swami, 2019;
Dietterich &
Bakiri, 1995).
Cissé et al.
(2012) proposed an autoencoder-based approach to design compact codes for multiclass classification problems with a large number of classes.
However, these approaches do not consider the task objective and classifiers’ nonuniform error probability distribution for regression.
Deep hashing approaches aim to find binary hashing codes for given inputs such that the hashing codes preserve the similarities in the inputs space (Luo et al., 2022;
Wang et al., 2018; Xia et al., 2014; Jin et al., 2019; Liu et al., 2016).
Deep supervised hashing approaches use the label information to design the loss function.
In deep hashing, loss functions are designed to decrease the hamming distance between binary codes for similar images (e.g., same label).
In contrast, label encoding design for regression aims to reduce the error between decoded output codes and target labels.
Further, deep hashing approaches are designed for classification datasets and do not account for the nonuniform error probability distribution of classifiers observed in regression.
As shown in prior work (Shah et al., 2022), classifiers’ nonuniform probability significantly affects the design of suitable codes for regression.
Thus, a naive adaptation of deep hashing approaches for regression problems performs poorly compared to codes designed by the proposed approach RLEL (Section A.1.5).
R EGULARIZED L ABEL E NCODING L EARNING
Regression aims to minimize the error between target labels y i and predictions
ˆ y i for a set of training samples i .
In regression by binary classification, the network learns M -bit binary-encoded labels
B i ∈ { 0 , 1 } M .
During inference, the predicted code ˆ B i is decoded to a real-valued label ˆ y i .
We propose to relax label encodings’ search space from a discrete ( { 0 , 1 } M ) to a continuous space ( R M ), enabling the use of traditional continuous optimization methods.
We propose regularizers to enable efficient search through this space.
This work automates the search for label encoding using an end-to-end training approach that learns the network parameters and label encoding together.
This section explains the proposed label encoding learning approach RLEL.
First, we explain the regression by binary classification formulation used in this work for end-to-end training of network parameters and label encoding.
Further, we introduce properties of suitable label encodings in continuous space.
Lastly, we explain the proposed regularizers and loss function that accelerate the search for label encodings by encouraging learned label encoding to exhibit the proposed properties.
3. L ABEL E NCODING L EARNING
Preliminaries:
Figure 3 represents the formulation used in this work for label encoding learning.
x i and y i represent the input and the real-valued target label for sample i , respectively.
We assume y i ∈
[1 , N ] for simplicity as the real-valued targets with any arbitrary numeric range can be scaled and shifted to this range.
Q i ∈ { 1 , 2 , ..., N } represents the quantized target label.
The input x i is [Figure] CorrelationVector
[Table]
passed through a feature extractor and fully connected (FC) layers to generate the predicted encoding ˆ
Z i ∈ R M 1 ⃝ .
Here, an FC layer of size θ ( θ < M ) is added between the feature vector and output code.
This layer reduces the number of parameters in FC layers and improves accuracy, as shown by previous work (Shah et al., 2022).
Each neuron of the output code is a binary classifier, and the magnitude ˆ Z ki gives a measure of the confidence of the classifier- k (Allwein et al., 2001).
The output code and a decoding matrix D ∈ R M × N are multiplied 2 ⃝ , and the output is passed through a softmax function to give a correlation vector
ˆ C i ∈ R N 3 ⃝ , where the value of ˆ C ki represents the probability that the predicted label ˆ y i
= k .
This correlation vector is then converted to a real-valued prediction by taking the expected value 4 ⃝ .
Table 1 summarizes the notations used in this work.
Prior works use custom-designed label encoding in this formulation.
For example, Shah et al.
(2022) proposed a series of suitable label encodings B i
=
E ( Q i ) ∈ { 0 , 1 } M .
The network can be trained using binary cross-entropy loss between B i and ˆ Z i , and these encodings are used as columns of the decoding matrix ( D : ,i =
E ( i ) ).
However, it is desirable to automatically find suitable encodings B i and decoding matrix D without searching through a set of hand-designed encodings.
The search space of binary label encodings is discrete and hence challenging to search using traditional continuous optimization methods (Darabi et al., 2018).
Hence, we relax the assumption of binarized label encodings and use a continuous search space.
This relaxation, coupled with the proposed formulation, enables the use of traditional optimizers to learn label encoding and the decoding matrix D with the entire network by optimizing the loss between targets and prediction.
Let S n represent the set of training samples with quantized target Q i = n , and
E ∈ R N
×
M represent a label encoding matrix, where each row E n, : is the encoding for target Q i = n .
E is defined as:
[Equation]
However, training the network solely with the loss between ˆ y i and y i does not constrain the search space of label encodings ( E ).
In regression, the label encoding ( E ) significantly impacts the accuracy, and label encodings that follow specific properties result in lower error (Shah et al., 2022).
The following section explains desirable characteristics of output codes for regression and how these properties can be encouraged in learned label encoding using regularization functions.
3. L ABEL E NCODING L EARNING WITH R EGULARIZERS
Section 2.1 summarizes the properties of suitable binary label encodings for regression proposed by prior works.
These properties constrain the vast search space of label encodings.
We further propose two properties applicable to real-valued label encodings to narrow its search space.
R1 - Distance between encodings: A binary classifier’s real-valued output represents its confidence (i.e., error probability).
The L1 distance between real-valued predicted encodings gives more weight to classifiers that are more confident (i.e., higher value).
Thus, by considering the L1 distance between real-valued codes instead of the hamming distance between binary codes, we can combine the second and third design properties of binary label encodings (Section 2.1) into a single rule for real-valued label encodings.
This gives the first regularization rule: L1 distance between encodings for two labels should increase with the difference between two labels , i.e., || E i, : − E j, : || 1 ∝ | i − j | .
R2 - Regularizing bit transitions: The number of bit transitions in a bit-position of label encoding gives a measure of the binary classifier’s decision boundary’s complexity.
There are no 0 → 1 or 1 → 0 transitions in real-valued label encodings.
Thus, we approximate the number of bit transitions by measuring the L1 distance between encodings for adjacent label values Q i = n and Q i = n + 1 .
The number of bit transitions for real-valued label encoding E can be approximated as:
[Equation]
This leads to the second regularization rule: The L1 distance between encodings for adjacent target label values should be regularized to find a balance between the complexity of the decision boundary and the error-correction capability of designed codes for a given benchmark.
3. L OSS F UNCTION F ORMULATION
We propose two regularizers applicable to learned label encoding ( E ) to limit its search space.
E is measured from the output codes ˆ Z i over the complete training dataset (Equation 1).
However, deep neural networks are trained using mini-batches, where each batch consists of K training examples sampled randomly from the (typically shuffled) training set.
We extend the proposed regularization rules to apply to a minibatch-based loss function.
[Equation]
: Regularizer R1 can be approximated as the following for a batch with K training examples:
[Equation]
The above regularization considers K 2 pairs in a minibatch of K samples, and penalizes a pair of training samples i and j if L1 distance between encodings ˆ Z i and ˆ Z j is less than twice the difference between corresponding label values.
The scaling parameter is set to two as it encourages at least one bit difference between two binary codes.
This encourages the L1 distance between encodings to be approximately proportional to the difference between corresponding target values.
R2:
Regularizer R2 minimizes the L1 distance between encodings of adjacent label values.
In a randomly formed minibatch consisting of only a subset of training examples, adjacent target labels might not be present.
Hence it is nontrivial to apply this regularizer to the label encoding.
However, we find that imposing this regularizer on the decoding matrix also helps with regularizing the bit transitions in the learned label encoding and can be added to the loss function irrespective of the batch formulation approach.
Theorerical analysis and empirical verification for this approach are provided in Section A.1.7 and Section 4.4.
Equation 4 represents the proposed regularizer.
[Equation]
Loss Function:
We use the cross-entropy loss between ˆ C i and soft target labels.
Here, each bit of label encoding resembles a binary classifier.
However, identifying the predicted label corresponding to the multi-bit label encoding can be treated as a multiclass classification problem.
Soft target labels are probability distributions generated using the distance between different classes.
Soft target labels can be used with cross-entropy loss and have shown improvement over typical classification loss between the correlation vector ˆ C i and quantized target label Q i or regression loss between the expected prediction ˆ y i and target label
y i for ordinal regression (Díaz & Marathe, 2019).
We use [Table] this loss function for RLEL and multiclass classification.
Complete loss function with regularizers (Equation 3 and 4) can be written as:
[Equation]
|−|| ˆ Z i − ˆ Z j || 1 ) , where ϕ j ( y i )
=
e −| j − y i | (cid:80) Nn =1 e −| n − y i | (5)
Here, the first term is the loss between target and predicted labels.
ϕ
i represents the target probability distribution generated from target
y
i .
The second and third terms are for regularizer R1 (Equation 3) and regularizer R2 (Equation 4), respectively.
A trade-off exists between the proposed desirable properties of label encodings: Encouraging one design property comes at the cost of relaxing constraints imposed by other design properties. As demonstrated by Shah et al. (2022), finding the right balance between these properties for a given benchmark is crucial to finding the best label encoding for a given problem.
Thus, these design properties can be naturally applied as regularizers, and the search for balance between different properties can be seen as tuning the regularization parameters α and β .
E VALUATION
This section first provides the experimental setup used to evaluate the proposed approach, then we compare RLEL with different label encoding design methods.
We also compare RLEL with different regression approaches to demonstrate its effectiveness as a generic regression approach.
Last, we provide an ablation study to show the impact of proposed regularizers.
4. E XPERIMENTAL S ETUP
Table 2 summarizes the regression tasks, feature extractors architecture (Figure 3), and datasets for benchmarks used for evaluation.
Selected benchmarks cover different tasks, datasets, and network architectures and have been used by prior works on regression due to the complexity of the task (Díaz & Marathe, 2019; Shah et al., 2022).
We also evaluated RLEL on facial landmark detection tasks with smaller datasets to demonstrate its generalization capability.
In this setup, a subset of training samples is used for training, whereas the complete test dataset is used to measure the test error.
Landmark-free 2D head pose estimation (LFH) takes a 2D image as input and directly finds the pose of a human head with three angles: yaw, pitch, and roll.
The facial landmark detection task focuses on finding ( x, y ) coordinates of key points in a face image.
The age estimation task is used to find a person’s age from the given face image.
In end-to-end autonomous driving, the car’s steering wheel’s angle is to be predicted for a given image of the road.
Normalized Mean Error (NME) or Mean
Absolute Error (MAE) with respect to raw real-valued labels are used as the evaluation metrics.
We compare with other encoding design approaches, including simulated annealing, autoencoder (summarized in Appendix A.2), and manually designed codes (Shah et al., 2022).
We also compare
[Table]
RLEL with generic regression approaches, such as direct regression and multiclass classification.
For direct regression, L1 or L2 loss functions with L2 regularization are used.
Label value scaling (hyperparameter) is used to change the numeric range of labels.
For multiclass classification, we use cross-entropy loss between the softmax output and target labels.
The feature extractor and regressor are trained end-to-end for all approaches.
The feature extractor architecture, data augmentation, and the number of training iterations are kept uniform across different approaches for a given benchmark.
There is no notable difference between the training time for all approaches.
The training dataset is divided into 70% training and 30% validation sets for tuning hyperparameters.
The network is trained using the full dataset after hyperparameter tuning.
We use the same values for quantization levels as prior work (Shah et al., 2022).
An average of five training runs with an error margin of 95% confidence interval is reported.
Appendix A.3 provides details on datasets, training parameters, related work (task-specific approaches), and evaluation metrics.
4. C OMPARISON OF RLEL WITH E NCODING D ESIGN A PPROACHES
Table 3 compares different encoding design approaches.
RLEL results in lower error than simulated annealing and autoencoder-based approaches for most benchmarks.
Both approaches are widely used for code design.
However, for regression tasks, the suitability of label encoding depends upon the problem, including the task, network architecture, and dataset (Shah et al., 2022).
Simulated annealing or autoencoder-based approaches do not optimize the encodings end-to-end with the regression problem. , resulting in higher error.
Furthermore, the gap between the error of learned label encoding with and without regularizers (RLEL and LEL) increases for smaller datasets, which suggests that RLEL-learned codes generalize better.
RLEL can not be used with binary-cross entropy loss for training.
We observe that for some benchmarks, the autoencoder-based approach outperforms (e.g., LFH1) as it can be used with binary-cross entropy loss.
The main objective of RLEL is to automatically learn label encoding that can reach the accuracies of manually designed codes (BEL), as using such codes is time and resource-consuming.
Hyperparameter search for RLEL can be performed by off-the-shelf hypermeter tuners/libraries without manual efforts (Li et al., 2017; Falkner et al., 2018).
In contrast, hand-designed codes need human intervention to design codes.
Also, multiple training runs are still required to find suitable codes for a given benchmark from a set of hand-designed codes.
As shown in Table 3, RLEL results in lower or comparable errors to hand-designed codes.
4. C OMPARISON OF RLEL WITH R EGRESSION A PPROACHES
RLEL is a generic regression approach that focuses on regression by binary classification and proposes a label encoding learning approach.
We compare RLEL with other generic regression approaches, including direct regression and multiclass classification as shown in Table 4.
RLEL problem formulation introduces more fully-connected layers after the feature extractor; hence, we also perform an ablation study on increasing the number of fully connected layers in Appendix A.1.4.
[Table]
[Table]
RLEL consistently lowers the error compared to direct regression and multiclass classification with 10 . 9% and 12 .
4% improvement on average.
4. A BLATION S TUDY
Figure 1a demonstrates that the use of regularizer R1 encourages the L1 distance between encodings to be proportional to the difference between target values.
The second regularizer R2 is introduced to regularize the number of bit transitions in encodings.
As mentioned in Section 3.3, we apply the regularization on the decoding matrix as it is nontrivial to apply this regularization on the output codes for randomly formed batches.
Table 5 summarizes the effect of α (i [Table]
e., the weight of R2) on the number of bit transitions in the decoding matrix and binarized/real-valued label encoding.
The second column is the number of bit transitions in the decoding matrix (Equation 4), which is used as the regularization function.
The third and fourth columns are the total number of bit transitions in binarized and real-valued encodings (Equation 2).
The table shows that the proposed regularizer on the decoding matrix also encourages fewer bit transitions in the label encoding.
Figure 1b shows the impact of regularizer R2 on learned binarized label encoding.
As pointed out by prior works (Shah et al., 2022), there is a trade-off between the error probability and error correction capability of classifiers for regression.
Hence, depending upon the benchmarks, more bit transitions can be added as the advantage of increased error correction outweighs the increase in classification error.
We observe a similar trend, where adding R2 does not improve error for some benchmarks (FLD1_s, FLD2_s), as it constrains the number of bit transitions.
C ONCLUSION
This work proposes an end-to-end approach, Regularized Label Encoding Learning, to learn label encodings for regression by binary classification setup.
We propose a combination of continuous approximation of binarized label encodings and regularization functions.
This combination enables an efficient and automated search of suitable label encoding for a given benchmark using traditional continuous optimization approaches.
The proposed regularization functions encourage label encoding learning with properties suitable for regression, and the learned label encodings generalize better, specifically for smaller datasets.
Label encodings designed by the proposed approach outperform simulated annealing- and autoencoder-designed label encodings by 12.6% and 2.1%, respectively.
RLEL-designed codes show lower or comparable errors to hand-designed codes.
RLEL reduces error on average by 10 .
9% and 12 .
4% over direct regression and multiclass classification.  A CKNOWLEDGEMENTS
This research has been funded in part by the National Sciences and Engineering Research Council of Canada (NSERC) through the NSERC strategic network on Computing Hardware for Emerging
Intelligent Sensory Applications (COHESA) and through an NSERC Strategic Project Grant.
Tor M.
Aamodt serves as a consultant for Huawei Technologies Canada Co. Ltd and recently served as a consultant for Intel Corp.
We have provided details on training hyperparameters, experimental setup, and network architectures in Appendix A.3.
Code is available at https://github.com/ ubc-aamodt-group/RLEL_regression .
We have provided the training and inference code with trained models.
Code of Ethics: Autonomous robotics and vehicles are major applications of deep regression networks.
Thus improvement of regression tasks can accelerate the progress of these fields, which may lead to some negative societal impacts such as loss of jobs, privacy, and ethical concerns.
A A PPENDIX
This supplemental material provides additional results and ablation studies (Section A.1), methodology for baseline encodings design approaches (Section A.2) and related work on task-specialized approaches and experimental setup (Section A.3) for RLEL.
Code is available at https://github. com/ubc-aamodt-group/RLEL_regression .
A. A BLATION S TUDY
Section A.1.1, Section A.1.
and Section A.1.3 provide an ablation study and supporting data on impact of proposed regularization functions and hyperparameters on label encoding learning.
Section A.1.4 covers an ablation study on the impact of the number of fully connected layers in direct regression and multiclass classification.
Section A.1.5 explains and compares deep hashing approaches (adapted for regression) with RLEL.
Section A.1.6 provides results for geometric mean and Pearson coefficient as evaluation metrics.
A.1. I MPACT OF R EGULARIZER R
(a) Without
[Figure]
) With
We proposed regularization function R1 to encourage the L1 distance between encodings to be proportional to the difference between corresponding label values.
Figure 4a and Figure 4b represent the L1 distance between pairs of learned encodings for FLD1_s benchmark without and with [Figure] [Figure] regularization, respectively.
The X -axis and Y -axis represent the label values.
Here, some columns and rows are replaced by white lines, as these label values are not present in the training dataset.
The data point at coordinates ( i, j ) represent the L1 distance between encodings for label i and j ,
i.e., || E i, : − E j, : || 1 .
For example, in Figure 4a, the L1 distance between encodings for label values 0 and 97 is ∼ 120 (light-blue coloured point at coordinate ( 0 , 97 )).
In Figure 4b, the L1 distance between encodings for label values 4 and 96 is ∼ 170 (red coloured point at coordinate ( 4 , 96 )).
The first design property (Section 3) states that the L1 distance between encodings should increase with the difference between corresponding label values.
The difference between label values for pairs of encodings increases with the distance from the diagonal of this plot.
Thus, the value of data points (i.e., the L1 distance between encodings) should increase with the distance from the diagonal of this plot.
As shown in Figure 4a, without regularization, the distance between encodings is less for faraway label values (blue-colored data points away from diagonal), which shows that learned encodings do not follow the proposed design property.
As shown in Figure 4b, the introduction of regularization function R2 remedies this and increases the L1 distance between encodings for faraway labels.
Similar observations are made for FLD2_s benchmarks, as shown in Figure 5a and Figure 5b.
[Figure]
Figure 6 plots the L1 distance between encodings versus the difference between corresponding label values for benchmarks FLD1_s and FLD2_s.
For both the benchmarks, the proposed regularizer
R1 helps enforce the first design property for real-valued label encodings and results in better label encodings with lower error (Table 3).
Effect of the scaling parameter in equation
We use the scaling parameter 2 in equation 3.
Our intuition behind using the scaling parameter 2 is based on binary-encoded labels.
For two adjacent labels (i.e., | y i − y
j
| = 1 ), the loss function encourages ||
ˆ Z i − ˆ Z j || 1 to be greater than 2 .
Here, ˆ Z [Table] is the output encodings.
In the case of binarized label encoding ( − 1 if Z < 0 and +1 if Z > 0 ), || Z i − Z j || 1 = 2 signifies that two encodings differ in at least one bit.
We also analyzed the effect of changing this parameter for two benchmarks.
Table 6 shows the impact of changing this scaling parameter for two benchmarks.
We observe that the error is higher if the scaling parameter is too low, as encodings for two adjacent labels can not be discriminated against.
If this parameter is set too high, the encoding space is more constrained and consequently the performance is degraded.
Based on this intuition and empirical verification on two benchmarks, we use the value 2 for all benchmarks.
A.1. I MPACT OF R EGULARIZER R
The regularization function R2 is proposed to reduce the number of bit transitions in the learned label encoding.
Figure 7 compares the label encodings learned for LFH1 benchmark for different values of α , where α is the weight of regularization function R2 (Equation 5).
Each row k is the encoding for label value k .
Each column k represents the output of the encoding position k for different label values.
The regularization function is proposed to decrease the transitions in an encoding bit (blue → red and red → blue) over the range of label values.
Section 4.4 provided quantitative results to demonstrate that increasing the value of α reduces the number of bit transitions.
We observe similar trends in the plots of learned label encodings shown in Figure 7; increasing the value of α decreases bit transitions in the learned label encodings and improves MAE.
A.1. E FFECT OF HYPERPARAMETERS IN RLEL
The RLEL approach introduces two hyperparameters.
We first evaluate the sensitivity to these hyperparameters to determine the complexity of hyperparameter tuning.
Figure 8 shows the NME for FLD1_s benchmark for different values of α and β values in Equation 5.
As shown in the figure, the error is not sensitive to small changes in these hyperparameters’ values, suggesting that a sparse search in the hyperparameter space suffices.
Furthermore, several approaches have been proposed for efficient hyperparameter search (Li et al., 2017; Falkner et al., 2018), and any off-the-shelf hypermeter tuners/libraries can be used to automatically find these values without manual efforts.
In contrast, hand-designed codes need human intervention to design codes.
Also, multiple training runs are still required to find suitable codes for a given benchmark from a set of hand-designed codes.
On the other hand, RLEL provides an end-to-end automated approach for label encoding learning.
A.1. I MPACT OF THE NUMBER OF FULLY - CONNECTED LAYERS :
For RLEL , we use an extra fully connected bottleneck layer in the regressor as proposed by the prior work on regression by binary classification (Shah et al., 2022).
We provide an ablation study (reproduced from (Shah et al., 2022)) to show the impact of additional fully connected layers in direct regression and multiclass classification.
Table 7 provides the error (MAE or NME) for direct regression and multiclass classification with one or two fully connected layers after the feature extractor.
As shown in the table, increasing the number of fully connected layers in direct regression and multiclass classification does not reduce the error for most benchmarks (possibly due to overparameterization). (a) Without regularization: α = 0 , MAE= 4 .       (b) α = 0 . 1 , MAE= 3 .
(c) α = 0 .
5 , MAE= 3 .
(d) α = 1 . 0 , MAE= 3 . 2.7 from Equation (5) 4. f r o m E q u a t
i o
[Figure]
A.1. C OMPARISON WITH D EEP H ASHING A PPROACHES
Deep supervised hashing approaches use neural networks as a hash function and learn hash codes in an end-to-end manner.
The loss function for deep supervised hashing is designed to preserve the similarity between inputs in the hashing space.
Often, these approaches use the label information to determine the similarity between images (i.e., same label)
(Liu et al., 2016; Xia et al., 2014).
Some deep hashing approaches have proposed to augment the loss function with classification loss to improve the performance.
We adapt two widely used deep-hashing approaches to regression and compare RLEL with deep hashing approaches.
Liu et al.
(2016) proposed a deep supervised hashing (DSH) approach with a loss function based on the pairwise similarity between images.
The proposed approach introduces a loss function to preserve the similarity between output codes for similar training images (e.g., same class) and maximize [Table] discriminability between output codes for different training images (e.g., different class).
Further, they propose using relaxation on the binary output and a regularizer to encourage the output code to be close to discrete values +1 / − 1 .
The hamming distance between output codes can be computed for binary-like outputs using the L2 norm.
We use DSH for regression with some modifications (DSH-reg).
We used the quantized label to determine the class of a training sample.
Lai et al.
(2015) proposed a triplet ranking loss to learn a hash function that preserves relative similarities between images (SFLH).
For images ( I, I + , I − ) , where I is closer to I + than I − , the loss function is designed to encourage higher hamming distance between codes for ( I, I − ) than ( I, I +) .
For classification datasets, triplets are typically formed using two images from the same class and one from a different class (Norouzi et al., 2012).
They proposed to use a piece-wise threshold function to encourage binary-like outputs.
We use the above approach (SFLH) for regression with a few modifications (SFLH-reg).
To generate triplets, we pick sets of three images from a given batch and determine the similarity between images using differences between the label values.
We use K 2 triplets for a minibatch of K training samples.
Further, for both DSH-reg and SFLH-reg, we augment the loss function with regression loss.
We add a fully-connected layer between the output code and prediction.
The MSE loss between the final outputs and target labels is added to the loss function (DSH-reg-L2, SFLH-reg-L2).
[Table]
Table 8 compares the modified deep hashing approaches with RLEL.
The gap between loss functions with and without regression loss is significant, which shows that a loss function that only aims to preserve the similarity between output codes is not sufficient and needs to account for the error between decoded output and target (i.e., regression loss).
RLEL results in a lower error as it is designed for regression problems that account for classifiers’ nonuniform error probability distribution.
Regularizer R1 encourages the distance between output codes for images to be proportional to the difference between label values, similar to pairwise or ranking-based loss functions proposed by deep hashing.
However, deep hashing approaches use the hamming distance between binary outputs.
As we show in Section 3.2, the hamming distance between codes does not account for the error probability of classifiers.
Thus we use the L1 distance between the real-valued outputs to account for the confidence of the classifiers.
R1 does not use regularizer or nonlinear activation on the output codes to encourage binary-like outputs, as typically done in deep hashing approaches.
In contrast, we show that suitable regression codes can be learned by not using this constraint.
Thus RLEL with only
R1 regularizer results in lower error than deep hashing approaches.
A.1. E VALUATION
[Table]
Table 9 compares RLEL with direct regression and multiclass classification using geometric mean and Pearson coefficient as evaluation metrics.
The geometric mean represents the geometric mean of absolute error for the test dataset.
The Pearson coefficient represents the correlation between the target and predicted labels for the test dataset.
As shown in the table, RLEL results in significant reduction in the error compared to other generic regression approaches [Table]
A.1. T HEORETICAL ANALYSIS OF PROPOSED REGULARIZATION FUNCTIONS
[Figure]
[Table]
[Figure]
[Table]
[Figure]
Regularization function R2:
We used matrix D instead of label encoding E to apply regularizer R2 in equation 4.
We insight into this decision as follows.
First, note the output encodings are multiplied with D to generate the correlation
vector ˆ C i (Figure 3).
We use the multiclass classification loss between ˆ C i and the target labels for training.
Due to this, label encoding E and decoding matrix D are related, and use of matrix D proves to be effective for regularizer R2.
We further explain this in detail below
[Figure]
Let E represent an encoding matrix of size N × M .
Each row E k : represents the encoding output when the label is k .
D is the decoding matrix of size M × N .
Let ˆ C k represent the output correlation row vector of size 1 × N when the target label is k .
Here, ˆ C k is obtained by multiplying E k, : with D (Figure 3).
[Equation]
Since we apply softmax on the output vector to find the predicted label (Figure 3), ideally, ˆ C kk should have the highest value as the target label value is k .
∴ [Equation]
, where,
[Equation] [Equation] [Equation] [Equation]
[Equation] ∴ E k, : .D : ,k > E k, : .D : ,x , where, x ̸ = k, x ∈ { 1 , 2 , ..., N } (Using equation 6).
Let θ k,x represent the angle between row vector E k, : and column vector D : ,x .
This leads to the below equation:
[Equation]
, x ̸ = k, x
[Equation]
[Equation]
N
[Equation]
Shah et al.
(2022) used a hand-crafted decoding matrix D with an equal number of 1 s and 0 s in each column for binary-encoded labels.
Hence the L2 norm of each column is the same.
In label encoding learning, parameters of matrix D are learned during training and are not constrained to have the same L2 norm for each column.
However, we observe a similar trend empirically.
Figure 9c plots the distribution of || D : ,x || for different benchmarks.
As shown in the figure, for most benchmarks, we observe a small variance in the distribution of || D : ,x || .
Based on this intuition and empirical validation, we assume that || D : ,x ||
≈
||
D : ,y || for x ∈ [1 , N ] and y ∈ [1 , N ] to simplify the analysis.
Using this assumption in equation 7 leads to the following inequality: cos ( θ k,k ) >
cos ( θ k,x ) , where, x ̸ = k, x ∈ { 1 , 2 , ..., N }
Thus the cosine similarity between E k, : and D : ,k should be the highest to predict the label k .
The optimization process to reduce the loss between the target and prediction will try to maximize this cosine similarity.
In the best case, the angle between E k, : and D : ,k will be zero, and both vectors are parallel.
This simplification leads to the following relation between E and D .
E k, : = tD : ,k , where t >
Similarly, E k +1 , : = t ′ D : ,k +1 , where t ′ >
Since t and t ′ both are positive values, reducing D i,k − D i,k +1 also reduces E k,i − E k
+1 ,i .
Regularizer rule R2 proposes to regularize the number of decision boundaries by regularizing (cid:80) Mi
=1 (cid:80) N − 1 j =1 | E j,i − E j +1 ,i | as per equation 2.
Based on the analysis above, regularizing (cid:80) Mi =1 (cid:80) N − 1 j {+=
+} | D i,j − D i,j +1 | helps with the above goal as E j,i − E j +1 ,i reduces with D i,j − D i,j +1 .
Regularization function R1:
The first property suggests ||
E i, : − E j, : || 1 ∝ | i − j | .
So ideally, || E i, : − E j, : ||
1 =
λ | i − j |
Since E x, : is average of ˆ Z i for samples with label value x (equation 1), the above condition leads to:
[Equation]
Based on this requirement, we add a regularization function max (0 , λ | y
i − y j | − ||
ˆ Z i − ˆ Z j || 1 ) , which penalizes the encodings if ||
ˆ Z i − ˆ Z j || 1 < λ | y
i − y j | .
It does not strictly impose equation 8.
However, [Table] it approximately imposes the constraint as per shown in empirical verification in Section A.1.1.
Our intuition behind using the scaling parameter 2 is based on binary-encoded labels.
For two adjacent labels (i.e., | y i − y
j
| = 1 ), the loss function encourages ||
ˆ Z i − ˆ Z j || 1 to be greater than 2 .
Here, ˆ Z is the output encodings.
In the case of binarized label encoding ( − 1 if Z < 0 and +1 if Z > 0 )
|| Z i − Z j || 1 = 2 signifies that two encodings differ in at least one bit.
A.1. I MPACT OF THE NUMBER OF QUANTIZATION LEVELS ( N )
The number of quantization buckets is treated as a design parameter for binary-encoded labels.
Shah et al.
(2022) showed that the error changes with the number of quantization levels.
Fewer levels introduce quantization error, and more levels increase the number of bits in the encoding.
They showed a trade-off between these two factors to decide the number of quantization levels.
Our work focuses on the design space of encoding and decoding functions.
Hence we use the same values for the quantization levels ( N ) as BEL Shah et al.
Parameter N tuning can be integrated into hyperparameter tuning or included in the optimization process.
We further analyze the effect of the number of quantization levels for RLEL.
Table 10 shows the
NME (Normalized Mean Error) for different values of N for FLD1 benchmark.
This suggests that the proposed method RLEL is less sensitive to the number of quantization levels for higher values.
For RLEL, the decoding matrix that converts the encodings to the predicted label is also learned during the training (Figure 3).
This matrix is of size M × N , where each column represents the weight parameters for one quantization level.
One possible reason for the above results is that matrix D adaptively learns the number of quantization levels suitable for this problem.
There is a potential to adaptively learn the number of quantization levels and non-uniform quantization using the proposed RLEL framework.
For example, in Figure 3- step (4), fixed parameters j are used to scale the correlation vector ˆ C ji and find the expected prediction ˆ y i .
These parameters represent quantization levels.
One possible approach to learning the quantization levels is to make these parameters trainable.
In this case, L1/L2 loss between the expected prediction ˆ y i and target labels y i can be used to train the network.
A.1. I MPACT OF DATASET SIZE ON ERROR FOR RLEL AND BEL
In order to compare the effect of dataset size on encoding design, we run BEL and RLEL approaches with the same training loss function (cross entropy loss in equation 5).
We take the dataset FLD and use a fraction of the dataset for training.
The entire test dataset is used for testing here.
Tablesummarizes the error achieved by RLEL and BEL for different fractions of the training dataset.
The evaluation shows that the gap between the performance of RLEL and BEL decreases with the increase in dataset size, which suggests that RLEL might be able to achieve lower error for larger datasets.
A.1. C OMPARISON OF LEARNED AND MANUALLY DESIGNED ENCODINGS
We visually compare the encoding learned by RLEL with BEL manually designed code for one benchmark.
Figure 10 shows the learned and manually designed encodings.
Here, row k represents an encoding for label k .
Column j represents the bit values for classifier-k over the numeric range of labels.
We notice some common characteristics between both encodings.
For example, the codes for
[Table]
[Figure]
[Table]
[Figure] [Table] [Figure] nearby labels differ by fewer bits than faraway labels.
Both the codes also have fewer bit transitions ( 0 → 1 and 1 → 0 transitions in a column).
These characteristics in the learned encodings are encouraged by the proposed regularizers R1 and R2.
There are a few differences between learned and hand-crafted encodings.
In contrast to hand-crafted labels, encodings for adjacent labels do not differ in some [-012345678910111213141516171819202122232425 Code 
 L a b e l (a) Learned-]cases, where hand-crafted encoding [-(RLEL) 
-]assures at least one or two bits of difference between adjacent labels
[-1 2 3 4 5 6 7 8 9 1011121314151617181920Code 
 L a b e l (b) Hand-crafted encoding (BEL) cases, where hand-crafted encoding assures at least one or two bits of difference between adjacent labels.-][Table]
A. L ABEL E NCODING D ESIGN
We evaluate different label encoding design approaches, including simulated annealing and autoencoder.
These approaches have been used to design encodings for multiclass classification by prior works (Song et al., 2021; Cissé et al., 2012).
We adapt these approaches to design encodings for regression tasks and compare RLEL with these code design techniques.
This section provides the methodology for simulated annealing and autoencoder-based label encoding design.
A.2. S IMULATED A NNEALING
Simulated annealing is a probabilistic approach to find a global optimum of a given function.
It is often used for combinatorial optimization, where the search space is discrete.
[-Output: C ∈ { 0 , 1 } M × N ; 1: C = C 0 ∈ { 0 , 1 } N × M , where Pr ( C 
-]
Algorithm 1 represents the pseudo-code for label encoding design using simulated annealing.
This algorithm takes two hyperparameters, K max (number of iterations) and T (initial temperature).
It designs a code matrix C of size N × M , where N is the number of values and M is the number of bits.
Each row k in this code matrix represents encoding for value k .
Code matrix C is initialized with a random matrix ofand 1 (Line 1).
For each iteration, a new code matrix C new is sampled from the current code matrix C using a Move function (Line 4).
For example, a move function can be designed to randomly flip a few bits in C .
10: end for
The difference between the errors of the current and new code matrix is measured (Line 5).
The error of a code matrix, i.e., expected regression error for this problem, is measured using function
E .
For example, E can be replaced by training a regression network for a given code matrix to measure the regression error.
Finally, the current code matrix C is updated with the new matrix C new probabilistically.
The probability is determined using the decrease in regression error and current temperature t (Line 6-8).
The current temperature is updated for each iteration (Line 9).
There are mainly two design parameters in the above algorithm: the error measurement function E and the move function Move .
We further explain the design of these functions.
Error measurement:
We used the expected absolute error between targets and decoded predictions for a given code matrix as its error, as the goal is to design a code matrix that results in lowest regression error.
However, training a regression network for each sample code matrix to measure its regression error is computationally expensive and time-consuming ( ∼ 200 training runs).
Hence we use an analytical model to estimate the regression error for a given code matrix.
Regression error is the absolute error between targets Q i and decoded predictions ˆ Q i .
For a given target Q i and target code B i =
C Q i , : , the predicted code ( ˆ B i ) will be erroneous due to classification errors.
This erroneous predicted code is decoded to a predicted value ( ˆ Q i ).
The following equation is used to predict ˆ Q i in expected-correlation-based decoding (Shah et al., 2022).
[Equation]
The regression error can be estimated given sufficient samples of B i and ˆ B i .
Shah et al.
(2022) provided an approximate model of classification errors.
They showed that for each classifier, its error probability distribution can be approximated using a combination of p Gaussian distributions, where p is the number of bit transitions.
Each Gaussian distribution is centered around a bit transition.
For example, for bit- k in unary code with bit transition between Q = k and Q = k + 1 , the error probability of the classifier- k for different target labels Q i can be approximated as:
[Equation]
ˆ B i can be sampled for the given Q i and C using the above error-probability model.
Equation 9 is then used to find the decoded prediction ˆ Q i .
We measure the expected absolute error between ˆ Q i and
Q i using 100 × N samples.
We further verify the validity of this analytical model by finding the correlation between regression error measured by this model and trained networks.
Figure 11 plots the analytical regression error versus actual regression error for FLD_1 benchmarks.
Here, each point is for a different code matrix.
[Figure]
na l y t i c a l m ode l ' s M AE [Figure] The Y -axis represents the absolute error approximated by the proposed analytical model.
The X -axis represents the absolute test error of a trained network for a given code matrix.
The figure shows that the proposed analytical model for error measurement approximates error with significant speedup.
Move function: The move function flips some bits in the current code matrix to sample a new one.
A naive approach would be to randomly flip b bits.
We further optimize the move function to consider the regression task objective.
For a given code matrix, using the proposed analytical model, we find a matrix F of size N × N , where F i,j = | i − j | × Pr ( Round ( D ( ˆ B, C )) =
j | Q =
i, B = C i, : ) .
Thus, each element represents a pair ( C i, : , C j, : ) of encodings’ contribution to expected error.
We select top- b pairs from this matrix.
For each pair of encodings, we find bit-positions that have equal bit-values between two encodings, and a randomly selected bit-position from this list is flipped in encoding C i, : .
This procedure increases the hamming distance between pairs of encodings that contribute the highest to the regression error.
Figure 12 compares the convergence of the proposed move function and a random-flip-based move function.
Here the Y -axis represents the approximated error for the current code matrix, and X -axis represents the iteration identifier.
The figure shows that the proposed error-based move function results in faster convergence and lower error.
We use the proposed move function with the analytical model to approximate regression error in
Algorithm 1 to design label encoding for regression using simulated annealing.
A.2. A UTOENCODER
Cissé et al.
(Cissé et al., 2012) proposed an autoencoder-based approach to design encodings for a multiclass classification problem.
Figure 13 represents the network architecture used for encodings [Figure] design.
Input S
i is an N -dimensional vector for class i .
Here, each element S i
[ j ] represents the similarity between class i and j .
The output of the bottleneck layer C
i is the designed encodings for class i .
For regression problems, we set S i
[ j ] = | i − j | .
Let W represent the weight parameters of the network.
The network is trained using SGD optimization, where each batch consists of randomly sampled i and j .
The following loss function is used for training:
[Equation]
Here, the first and second terms represent reconstruction losses for inputs S i and S j .
The third term encourages a minimum distance of b between any pair of encodings to yield unique encodings for different classes.
The fourth term is an L2-regularizer.
Once the network is trained, the real-valued encodings C i are converted to binary encodings such that it has equal numbers of 0 s and 1 s. This formulation introduces three hyperparameters.
We determine the number of bit transitions in the designed label encodings and select hyperparameters that result in the lowest number of bit transitions.
Note that this autoencoder network is decoupled from the regression network and design codes agnostic to classifiers’ characteristics for a given regression problem.
A. E XPERIMENTAL M ETHODOLOGY
We use 11 benchmarks covering four different regression tasks for evaluation.
This section summarizes the experimental setup, including datasets, evaluation metrics, hyperparameters, and related work for each of these tasks.
We also report the training time using an Nvidia RTX 2080 Ti GPU with 11GB of memory for each benchmark.
A.3. H EAD P OSE E STIMATION
In landmark-free 2D head pose estimation, for a given 2D image, the head pose of a human is directly estimated in terms of three angles: yaw, pitch, and roll.
We use loose cropping around the center with random flipping for data augmentation.
We use the ResNet50 network as the feature extractor.
This network is initialized using pre-trained parameters for ImageNet (Russakovsky et al., 2015) dataset.
During the training for RLEL the entire network, including the feature extractor, is trained.
Datasets: We use the evaluation methodology followed by prior works (Ruiz et al., 2018; Yang et al., 2019).
Two protocols are used for evaluation.
Protocol 1 (LFH1): This protocol uses the BIWI (Fanelli et al., 2013) dataset for training and evaluation.
This dataset consists of 15 , 128 frames of 20 subjects.
Random 70% − 30% splits are used for training and evaluation.
The ranges of yaw, pitch, and roll angles are [ − 75 ◦ , 75 ◦ ] , [ − 65 ◦ , 85 ◦ ] , and [ − 55 ◦ , 45 ◦ ] , respectively.
Protocol 2 (LFH2):
In this protocol, the network is trained using the 300W-LP (Zhu et al., 2016)
dataset consisting of 122 , 450 samples.
AFLW2000 (Zhu et al., 2016) dataset is used for evaluation.
The range of all labels is [ − 99 ◦ , 99 ◦ ] in this setting.
Evaluation metrics: We report the Mean Absolute Error (MAE) between the targets ( y i ) and predictions ( ˆ y i ).
Let N represent the number of samples, and P represent the number of labels (three in head pose estimation).
The MAE is defined as:
[Equation]
Network architecture and training parameters:
Table 12 summarizes the hyperparameters used for RLEL .
The learning rate of the decoding matrix D is kept 10 × higher than the learning rate of the feature extractor.
L2 regularization with weight of 0.0001 is used for direct regression.
[Table]
Related work Head pose estimation is a widely studied problem.
Existing task-specialized approaches propose different loss formulations or feature extractors to improve the error.
HopeNet (Ruiz et al., 2018) proposed a combination of regression and classification loss.
SSR-Net
(Yang et al., 2018) and FSA-Net (Yang et al., 2019) proposed stage-wise soft regression.
QuatNet (Hsu et al., 2019) proposed to use MSE loss with custom ordinal regression loss.
RAFA-Net (Behera et al., 2021) proposed an attention-based feature extractor architecture.
Table 13 and Table 14 compare the performance of RLEL with related work.
[Table]
A.3. F ACIAL L ANDMARK D ETECTION
Facial landmark detection focuses on finding the ( x, y ) coordinates of facial keypoints for a given 2D image.
Evaluation metrics: We report the Normalized Mean Error (NME) between the targets y i and predictions ˆ y i .
Inter-ocular distance normalization is used for all datasets.
For N test samples, P facial landmarks, and L normalization factor, the NME is defined as:
[Equation]
[Table]
Datasets:
We use three datasets widely used for facial landmark detection: COFW (Burgos-Artizzu et al., 2013), 300W (Sagonas et al., 2013), and WFLW (Wu et al., 2018).
HRNetV2-W18 network architecture for feature extraction (Wang et al., 2020) and the modified regressor architecture for label encoding proposed by BEL (Shah et al., 2022) are used in this work.
Random flipping, scaling ( 0 .
75 − 1 . 25 ), and rotation ( ± 30 ) are used for data augmentation.
The COFW dataset consists of 1 , 345 training and 507 testing images annotated with 29 landmarks.
The training set of the 300W dataset has 3 , 148 images annotated with 68 facial landmarks.
This dataset provides four test sets: full test set, common subset, challenging subset, and the official test set with 300 indoor andoutdoor images.
WFLW dataset is a comparatively larger dataset with 7 , 500 training and 2 ,testing images.
Each image is annotated with 98 facial landmarks.
The test set is divided into six subsets: large pose, expression, illumination, make-up, occlusion, and blur.
Training parameters:
Table 15 provides a summary of all the training parameters.
The learning rate of the decoding matrix D is kept 20 × higher than the learning rate of the feature extractor.
The
HRNetV2-W18 network is initialized with pretrained weight parameters for the ImageNet dataset.
We refer to HRNetV2-W18 evaluated on COFW as FLD1/FLD1_s , on 300W as FLD2/FLD2_s [Table] and on WFLW as FLD3/FLD3_s .
[Table]
Related work Facial landmark detection is a widely studied problem.
A common approach is to use heatmap regression, where the target heatmaps are generated by assuming a Gaussian distribution around the ground truth landmark location.
Prior works proposed the use of binary heatmaps with pixel-wise binary cross-entropy loss (Bulat & Tzimiropoulos, 2016).
HRNet (Wang et al., 2020) proposed a feature extractor that maintains high-resolution representations and uses heatmap regression.
AWing (Wang et al., 2019) proposed a modified heatmap regression loss function with adapted wing loss.
AnchorFace (Xu et al., 2020) used anchoring of facial landmarks on templates.
LUVLi (Kumar et al., 2020) proposed a landmark’s location, uncertainty, and visibility likelihood-based loss.
Table
16- 18 compare RLEL with related work
[Table] [Table]
[Table]
[Table]
A.3. A GE E STIMATION
This task focuses on predicting a person’s age from a given 2D image.
MAE (Equation 12) and
Cumulative Score (CS) are used as the evaluation metrics [Table] and ResNet50 (He et al., 2016) is used as the feature extractor.
CS θ is the percentage of test samples with absolute error less than θ years
[Table] Datasets MORPH-II (Ricanek & Tesafaye, 2006) and AFAD (Niu et al., 2016) datasets are used for evaluation.
We follow the protocols for preprocessing and data augmentation of datasets as per prior works (Shah et al., 2022; Raschka, 2018).
MORPH-II dataset consists of 55 , 608 images with random split of 39 , 617 training, 4 , 398 validation, and 11 , 001 test images.
The AFAD dataset consists of 164 , 432 images with random split of 118 , 492 training, 13 , 166 validation, and 32 , 763 test images.
Training parameters:
Table 19 summarizes the training parameters for AE1 (MORPH-II) and
AE2 (AFAD) benchmarks.
The learning rate of the decoding matrix D is kept 10 × higher than the learning rate of the feature extractor.
L2 regularization with weight of 0.001 is used for direct regression.
Training for AE1 and AE2 consumes ∼ 2 and ∼ 7 hours, respectively.
[Table]
Related work Different approaches including ordinal regression (Niu et al., 2016; Cao et al., 2020; Pan et al., 2018; Gao et al., 2018), soft stage-wise regression (Yang et al., 2018; 2019), soft labels (Díaz & Marathe, 2019) have been proposed for age estimation.
OR-CNN (Niu et al., 2016) and CORAL-CNN (Cao et al., 2020) proposed ordinal regression by binary classification with threshold-based encodings (i.e., unary codes).
DLDL (Gao et al., 2018) augmented the loss function with KL-divergence between softmax output and soft target probability distributions.
MV-Loss (Pan et al., 2018) proposed to penalize the prediction based on the variance of the age distribution.
We compare CLL with related work in Table 20 and Table
[Table]
[Table] [Table]
A.3. E ND - TO - END S ELF D RIVING
For the regression task of end-to-end autonomous driving, we use the NVIDIA PilotNet dataset, and
PilotNet model (Bojarski et al., 2016).
In this task, for a given image of the road, the angle of the steering wheel that should be taken next is predicted.
MAE (Equation 12) is used as the evaluation metric.
Dataset The PilotNet driving dataset consists of 45 , 500 images taken around Rancho Palos Verdes and San Pedro, California (Chen).
We use the data augmentation technique used by prior works (Bojarski et al., 2016).
Training parameters Table 22 summarizes the training parameters.
The learning rate of the decoding matrix D is kept 10 × higher than the learning rate of the feature extractor.
[Table]
Related work End-to-end autonomous driving is an interesting task with increasing attention.
PilotNet (Bojarski et al., 2017) used a small, application-specific network.
We compare RLEL with the baseline PilotNet architecture in Table 23.
[Table]