R ELATIONAL S URROGATE L OSS L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Evaluation metrics in machine learning are often hardly taken as loss functions, as they could be non-differentiable and non-decomposable, e.g. , average precision and F1 score.
This paper aims to address this problem by revisiting the surrogate loss learning, where a deep neural network is employed to approximate the evaluation metrics.
Instead of pursuing an exact recovery of the evaluation metric through a deep neural network, we are reminded of the purpose of the existence of these evaluation metrics, which is to distinguish whether one model is better or worse than another.
In this paper, we show that directly maintaining the relation of models between surrogate losses and metrics sufﬁces, and propose a rank correlation-based optimization method to maximize this relation and learn surrogate losses.
Compared to previous works, our method is much easier to optimize and enjoys signiﬁcant efﬁciency and performance gains.
Extensive experiments show that our method achieves improvements on various tasks including image classiﬁcation and neural machine translation, and even outperforms state-of-theart methods on human pose estimation and machine reading comprehension tasks.
I NTRODUCTION
Evaluation metrics matter in machine learning since it depicts how well we want the models to perform.
Nevertheless, most of them are non-differentiable and non-decomposable, thus we can not directly optimize them during training but resort to loss functions (or surrogate losses), which serve exactly as a proxy of task metrics.
For example, pose estimation task uses percentage of correct keypoints (PCK) (Yang & Ramanan, 2012) to validate point-wise prediction accuracy, but it often adopts mean square error (MSE) as loss function.
Neural machine translation task takes the sentence-level metric BLEU (Papineni et al., 2002) to evaluate the quality of predicted sentences, while using word-level cross-entropy loss (CE Loss) in training.
Besides this manual proxy, some works (Grabocka et al., 2019; Patel et al., 2020) propose to learn surrogate losses which approximate the metrics using deep neural networks (DNN), so the optimization of metrics can be relaxed to a differentiable space.
For example, taking predictions and labels as input, (Grabocka et al., 2019) approximates the outputs of surrogate losses and evaluation metrics by minimizing their L2 distances.
Moreover, recent work even involves the prediction networks into the surrogate loss learning by alternatively updating the loss and predictions, i.e. , they train the surrogate losses after every epoch during training, then use the latest optimized losses to train prediction networks in the next epoch.
For instance, (Grabocka et al., 2019) mainly focuses on the simple binary classiﬁcation while LS-ED (Patel et al., 2020) chooses to adopt the surrogate losses in the post-tuning stage (ﬁne-tuning the models learned by original losses) and achieves promising improvements.
However, these methods often suffer from heavy computational consumption and do not perform well on large-scale challenging datasets.
Both manual and learned surrogate losses follow an exact recovery manner; namely, the surrogate losses should approximate the target metrics rigorously, and optimizing the surrogate loss is supposed to improve the evaluation metrics accordingly.
However, this assumption does not always hold due to the approximation gap, bringing bias to the optimization and leading to sub-optimal results.
Instead of pursuing an exact recovery of the evaluation metric, we are reminded of the purpose of metrics, which is to distinguish the performance of models.
If a model has a smaller loss than the other model, its metric ought to be better.
Nevertheless, current surrogate losses usually have weak relation with the evaluation metrics ( e.g. , CE Loss & BLEU in Figure 1 (b)).
Ideally, the surrogate loss should maintain strong relation of evaluation metric to all models.
[Figure]
In this paper, we leverage the ranking correlation as the relation between surrogate losses and evaluation metrics.
Then a natural question raises, if the loss functions only require accurate relative rankings to discriminate the models, why do we need to approximate the metrics exactly?
In this way, we propose a method named Relational Surrogate Loss (ReLoss) to maximize this rank correlation directly.
Concretely, our ReLoss directly leverages the simple Spearman’s rank correlation (Dodge, 2008) as the learning objective.
By adopting differentiable ranking method, the ranking correlation coefﬁcient can be maximized through gradient descent.
Compared to exactly recovering the metrics, our correlation-based optimization is much easier to learn, and our ReLoss, which is simply constructed by multi-layer perceptions, aligns well with the metrics and obtains signiﬁcantly better correlations compared to the original losses.
For example, the commonly used loss MSE in pose estimation only has 46 .
71% Spearman’s rank correlation coefﬁcient with the evaluation metric
PCK, while our ReLoss enjoys 84 .
72% relative improvement (see Table 1 and Figure 1).
Our ReLoss generalizes well to various tasks and datasets.
We learn ReLoss using randomly generated data and pre-collected network outputs, then the learned losses are integrated into the training of prediction networks as normal loss functions ( e.g. , cross-entropy loss), without any further ﬁnetuning.
Note that we use the same surrogate losses with the same weights in each task, and we ﬁnd that it is sufﬁcient to obtain higher performance.
Compared to previous works, our method is much easier to optimize and enjoys signiﬁcant efﬁciency and performance improvements.
Extensive experiments on the synthetic dataset and large-scale challenging datasets demonstrate our effectiveness.
Moreover, our method outperforms the state-of-the-art methods in human pose estimation and machine reading comprehension tasks.
For example, on human pose estimation task, our ReLoss outperforms the state-of-the-art method DARK (Zhang et al., 2020) by 0 .
2% on COCO test-dev set; on machine reading comprehension task, we achieve new state-of-the-art performance on DuReader 2.0 test set, outperforming all the competitive methods, and even obtain 7 . 5% better ROUGE-L compared to human performance.
R ELATED W ORK
Surrogate loss learning.
Since most of the metrics in deep learning tasks are non-differentiable and non-decomposable ( e.g. , accuracy, F1, AUC, AP, etc. ), surrogate losses aim to approximate the metrics to make them differentiable using neural networks.
(Grabocka et al., 2019)
ﬁrst proposes to learn surrogate losses by approximating the metrics of tasks through a neural network, and the losses are optimized jointly with the prediction model via bilevel optimization.
(Patel et al., 2020) learns the surrogate losses via a deep embedding where the Euclidean distance between the prediction and ground truth corresponds to the value of the metric.
However, it is hard to obtain a precise prediction by directly optimizing the surrogate loss with such a strong constraint.
We remind that the role of loss functions is to determine which model is better, but with the unavoidable existence of approximation gap, this determinability does not always hold.
In addition, these methods both train the surrogate losses alternately with prediction networks, resulting in noticeable efﬁciency and generability deduction compared to regular losses.
In our paper, instead of only focusing on point-to-point recovery, which ignores the rankings between relative values of metrics, we ease the
[Figure]
optimization constraint by explicitly learning our ReLoss with rank correlation, and enjoy signiﬁcant performance and efﬁciency improvements.
Differentiable sorting & ranking.
Differentiable sorting and ranking algorithms (Adams & Zemel, 2011; Grover et al., 2018; Blondel et al., 2020; Petersen et al., 2021) can be used in training neural networks with sorting and ranking supervision.
Recent approach (Blondel et al., 2020) proposes to construct differentiable sorting and ranking operators as projections onto the permutahedron, i.e. , the convex hull of permutations, and using a reduction to isotonic optimization.
(Petersen et al., 2021) proposes differentiable sorting networks by relaxing their pairwise conditional swap operations.
In this paper, we can use any of these differentiable ranking algorithms to generate differentiable ranking vectors, then directly optimize the rank correlation coefﬁcient for the supervision of our surrogate losses.
The algorithm in (Petersen et al., 2021) is adopted for better performance.
P RELIMINARIES
For a given task with a metric function M ( y , ˆ y ) , where y and ˆ y denote the predicted labels and ground-truth labels, respectively, its loss function L ( y , ˆ y ) can be formulated as:
[Equation]
where f can be any function with output ∈ R 1 .
In this paper, we tend to use a learned DNN ( f DNN ) with weights θ l as a surrogate loss, i.e.,
[Equation]
The surrogate losses are learned with the networks’ outputs y and the corresponding metric values
M ( y , ˆ y ) , i.e. ,
[Equation]
where O s is the learning objective of surrogate loss.
The prediction networks with weights θ m are then optimized by descending the learned surrogate losses L ( y , ˆ y ; θ ∗ l ) , i.e.
[Equation]
Approximation-based optimization To learn a surrogate loss w.r.t.
a metric, an intuitive idea is to approximate the metric’s outputs, i.e. , learn the surrogate losses by minimizing the distances between the outputs of surrogate losses and their corresponding metric values, which is adopted in previous works (Grabocka et al., 2019; Patel et al., 2020), their learning objective O s is
[Equation]
we call this optimization as approximation-based optimization.
However, it is hard for a DNN to fully recover the evaluation metric.
We conduct toy experiments using a random weighted DNN with output ∈ R 1 as an evaluation metric, and then the surrogate loss is learned using limited observations of the metric.
As illustrated in Figure 2 (a), since approximating a random network with random inputs is challenging, the errors between surrogate loss learned by approximation-based optimization and metric values are noticeably large.
In order to validate the effectiveness of losses in training, we then train the input data with metric or learned losses, as shown in Figure 2 (c), we illustrate the curves of metric values w.r.t.
learned input data during training, and directly using metric as loss function obtains best metric value (lower is better), but the performance of input data using approximation-based loss is getting worse.
L EARNING R ELATIONAL S URROGATE L OSS 4. R ELATION AS R ANK C ORRELATION
Based on the previous discussion, the prior works adopt an unnecessary constraint by enforcing the surrogate losses to fully recover the evaluation metrics.
However, the loss function only needs to have the same ranking relation to the metrics, i.e. , we just need to make the surrogate losses have the same ranking as metrics.
In this paper, we obtain the relation between surrogate losses and evaluation metrics by using rank correlation as the learning objective, which we call correlationbased optimization.
The relation between surrogate losses and evaluation metrics is measured by ranking correlation, which is a statistic that measures the relationship between rankings of the same variable.
A ranking correlation coefﬁcient measures the degree of similarity between two rankings and can be used to assess the relation’s signiﬁcance.
If the surrogate loss fully correlates to the evaluation metric, the descent of loss value will always obtain better metric values.
Spearman’s rank correlation For optimization of surrogate losses, we use the most commonly used Spearman’s rank correlation (Dodge, 2008).
For two vectors a and b with size n , the Spearman’s rank correlation is deﬁned as:
[Equation]
where r a is the rank vector of a , Cov ( r a , r b ) is the covariance of the rank vectors, Std ( r a ) denotes the standard derivation of r a .
4. L EARNING L OSSES BY M AXIMIZING R ANK C ORRELATION
Correlation-based optimization We use Spearman’s rank correlation as the objective to learn our surrogate losses, since the loss should have a negative correlation w.r.t.
the metric (higher is better), our objective is to minimize the Spearman’s rank correlation coefﬁcient, i.e. ,
[Equation]
Since the computation of rank vectors r a and r b in Eq.(6) is not differentiable, we adopt one of the differentiable ranking methods (Petersen et al., 2021) to obtain differentiable ranking vectors, and empirically ﬁnd that the errors in differentiable approximation is negligible and our learned correlation can be very close to the optimal value, i.e. , O s = − 1 .
As shown in Figure 2, compared to approximation-based optimization, the surrogate loss learned by our correlation-based optimization obtains higher rank correlation and faster convergent speed.
Besides, optimizing with our correlation-based loss achieves signiﬁcantly better performance than approximation-based optimization, and is more stable than the original loss (evaluation metric).
Learning with Gradient Penalty In our paper, we can directly backward through the surrogate loss to obtain its gradients.
However, we ﬁnd the ﬁrst-order derivative of the learned ReLoss w.r.t.
the prediction y changes rapidly since we only constrain the correlation in Eq.(3), which result in either vanishing or exploding gradients.
Nevertheless, in the optimization of networks, we want a loss with smooth gradients to train the networks steadily.
Following (Gulrajani et al., 2017; Patel et al., 2020), we now propose an alternative way to smooth the gradients by enforcing the Lipschitz constraint.
A differentiable function is 1-Lipschitz if and only if it has gradients with norm at most 1 everywhere, so we consider directly constraining the gradient norm of the loss’s output w.r.t.
its input, i.e.
[Equation]
This penalty of gradients has been shown to enhance the training stability for generative adversarial networks (Gulrajani et al., 2017).
Our objective of surrogate loss learning in Eq.(7) becomes
[Equation]
we use λ = 10 in our experiments.
4. P IPELINE
Now we illustrate how to learn our ReLoss.
Different from previous works (Grabocka et al., 2019;
Patel et al., 2020) which train the surrogate loss and prediction network alternatively as bilevel optimization, we want our surrogate loss to be general as vanilla loss ( e.g. , cross-entropy loss).
Since we learn the surrogate loss with a much weaker constraint, our surrogate loss can generalize better to the whole distribution of outputs and metric values to train the surrogate loss once for all, without further ﬁne-tuning.
Our training strategy of surrogate loss is summarized in Algorithm 1.
The training data of surrogate losses is the combination of randomly generated data G R and the outputs of models G M .
Concretely, we design a random generator to produce random outputs and labels uniformly for G R , while for
G M , we use the intermediate checkpoints of the prediction networks trained by original loss to predict the outputs of train data.
Each batch of training data is generated from G R or G M with probabilities p and 1 − p , respectively.
Input: surrogate loss L with random weights θ l , batch size N , metric function M , data generators G M and G R , sample probability p .
Output: learned surrogate loss with highest correlation.
do 2:
4: generate a batch of predictions and ground-truth labels ( y i , ˆ y i ) from G R with probability p or
M
; 5: compute
.t.
; 6: compute
; 7: compute
9: end
; 11: optimize by
; 12: end while 13: return surrogate loss with l .
Usage of learned ReLoss.
The learned ReLoss can be ﬁxed and then integrated into the training of prediction networks, i.e. , we only change the loss function in training, without any modiﬁcation on training strategy, network architecture, etc.
Besides, we emprically ﬁnd that ReLoss would achieve better performance if combined with the regular loss.
In this case, the regular loss might act as a regularization term, and bring a decent prior for the ReLoss to enhance the optimization.
E XPERIMENTS
To fully experiment with the effectiveness and generability of our ReLoss, we conduct experiments on both computer vision and natural language processing tasks.
In computer vision, we experiment
[Table] [Table]
on image classiﬁcation and human pose estimation tasks, while in natural language processing, we experiment on machine reading comprehension and neural machine translation tasks.
We ﬁrst show the rank correlations of original losses and our learned surrogate losses to the metrics in Table 1.
Spearman’s (Dodge, 2008) and Kendall’s Tau (Kendall, 1938) are two commonly used coefﬁcients to measure the ranking correlations between two vectors.
We can see that in all our experimented tasks, our ReLoss achieves higher correlations compared to the original losses.
Notably, even for cross-entropy (CE) loss, which has been shown to align well with the misclassiﬁcation rate, our surrogate loss still performs better on classiﬁcation tasks with the metric accuracy (ACC).
5. C OMPUTER V ISION
Image classiﬁcation We conduct experiments on three benchmark datasets CIFAR-10, CIFAR100 (Krizhevsky et al., 2009), and ImageNet (Deng et al., 2009).
On CIFAR-10 and CIFARdatasets, we train ResNet-56 (He et al., 2016) with original CE loss and our surrogate loss and report their accuracies on the test set with mean and standard derivation of 5 runs.
While on ImageNet dataset, we train ResNet-50 and MobileNet V2 (Sandler et al., 2018), their accuracies on validation set are reported.
Notably, all experiments use the same surrogate loss with the same weights.
Table 2 shows the evaluation results.
We can see that, though the original CE loss obtains a very high correlation ( ∼ 0 .
96 in Table 1), by integrating our surrogate loss with higher correlation, the performance can still be improved.
Note that we use the same surrogate loss with ﬁxed weights in these three datasets, which means that our loss can generalize to different image classiﬁcation datasets and gain the improvements with negligible additional cost.
Human pose estimation Human pose estimation (HPE) aims to locate the human body and build body skeleton from images.
It is difﬁcult to precisely evaluate the performance of HPE since many features need to be considered ( e.g. , the quality of body parts, the precision of each keypoints).
As a result, many metrics are proposed for HPE.
Percentage of correct keypoints (PCK) (Yang &
Ramanan, 2012) and Average Precision (AP) are two of the most commonly used ones.
However, current methods usually adopt mean square error (MSE) to minimize the distance between predicted heatmap and target heatmap, which correlates weakly with the evaluation metrics.
In our experiments, we choose to approximate PCK@0.05 since it better reﬂects the quality of each keypoint, and our ReLoss achieves signiﬁcant improvement on rank correlation compared to the original MSE loss.
We use the most widely used large-scale dataset COCO (Lin et al., 2014) to evaluate our performance, and the results are summarized in Table 3.
We can see that, on validation set, our ReLoss signiﬁcantly improves the baseline methods, and the AP 75 improves the most since our ReLoss aligns PCK for better keypoint localization.
On test-dev set, we integrate our ReLoss into state-of-the-art method DARK (Zhang et al., 2020) and achieve improvements on all the metrics.
[Table]
N ATURAL L ANGUAGE P ROCESSING
The gaps between loss functions and evaluation metrics on natural language processing tasks are severer since the tasks often use sentence-level evaluation metrics ( e.g. , BLEU and ROUGE-L) but adopt word-level cross-entropy loss in training
[Table]
Machine reading comprehension The task of machine reading comprehension (MRC) aims to empower machines to answer questions after reading articles.
Concretely, with a given question, the models are required to locate a segment of text from the corresponding reading passage, which is most probably the answer.
We use F1 score as the evaluation metric to learn surrogate loss, and experiment on two typical MRC datasets SQuAD (Rajpurkar et al., 2016) and DuReader (He et al., 2018).
SQuAD evaluates performance using F1-score, and DuReader uses ROUGE-L (Lin, 2004) and BLEU-4 (Papineni et al., 2002).
The evaluation results are summarized in Table 4 and Table 5.
On DuReader 2.0 dataset, our ReLoss gains improvements on dev set, and achieves state-of-the-art performance on test set 1 .
On SQuAD 1.1 dataset, we also achieve improvements compared to the baseline method
[Table] [Table]
[Table] [Table]
[Table]
Neural machine translation Neural machine translation (NMT) aims to translate a sentence from the source to the target language with an end-to-end neural model.
The evaluation metric of NMT is
BLEU (Papineni et al., 2002), which measures the n-gram overlap between the generated translation and the reference.
We conduct experiments on the Non-Autoregressive neural machine Translation (NAT) task, in which the model generates target words independently and simultaneously.
Since the output of NAT cannot be properly evaluated through word-level cross-entropy loss due to the multi-
[Table] [Table]
modality problem in language, the correlation between cross-entropy loss and translation quality is weak, limiting the NAT performance.
Table 6 shows the NAT evaluation results on WMT-16 EN → RO and RO → EN datasets.
We conduct experiments based on the NAT-base and a strong baseline BoN- L 1 (N=2)
(Shao et al., 2021), which introduce the BoN loss to ﬁne tune NAT-base by modeling the bag of ngrams in the sentence.
We integrate ReLoss into the baseline methods NAT-base (Gulrajani et al., 2017) and BoN- L 1 (N=2), and the evaluation results show that our ReLoss can improve both of them.
5. A BLATION S TUDIES
Compare with LS-ED.
Prior work LS-ED (Patel et al., 2020) aims to post-tune the scene text recognition (STR) model using a surrogate loss, which is learned with approximation-based optimization.
In order to compare our method with LS-ED, we conduct experiments on the same settings.
Following LS-ED, we learn the surrogate loss using edit distance as the metric, then ﬁne-tune the trained model using our learned loss (without using original loss for fair comparisons).
The results in Table 7 show that our ReLoss signiﬁcantly outperforms the baselines CE and LS-ED.
Note that we only train our ReLoss once then integrate it into training, indicating that our loss is more efﬁcient and general.
Transferability of learned ReLoss.
In all our experiments, we use the same surrogate loss in each task.
If we learn different surrogate losses on speciﬁc datasets, would the performance be better?
To validate this, we conduct experiments to train ReLoss independently on each dataset, as shown in
Table 9.
The ReLoss transferred from ImageNet dataset performs similar to the consistent ReLoss learned on corresponding datasets.
It might be because we train the ReLoss using predicted and randomly generated data, and it is sufﬁcient to cover different distributions of datasets on image classiﬁcation.
We also experiment on NMT task.
As shown in Table 6, we train ReLoss on both
EN → RO and RO → EN, and the results using either of them to train the networks are similar, which demonstrates that the learned ReLoss is language-independent and can bring similar improvements on the other translation direction.
[Table]
[Table] [Table]
Comparison of approximation-based and our correlation-based optimization.
In Figure 2, we compare our ReLoss with approximation-based methods on the synthetic dataset.
Now we further conduct experiments on image classiﬁcation task to show our superiority.
Concretely, we learn the surrogate losses with the same architecture using approximation-based or our correlation-based optimization, then integrate them to train networks on CIFAR datasets.
As shown in Table
[Table]
our ReLoss with correlation-based optimization obtains the highest accuracies compared to the CE loss and approximation-based loss.
Note that the standard derivations of accuracies of approximation-based loss are much larger than CE loss and correlation-based loss; this might be because the imprecise rankings and gradients in approximation-based loss weaken the training stability.
Integrating ReLoss with / without regular losses.
We empirically ﬁnd that the prediction networks using our ReLoss converge very fast at the beginning of training, then the performance will increase very slowly or even get worse, the experiments on synthetic dataset show the similar trend (see
Figure 2 (c)).
A possible reason is that there exist some data points that surrogate losses can not predict accurately, making the optimization fall into local minima.
For better performance, we use the regular loss as a regularization term to help the surrogate losses jump out local minima.
We conduct experiments to show the differences by integrating ReLoss with or without regular loss.
As summarized in Table 10, the performance drops if not adding regular loss in training, showing that the regular losses can bring a decent prior for ReLoss to achieve better performance.
5. C OMPLEXITY A NALYSIS
Denoting the training iterations of surrogate losses as T l , the training epochs and iterations in each epoch of prediction networks are E m and T m , respectively, the runtime complexity of our ReLoss is O
( T l + E m × T m ) .
For comparison, the runtime complexity of regular loss is O
( E m × T m ) , while for previous surrogate loss learning method (Grabocka et al., 2019), it trains the surrogate losses after every iteration and has a runtime complexity of O ( E m × T l + E m × T m ) .
Note that our additional cost
O ( T l ) of learning ReLoss costs only 0.5 GPU hour on image classiﬁcation with a single NVIDIA TITAN
Xp GPU, and we only need to train ReLoss once for each task, reducing much computational cost compared to previous works.
C ONCLUSION
As a proxy of the evaluation metric, loss function matters in machine learning since it controls the optimization of networks.
However, it is often hard to design a loss function with strong relation to the evaluation metric.
In this paper, we aim to address this problem by learning surrogate losses using deep neural networks.
Unlike previous works that pursue an exact recovery of the evaluation metric, we are reminded of the essence of the loss function and evaluation metric, which is to distinguish the performance of models.
We show that directly maximizing the relation between surrogate losses and evaluation metrics through rank correlation can learn better losses.
Our ReLoss achieves signiﬁcant improvements on various tasks, including computer vision tasks and natural language processing tasks.
Besides, we even achieve new state-of-the-art performance on human pose estimation and machine reading comprehension tasks.
A A PPENDIX A. N EURAL A RCHITECTURES OF O UR S URROGATE L OSSES
Image classiﬁcation In order to make the learned surrogate loss generalize to all the classiﬁcation tasks which take accuracy as the metric, we use the logits y pos with positive labels as the input of our neural network, which is the same as cross-entropy loss, and the outputs of the surrogate loss are simply computed through 4-layer perceptions with intermediate activations, i.e. , l = Mean ( FC ( ELU ( FC ( ELU ( FC ( ELU ( FC ( y pos )))))))) , (10) we use ELU activation (Clevert et al., 2015) for stable gradients since it is C ∞ continuous.
Machine reading comprehension and neural machine translation Since the evaluation metrics of MRC and NMT are computed by a sequence of texts, based on the architecture in image classiﬁcation, we use additional self-attention mechanisms (Vaswani et al., 2017) to extract sequential information.
Human pose estimation Given the prediction heatmap and target heatmap, the original MSE loss is used to minimize the distance between these two heatmaps.
Our ReLoss ﬁrst embeds these two heatmaps into two hidden vectors, then computes the MSE loss between them as the ﬁnal loss.
A. T RAINING S TRATEGIES
Surrogate loss We train the surrogate losses using Adam optimizer with a ﬁxed learning rate of 0.01, and the weight decay is set to 1e-4.
Image classiﬁcation The reported models are trained using the same code, with the only difference in the loss function.
On CIFAR-10 and CIFAR-100 datasets, we train ResNet-20 for 200 epochs with an initial learning rate of 0.1, which decays 0.1 at 100th and 150th epochs, the batch size is set to 128 with cutout (DeVries & Taylor, 2017) data augmentation, we run each experiment 5 times with different random seeds and report their mean accuracy with standard derivation.
On ImageNet, we follow the same training strategy as in torchvision 2 (Marcel & Rodriguez, 2010).
Concretely, we train ResNet-50 for 120 epochs with an initial learning rate of 0.1, a step learning rate scheduler which decays 0.1 every 30 epochs is adopted.
While for MobileNet V2, we train it for 300 epochs with 4e-5 weight decay, a cosine learning rate scheduler is adopted with an initial learning rate of 0.045.
The batch sizes for ResNet-50 and MobileNet V2 are both set to 32.
We use SGD optimizer with 0.9 momentum on all datasets.
Note that all the experiments use the same surrogate loss with the same weights.
Human pose estimation We train ResNet-50, HRNet-W32, and
DARK-HRNet-W48 following the default conﬁgurations in MMPose (Contributors, 2020).
Concretely, the models are trained with
Adam optimizer for 210 epochs, and a step learning rate scheduler is adopted with initial value 5e4, which decays 0.1 at 170th and 200th epochs.
The total batch sizes of 8 GPUs with input size 256 × 192 and 384 × 288 are 512 and 256 , respectively.
Machine reading comprehension We train the networks using Adam optimizer with weight decay 0 .
01 , a linear learning rate strategy which warmups 0 .
1 epoch and decays 2 epochs is adopted.
On
DuReader 2.0 dataset, the batch size is set to 32 ; we train MacBERT-base and MacBERT-large with learning rates 3e-5 and 2e-5, respectively.
On SQuAD 1.1 dataset, we train BERT-base and
BERT-large with batch sizes 32 and 2 , and the learning rates are 5e-5 and 1e-5, respectively.
Neural machine translation For WMT16 EN-RO, we use the WMT 2016 corpus, which consists of 610K sentence pairs for training.
We take news-dev-2016 and news-test-2016 as development and test sets.
We learn a joint BPE model with 32K operations and share the vocabulary for source and target languages.
As knowledge distillation (Hinton et al., 2015; Kim & Rush, 2016) has been proven to be crucial for training NAT models, we ﬁrst train an auto-regressive transformer model (Vaswani et al., 2017) as the teacher and then apply sequence-level knowledge distillation to construct the corpus for training NAT models.
The NAT-base takes the same architecture as the base transformer model except that we modify the attention mask of the decoder for not masking the future tokens.
We use a target length predictor to predict the length of the target sentence.
We use golden length during the training and the predicted length during the inference.
For training ReLoss, we only use the outputs of the model.
For NAT-base, the number of training steps is 200K. We select the checkpoint based on the validation set.
We add ReLoss with a factor of 1 on the CE loss to ﬁne-tune the NAT-base for 10k steps with a batch size of 32 and a ﬁxed learning rate of 1e-5.
For BoN-L (N=2) (Shao et al., 2021), we reproduce the results using the public repo 3 .
We combine ReLoss with the BoN Loss to ﬁne-turn the model for 3K steps with a batch size of 512, which keeps the same with the BoN-L1 (N=2).
A. M ORE A BLATION S TUDIES
Performance of losses with different rank correlations To show the inﬂuence of the rank correlations on the performance, we choose the surrogate losses with different rank correlations to train models on CIFAR-10 dataset.
The results in Figure 3 (a) clearly show that the increase of rank correlation boosts the performance, and the losses with lower rank correlations will disturb the training of networks.
Performance of losses with different network capacities Our surrogate losses are constructed with fully connected layers.
To validate the inﬂuence of capacities of loss model, we conduct experiments to learn losses on different capacities on CIFAR-10 dataset.
As shown in Figure 3 (b), our original loss model has 33.4K parameters, we adjust the number of layers or hidden dimensions to change the network capacity.
The results show that the performance of our ReLoss gets saturated on a small number of parameters.
With this small network capacity, its computational cost is negligible in the training of prediction networks.
[Figure]
[Table]
Compare with alternately learning losses The outputs of prediction networks change during the training.
For surrogate losses using approximation-based optimization, it is hard to obtain accurate predictions for all the possible predictions, so the previous works (Grabocka et al., 2019; Patel et al., 2020) learn surrogate losses alternately with prediction networks during training.
In our paper, we show that our ReLoss achieves higher performance using only pre-trained surrogate losses.
We further conduct experiments to train our ReLoss alternately with prediction networks.
As shown in
Table 11, the alternate training obtains higher accuracies but gets more unstable on CIFAR datasets.
We think this might be because the update of weights in surrogate losses will disturb the gradients, while our ReLoss without alternate training provides the same gradients for the same predictions and labels, thus more stable.
Since the performance improvements of alternate training are marginal, we can use pre-trained losses for better generalization and efﬁciency.
Effect of gradient penalty Our paper aims to stabilize the gradients of surrogate losses by introducing a gradient penalty regularization in loss learning.
We conduct experiments to show the effectiveness of the gradient penalty.
As summarized in Table 12, we train the models on CIFAR datasets using the surrogate losses with or without gradient penalty.
The results show that the ReLoss without gradient penalty performs poorly compared to the one with gradient penalty and even the original loss, although it obtains a good rank correlation.
It indicates that the regularization of gradients of the surrogate losses is necessary and contributes a lot to the performance.
[Table] As for the weight λ in Eq.(9), we empirically ﬁnd that this regularization is easy to achieve since our learning objective of correlation is weak.
We have tried different values of λ , the term of gradient penalty is always very small ( ∼ 1e-3), so we directly follow previous work (Gulrajani et al., 2017) and use λ = 10 .
Robustness of performance in multiple independent runs To validate the robustness of the training of ReLoss, we conduct experiments to train the ReLoss multiple times independently, and leverage these learned surrogate losses to train ResNet-56 on CIFAR datasets.
As the results summarized in Table 13, the accuracies of multiple runs are similar (with low standard variance), showing that our ReLoss can obtain stable results.
We believe that our ReLoss is easy to learn, and the regularization term of gradient penalty could obtain stable gradients of surrogate losses w.r.t.
the logits.
As a result, the performance would be robust.
[Table] [Figure] V ISUALIZATION OF C ONVERGENCE C URVES IN T RAINING
We visualize the convergence curves of CE loss and our surrogate loss in Figure 4.
We can see that our loss obtains higher validation accuracies over the whole training procedure.
[Figure]