A UTO S EG -L OSS : S EARCHING M ETRIC S URROGATES FOR S EMANTIC S EGMENTATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Designing proper loss functions is essential in training deep networks. Especially in the ﬁeld of semantic segmentation, various evaluation metrics have been proposed for diverse scenarios.
Despite the success of the widely adopted crossentropy loss and its variants, the mis-alignment between the loss functions and evaluation metrics degrades the network performance.
Meanwhile, manually designing loss functions for each speciﬁc metric requires expertise and signiﬁcant manpower. In this paper, we propose to automate the design of metric-speciﬁc loss functions by searching differentiable surrogate losses for each metric. We substitute the non-differentiable operations in the metrics with parameterized functions, and conduct parameter search to optimize the shape of loss surfaces. Two constraints are introduced to regularize the search space and make the search efﬁcient.
Extensive experiments on PASCAL VOC and Cityscapes demonstrate that the searched surrogate losses outperform the manually designed loss functions consistently. The searched losses can generalize well to other datasets and networks.
Code shall be released.
I NTRODUCTION
Loss functions are of indispensable components in training deep networks, as they drive the feature learning process for various applications with speciﬁc evaluation metrics.
However, most metrics, like the commonly used 0-1 classiﬁcation error, are non-differentiable in their original forms and cannot be directly optimized via gradient-based methods.
Empirically, the cross-entropy loss serves well as an effective surrogate objective function for a variety of tasks concerning categorization.
This phenomenon is especially prevailing in image semantic segmentation, where various evaluation metrics have been designed to address the diverse task focusing on different scenarios.
Some metrics measure the accuracy on the whole image, while others focus more on the segmentation boundaries.
Although cross-entropy and its variants work well for many metrics, the mis-alignment between network training and evaluation still exist and inevitably leads to performance degradation.
Typically, there are two ways for designing metric-speciﬁc loss functions in semantic segmentation.
The ﬁrst is to modify the standard cross-entropy loss to meet the target metric (Ronneberger et al., 2015; Wu et al., 2016).
The other is to design other clever surrogate losses for speciﬁc evaluation metrics (Rahman & Wang, 2016; Milletari et al., 2016).
Despite the improvements, these handcrafted losses need expertise and are non-trivial to extend to other evaluation metrics.
In contrast to designing loss functions manually, an alternative approach is to ﬁnd a framework that can design proper loss functions for different evaluation metrics in an automated manner, motivated by recent progress in AutoML (Zoph & Le, 2017; Pham et al., 2018; Liu et al., 2018; Li et al., 2019).
Although automating the design process for loss functions is attractive, it is non-trivial to apply an
AutoML framework to loss functions. Typical AutoML algorithms require a proper search space, in which some search algorithms are conducted. Previous search spaces are either unsuitable for loss design, or too general to be searched efﬁciently. Recently Li et al.
(2019) and Wang et al.
(2020) proposed search spaces based on existing handcrafted loss functions.
And the algorithm searches for the best combination.
However, these search spaces are still limited to the variants of cross-entropy loss, and thus do not address the mis-alignment problem well.
In this paper, we propose a general framework for searching surrogate losses for mainstream nondifferentiable segmentation metrics.
The key idea is that we can build the search space according to the form of evaluation metrics.
In this way, the training criteria and evaluation metrics are uniﬁed.
Meanwhile, the search space is compact enough for efﬁcient search.
Speciﬁcally, the metrics are ﬁrst relaxed to the continuous domain by substituting the one-hot prediction and logical operations, which are the non-differentiable parts in most metrics, with their differentiable approximations.
Parameterized functions are introduced to approximate the logical operations, ensuring that the loss surfaces are smooth while accurate for training.
The loss parameterization functions can be of arbitrary families deﬁned on [0 , 1] .
Parameter search is further conducted on the chosen family so as to optimize the network performance on the validation set with the given evaluation metric.
Two essential constraints are introduced to regularize the parameter search space.
We ﬁnd that the searched surrogate losses can effectively generalize to different networks and datasets.
Extensive experiments on Pascal VOC (Everingham et al., 2015) and Cityscapes (Cordts et al., 2016) show our approach delivers accuracy superior than the existing losses speciﬁcally designed for individual segmentation metrics with a mild computational overhead.
Our contributions can be summarized as follows: 1) Our approach is the ﬁrst general framework of surrogate loss search for mainstream segmentation metrics.
We propose an effective parameter regularization and parameter search algorithm, which can ﬁnd loss surrogates optimizing the target metric performance with mild computational overhead.
The surrogate losses obtained via the proposed searching framework promote our understandings on loss function design and by themselves are novel contributions, because they are different from existing loss functions speciﬁcally designed for individual metrics, and are transferable across different datasets and networks.
R ELATED W ORK
Loss function design is an active topic in deep network training (Ma, 2020).
In the area of image semantic segmentation, cross-entropy loss is widely used (Ronneberger et al., 2015; Chen et al., 2018).
But the cross-entropy loss is designed for optimizing the global accuracy measure (Rahman & Wang, 2016;
Patel et al., 2020), which is not aligned with many other metrics.
Numerous studies are conducted to design proper loss functions for the prevalent evaluation metrics.
For the mIoU metric, many works (Ronneberger et al., 2015; Wu et al., 2016) incorporate class frequency to mitigate the class imbalance problem.
For the boundary F1 score, the losses at boundary regions are up-weighted (Caliva et al., 2019; Qin et al., 2019), so as to deliver more accurate boundaries.
These works carefully analyze the property of speciﬁc evaluation metrics, and design the loss functions in a fully handcrafted way, which needs expertise.
By contrast, we propose a uniﬁed framework for deriving parameterized surrogate losses for various evaluation metrics.
Wherein, the parameters are searched by reinforcement learning in an automatic way.
The networks trained with the searched surrogate losses deliver accuracy on par or even superior than those with the best handcrafted losses.
Direct loss optimization for non-differentiable evaluation metrics has long been studied for structural SVM models (Joachims, 2005;
Yue et al., 2007; Ranjbar et al., 2012).
However, the gradients w.r.t.
features cannot be derived from these approaches.
Therefore, they cannot drive the training of deep networks through back-propagation.
Hazan et al. (2010) proposes to optimize structural SVM with gradient descent, where loss-augmented inference is applied to get the gradients of the expectation of evaluation metrics. Song et al.
(2016) further extends this approach to non-linear models (e.g., deep neural networks).
However, the computational complexity is very high during each step in gradient descent.
Although Song et al.
and
Mohapatra et al.
(2018) have designed efﬁcient algorithms for the Average Precision (AP) metric, other metrics still need specially designed efﬁcient algorithms.
Our method, by contrast, is general for the mainstream segmentation metrics.
Thanks to the good generalizability, our method only needs to perform the search process once for a speciﬁc metric, and the searched surrogate loss can be directly used henceforth.
Applying the searched loss for training networks brings very little additional computational cost.
Surrogate loss is introduced to derive loss gradients for the non-differentiable evaluation metrics.
There are usually two ways for designing surrogate losses.
The ﬁrst is to handcraft an approximated differentiable metric function.
For the IoU measure, Rahman & Wang (2016) propose to approximate the intersection and union seperately using the softmax probabilities in a differentiable form, and show its effectiveness on binary segmentation tasks.
Berman et al.
(2018) further deal with multi-class segmentation problems by extending mIoU from binary inputs to the continuous domain with the convex Lov`asz extension, and their method outperforms standard cross entropy loss in multi-class segmentation tasks.
For the F1 measure, dice loss is proposed by Milletari et al.
as a direct objective by substituting the binary prediction with the softmax probability.
In spite of the success, they do not apply for other metrics.
The second solution is to train a network to approximate the target metric.
Nagendar et al.
(2018) train a network to approximate mIoU. Patel et al.
(2020) design a neural network to learn embeddings for predictions and ground truths for tasks other than segmentation.
This line of research focuses on minimizing the approximation error w.r.t.
the target metrics.
But there is no guarantee that their approximations provide good loss signals for training.
These approximated losses are just employed in a post-tuning setup, still relying on cross-entropy pre-trained models.
Our method signiﬁcantly differs in that we search surrogate losses to directly optimize the evaluation metrics in applications.
AutoML is a long-pursued target of machine learning (He et al., 2019).
Recently a sub-ﬁeld of
AutoML, neural architecture search (NAS), has attracted much attention due to its success in automating the process of neural network architecture design (Zoph & Le, 2017; Pham et al., 2018;
Liu et al., 2018).
As an essential element, loss function has also raised the interest of researchers to automate its design process.
Li et al.
(2019) and Wang et al.
(2020) design search spaces based on existing human-designed loss functions and search for the best combination parameters.
There are two issues: a) the search process outputs whole network models rather than loss functions.
For every new network or dataset, the expensive search procedure is conducted again, and b) the search space are ﬁlled with variants of cross-entropy, which cannot solve the mis-alignment between cross-entropy loss and many target metrics.
By contrast, our method outputs the searched surrogate loss functions of close form with the target metrics, which are transferable between networks and datasets.
R EVISITING E VALUATION M ETRICS FOR S EMANTIC S EGMENTATION
Various evaluation metrics are deﬁned for semantic segmentation, to address the diverse task focusing on different scenarios.
Most of them are of three typical classes:
Acc-based, IoU-based, and
F1-score-based.
This section revisits the evaluation metrics, under a uniﬁed notation set.
Table 1 summarizes the mainstream evaluation metrics.
The notations are as follows: suppose the validation set is composed of N images, labeled with categories from C classes (background included).
Let I n , n ∈ { 1 , . . .
, N } be the n -th image, and Y n be the corresponding ground-truth segmentation mask.
Here Y n = { y n,c,h,w } c,h,w is a one-hot vector, where y n,c,h,w ∈ { 0 , 1 } indi- cates whether the pixel at spatial location ( h, w ) belongs to the c -th category ( c ∈ { 1 , . . .
, C } ).
In evaluation, the ground-truth segmentation mask Y n is compared to the network prediction ˆ Y n = { ˆ y n,c,h,w } c,h,w , where ˆ y n,c,h,w ∈ { 0 , 1 } .
ˆ y n,c,h,w is quantized from the continuous scores produced by the network (by argmax operation).
Acc-based metrics.
The global accuracy measure (gAcc) counts the number of pixels correctly classiﬁed.
It can be written with logical operator AND as Eq.
The gAcc metric counts each pixel equally, so the results of the long-tailed categories have little impact on the metric number.
The mean accuracy (mAcc) metric mitigates this by normalizing within each category as in Eq.
IoU-based metrics.
The evaluation is on set similarity rather than pixel accuracy.
The intersectionover-union (IoU) score is evaluated between the prediction and the ground-truth mask of each category.
The mean IoU (mIoU) metric averages the IoU scores of all categories, as in Eq.
In the variants, the frequency weighted IoU (FWIoU) metric weighs each category IoU score by the category pixel number, as in Eq.
The boudary IoU (BIoU) (Kohli et al., 2009) metric only cares the segmentation quality around the boundary, so it picks the boundary pixels out in evaluation and ignores the rest pixels.
It can be calculated with Eq.
(5), in which BD( y n ) denotes the boundary region in map y n .
BD( y n ) is derived by applying XOR operation on the min-pooled ground-truth mask.
The stride of the Min - Pooling( · ) is 1.
F1-score-based metrics.
F1-score is a criterion that takes both precision and recall into consideration.
A well-known metric of this type is boundary F1-score (BF1-score)
(Csurka et al., 2013), which is widely used for evaluating boundary segmentation accuracy.
The computation of precision and recall in BF1-score is as in Eq.
(6), where BD(ˆ y n ) and BD( y n ) are derived from Eq.
Max pooling with stride 1, Max - Pooling( · ) , is applied on the boundary regions to allow error tolerance.
[Table]
y XOR Min - Pooling( y ) (5)
F1-score-based Boundary F1 Score ∗
BF1 - score = 1 C (cid:88) c 2 × prec c × recall c (prec c + recall c ) where prec c = (cid:80) n,h,w BD(ˆ y n ) c,h,w AND Max - Pooling(BD( y n ) c,h,w ) (cid:80) n,h,w BD(ˆ y n ) c,h,w , recall c = (cid:80) n,h,w Max - Pooling(BD(ˆ y n ) c,h,w ) AND(BD( y n ) c,h,w ) (cid:80) n,h,w BD( y n ) c,h,w (6)
A UTO S EG -L OSS F RAMEWORK
In the Auto Seg-Loss framework, the evaluation metrics are transferred into continuous surrogate losses with learnable parameters, which are further optimized.
Fig.
1 illustrates our approach.
4. E XTENDING METRICS TO SURROGATES
As shown in Section 3, most segmentation metrics are non-differentiable because they take onehot prediction maps as input, and contain binary logical operations.
We extend these metrics to be continuous loss surrogates by smoothing the non-differentiable operations within.
Extending One-hot Operation.
The one-hot prediction map, ˆ Y n = { ˆ y n,c,h,w } c,h,w , is derived by picking the highest scoring category at each pixel, which is further turned into one-hot form.
Here, we approximate the one-hot predictions with softmax probabilities, as,
[Equation]
(7) where z n,c,h,w ∈ R is the category score output by the network (without normalization).
The approximated one-hot prediction is denoted by (cid:101) y n,c,h,w .
Extending Logical Operations.
As shown in Table 1, the non-differentiable logical operations, f AND ( y 1 , y 2 ) , f OR ( y 1 , y 2 ) , and f XOR ( y 1 , y 2 ) , are of indispensable components in these metrics.
Because the XOR operation can be constructed by AND and OR , f XOR ( y 1 , y 2 ) =
f OR ( y 1 , y 2 ) − f AND ( y 1 , y 2 ) , we focus on extending f AND ( y 1 , y 2 ) and f OR ( y 1 , y 2 ) to the continuous domain.
Following the common practice, the logical operators are substituted with arithmetic operators
[Equation]
where y 1 , y 2 ∈ { 0 , 1 } .
Eq.
(8) can be directly extended to take continuous y 1 , y 2 ∈ [0 , 1] as in- puts.
By such an extension, together with the approximated one-hot operation, a na¨ıve version of differentiable surrogate losses can be obtained.
The strength of such surrogates is that they are directly derived from the metrics, which signiﬁcantly reduces the gap between training and evaluation.
However, there is no guarantee that the loss surfaces formed by na¨ıvely extending Eq.
(8) provide trainingset 𝒮𝒮 hold-outsubset trainingsubset categoryscore proxynetwork one-hot Softmax ℎ AND ( �𝑦𝑦 1 , �𝑦𝑦 2 ; Θ AND ) ··· ··· ̃𝜉𝜉 Θ (N 𝜔𝜔 ; 𝒮𝒮 train ) Softmax segmentationnetwork Surrogate Parameter Search Space gradients gradients
Optimal Parameterization category score softmaxprobability one-hotprediction
AND
OR ℎ OR ( �𝑦𝑦 1 , �𝑦𝑦 2 ; Θ OR ) surrogate metric surrogate metric ̃𝜉𝜉 Θ ∗ (N 𝜔𝜔′ ; 𝒮𝒮 )
ℎ
AND ( �𝑦𝑦 1 , �𝑦𝑦 2 ; Θ AND∗ ) ℎ
OR ( �𝑦𝑦 1 , �𝑦𝑦 2 ; Θ OR∗ ) targetmetric rewards
Auto Seg-Loss Parameter Search
Searchedparameters Θ ∗
Searched Surrogate Loss 𝒮𝒮 holdout split
N 𝜔𝜔 𝒮𝒮 train proxynetwork
N 𝜔𝜔
N 𝜔𝜔′ 𝑍𝑍 �𝑌𝑌 𝑍𝑍 �𝑌𝑌 categoryscore softmax probability 𝑍𝑍 �𝑌𝑌 𝜉𝜉 (N 𝜔𝜔 ; 𝒮𝒮 holdout ) accurate loss signals.
To adjust the loss surfaces, we parameterize the AND and OR functions as h AND ( y 1 , y 2 ; θ AND ) = g ( y [Equation]
; θ AND ) g ( y
[Equation]
; θ AND ) [Equation] 9) where g ( y ; θ ) : [0 , 1] → R is a scalar function parameterized by θ .
The parameterized function g ( y ; θ ) can be from arbitrary function families deﬁned on [0 , 1] , e.g., piecewise linear functions and piecewise B´ezier curves.
With a chosen function family, the parameters θ control the shape of loss surfaces.
We seek to search for the optimal parameters θ so as to maximize the given evaluation metric.
Meanwhile, optimal parameter search is non-trivial.
With the introduced parameters, the plasticity of loss surfaces is strong.
The parameterized loss surfaces may well be chaotic, or be far away from the target evaluation metric even at the binary inputs.
For more effective parameter search, we regularize the loss surfaces by introducing two constraints on g ( y ; θ ) .
Truth-table constraint is introduced to enforce the surrogate loss surfaces taking the same values as the evaluation metric score at binary inputs.
This is applied by enforcing
[Equation]
Thus, the parameterized functions h ( y 1 , y 2 ; θ ) preserve the behavior of the corresponding logical operations f ( y 1 , y 2 ) on binary inputs y 1 , y 2 ∈ { 0 , 1 } .
Monotonicity constraint is introduced based on the observation of monotonicity tendency in the truth tables of AND and OR .
It pushes the loss surfaces towards a benign landscape, avoiding dramatic non-smoothness.
The monotonicity constraint is enforced on h AND ( y 1 , y 2 ) and h OR ( y 1 , y 2 ) , as [Equation]
Applying the chain rule and the truth table constraint, the monotonicity constraint implies
[Equation]
Empirically we ﬁnd it important to enforce these two constraints in parameterization.
Extending Evaluation Metrics.
Now we can extend the metrics to surrogate losses by a) replacing the one-hot predictions with softmax probabilities, and b) substituting the logical operations with parameterized functions.
Note that if the metric contains several logical operations, their parameters will not be shared.
The collection of parameters in one metric are denoted as Θ .
For a segmentation network N and evaluation dataset S , the score of the evaluation metric is denoted as ξ (N; S ) .
And the parameterized surrogate loss is denoted as (cid:101) ξ Θ (N; S ) .
4. S URROGATE PARAMETERIZATION
The parameterized function can be from any function families deﬁned on [0, 1], such as picewise
B´ezier curve and piecewise linear functions.
Here we choose the piecewise B´ezier curve for parameterizing g ( y ; θ ) , which is widely used in computer graphics and is easy to enforce the constraints via its control points.
We also verify the effectiveness of parameterizing g ( y ; θ ) by piecewise linear functions.
See Fig.
2 for visualization and Appendix B for more details.
A piecewise B´ezier curve consists of a series of quadratic B´ezier curves, where the last control point of one curve segment coincides with the ﬁrst control point of the next curve segment.
If there are n segments in a piecewise B´ezier curve, the k -th segment is deﬁned as
[Equation]
where s transverses the k -th segment, B 2 k
+ i =
( B (2 k + i ) ,u , B (2 k + i ) ,v ) ( i = 0 , 1 , 2) denotes the i -th control point on the k -th segment, in which u, v index the 2-d plane axes.
A piecewise B´ezier curve with n segments has 2 n + 1 control points in total.
To parameterize g ( y ; θ ) , we assign
[Equation]
where θ is the control point set, B 2 k,u < B (2 k +1) ,u < B (2 k +2) ,u , 0 ≤ k ≤ n − 1 .
Given an input y , the segment index k and the transversal parameter s are derived from Eq.
(13c) and Eq.
(13a), respectively.
Then g ( y ; θ ) is assigned as Eq.
(13b).
Because g ( y ; θ ) is deﬁned on y ∈ [0 , 1] , we arrange the control points in the u -axis as, B 0 ,u = 0 , B 2 n,u = 1 , where the u -coordinate of the ﬁrst and the last control points are at 0 and 1 , respectively.
The strength of the piecewise B´ezier curve is that the curve shape is deﬁned explicitly via the control points.
Here we enforce the truth-table and the monotonicity constraints on the control points via,
[Equation]
To fulﬁll the above restrictions in optimization, the speciﬁc form of the parameters is given by
[Equation]
with B 0 = (0 , 0) and B 2 n = (1 , 1) ﬁxed.
So every θ i =
( θ i,u , θ i,v ) is in range [0 , 1]
and it is straight-forward to compute the actual coordinates of control points from this parameterized form.
Such parameterization makes each θ i independent with each other, and thus simpliﬁes the optimization.
By default, we use piecewise B´ezier curve with two segments to parameterize g ( y, θ ) .
y 0.
g ( y ; θ ) 0.
s 1 − s ( B 2 ,u ,B 2 ,v ) ( B 3 ,u ,B 3 ,v ) ( B 4 ,u ,B 4 ,v ) v u y =(
1 − s )
2 B 2 ,u +2 s (1 − s ) B 3 ,u + s 2 B 4 ,u g ( y ; θ )=(1 − s ) 2 B 2 ,v +2 s (1 − s ) B 3 ,v + s 2 B 4 ,v using Piecewise B´ezier curve with four segments.
The red points are control points. The purple point is on the curve, which shows the relationship among y , g ( y ; θ ) and the transversal parameter s .
Algorithm
Input: Initialized network N ω 0 , initialized distribution µ 1 and σ 2 , target metric ξ , training set S train and hold-out training set S hold-out Result: Obtained optimal parameters Θ ∗ for t = 1 to T do for i = 1 to M do
Sample parameter Θ ( t ) i ∼ N trunc [0 , 1] ( µ t , σ 2 I ) ;
Network training ω
[Equation]
arg
[Equation]
[Equation]
(N
[Equation]
; S train ) , with w initialized from w 0 ; Compute the evaluation metric score ξ (Θ ( t ) i ) =
ξ (N ω ∗ (Θ (t)i ) ; S hold-out ) ; end Update µ t
[Equation]
= arg
[Equation] [Equation]
, Θ
[Equation]
t ) i ) ; end return Θ ∗ = arg max µ
[Equation]
M
[Equation]
t )
i
[Equation]
, ∀
[Equation]
, T +
[Figure]
4. S URROGATE PARAMETER OPTIMIZATION
Algorithm 1 describes our parameter search algorithm.
The training set is split into two subsets, S train for training and S hold-out for evaluation in the search algorithm, respectively.
Speciﬁcally, suppose we have a segmentation network N ω with weights ω , our search target is the parameters that maximize the evaluation metric on the hold-out training set ξ (N ω ;
S hold-out )
[Equation]
To optimize Eq.
(14), the segmentation network is trained with SGD as the inner-level problem.
At the outer-level, we use reinforcement learning as our searching algorithm, following the common practice in AutoML (Zoph & Le, 2017; Pham et al., 2018).
Other searching algorithms, such as evolutionary algorithm, may also be employed.
Speciﬁcally, the surrogate parameters are searched via the PPO2 algorithm (Schulman et al., 2017).
The process consists of T sampling steps.
In the t -th step, we aim to explore the search space around that from t − 1 .
Here M parameters { Θ ( t ) i } Mi =1 are sampled independently from a truncated normal distribution (Burkardt, 2014), as Θ ∼ N trunc [0 , 1] ( µ t , σ 2 I ) , with each variable in range [0 , 1] .
In it, µ t and σ 2 I denote the mean and covariance of the parent normal distribution ( σ is ﬁxed as 0.2 in this paper).
µ t summarizes the information from the ( t − 1) -th step.
M surrogate losses are constructed with the sampled parameters, which drive the training of M segmentation networks separately.
To optimize the outer-level problem, we evaluate these models with the target metric and take the evaluation scores as rewards for PPO2.
Following the PPO2 algorithm, µ t +1 is computed as µ t +1 = arg max µ 1 M (cid:80) Mi
=1 R ( µ, µ t , Θ i ) ,where the reward R ( µ, µ t , Θ i ) is as R ( µ, µ t , Θ i ) =
min
[Equation]
2 I )
[Equation]
p
[Equation]
(cid:19) [Equation] (Θ
[Equation]
) (cid:19) , where min( · , · ) picks the smaller item from its inputs, CLIP ( x, 1 − (cid:15), 1 + (cid:15) )
clips x to be within 1 − (cid:15) and 1 + (cid:15) , and p (Θ i ; µ, σ 2 I ) is the PDF of the truncated normal distribution.
Note that the mean reward of the M samples is subtracted when computing ξ (Θ i ) for better convergence.
After
T steps, the mean µ t with the highest average evaluation score is output as the ﬁnal parameters Θ ∗ .
Empirically we ﬁnd the searched losses have good transferability, i.e., they can be applied for different datasets and networks.
Beneﬁting from this, we use a light proxy task for parameter search.
In it, we utilize a smaller image size, a shorter learning schedule and a lightweight network.
Thus, the whole search process is quite efﬁcient (8 hours on PASCAL VOC with 8 NVIDIA Tesla V
GPUs).
More details are in Appendix A.
In addition, the search process can be conducted only once for a speciﬁc metric and the resulting surrogate loss can be directly used for training henceforth.
E XPERIMENTS
We evaluate on the PASCAL VOC 2012 (Everingham et al., 2015) and the Cityscapes (Cordts et al., 2016) datasets.
We use Deeplabv
+ (Chen et al., 2018) with ResNet-50/101 (He et al., 2016) as the network model.
During the surrogate parameter search, we randomly sample 1500 training images in
PASCAL VOC and 500 training images in Cityscapes to form the hold-out set S hold-out , respectively.
The remaining training images form the training set S train in search.
µ 0 is set to make g ( y ; θ )
=
y .
The backbone network is ResNet-50.
The images are down-sampled to be of 128 × 128 resolution.
SGD lasts only 1000 iterations with a mini-batch size of 32.
After the search procedure, we re-train the segmentation networks with ResNet-101 using the searched losses on the full training set and evaluate them on the actual validation set.
The re-train settings are the same as Deeplabv
+ (Chen et al., 2018), except that the loss function is substituted by the obtained surrogate loss.
The search time is counted on 8 NVIDIA Tesla V100 GPUs.
More details are in Appendix A.
5. S EARCHING FOR D IFFERENT M ETRICS
In Table 2, we compare our searched surrogate losses against the widely-used cross-entropy loss and its variants, and some other metric-speciﬁc surrogate losses.
We also seek to compare with the
AutoML-based method in Li et al.
(2019), which was originally designed for other tasks.
But we cannot get reasonable results due to convergence issues.
The results show that our searched losses are on par or better the previous losses on their target metrics.
It is interesting to note that the obtained surrogates for boundary metrics (such as BIoU and BF1) only focus on the boundary areas, see Appendix C for further discussion.
We also tried training segmentation networks driven by both searched mIoU and BIoU/BF1 surrogate losses.
Such combined losses reﬁne the boundaries while keeping reasonable global performance.
[Table]
5. G ENERALIZATION OF THE L OSS
Generalization among datasets.
Table 3 evaluates the generalization ability of our searched loss surrogates among different datasets.
Due to limited computational resource, we train networks only with the searched mIoU, BF1 and mAcc surrogate losses.
The results show that our searched surrogate losses generalize well between these two datasets with quite different scenes and categories
[Table] [Table]
Generalization among segmentation networks.
The surrogate losses are searched with ResNet50 + DeepLabv3+ on PASCAL VOC.
The searched losses drive the training of ResNet-101 +
DeepLabv3+, PSPNet (Zhao et al., 2017) and HRNet (Sun et al., 2019) on PASCAL VOC.
Table 4 shows the results.
The results demonstrate that our searched loss functions can be applied to various semantic segmentation networks.
5. A BLATION
Parameterization and constraints.
Table 5 ablates the parameterization and the search space constraints.
In it, a surrogate without parameters refers to Eq. (8), with the domain extended from discrete points { 0, 1 } to continuous interval [0, 1]. This naive surrogate deliver much lower accuracy, indicating the essence of parameterization. Without the truth-table constraint, the training process diverges at the very beginning, where the loss gradients become “NaN”.
And the performance drops if the monotonicity constraint is not enforced.
The performance drops or even the algorithm fails without the constraints. [Table]
Proxy tasks for parameter search.
Table 6 ablates this.
The bottom row is our default setting with a light-weight backbone, down-sampled image size and shorter learning schedule.
The default setting delivers on par accuracy with heavier settings.
This is consistent with the generalization ability of our surrogate losses.
Thus we can improve the search efﬁciency via light proxy tasks.
Parameter search algorithm.
Fig.
3 compares the employed PPO2 (Schulman et al., 2017) algorithm with random search.
The much better performance of PPO2 suggests that surrogate loss search is non-trivial and reinforcement learning helps to improve the search efﬁciency.
[Table]
[Figure]
[Figure]
C ONCLUSION
The introduced Auto Seg-Loss is a powerful framework to search for the parameterized surrogate losses for mainstream segmentation evalutation metrics.
The non-differentiable operators are substituted by their parameterized continuous counterparts.
The parameters are optimized to improve the ﬁnal evaluation metrics with essential constraints.
It would be interesting to extend the framework to more tasks, like object detection, pose estimation and machine translation problems.