P OLY L OSS : A P OLYNOMIAL E XPANSION P ERSPECTIVE OF C LASSIFICATION L OSS F UNCTIONS
A BSTRACT
Cross-entropy loss and focal loss are the most common choices when training deep neural networks for classiﬁcation problems.
Generally speaking, however, a good loss function can take on much more ﬂexible forms and should be tailored for different tasks and datasets.
Motivated by how functions can be approximated via Taylor expansion we propose a simple framework, named PolyLoss , to view and design loss functions as a linear combination of polynomial functions.
Our
PolyLoss allows the importance of different polynomial bases to be easily adjusted depending on the targeting tasks and datasets, while naturally subsuming the aforementioned cross-entropy loss and focal loss as special cases.
Extensive experimental results show that the optimal choice within the PolyLoss is indeed dependent on the task and dataset.
Simply by introducing one extra hyperparameter and adding one line of code, our Poly-1 formulation outperforms the crossentropy loss and focal loss on 2D image classiﬁcation, instance segmentation, object detection, and 3D object detection tasks, sometimes by a large margin
[Table]
I NTRODUCTION
Loss functions are important in training neural networks.
In principle, a loss function could be any (differentiable) function that maps predictions and labels to a scalar.
Therefore, designing a good loss function is generally challenging due to its large design space and designing a universal loss function that works across different tasks and datasets is even more challenging: for example, L1 /
L2 losses are commonly used for regression tasks, but they are rarely used for classiﬁcation tasks; focal loss is often used to alleviate the overﬁtting issue of cross-entropy loss for imbalanced object detection datasets (Lin et al., 2017), but it is not shown to consistently help other tasks.
Many recent works have also explored new loss functions via meta-learning, ensembling or compositing different losses (Hajiabadi et al., 2017; Xu et al., 2018; Gonzalez & Miikkulainen, 2020b;a; Li et al., 2019).
In this paper we propose PolyLoss : a novel framework for understanding and designing loss functions.
Our key insight is to decompose commonly used classiﬁcation loss functions, such as crossentropy loss and focal loss, into a series of weighted polynomial bases.
They are decomposed in the form of (cid:80) ∞
j =1 α j (1 − P t ) j , where α j ∈ R + is the polynomial coefﬁcient and P t is the prediction probability of the target class label.
Each polynomial base (1 − P t )
j is weighted by a corresponding polynomial coefﬁcient α j , which enables us to easily adjust the importance of different bases for different applications.
When α j = 1 /j for all j , our PolyLoss becomes equivalent to the commonly used cross-entropy loss, but this coefﬁcient assignment may not be optimal.
Our study shows that, in order to achieve better results, it is necessary to adjust polynomial coefﬁcients α j for different tasks and datasets.
Since it is impossible to adjust an inﬁnite number of α j we explore various strategies with a small degree of freedom.
Perhaps surprisingly, we observe that simply adjusting the single polynomial coefﬁcient for the leading polynomial, which we denote
L Poly-1 , is sufﬁcient to achieve signiﬁcant improvements over the commonly used cross-entropy loss and focal loss.
Overall, our contribution can be summarized as:
• Insights on common losses : We propose a uniﬁed framework, named PolyLoss , to rethink and redesign loss functions.
This framework helps to explain cross-entropy loss and focal loss as two special cases of the PolyLoss family (by horizontally shifting polynomial coefﬁcients), which was not recognized before.
This new ﬁnding motivates us to investigate new loss functions that vertically adjust polynomial coefﬁcients, shown in Figure 1.
• New loss formulation: We evaluate different ways of vertically manipulating polynomial coefﬁcients to simplify the hyperparameters search space.
We propose a simple and effective Polyloss formulation which only introduces one hyperparameter and one line of code.
• New ﬁndings: We identify that focal loss, though effective for many detection tasks, is suboptimal for the imbalanced ImageNet-21K.
We ﬁnd the leading polynomial contributes to a large portion of the gradient during training, and its coefﬁcient correlates to the prediction conﬁdence P t .
In addition, we provide an intuitive explanation on how to leverage this correlation to design good PolyLoss tailored to imbalanced datasets.
• Extensive experiments: We evaluate our PolyLoss on different tasks, models, and datasets.
Results show PolyLoss consistently improves the performance on all fronts, summarized in Table 1, which includes the state-of-the-art classiﬁers EfﬁcientNetV2 and detectors RSN.
R ELATED W ORK
Cross-entropy loss is used in many popular and current state-of-the-art models for perception tasks such as classiﬁcation, detection and semantic segmentation (Tan & Le, 2021;
He et al., 2017; Zoph et al., 2020; Tao et al., 2020).
Instead of trying to improve cross-entropy loss for speciﬁc tasks like many other prior works, the goal of this paper is to provide a uniﬁed framework for systematically designing a better classiﬁcation loss function.
Loss for class imbalance Training detection models, especially single-stage detectors, is difﬁcult due to class imbalance.
Common approaches such as hard example mining and reweighing are developed to address the class imbalance issue (Sung, 1996; Viola & Jones, 2001; Felzenszwalb et al., 2010;
Shrivastava et al., 2016;
Liu et al., 2016; Bulo et al., 2017).
As one of these approaches, focal loss is designed to mitigate the class imbalance issue by focusing on the hard examples and is used to train state-of-the-art 2D and 3D detectors (Lin et al., 2017; Tan et al., 2020; Du et al., 2020; Shi et al., 2020; Sun et al., 2021).
In our work, we found that focal loss is suboptimal for the imbalanced ImageNet-21K. Using the PolyLoss framework, we discover a better loss function, which performs the opposite role of focal loss.
We further provide intuitive understanding of why it is important to design different loss functions tailored to different imbalanced datasets using the
PolyLoss framework.
Robust loss to label noise Another direction of research is to design loss functions that are robust to label noise (Ghosh et al., 2015; 2017; Zhang & Sabuncu, 2018;
Wang et al., 2019; Oksuz et al., 2020; Menon et al., 2020).
A commonly used approach is to incorporate noise robust loss function such as Mean Absolute Error (MAE) into cross-entropy loss.
In particular, Taylor cross entropy loss is proposed to unify MAE and cross-entropy loss by expanding the cross-entropy loss in (1 − P t )
j polynomial bases (Feng et al., 2020).
By truncating the higher-order polynomials, they show truncated cross-entropy loss function is closer to MAE, which is more robust to label noise on datasets with synthetic label noise.
In contrast, our PolyLoss provides a more general framework to design loss functions for different datasets by manipulating polynomial coefﬁcients, which includes dropping higher-order polynomials proposed in Feng et al.
Our experiments in subsection 4. show the loss proposed in Feng et al. (2020) performs worse than cross-entropy loss on the clean
ImageNet dataset.
Several recent works demonstrate learning the loss function during training via gradient descent or meta learning (Hajiabadi et al., 2017; Xu et al., 2018; Gonzalez & Miikkulainen, 2020a; Li et al., 2019; 2020).
Notably, TaylorGLO utilizes CMA-ES to optimize multivariate Taylor parameterization of a loss function and learning rate schedule during training (Hansen & Ostermeier, 1996; Gonzalez & Miikkulainen, 2020b).
Due to the search space scale with the order
[Figure]
of polynomials
, the paper demonstrates that using the third-order parameterization (8 parameters), the learned loss function schedule outperforms cross-entropy loss on 10-class classiﬁcation problems.
Our paper (Figure 2a), on the other hand shows for 1000-class classiﬁcation tasks, hundreds of polynomials are needed.
This results in a prohibitively large search space.
Our proposed Polyformulation mitigates the challenge of the large search space and do not rely on advanced black-box optimization algorithms.
Instead, we show a simple grid search over one hyperparameter can lead to signiﬁcant improvement on all tasks that we investigate.
P OLY L OSS
PolyLoss provides a framework for understanding and improving the commonly used cross-entropy loss and focal loss, visualized in Figure 1.
It is inspired from the Taylor expansion of cross-entropy loss (Equation 1) and focal loss (Equation 2) in the bases of (1 − P t )
j :
[Equation]
where P t is the model’s prediction probability of the target ground-truth class.
Cross-entropy loss as PolyLoss Using the gradient descent method to optimize the cross-entropy loss requires taking the gradient with respect to P t .
In the PolyLoss framework, an interesting observation is that the coefﬁcients 1 /j exactly cancel the j th power of the polynomial bases, see
Equation 1.
Thus, the gradient of cross-entropy loss is simply the sum of polynomials (1 − P t ) j , shown in Equation 3.
[Equation]
The polynomial terms in the gradient expansion capture different sensitivity with respect to P t .
The leading gradient term is 1 , which provides a constant gradient regardless of the value of P t .
On the contrary, when j (cid:29) 1 , the j th gradient term is strongly suppressed when P t gets closer to 1 .
Focal loss as PolyLoss In the PolyLoss framework, Equation 2, it is apparent that the focal loss simply shifts the power j by the power of a modulating factor γ .
This is equivalent to horizontally shifting all the polynomial coefﬁcients by γ as shown in Figure 1.
To understand the focal loss from a gradient prospective, we take the gradient of the focal loss (Equation 2) with respect to P t : − [Equation] FL
[Equation]
For a positive γ , the gradient of focal loss drops the constant leading gradient term, 1 , in the crossentropy loss, see Equation 3.
As discussed in the previous paragraph, this constant gradient term causes the model to emphasize the majority class, since its gradient is simply the total number of
[Table]
examples for each class.
By shifting the power of all the polynomial terms by γ , the ﬁrst term then becomes (1 − P t ) γ , which is suppressed by the power of γ to avoid overﬁtting to the already conﬁdent (meaning P t close to 1 ) majority class.
More details are shown in Appendix F.
Connection to regression and general form Representing the loss function in the PolyLoss framework provides an intuitive connection to regression.
For classiﬁcation tasks where y =is the effective probability of the ground-truth label, the polynomial bases (1 − P t )
j can be expressed as ( y − P t ) j .
Thus both cross-entropy loss and focal loss can be interpreted as a weighted ensemble of distances between the prediction and label to the j th power.
However, a fundamental question in those losses: Are the coefﬁcients in front of the regression terms optimal?
In general, PolyLoss is a monotone decreasing function 1 on [0 , 1] which can be expressed as (cid:80) ∞ j =1 α j (1 − P t ) j , which provide a ﬂexible framework to adjust each coefﬁcient 2 .
PolyLoss can be generalized to non-integer j , but for simplicity we only focus on integer power ( j ∈ Z + ) in this paper.
In the next section, we investigate several strategies on designing better loss functions in the
PolyLoss framework via manipulating α j .
U NDERSTANDING THE E FFECT OF P OLYNOMIAL C OEFFICIENTS
In the previous section, we established the PolyLoss framework and showed that cross-entropy loss and focal loss simply correspond to different polynomial coefﬁcients, where focal loss horizontally shifts the polynomial coefﬁcients of cross-entropy loss.
In this section, we propose the ﬁnal loss formulation Poly-1 .
We study in depth how vertically adjusting polynomial coefﬁcients, shown in Figure 1, may affect training.
Speciﬁcally, we explore three different strategies in assigning polynomial coefﬁcients: dropping higher-order terms; adjusting multiple leading polynomial coefﬁcients; and adjusting the ﬁrst polynomial coefﬁcient, summarized in Table 2.
We ﬁnd adjusting the ﬁrst polynomial coefﬁcient (Poly-1 formulation) leads to maximal gain while requiring minimal code change and hyperparameter tuning.
In these explorations, we experiment with 1000-class ImageNet (Deng et al., 2009) classiﬁcation.
We abbreviate it as ImageNet-1K to differentiate it from the full version, which contains 21K classes.
We use ResNet-50 (He et al., 2016) and its training hyperparameters without modiﬁcation.
4. L Drop : R EVISITING DROPPING HIGHER - ORDER POLYNOMIAL TERMS
Prior works (Feng et al., 2020; Gonzalez & Miikkulainen, 2020b) have shown dropping the higherorder polynomials and tuning the leading polynomials can improve model robustness and performance.
We adopt the same loss formulation L Drop = (cid:80) Nj =1 1 /j (1 − P t ) j , as in Feng et al.
(2020), and compare their performance with the baseline cross-entropy loss on ImageNet-1K. As shown in Figure 2a, we need to sum up more than 600 polynomial terms to match the accuracy of crossentropy loss.
Notably, removing higher-order polynomials cannot simply be interpreted as adjusting the learning rate.
To verify this, Figure 2b compares the performance for different learning rates with various cutoffs:
no matter we increase or decrease the learning rate from the original value of 0 . 1 , the accuracy worsens.
Additional hyperparameter tuning is shown in Appendix C.
[Figure]
8 Cutoff index N
A cc u r a c y lr =6.4 lr =1.6 lr =0. lr =0. lr =0.
[Figure]
[Figure]
To understand why higher-order terms are important, we consider the residual sum after removing the ﬁrst N polynomial terms from cross-entropy loss: R N [Equation] j .
Theorem 1.
For any small ζ > 0 , δ > 0 if N > log 1 − δ ( ζ · δ ) , then for any p ∈ [ δ, 1] , we have | R N ( p ) | < ζ and | R (cid:48)
N ( p ) | < ζ .
(Proof in Appendix A)
Hence, taking a large N is necessary to ensure L Drop is uniformly close to L CE in the perspectives of loss and loss derivative on [ δ, 1] .
For a ﬁxed ζ , as δ approaches 0 , N grows rapidly.
Our experimental results align with the theorem.
The higher-order ( j > N + 1 ) polynomials play an important role during the early stages of training, where P t is typically close to zero.
For example, when
P t ∼ 0 .
001 , according to Equation 3, the coefﬁcient of the 500th term’s gradient is 0 .
999 499 ∼ 0 .
6 , which is fairly large.
Different from aforementioned prior works, our results show that we cannot easily reduce the number of polynomial coefﬁcients α j by excluding the higher-order polynomials.
Dropping higher order polynomials is equivalent to pushing all the higher order ( j > N +1 ) polynomial coefﬁcients α j vertically to zero in the PolyLoss framework.
Since simply setting coefﬁcients to zero is suboptimal for training ImageNet-1K, in the following sections, we investigate how to manipulate polynomial coefﬁcient beyond setting them to zero in the PolyLoss framework.
In particular, we aim to propose a simple and effective loss function that requires minimal tuning.
4. L P OLY -N : P ERTURBING LEADING POLYNOMIAL COEFFICIENTS
In this paper, we propose an alternative way of designing a new loss function in the PolyLoss framework, where we adjust the coefﬁcients of each polynomial.
In general, there are inﬁnitely many polynomial coefﬁcients α j need to be tuned.
Thus, it is infeasible to optimize the most general loss:
[Equation]
The previous section (subsection 4.1) has shown that hundreds of polynomials are required in training to do well on tasks such as ImageNet-1K classiﬁcation.
If we naively truncate the inﬁnite sum in
Equation 5 to the ﬁrst few hundreds terms, tuning coefﬁcients for so many polynomials still results in a prohibitively large search space.
In addition, collectively tuning many coefﬁcients also does not outperform cross-entropy loss, details in Appendix D.
To tackle this challenge, we propose to perturb the leading polynomial coefﬁcients in cross-entropy loss, while keeping the rest the same.
We denote the proposed loss formulation as Poly-N, where N stands for the number of leading coefﬁcients that will be tuned.
[Equation]
[Equation] [Equation]
P t )
+ ...
+
[Equation]
N + 1 /N )
[Equation]
1 − P
[Equation]
N
[Equation]
(cid:123)(cid:122) (cid:125) perturbed by
[Equation]
j
[Equation]
same as
[Equation]
[Table]
N=
[Table]
N=
[Equation]
= [Equation] [Table] Here, we replace the j th polynomial coefﬁcient in crossentropy loss 1 /j with 1 /j + (cid:15) j , where (cid:15) j ∈ [ − 1 /j, ∞ ) is the perturbation term.
This allows us to pinpoint the ﬁrst N polynomials without the need to worry about the inﬁnitely many higher-order ( j > N + 1 ) coefﬁcients, as in Equation 5.
[Figure]
+
[Figure]
P t ) , where (cid:15) 1 ∈ {− 1 , 0 , 1 , . . .
[Figure]
Table 3 shows L Poly-N outperforms the baseline cross-entropy loss accuracy.
We explore Ndimensional grid search and greedy grid search of (cid:15) j in L Poly-N up to N =
and ﬁnd that simply adjusting the coefﬁcient of the ﬁrst polynomial ( N = 1 ) leads to better classiﬁcation accuracy.
Performing 2D grid search ( N = 2 ) can further boost the accuracy.
However, the additional gain is small (+0.1) compared to adjusting only the ﬁrst polynomial (+0.4).
4. L P OLY -1 : SIMPLE AND EFFECTIVE
As shown in the previous section, we ﬁnd tuning the ﬁrst polynomial term leads to the most signiﬁcant gain.
In this section, we further simplify the Poly-N formulation and focus on evaluating
Poly-1, where only the ﬁrst polynomial coefﬁcient in cross-entropy loss is modiﬁed.
[Equation]
We study the effect of different ﬁrst term scaling on the accuracy and observe that increasing the ﬁrst polynomial coefﬁcient can systematically increase the ResNet-50 accuracy, as shown in Figure 3a.
This result suggests that the cross-entropy loss is suboptimal in terms of polynomial coefﬁcient values, and increasing the ﬁrst polynomial coefﬁcient leads to consistent improvement, which is comparable to other training techniques (Appendix E).
Figure 3b shows the leading polynomial contributes to more than half of the cross-entropy gradient during training for the majority of the time, which highlights the signiﬁcance of the ﬁrst polynomial term ( 1 − P t ) compared to the rest of the inﬁnite many terms.
Therefore, in the remaining of the paper, we adopt the form of L Poly-1 and primarily focus on adjusting the leading polynomial coefﬁcient.
As is evident from Equation 7, it only modiﬁes the original loss implementation by a single line of code (adding a (cid:15) 1 (1 − P t ) term on top of cross-entropy loss).
Note that, all the training hyperparameters are optimized for cross-entropy loss.
Even so, a simple grid search on the ﬁrst polynomial coefﬁcients in the Poly-1 formulation signiﬁcantly increases the classiﬁcation accuracy.
We ﬁnd optimizing other hyperparameters for L Poly-1 leads to higher accuracy, and show more details in Appendix B.
E XPERIMENTAL R ESULTS
In this section, we compare our PolyLoss against the commonly used cross-entropy loss and focal loss on various tasks, models, and datasets.
For the following experiments, we adopt the default training hyperparameters in the public repositories without any tuning.
Nevertheless, Poly-1 formulation leads to consistent advantage over default loss functions at the cost of a simple grid search.
5. L P OLY -1 IMPROVES 2D IMAGE CLASSIFICATION ON I MAGE N ET
Image classiﬁcation is a fundamental problem in computer vision, and progress on image classiﬁcation has led to progress on many related computer vision tasks.
In terms of the network architecture, in addition to the ResNet-50 already used in section 4, we also experiment with the state-of-the-art
EfﬁcientNetV2 (Tan & Le, 2021).
We use the ImageNet settings in (Tan & Le, 2021) except for replacing the original cross-entropy loss with our PolyLoss L Poly − 1 with different values of (cid:15) 1 .
In terms of the dataset, in addition to the ImageNet-1K dataset already used in section 4, we also consider ImageNet-21K, which has about 13M training images with 21,841 classes.
We will study both the ImageNet-21K pretraining results and the ImageNet-1K ﬁnetuning results.
Pretraining EfﬁcientNetV2-L on ImageNet-21K, then ﬁnetuning it on ImageNet-1K can improve classiﬁcation accuracy (Tan & Le, 2021).
Here, we follow the same pretraining and ﬁnetuning schedule as reported in Tan & Le (2021) without modiﬁcation 4 but replace the cross-entropy loss with L Poly-1 = − log( P t )
+ (cid:15) 1 (1 − P t ) .
We reserve 25,000 images from the training set as minival to search the optimal (cid:15) 1 .
[Figure]
Validation accuracy of EfﬁcientNetV2 models pretrained on ImageNet-21K are plotted
Poly
Loss outperforms cross-entropy loss with about
2 speed up Pretraining on ImageNet-21K Figurehighlights the importance of using tailored loss function when pretraining model on ImageNet21K dataset.
A simple grid search over (cid:15) 1 ∈ { 0 , 1 , 2 , . . .
, 7 } in L Poly-1 without changing other default hyperparameters leads to around 1% accuracy gain for all SOTA EfﬁcientNetV models with different sizes.
The accuracy improvement of using a better loss function nearly matches the improvement of scaling up the model architecture (S to M and M to L).
Surprisingly, see Figure 5a, increasing the weight of the leading polynomial coefﬁcient improves the accuracy of pretraining on
ImageNet-21K (+0.6), whereas reducing it lowers the accuracy (-0.9).
Setting (cid:15)
= − 1 truncates the leading polynomial term in the cross-entropy loss (Equation 1), which is similar to having a focal loss with γ = 1 (Equation 2).
However, the opposite change, where (cid:15) 1 > 0 , improves the accuracy on the imbalanced ImageNet-21K.
We hypothesize the prediction of the imbalanced ImageNet-21K is not conﬁdent enough ( P t is small), and using positive (cid:15) 1 PolyLoss leads to more conﬁdent predictions.
To validate our hypothesis, we plot P t as a function of training steps in Figure 5b.
We observe that (cid:15) 1 directly controls the mean P t over all classes.
Using positive (cid:15) 1 PolyLoss leads to more conﬁdent prediction (higher P t ).
On the other hand, negative (cid:15) 1 PolyLoss lowers the conﬁdence.
[Figure]
[Figure] [Figure]
[Figure] [Figure]
[Table]
set .
We set (cid:15) 1 = 2 for both.
Fine tuning on ImageNet-1K After pretraining on
ImageNet-21K, we take the EfﬁcientNetV2-L checkpoint and ﬁnetune it on ImageNet-1K, using the same procedure as Tan & Le (2021) except for replacing the original cross-entropy loss with the Poly-1 formulation.
PolyLoss improves the ﬁnetuning accuracy by 0.4%, advancing the
ImageNet-1K top-1 accuracy from 86.8% to 87.2%.
5. L P OLY -1 IMPROVES 2D INSTANCE SEGMENTATION AND OBJECT DETECTION ON COCO
Instance segmentation and object detection require localizing objects in an image in addition to recognizing them: the former in the form of arbitrary shapes and the latter in the form of bounding boxes.
For both instance segmentation and object detection, we use the popular COCO (Lin et al., 2014) dataset, which contains 80 object classes.
We choose Mask R-CNN
(He et al., 2017) as the representative model for instance segmentation and object detection.
These models optimize multiple losses, e.g. L MaskRCNN =
L cls + L box + L mask .
For the following experiments, we only replace the L cls with PolyLoss and leave other losses intact.
Results are summarized in Table 5.
[Table]
Reducing the leading polynomial coefﬁcient improves Mask R-CNN AP and AR.
In training
Mask R-CNN, we use the training schedule optimized for cross-entropy loss, 5 and replace the crossentropy loss with L Poly − 1 = − log( P t )
+ (cid:15) 1 (1 − P t ) for the classiﬁcation loss L cls , where (cid:15) 1 ∈ {− 1 . 0 , − 0 .
8 , − 0 .
6 , − 0 .
4 , − 0 .
We ensure the leading coefﬁcient is positive, i.e. (cid:15) 1 ≥ − 1 .
Our results in Figure 6a show systematic improvements of box AP, box AR, mask AP, and mask
AR as we reduce the weight of the ﬁrst polynomial by using negative (cid:15) 1 values.
Note that Poly- ( (cid:15)
= − 1 ) not only improves AP but also signiﬁcantly increases AR, shown in Table [Figure]
[Figure]
Tailoring loss function to datasets and tasks is important.
ImageNet-21K and COCO are both imbalanced but the optimal (cid:15) for PolyLoss are opposite in sign, i.e. (cid:15)
= 2 for ImageNet-21K classiﬁcation and (cid:15) = − 1 for Mask R-CNN detection.
We plot the P t of the Mask R-CNN classiﬁcation head and found the original prediction is overly conﬁdent ( P t is close to 1 ) on the imbalanced COCO dataset, thus using a negative (cid:15) lowers the prediction conﬁdence, as shown in Figure 6b.
This effect is similar to label smoothing, but unlike label smoothing, as long as (cid:15) > − 1 , PolyLoss reduces the gradient but will not penalize over conﬁdent prediction.
5. L P OLY -1 IMPROVES 3D OBJECT DETECTION ON W AYMO O PEN D ATASET
[Table]
[Figure]
[Table] [Figure] [Table] Therefore, for RSN, we drop the ﬁrst term and tune the new leading polynomial (1 − P t ) γ +2 .
Detecting 3D objects from LiDAR point clouds is an important topic and can directly beneﬁt autonomous driving applications.
We conduct these experiments on the Waymo Open Dataset (Sun et al., 2020).
Similar to 2D detectors, 3D detection models are commonly based on single-stage and two-stage architectures.
Here, we evaluate our PolyLoss on two models: a popular single-stage
PointPillars model (Lang et al., 2019); and a state-of-the-art two-stage Range Sparse Net (RSN) model (Sun et al., 2021).
Both models rely on multi-task loss functions during training.
Here, we focus on improving the classiﬁcation focal loss by replacing it with PolyLoss.
Similar to the 2D perception cases, we adopt the Poly-1 formulation to improve upon focal loss, shown in Table 6.
PolyLoss improves single-stage PointPillars model.
The PointPillars model converts the raw 3D point cloud to a 2D top-down pseudo image, and then detect 3D bounding boxes from the 2D image in a similar way to RetinaNet (Lin et al., 2017).
Here, we replace the classiﬁcation focal loss ( γ = 2 ) with L FLPoly-1 = − (1 − P t ) 2 log P t + (cid:15) 1 (1 − P t ) 3 and adopt the same train-
[Table]
ing schedule optimized for focal loss without any modiﬁcation 6 .
Table 7 shows that L FLPoly-1 with (cid:15)
=
− 1 leads to signiﬁcant improvement on all the metrics for both vehicle and pedestrian models.
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Figure]
[-L FLPoly 1 Pillars (1 P t ) 3 (1 P t ) 4 (1 P t ) 5 (1 P t ) 
-][Table] [Figure] [Table]
[Figure]
P o l y c o e ff i
c i e n t s +0.2(1 P t )
0.4(1 P t )
[Figure]
Advancing the state-of-the-art with RSN.
RSN segments foreground points from the 3D point cloud in the ﬁrst stage, and then applies sparse convolution to predict 3D bounding boxes from the selected foreground points.
RSN uses the same focal loss as the PointPillars model, i.e., L FL = − (1 − P t ) 2 log P t .
Since the optimal
L FLPoly-1 for PointPillars ( (cid:15) 1 = − 1 ) is equivalent to dropping the ﬁrst polynomial, we adapt the same loss formulation for RSN and tune the new leading polynomial (1 − P t ) 4 by deﬁning L FLPoly-1 ∗ = − (1 − P t ) 2 log( P t ) − (1 − P t )
+ (cid:15) 2 (1 − P t ) 4 , shown in Figure 7.
We follow the same training schedule optimized for focal loss described in Sun et al. (2021) without adjustment.
Our results, in Table 7, show that tuning the new leading polynomial improves all metrics (except vehicle detection BEV APH L2) for the SOTA 3D detector.
C ONCLUSION
In this paper, we propose the PolyLoss framework, which provides a uniﬁed view on common loss functions for classiﬁcation problems.
We recognize that, under polynomial expansion, focal loss is a horizontal shift of the polynomial coefﬁcients compared to the cross-entropy loss.
This new insight motivates us to explore an alternative dimension.
i.e. vertically modify the polynomial coefﬁcients.
Our PolyLoss framework provides ﬂexible ways of changing the loss function shape by adjusting the polynomial coefﬁcients.
In this framework, we propose a simple and effective Poly-1 formulation.
By simply adjusting the coefﬁcient of the leading polynomial coefﬁcient with just one extra hyperparameter (cid:15) 1 , we show our simple Poly-1 improves a variety of models across multiple tasks and datasets.
We hope Poly-1 formulation’s simplicity (one extra line of code) and effectiveness will lead to adoption in more applications of classiﬁcation than the ones we have managed to explore.
More importantly, our work highlights the limitation of common loss functions, and simple modiﬁcation could lead to improvements even on well established state-of-the-art models.
We hope these ﬁndings will encourage exploring and rethinking the loss function design beyond the commonly used cross-entropy and focal loss, as well as the simplest Poly-1 loss proposed in this work.
A CKNOWLEDGEMENTS
We thank James Philbin, Doug Eck, Tsung-Yi Lin and the rest of Waymo Research and Google
Brain teams for valuable feedback.
R EPRODUCIBILITY STATEMENT
Our experiments are based on public datasets and open source code repositories, shown in footnote 3-6.
We do not tune any default training hyperparameters and only modify the loss functions, which are shown in Table 2-7.
The proposed ﬁnal formulation L Poly-1 requires one line of code modiﬁcation .
Example code for L CEPoly-1 with softmax activation is shown below.
def poly1_cross_entropy(logits, labels, epsilon=1.0): # pt has shape (batch). pt = tf.reduce_sum(labels * tf.nn.softmax(logits), axis=-1) # CE and Poly1 have shape (1). CE = tf.nn.softmax_cross_entropy_with_logits(labels, logits) Poly1 = CE + epsilon * tf.reduce mean(1 - pt) return Poly
Example code for L FLPoly-1 with sigmoid activation is shown below.
def poly1_focal_loss(logits, labels, epsilon=1.0, gamma=2.0): # p, pt, FL, and Poly1 have shape [batch, num of classes]. p [Equation] ) pt
[Equation] labels [Equation] labels
[Equation]
FL = focal_loss(pt, gamma) Poly1 =
FL + epsilon * tf.math.pow(
[Equation]
pt, gamma + 1) return Poly
S UPPLEMENTARY MATERIAL A P ROOF OF THEOREM
Theorem 1.
For any small ζ > 0 , δ > 0 if N > log 1 − δ ( ζ · δ ) , then for any p ∈ [ δ, 1] , we have | R N ( p ) | < ζ and | R (cid:48)
N ( p ) | < ζ .
Proof.
[Equation]
B A DJUSTING OTHER TRAINING HYPERPARAMETERS LEADS TO HIGHER GAIN .
All the experiments shown in the main text are based on hyperparameters optimized for the baseline loss function, which actually puts PolyLoss at a disadvantage.
Here we use weight decay rate for
ResNet50 as an example.
The default weight decay (1e-4) is optimized for cross-entropy loss.
Adjusting the decay rate may reduce the model performance of cross-entropy loss but leads to much higher gain for PolyLoss (+0.8%), which is better than the best accuracy (76.3%) trained using cross-entropy loss (+0.8%).
[Table]
weight decay value is 1e-4.
C L D ROP WITH MORE HYPERPARAMETER TUNING
For L Drop (N = 2), besides adjusting the learning rate, we further tune the coefﬁcient ( α ) of the second polynomial, similar to a prior work (Gonzalez & Miikkulainen, 2020b), and weight decay.
[Equation]
Unlike (Feng et al., 2020), where α = 0 .
5 after dropping all higher-order polynomial, we ﬁnd the optimal α = 8 , while the optimal learning rate is the same as the default setting (0.1).
This alone increases the accuracy to 70.9, which shows simply dropping polynomial terms is not enough and adjusting the polynomial coefﬁcients is critical.
Further tuning weight decay leads to less than 0.1% model quality improvement.
Comparing to prior works (Gonzalez & Miikkulainen, 2020b; Feng et al., 2020), Poly-1 is more effective and only contains one hyperparameter.
Tuning weight decay of Poly-1 further increases the accuracy while having less hyperparameters compared to L Drop ∗ , shown in Table 9.
D C OLLECTIVELY TUNING MULTIPLE POLYNOMIAL COEFFICIENTS
Besides adjusting individual polynomial coefﬁcients, in this section, we explore collectively tuning multiple polynomial coefﬁcients in the PolyLoss framwork.
In particular, we change the coefﬁcients [Table] in the original cross-entropy loss from 1 /j
(Equation 1) to exponential decay.
Here, we deﬁne
L exp = 2 N
[Table]
j =
[Equation]
(9) where we cut off the inﬁnite sum at twice the decay factor N .
We performed 2D grid search on
N ∈ { 5 , 20 , 80 , 320 } and learning rate ∈ { 0 . 1 , 0 . 4 , 1 . 6 , 6 . 4 } .
The best accuracy is 72.3, where
N = 80 and learning rate = 1 . 6 , shown in Table 10.
[Table]
Though Poly-1 is better than using L exp , there are a lot more possibilities besides using exponential decay.
We believe understanding how collectively tuning multiple coefﬁcients affects the training is an important topic.
E C OMPARING TO OTHER TRAINING TECHNIQUES
As shown in recent works (He et al., 2019; Bello et al., 2021; Wightman et al., 2021), though independent novel training techniques often lead to sub 1% improvement, combining them could lead to signiﬁcant overall improvements.
To put things into perspective, Poly-1 achieves similar improvements as other commonly used training techniques, such as label smoothing and dropout on
FC, shown in Table 11.
Cross
[Table]
entropy
Poly
[Table]
Num. of parameters
[Table]
F R EDISCOVERING FOCAL LOSS FROM P OLY L OSS
[Figure] terms can improve RetinaNet Focal loss was ﬁrst developed for single-stage detector RetinaNet to address strong class imbalance presented in object detection (Lin et al., 2017).
Here, we provide an additional ablation study on how to systemically discover focal loss in the PolyLoss framework and investigate how the leading terms affect training in the presence of class imbalance.
Rediscovering the concept of focal loss from crossentropy loss.
Here, we take a step back and attempt to systematically rediscover the concept of focal loss via our PolyLoss framework.
Focal loss is commonly used for training detection models.
Coming up with such an insight to address the class imbalance issue in detection requires strong domain expertise.
We start with the PolyLoss representation of cross-entropy loss and improve it from the PolyLoss gradient perspective.
[Figure]
We start with the cross-entropy loss and deﬁne PolyLoss by dropping the ﬁrst N polynomials in cross-entropy loss, i.e. L Drop-front = (cid:80) ∞
j = N +1 1 /j
(1 − P t )
j = L CE − (cid:80) Nj =1 1 /j (1 − P t ) j .
Dropping the ﬁrst two polynomial terms (1 − P t ) signiﬁcantly improves both the detection
AP and AR, see Figure 8.
Dropping the ﬁrst two polynomials ( N = 2 ) leads to the best RetinaNet performance, which is similar to setting γ = 2 in focal loss, i.e. focal loss γ = 2 pushes all the polynomial coefﬁcients to the right by 2, shown in Figure 1 right, which is similar to truncating the ﬁrst two polynomial terms.
Leading polynomials cause overﬁtting to the majority class.
In the PolyLoss framework, the leading polynomial of cross-entropy loss is a constant, shown in
Equation 3.
For binary classiﬁcation, the leading gradient for each class is simply N background − N object , where
N background and N object are the counts of background and object instances in the training mini-batch.
When the class counts are extremely imbalanced, the majority class will dominate the gradient which will lead to signiﬁcant bias towards optimizing the majority class.
Dropping polynomials reduces the extremely conﬁdent prediction P t , see Figure 9.
To examine the composition of the overall prediction conﬁdence, we also plot the P t for background only and P t for object only.
Due to the extreme imbalance between the background and the object class, the overall P t is dominated by the background only P t .
So reducing the overall
P t decreases the background P t .
On the other hand, reducing overﬁtting to the majority background class leads to more conﬁdent prediction P t on the object class.