T HE WEIGHTED MEAN TRICK – O PTIMIZATION STRATEGIES FOR ROBUSTNESS
Anonymous authors
Paper under double-blind review
A BSTRACT
We prove that minimizing a weighted mean results in optimizing the higher-order moments of the loss distribution such as the variance, skewness, and kurtosis.
By optimizing the higher-order moments one can tighten the upper bound of the loss mean deviating from the true expectation and improve the robustness against outliers.
Such types of optimization problems often lead to non-convex objectives therefore we explore the extent to which the proposed weighted mean trick preserves convexity, albeit at times at a decrease in efﬁciency.
Experimental results show that the weighted mean trick exhibits similar performance with other specialized robust loss functions when training on noisy datasets while providing a stronger theoretical background
The proposed weighted mean trick is a simple yet powerful optimization framework that is easy to integrate into existing works.
I NTRODUCTION
The most common objective in machine learning is to minimize the average loss.
However, by doing this it implies that all samples are equal.
The goal of this work is to convince the reader that from the perspective of the model, not all samples are equal and optimizing a weighted mean is a more progressive approach.
For instance, at the end of the training, few samples are left uncaptured by the model which yield large loss values.
One might choose to put more weight on those samples to help the model learn (individual fairness), or consider them noise and assign them less weight (robustness).
The decision will also be reﬂected by the loss variance; it will decrease when hard samples are weighted more and increase otherwise.
Before introducing more formally the weighted mean, we ﬁrst justify theoretically the impact of variance penalization.
Using the empirical Bernstein bound (Maurer & Pontil, 2009) for i.i.d.
loss values Z and bounded variance we have with probability 1 − δ :
[Equation]
where C 1 , C 2 are problem depended constants.
This inequality reveals two things.
First, we have with high probability that the empirical mean is close to the theoretical value.
This bound along with similar PAC-Bayes bounds (Seldin et al., 2012; Tolstikhin & Seldin, 2013) justiﬁes the practical success of the empirical risk minimization (ERM) which has the objective to minimize the mean loss value (Namkoong & Duchi, 2017).
Secondly, the difference between the two is bounded by the empirical variance.
This second implication led to a large and growing body of studies that investigate variance penalization (Maurer & Pontil, 2009; Namkoong & Duchi, 2017; Duchi & Namkoong, 2019; Staib et al., 2019; Lam, 2019; Heinze-Deml & Meinshausen, 2021;
Hu et al., 2018).
Moreover, by penalizing the variance, the intrinsic bias-variance tradeoff of the ERM can be controlled.
Two studies inﬂuential for this paper are that of Duchi & Namkoong (2019) and Li et al.
Duchi &
Namkoong (2019) proposed taking the expectation with respected to a different distribution which allowed penalizing variance while preserving the convexity of the objective.
Similarly, Li et al.
(2021) investigated optimizing a tilted empirical risk which is equivalent to penalizing all the higher-order moments simultaneously.
In summary, previous methods either penalize only one moment or all the higher-order moments but are not ﬂexible enough to penalize any desired combination of the higher-order moments.
Inspired by the above works, we propose to optimize a weighted mean and prove that for certain weights, it is equivalent to optimizing any higher-order moments of the loss distribution such as variance, skewness, and kurtosis.
Our approach generalizes that of Duchi & Namkoong (2019);
Li et al.
(2021) while simplifying the optimization procedure and enabling separate penalization of the higher-order moments.
In particular, we will construct the weights w such that optimizing the weighted mean of the loss (cid:96) is equivalent to applying a variance penalization:
[Equation]
or penalization of other central moments (e.g., skewness, kurtosis).
Construction of these weights are treated by Theorems 1 and 3 from section 3, which require minimal computational resources.
Important note, penalizing directly the variance preserves convexity only for values of λ within a very narrow range (Maurer & Pontil, 2009).
However, the weighted mean method yields a convex objective for any positive values of λ .
In detail, our contributions are as follows:
( C1 )
We build upon the work of Duchi & Namkoong (2019) and prove that optimizing a weighted mean is equivalent to reducing or amplifying both variance (Theorem 1) and higher-order moments (Theorem 3).
( C2 )
We derive the limits of the penalization interval which preserves convexity (Lemma 2) and also show how to penalize with values outside this interval while still maintaining convexity (Lemma 4).
( C3 )
We connect the variance and higher-order moments penalization using weighted mean to the robustiﬁcation of loss functions (Lemma 5 and Lemma 6).
( C4 )
We develop a convex version of the variance penalized cross-entropy loss which provides a higher accuracy in high noise scenarios with class dependent noise.
( C5 )
We show experimentally that a negative variance penalization improves model accuracy when training with noisy labels.
The implications of the weighted mean are much broader than what we investigate in this work.
We limit the scope of this paper to classiﬁcation using deep neural networks trained with noisy labels.
But the mathematical framework also covers the control of the bias-variance trade-off and can also be applicable to regression problems.
In the sequel, we start by introducing the notation in section 2, then in section 3 we present the moment penalization strategy and the weighted mean trick which is a computational technique to make moment penalization practical.
Next, in section 4, we illustrate the application of moment penalization using the weighted mean formulation to optimize for robustness when training with noisy labels.  N OTATIONS In subsequent sections we will use the following notation.
A training data set is deﬁned as D = { ( x i , y i ) } ni =1 where x i ∈
X are the features and y i ∈ Y = { 1 , . . .
, k } represent the class labels.
A classiﬁer f ( x ; θ ) is a mapping f : X × Θ → V from the feature space to the probability simplex V parameterized by θ . A loss function (cid:96) : V × Y → [0 , ∞ ) gives a penalty (cid:96) ( v, y ) when the model predicted the value v and label y was observed. A weight function w : V × Y → R assigns to each sample a weight. As we are interested in optimizing the model output value v that minimizes the penalty, we will focus our investigation on loss functions (cid:96) ( v, y ) that are convex in v and seek to preserve the convexity when optimizing the weighted mean.
In ERM, we are interested with ﬁnding the model parameters θ that minimize the empirical risk calculated as E D [ (cid:96) ( f
( x i ; θ ) , y i )] and the weighted form E D [ w ( (cid:96) ( f ( x i ; θ ) , y i ))
(cid:96) ( f ( x i ; θ ) , y i )] .
To simplify the notation, we will drop the dataset D from the expectation subscript and the arguments of the loss and the weight function, i.e., E [ (cid:96) ] ⇔ E D
[ (cid:96) ( f ( x i ; θ ) , y i )] and E [ w(cid:96) ] ⇔ E D [ w ( (cid:96) ( f ( x i ; θ ) , y i ))
(cid:96) ( f ( x i ; θ ) , y i )] .
Similarly, the notation for minimum is simpliﬁed as min (cid:96) ⇔ min x i ,y i ∈D (cid:96) ( f ( x i ; θ ) , y i ) .
[Figure]
[Figure]
[Figure]
[Figure]
3 .−  0 .0 .0 .
5 0 .1 .1 .1 .1 .−
3 λ min 3 λ max λ max >λ min <
In ERM, the impact on the model parameters of a sample is determined by the balance between its loss value and the average of the training batch.
Especially observed in the latter stages of the training when majority of samples have a small loss value except few which are left uncaptured by the model.
Case in which we consider them as unlearned samples and amplify their impact or as outliers and suppress their impact on the model.
The weighted mean trick consist in assigning weights to each sample based on the loss value to either amplify or suppress their impact on training.
This is similar to dropout (Hinton et al., 2012) but for losses.
However, weighted mean trick is a deterministic process and the weights are not restricted to 0 or 1 but can take any non-negative value.
In what follows, we apply the weighted mean trick to extend the ERM framework to multi-objective optimization of the mean and higher-order moments of the loss function.
First, we show that for some distinct weights, optimizing the weighted mean is equivalent to a simultaneous optimization of the mean and variance.
Theorem 1 (Variance Expansion) .
Let (cid:96) be a loss function with ﬁnite E [ (cid:96) ] and ﬁnite V [ (cid:96) ] and let w ( v, y )
= 1 + λ ( (cid:96) ( v, y ) − E [ (cid:96) ( v, y )]) , then we have:
[Equation]
Proof.
Replacing w with the deﬁnition and using the linearity property of the expectation along with
Proposition 7 we get: E [ w(cid:96) ]
[Equation]
Thus, switching from mean to weighted mean allows us to control the bias-variance tradeoff through λ to improve the distributional robustness of the model (Maurer & Pontil, 2009).
However, the range of λ values that preserve the convexity of the objective depends on the average and the minimum penalty returned by the loss function (cid:96) as shown by the next lemma.
Lemma 2.
As introduced in Theorem 1, the variance expansion of a convex loss function (cid:96) ( v, y ) in v yields a new objective that is also convex in v if λ ∈ [0 , λ max ] , where λ max = 1 / ( E [ (cid:96) ] − min (cid:96) ) .
Proof is provided in Appendix A.
This lemma precisely shows why directly penalizing the variance does not preserve convexity besides when λ takes values in a narrow interval.
Moreover, the upper limit of this interval is not constant and changes with each iteration of the optimization algorithm.
Note that the further the minimum value min (cid:96) is from the average loss value E [ (cid:96) ] , the narrower the interval is.
However, when using the weighted mean trick to penalize the variance and clip the negative weights to 0 then the objective remains convex for any positive λ .
Figure 1 shows the weights distribution for different values of λ .
When λ is positive (left plot) samples closer to the decision boundary (which also means with larger loss values) receive more weight.
On the other hand, when λ is negative (center plot) samples with larger loss values receive less weight.
Of note, for λ <
the objective is not convex, however as we will show later will still converge to an optimal solution.
The right plot shows a binary classiﬁcation problem where each class consists of two clusters, one cluster is diagonally aligned containing the majority of points and another consisting of the three central points aligned across a horizontal line.
The parameter λ controls the placement of the decision boundary with respect to these two clusters.
Positive λ values place more weight on the three horizontal points as they are closer to the decision boundary and favors the model to learn how to perfectly separate the two classes.
On the other hand, negative λ values place less weight on the horizontal points close to the boundary and more on the farther ones.
Therefore, this aligns the decision boundary only with respect to the two large clusters.
Variance is not the only central moment that can be penalized using the weighted mean.
In fact, any combination of the moments can be penalized.
The next theorem generalizes theorem 1.
Theorem 3 (Moments Expansion) .
Let (cid:96) be a loss function with ﬁnite ﬁrst m central moments and deﬁne w [Equation] [Equation]
[-=
-]=
[Equation]
[-i − 1 , then taking the weighted mean of (cid:96) with weights W is equivalent − 1 −     λ 1 =2 λ 0 =1 λ 3 =1 λ 2 =2 . 5 λ 1 =3 λ 
 = −  λ 1 = − 2 λ 0 = −  λ 2 =1 λ 1 =0 .-][Equation]
[Equation]
i
[Equation]
1 , then by taking the weighted mean of (cid:96) with weights w is equivalent to:
[Equation]
where ˜ λ i
=
λ
i
+ λ
i +1 E
[ (cid:96) ] for i < m and ˜ λ m = λ m
Proof is provided in Appendix A.
We notice that penalizing moments higher than two incurs an additional penalization of the previous moment.
For example, penalizing skewness by λ 3 incurs a variance penalization of λ 3 E [ (cid:96) ] .
Theorem 3 also has an algebraic interpretation.
Note that the formula for weights is nothing more than a polynomial in (cid:96) ( v, y ) translated by E [ (cid:96) ] .
When penalizing the variance, λ 2 controls the slope of the linear equation, and when penalizing the skewness, λ 3 controls the curvature of the quadratic equation.
Moreover, the penalization factors λ
i also deﬁne the placement of the roots of the polynomial and the convexity of the weighted mean objective.
Lemma 4 (Convexity of Moments Expansion) .
Let (cid:96) ( v, y ) be a loss function convex in v and p : R → [0 , ∞ ) be a non negative convex function and M ≥ 0 , then the weighted objective w ( v, y ) (cid:96) ( v, y ) with w ( v, y ) = p ( (cid:96) ( v, y ) − M ) is convex in v if p is non decreasing.
Proof is provided in Appendix A.
The above theorem lists the two requirements on the polynomial function p such that the weighted mean objective remains convex.
First, the function must be nonnegative, thus, the negative weights must be clipped to 0.
Secondly, when the function takes positive values, p must be non decreasing and convex.
Figure 2 shows several examples of polynomials.
Variance penalization implies p is an afﬁne function, and thus, convex.
However, for λ 1 > 0 the polynomial is non decreasing (left plot) and for λ 1 < 0 the polynomial is non increasing (center plot).
Thus, only positive variance penalization will result in a convex objective.
Of note, Lemmaupper bounds λ 1 if weights are not clipped to 0 , however, if clipping is used λ 1 is not upper bounded.
When penalizing skewness, p is a quadratic function (right plot).
In this case, if λ 2 > 0 then p is convex and non decreasing only on a restricted interval instead of the entire real line.
Thus, convexity can be preserved by appropriately adjusting the roots of the polynomial such that p is convex on [min (cid:96) − E [ (cid:96) ] , + ∞ ] .
When clipping negative weights to 0 it prevents the optimization objective to switch from minimization to maximization, usually an undesirable behavior.
Consequently, the samples with a corresponding zero weight reach their maximum contribution in the moments penalization.
Further increasing the factors λ
i , will have no effect on those samples and only samples with non-zero weights will participate in the training.
As a result, the efﬁciency of the moments penalization will slightly fall.
Moreover, the samples with zero weight will be excluded from training, technique used in the past by multiple studies.
Rockafellar & Uryasev (2000) used a similar clipping technique to optimize only for samples that are part of the tail of the distribution.
The Moments Expansion theorem extends the variance expansion proposed by Duchi & Namkoong (2019) to include higher-order moments.
However, even when only the variance is penalized, the two methods are still slightly different.
Moments expansion theorem computes the weights directly from the loss values, whereas the variance expansion of Duchi & Namkoong (2019) solves a secondary optimization problem to ﬁnd the weights.
The use of a secondary optimization problem has the advantage of penalizing the variance more consistently despite being more computationally expensive.
On the contrary, when using the moments expansion, the penalized variance can be slightly lower − [Figure] 1 −     λ 1 =2 λ 0 =1 λ 3 =1 λ 2 =2 .
{+5 λ 1 =3 λ 
 = −  λ 1 = − 2 λ 0 = −  λ 2 =1 λ 1 =0 . 5 λ 0 = − 0 . [Figure]+}
Center plot shows a convex but non increasing polynomial and thus will result in a non convex objective.
Right plot shows a convex and non decreasing polynomial on the interval [ − 1 , + ∞ ] , and thus will yield a convex objective when min( (cid:96) )
− E [ (cid:96) ] ≥ − 1 .
depending on the number of weights that are 0 .
However, the advantage is that the direct computation of the weights makes it easier to include the method into existing analysis frameworks.
A similar method that extends the optimization objective to include penalization factors for higher-order moments was proposed by Li et al.
The proposed method replaces the ERM objective with a tilted version calculated as 1 t K ( t ) = 1 t log E [ e t(cid:96) ] where K ( t ) is the cumulant-generating function of the loss (cid:96) .
The penalization of the higher order moments of the tilted ERM can be recovered from the power series expansion of K ( t ) .
The distinction between the two is that the moments penalization introduced in this paper represents a generalization of the tilted ERM as it allows any combination of the higher order moments to be penalized whereas tilted ERM uses a single parameter that governs the penalization factors.
In summary, the moments penalization implemented using the weighted mean trick is more ﬂexible, however, it comes at a cost as there are more parameters to tune when penalizing multiple moments compared to tilted ERM of Li et al.
Weighted mean trick in practice.
To apply the method in practice, the classical batch training algorithm must be extended to include an additional step, the weights calculation.
Instead of directly calculating the average loss, the user will calculate the loss value for each sample in the batch and then use the expression from Theorem 3 to compute the weights and the weighted mean.
The moments penalization factors λ
i are the hyper-parameters and are tuned in ascending order with respect to i .
However, penalizing higher-order moments might affect the impact of the lower-order ones, and thus, it might require a few iterations to ﬁnd the optimal combination.
The implementation of this algorithm in Pytorch (Paszke et al., 2019) is available on GitHub.
1 Since the gradient of the weighted mean is the weighted gradient of the elements, this allows weights to control the impact of each sample on the model parameters.
Of note, the theoretical results hold when switching from expectation to sample expectation, E n [ (cid:96) ]
= 1 n (cid:80) ni
=1 (cid:96) ( f ( x i , θ ) , y i ) .
Algorithm input : f ( x ; θ ) – model to be trained { x i , y i } n– batch of training data { λ } m– penalization factors (cid:96) ( v, y ) – loss function while stopping criteria not reached do for i ← 1 to n do z i ← (cid:96) ( f ( x i , θ ) , y i ) ; /* sample loss */ w ← (cid:104)(cid:80) m j =1 λ j ( z − E n [ z ])
j − 1 (cid:105)
+
L w ← 1 n (cid:80) ni
=
w i z i ; /* weighted mean */ θ ← θ − γ ∇ θ L w ; /* update model parameters */
R OBUST C LASSIFICATION
In this section, we will explore the robustiﬁcation of the cross-entropy function under label noise by bounding the loss values using negative variance penalization.
In this case, the resulting objective will not be convex.
However, for moderately small penalization factors the objective remains convex on almost the entirety of the domain and does not hinder the convergence.
Moreover, we also develop a convex version of the variance penalized cross-entropy though for a minor price in performance.
The objective of robust classiﬁcation is to learn an optimal classiﬁer f ∗ that minimize the average loss for both clean and noisy data.
Formally, f ∗ = argmin E
[ (cid:96) ( f ( x, θ ) , ˆ y )] where ˆ y represents noisy labels.
The noise considered in this paper corrupts the class labels with probability P (ˆ y i
(cid:54) =
y i ) =
η or preserves it with probability P (ˆ y i
=
y i )
= 1 − η , where η ∈ [0 , 1] .
We investigate two scenarios: when η does not depend on the class label (class independent or uniform noise), and when η depends on the class label (class dependent or asymmetric noise).
In what follows, we show that using a negative variance penalization factor, λ 2 < 0 , bounds the loss function.
By bounding the loss function the impact of misclassiﬁcation of noisy samples on the average loss will decrease, and thus reduce the noise impact on the model during training.
Ghosh et al.
(2017) outlined the distribution independent sufﬁcient conditions for a loss function to be robust under both, class dependent and class independent noise.
Speciﬁcally, if the loss function satisﬁes the symmetry constraint:
[Equation]
where C is a constant and it holds ∀ f, ∀ θ, ∀ x ∈ X , then (cid:96) is noise tolerant for class independent noise
when η < k − 1 k
(see Theorem 1 of Ghosh et al.
Moreover, if the population risk of the optimum classiﬁer E [ (cid:96) ( f ∗ ( x, θ ) , ˆ y )]
= 0 then (cid:96) is also noise tolerant for class dependent noise.
Among the commonly used losses, only the mean absolute error (MAE) satisﬁes the symmetry constraint.
On the other end, the cross-entropy (CE) loss which is widely used for classiﬁcation, not only does not satisfy the above constraint, but is also an unbounded loss making it extremely susceptible to noise.
Speciﬁcally, as noise makes the predicted probability of the correct class approach zero, its cross-entropy loss will approach inﬁnity.
Following the work of Ghosh et al.
(2017), many studies (Zhang & Sabuncu, 2018; Feng et al., 2020; Wang et al., 2019b; Ma et al., 2020; Wang et al., 2019a) found that models trained with MAE struggle to converge and proposed novel losses that rely on boundedness to achieve robustness instead of the symmetry constraint to avoid convergence problems.
Zhang & Sabuncu (2018) proposed a generalized cross-entropy loss parameterized by q that recovers CE loss for q → 0 and MAE for q = 1 .
For intermediate values of q , the loss is bounded and trades robustness for convergence.
Feng et al.
(2020) found that the ﬁrst two terms of the Taylor series expansion for the CE loss are the MAE and MSE, respectively.
And suggested adjusting the number of terms in the Taylor expansion for the
CE to balance noise robustness and convergence.
Wang et al.
(2019b) proposed using a two term loss, complimenting the CE loss with a secondary reverse cross-entropy term that satisﬁes the symmetry constraint.
The research by Ma et al.
(2020) extended the concept of two term loss combining a robust “active” loss and a robust “passive” loss.
In the following, we use the weighted mean trick to bound a loss function to improve its noise robustness.
The next lemma specify the requirements for the weights:
Lemma 5.
Let (cid:96) ( v, y ) represent an unbounded loss function and w ( v, y ) a corresponding nonnegative and bounded weight function, then the product w ( v, y ) (cid:96) ( v, y ) is bounded if there exists a ﬁnite threshold [-(cid:96) 
-]L 0 such that when (cid:96) ( v, y ) ≥ L 0 the corresponding weights w ( v, y ) are 0.
Proof.
Since when (cid:96) ( v, y ) ≥ L 0 the corresponding weights w ( v, y )
{+= 
+}
the product w ( v, y ) (cid:96) ( v, y ) is also 0. When (cid:96) ( v, y ) < L 0 since the weights are bounded the product w ( v, y ) (cid:96) ( v, y ) is also bounded and thus 0 ≤ w ( v, y ) (cid:96) ( v, y ) ≤ B where B is a positive constant that depends on the weight and loss function.
The simplest solution that satisﬁes the above requirement is the negative variance penalization, λ 2 < 0 , as the next lemma shows:
[-For non-negative weights W computed using Theorem 3 (Moments Expansion) with λ 0 > 0 , λ 1 < 
-]
[-and λ i = 0 , ∀ i > 1 , loss values (cid:96) greater or equal to (cid:96) 
-]
E [ ‘ ce ] = 1 .λ 2 = − 0 .
1 λ 1 =1 λ 2 = − 0 .
2 λ 1 =
E [ ‘ ce ] = 2 .
E [ ‘ ce ] = 0 .λ 2 = − 0 .
15 λ 1 =1 λ 2 = − 0 . 15 λ 1 =   λ 2 = − 0 .
15 λ 1 =1 λ 2 = − 0 .
15 λ 1 =
[Figure]
Lemma 6.
For weights w ( v, y ) computed using Theorem 3 with λ 1 > 0 , λ 2 < 0 , λ i = 0 , ∀ i >and with negative weights clipped to 0, penalties (cid:96) ( v, y ) ≥ L 0 with L 0 = E [ (cid:96) ] − λ 1 λ 2 will have an associated weight of 0 .
Proof.
From Theorem 3 the weights are calculated as w ( v, y ) = λ 1 + λ 2 ( (cid:96) ( v, y ) − E
[ (cid:96) ( v, y )]) from which we can determine the threshold value [-(cid:96) 
-]L 0 =
E [ (cid:96) ] − λ 1 λ 2 .
Thus due to the non-negativity constraint (cid:96) ( v, y ) ≥ L 0 will have an associated weight of 0.
Of note, bounding an unbounded loss function is achieved through clipping, however, the resulting loss function is not convex as shown in Figure 3.
With parameters λ [-
-]1 and λ 2 along with E [ (cid:96) ] establishing which samples are considered noise and excluded from training.
Since for λ 2 < 0 the threshold [-(cid:96) 
-]L 0 = E [ (cid:96) ] − λ 1 λ 2 is greater than E [ (cid:96) ] , thus all excluded samples have a loss value above average.
Moreover, the higher the magnitude of λ 2 , the closer the threshold L 0 is to E [ (cid:96) ] .
This can be seen in the left plot of Figure 3.
Moreover, the magnitude of λ 2 dictates by how much the loss values below average are ampliﬁed and the ones above average suppressed.
In practice, using larger values for λ 2 will decrease the impact of misclassiﬁed samples and those near the separating hyperplane on the placement of the decision boundary.
This effect is shown in Figure 1 by the decision boundary 3 λ min which correspond to a negative variance penalization and which is aligned with the diagonal clusters far from the boundary.
Lemma 6 can be extended to include other higher-order moments.
In this case, the roots of the weights polynomial will determine which samples participate in the optimization problem and which are considered noise.
For example, if the decision boundary should be decided by samples with average loss values then a possible solution is to penalize skewness.
Since in this case the weights are deﬁned by a quadratic equation, and using a negative value for λwill make the parabola open downwards and thus assign non-zero weights only to samples around the mean.
Moreover, we also develop a convex version of the variance penalized cross-entropy by constraining the second derivative of the weighted loss function to be non-negative.
The cross-entropy loss is deﬁned as (cid:96) ce ( v, y ) = − log v [ y ] where v [ y ] represents the predicted probability of sample having the label y . To simplify the notation, we will rewrite the cross-entropy loss as (cid:96) ce ( u ) =
− log u where u
= v [ y ] and similarly rewrite the weight function as w ( u ) = λ 1 + λ 2 ( (cid:96) ( u ) − E [ (cid:96) ( u )]) .
We note that the second derivative of the weighted loss w ( u ) (cid:96) ( u ) is 0 for u = u c with u c = exp (cid:16) 1 + λ 1 2 λ 2 − E [ (cid:96) ] 2 (cid:17) and is negative for u < u c . To constrain the second derivative be non-negative we linearly interpolate w ( u ) (cid:96) ( u ) for u < u c using the derivative of w ( u ) (cid:96) ( u ) at u c .
The weight function for u < u c is w ( u ) = (cid:2) w ( u c ) (cid:96) ( u c ) + dd u w ( u ) (cid:96) ( u ) | u
= u c ( u − u c ) (cid:3) /(cid:96)
( u ) with the resulting convex objective shown on the right plot of Figure 3.
The black dot marks the transition point u c from the weighted cross-entropy on the right to the linear interpolation on the left of the dot.
Of note, the resulting objective is still bounded albeit with a higher upper bound.
In general, methods for training of neural networks in the presence of noise can be classiﬁed into two broad categories: noise model-free and noise model-based strategies.
Noise model-free methods
Sabuncu Wang 2019b
Mnih & Patrini al., 2016; Rooyen
Feng et al., 2020; Ma et al., 2020) and learning management such as meta-learning or regularization (Li et al., 2017; Szegedy et al., 2016;
Hendrycks et al., 2019; Liu et al., 2020;
Harutyunyan et al., 2020).
On the other hand, noise model-based methods estimate the properties of the noise and use this information when train the model.
Noise model-based methods can be further divided into: using noise transition matrix (Patrini et al., 2017; Hendrycks et al., 2018; Chen & Gupta, 2015; Bekker & Goldberger, 2016; Goldberger & Ben-Reuven, 2017; Sukhbaatar et al., 2015; Xia et al., 2019;
Yao et al., 2019; Wang et al., 2020; Xia et al., 2020; Yao et al., 2020; Lukasik et al., 2020), noise mitigation requiring a clean dataset (Jiang et al., 2018;
Ren et al., 2018; Veit et al., 2017; Zhang et al , 2020; Yuan et al., 2018; Vahdat, 2017; Mirzasoleiman et al., 2020), or incrementally estimating the distribution of clean data (Liu et al., 2017; Zheng et al., 2020; Arazo et al., 2019; Zhang et al., 2017;
Ghosh & Lan, 2021; Zhang et al., 2021; Wu et al., 2020).
Experimental results.
The procedure adopted for all the experiments is similar to other studies investigating robust losses (Zhang & Sabuncu, 2018;
Feng et al., 2020; Wang et al., 2019b; Ma et al., 2020).
The source code to reproduce our results is available online.
1 After we corrupt the training datasets (CIFAR-10, CIFAR-100, Fashion-MNIST) with noise we retain 10% as a secondary validation dataset.
This allows us to detect when the model is overﬁtting on noise by comparing the performance on clean versus noisy validation data.
For the class independent noise, we ﬂip the label to any other class with equal probability such that the ratio of noisy labels is η .
For class dependent noise, for CIFAR-10 and Fashion-MNIST we only ﬂip the label in the source classes { 9 , 2 , 3 , 5 , 4 } to the corresponding target class { 1 , 0 , 5 , 3 , 7 } given the noise ratio η .
And for CIFAR-100 we ﬂip between two randomly selected subclasses withing each superclass.
We use the same modes as in Wang et al.
(2019b) when training on CIFAR-10 which is an 8 layer network composed ofconvolutional layers followed by 2 fully connected layers.
For Fashion-MNIST the model contains 4 convolutional layers followed by 3 fully connected layers.
For CIFAR-100 the model we use is
ResNet-34 (He et al., 2016).
We train using SGD with 0 .
9 momentum, 0 .
005 weight decay for the convolutional layers and 0 .
01 for the fully connected layers, and a starting learning rate of 0 .
01 which we divide by 10 every 20 epochs for a total of 60 training epochs for CIFAR datasets and everyepochs for a total of 15 training epochs for Fashion-MNIST.
The training batch size is 128 samples.
We compare our solution for robustiﬁcation of the loss function through moments penalization against ﬁve other state of the art methods: i )
Tilted empirical risk minimization (TERM) introduced by Li et al.
(2021), ii ) Taylor expansion of cross entropy (Taylor) proposed by Feng et al.
(2020), iii )
Normalized cross entropy coupled with reverse cross entropy (Normalized) investigated by Ma et al.
(2020), iv )
Symmetric cross entropy (Symmetric) studied by Wang et al.
(2019b), and v ) Generalized cross entropy (Generalized) explored by Zhang & Sabuncu (2018).
For these methods, when possible we used the same parameters suggested by their authors, however, when the method underperfomed we used cross-validation to ﬁnd better ones.
When training on CIFAR-10 and Fashion-MNIST datasets we used: λ 1 = 1 , λ 2 = − 0 .
5 for Moments and its convex version, t = − 0 .
5 for TERM, t = 2 for Taylor, α = 10 , β
=
1 for Normalized, α = 0 .
1 , β = 1 for Symmetric, and q = 0 .
7 for
Generalized.
When training on CIFAR-100 the parameters where: λ 1 = 1 , λ 2 = − 0 .
5 for Moments and its convex version, t = − 0 .
5 for TERM, t = 6 for Taylor, α = 10 , β = 0 .
1 for Normalized, α = 6 . 0 , β = 0 .
1 for Symmetric, and q = 0 .
7 for Generalized.
The model accuracy for clean data when training with both, class independent and class dependent noise, are summarized in Table 1.
Of note, we report the accuracy results of the model at the end of the training to capture the overﬁtting on noise for some models.
We observe that the training with moments penalization outperforms other methods in low to moderate noise level scenarios and falls behind in high noise scenarios but still anove the classical cross-entropy.
The performances of the convex version and non-convex version are very similar, however, the convex version shows better results in high noise scenarios under class dependent noise.
In case of CIFAR datasets, for class independent noise and for η = 0 .
2 the proposed method improves the accuracy of the classical cross-entropy by 6% , the highest improvement among the investigated methods.
However, this improvement decreases as the noise ratio increases and approaches the [Table] Classical [Table]
Generalized
[Table]
accuracy of the classical cross entropy loss for η = 0 . 8 .
To note, the convex version of the moments method for η = 0 .
8 underpeformed by 0.8% compared to the classical cross-entropy, however, in all other scenarios it had a higher accuracy.
High noise scenarios turned out to be challenging for all losses with the Taylor and Generalized cross-entropy losses not converging on CIFAR-10 for η = 0 . 8 and the best accuracy was registered by Symmetric cross-entropy loss proposed by Wang et al. (2019b).
For class dependent noise, the moments penalization registered the highest accuracy for CIFAR-10 when η = 0 .
1 and η = 0 .
but had the lowest accuracy for η = 0 .
To further investigate this behavior, we reran the experiments and monitored the accuracy for each individual class.
The classical cross-entropy recorded similar accuracy as the robust methods on classes affected by noise and outperformed them on classes 6 and 8, both not targeted by class dependent noise.
The poor performance of robust methods can be justiﬁed by using parameters tuned for low and moderate noise.
In case of Fashion-MNIST dataset, as similar behavior as with CIFAR datasets was observed.
For low to moderate noise level scenarios the moments penalization methods had the highest or the second highest accuracy.
Moreover, when the noise was class dependent the convex version outperformed the non-convex version by at least 6%.
C ONCLUSION
The main goal of the current work was to investigate the optimization of a weighted mean and the ﬂexibility it provides in enforcing properties such as robustness when training with noisy labels.
In addition, we extended previous variance penalization methods to include higher-order moments while eliminating some of their limitations.
One of the signiﬁcant ﬁndings to emerge from this study was that we can control the ﬁnal distribution of the loss values by penalizing higher-order moments.
In particular, by enforcing the distribution of the cross-entropy to have a higher variance through negative variance penalization, we improved the models’ accuracy when trained with noisy labels.
Although this paper centers on classiﬁcation problems, the framework can also control the bias-variance trade-off and can also be applicable to regression problems.
A P ROOFS AND A DDITIONAL T HEORETICAL R ESULTS
First, we present a proposition used in the proof of Theorem 1.
Proposition 7.
Let U and V be two random variables, and let U and V denote their expectation, then:
[Equation]
Proof.
To prove we write V as (cid:0) V − V (cid:1) + V and then use the linear property of the expectation operator:
[Equation]
Next we present two propositions used in the proofs of the main theorems of this paper.
Proposition 8.
Let (cid:96) : R n → [0 , ∞ ) and p : R → R two convex functions, then the composition w ( x ) = p ( (cid:96) ( x ) − M ) where M ≥ 0 is convex if p is non decreasing.
Proof.
Using the convexity of (cid:96) for any x, y ∈ R n and α ∈ [0 , 1] we have:
[Equation]
Using the fact that p is non decreasing along we the above inequality we obtain:
[Equation]
and using convexity of p for the term on the right we get:
[Equation]
Combining the two inequalities leads to the following result ( (cid:96) ( αx + [Equation] 1 − [Equation]
y )) ≤ αp ( [Equation]
[Equation]
) + (1 − α ) p
(cid:96) ( y )) which proves the composition p ( (cid:96) ( x )) is convex.
Moreover, since p is convex and (cid:96) ( x ) − M is in the domain of p then p ( (cid:96) ( x ) − M ) is also convex as it is a composition with an afﬁne mapping.
This proves that w ( x ) is convex.
Proposition 9.
Let f, g : R n → [0 , ∞ ) be two convex function taking non-negative values then their product h ( x ) =
f ( x ) g ( x ) is convex if [ f ( x ) − f ( y )]
[ g ( x ) − g ( y )] ≥ 0 , ∀ x, y ∈ R n .
Proof.
The function h is convex if it satisﬁes the following inequality where α ∈ [0 , 1] :
[Equation]
Note that since the two functions are convex the following inequalities hold:
[Equation]
[Equation]
Substituting ( fg )( αx + (1 − α ) y ) from 15 in inequality 10 we obtain:
[Equation]
Multiplying the two square brackets and then grouping the terms we get:
[Equation]
We complete the proof by noting that α ≥ 0 and thus the product of two convex functions is convex if [ f ( x ) − f ( y )]
[ g ( x ) − g ( y )] ≥ 0 .
Lemma 2.
As introduced in Theorem 1, the variance expansion of a convex loss function (cid:96) ( v, y ) in v yields a new objective that is also convex in v if λ ∈ [0 , λ max ] , where λ max = 1 / ( E [ (cid:96) ] − min (cid:96) ) .
Proof.
Since y are constants and take ﬁnite values we can index both, loss and weight functions, using y as (cid:96) y
( v ) =
(cid:96) ( v, y ) and w y ( v ) = w ( v, y ) . We note that w y ( v ) = p ( (cid:96) y ( v ) − E [ (cid:96) ]) where p is a polynomial of degree 1 with p ( t ) =
λt +1 . First we use the result of Proposition 8 and get that w y ( v ) is convex if λ is non negative thus the lower bound of the interval.
Next, we use Proposition 9 which states that w y ( v ) (cid:96) y ( v ) is convex if [ w y ( v ) − w y ( u )]
[ (cid:96) y ( v ) − (cid:96) y ( u )] ≥ 0 and w y ( v ) is non negative.
Since p is convex we can use the ﬁrst order condition, p ( a ) − p ( b ) ≥ λ ( a − b ) , with a = (cid:96) y ( v ) − E [ (cid:96) ] and b = (cid:96) y ( u ) − E
[ (cid:96) ] and replace w y in the previous inequality to obtain λ [ (cid:96) ( x ) − (cid:96) ( y )] 2 ≥ 0 which is always true for λ > 0 . However, w y ( v ) is non negative only for λ ≤ 1 / (min (cid:96) − E [ (cid:96) ]) which proves the upper bound of the interval.
Theorem 3 (Moments Expansion) .
Let (cid:96) be a loss function with ﬁnite ﬁrst m central moments and deﬁne w ( [Equation] , y )
[-=
-][Equation]
[Equation] E [ [Equation] ( v, y )])
i − 1 , then by taking the weighted mean of (cid:96) with weights w is equivalent to:
[Equation]
where ˜ λ i
=
λ
i
+ λ
i +1 E
[ (cid:96) ] for i < m and ˜ λ m = λ m
Proof.
In this case, for i ≥ 2 we cannot apply Lemma 7 since the expression ( (cid:96) − E
[ (cid:96) ])
i is not guaranteed to have zero mean and as a result it incurs an additional penalization of the previous moment.
The proof follows the same steps as Theorem 1:
[Equation]
=
[Equation]
i = λ i E
[Equation]
]) i − 1 (cid:96) (cid:3) =
λ 1 E
[ (cid:96) ]
+ m (cid:88) i =
λ i E (cid:2) ( (cid:96) − E
[ (cid:96) ])
i − 1 (cid:96) (cid:3) =
λ 1 E
[ (cid:96) ]
+ m (cid:88) i =
λ i E (cid:2) ( (cid:96) − E
[ (cid:96) ])
i − 1 ( (cid:96) − E
[ (cid:96) ]
+ E [ (cid:96) ]) (cid:3) =
λ 1 E
[ (cid:96) ]
+ m (cid:88) i =
λ i E (cid:2) ( (cid:96) − E
[ (cid:96) ]) i (cid:3) + m (cid:88) i
= λ i E
[ (cid:96) ] E (cid:2) ( (cid:96) − E
[ (cid:96) ])
i − 1 (cid:3) =
λ 1 E
[ (cid:96) ]
+ m − 1 (cid:88) i =
˜
λ i E (cid:2) ( (cid:96) − E [ (cid:96) ])
i (cid:3) + λ m E
[( (cid:96) − E [ (cid:96) ]) m ]
For the last step, we combine the two sums by matching the E (cid:2) ( (cid:96) − E
[ (cid:96) ]) i (cid:3) terms and consolidate the penalization factors as ˜ λ
i =
λ
i
+ λ
i +1 E
[ (cid:96) ] .
Lemma 4 (Convexity of Moments Expansion) .
Let (cid:96) ( v, y ) be a loss function convex in v and p : R → [0 , ∞ ) be a non negative convex function and M ≥ 0 , then the weighted objective w ( v, y ) (cid:96) ( v, y ) with w ( v, y ) = p ( (cid:96) ( v, y ) − M ) is convex in v if p is non decreasing.
Proof.
Since y are constants and take ﬁnite values we can index both, loss and weight functions, using y such as (cid:96) y ( v ) =
(cid:96) ( v, y ) and w y ( v ) =
w ( v, y ) . With w y ( v ) = p ( (cid:96) y ( v ) − M ) and using the result of Proposition 8 we obtain that w y ( v ) is convex as p is non decreasing.
Next, from Proposition 9 and since p takes non negative values we get that the product w y ( v ) (cid:96) y ( v ) is convex if [ w y ( v ) − w y ( u )]
[ (cid:96) y ( v ) − (cid:96) y ( u )] ≥ 0 .
Since p is convex we use the ﬁrst order condition p ( a ) − p ( b ) ≥ p (cid:48) ( b )( a − b ) with a = (cid:96) y ( v ) − M and b = (cid:96) y ( u ) − M to obtain:
[Equation]
Substituting this result in the requirement from Proposition 9 we obtain:
[Equation]
Given that p is non decreasing implies p (cid:48) ( [Equation] ( y ) − M ) ≥ {+
 and+} proves that the inequality always holds.