Original: C AN GRADIENT CLIPPING MITIGATE LABEL NOISE ?
Revised: C AN GRADIENT CLIPPING MITIGATE LABEL NOISE ?

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Gradient clipping is a widely-used technique in the training of deep networks, and is generally motivated from an optimisation lens: informally, clipping controls the dynamics of iterates, thus enhancing the rate of convergence to a local minimum.
Revised: Gradient clipping is a widely-used technique in the training of deep networks, and is generally motivated from an optimisation lens: informally, clipping controls the dynamics of iterates, thus enhancing the rate of convergence to a local minimum.

Original: This intuition has been made precise in a line of recent works, which show that suitable clipping can yield signiﬁcantly faster convergence than vanilla gradient descent.
Revised: This intuition has been made precise in a line of recent works, which show that suitable clipping can yield signiﬁcantly faster convergence than vanilla gradient descent.

Original: In this paper, we propose a new lens for studying gradient clipping, namely, robustness : informally, one expects clipping to mitigate the effects of noise, since one does not overly trust any single sample.
Revised: In this paper, we propose a new lens for studying gradient clipping, namely, robustness : informally, one expects clipping to mitigate the effects of noise, since one does not overly trust any single sample.

Original: Surprisingly, we prove that for the common problem of label noise in classiﬁcation, standard gradient clipping does not in general provide robustness.
Revised: Surprisingly, we prove that for the common problem of label noise in classiﬁcation, standard gradient clipping does not in general provide robustness.

Original: On the other hand, we show that a simple variant of gradient clipping is robust, and is equivalent to suitably modifying the underlying loss function.
Revised: On the other hand, we show that a simple variant of gradient clipping is robust, and is equivalent to suitably modifying the underlying loss function.

Original: As a special case, this yields a simple, noise-robust modiﬁcation of the standard cross-entropy loss which performs well empirically.
Revised: As a special case, this yields a simple, noise-robust modiﬁcation of the standard cross-entropy loss which performs well empirically.

Original:  I NTRODUCTION : TWO FACES OF GRADIENT CLIPPING
Revised:  I NTRODUCTION : TWO FACES OF GRADIENT CLIPPING

Original: Gradient clipping is a widely-used technique in the training of deep networks (Mikolov, 2012; Pascanu et al., 2012; 2013; Mnih et al., 2015; Gehring et al., 2017; Merity et al., 2018;
Revised: Gradient clipping is a widely-used technique in the training of deep networks (Mikolov, 2012; Pascanu et al., 2012; 2013; Mnih et al., 2015; Gehring et al., 2017; Merity et al., 2018;

Original: Peters et al., 2018), and in its simplest form, involves capping the global parameter gradient norm at a speciﬁed threshold.
Revised: Peters et al., 2018), and in its simplest form, involves capping the global parameter gradient norm at a speciﬁed threshold.

Original: Constraining the gradient norm is paramount in training models such as recurrent neural networks, which may otherwise suffer from an “exploding gradient” problem (Bengio et al., 1994).
Revised: Constraining the gradient norm is paramount in training models such as recurrent neural networks, which may otherwise suffer from an “exploding gradient” problem (Bengio et al., 1994).

Original: Clipping has seen broader value in other neural models, and is abstractly understood as being valuable from an optimisation lens: intuitively, by ensuring that the gradient norm is not too large, the dynamics of iterates are more well-behaved (Hazan et al., 2015; Levy, 2016; Zhang et al., 2019).
Revised: Clipping has seen broader value in other neural models, and is abstractly understood as being valuable from an optimisation lens: intuitively, by ensuring that the gradient norm is not too large, the dynamics of iterates are more well-behaved (Hazan et al., 2015; Levy, 2016; Zhang et al., 2019).

Original: A distinct line of work has also explored the value of gradient clipping from a privacy lens (Abadi et al., 2016;
Revised: A distinct line of work has also explored the value of gradient clipping from a privacy lens (Abadi et al., 2016;

Original: Pichapati et al., 2019): intuitively, it prevents any single instance from overly inﬂuencing parameter updates, which is a core requirement in ensuring differential privacy (Dwork et al., 2006).
Revised: Pichapati et al., 2019): intuitively, it prevents any single instance from overly inﬂuencing parameter updates, which is a core requirement in ensuring differential privacy (Dwork et al., 2006).

Original: In this paper, we propose a new lens with which to study gradient clipping, namely, robustness : intuitively, clipping the gradient prevents over-conﬁdent descent steps, which is plausibly beneﬁcial in the presence of noise.
Revised: In this paper, we propose a new lens with which to study gradient clipping, namely, robustness : intuitively, clipping the gradient prevents over-conﬁdent descent steps, which is plausibly beneﬁcial in the presence of noise.

Original: Given this intuition, our interest is whether gradient clipping can mitigate the problem of label noise in classiﬁcation, which has received signiﬁcant recent interest (Scott et al., 2013; Natarajan et al., 2013; Menon et al., 2015; Liu & Tao, 2016; Patrini et al., 2017; Ghosh et al., 2017; Han et al., 2018; Zhang & Sabuncu, 2018; Song et al., 2019; Thulasidasan et al., 2019;
Revised: Given this intuition, our interest is whether gradient clipping can mitigate the problem of label noise in classiﬁcation, which has received signiﬁcant recent interest (Scott et al., 2013; Natarajan et al., 2013; Menon et al., 2015; Liu & Tao, 2016; Patrini et al., 2017; Ghosh et al., 2017; Han et al., 2018; Zhang & Sabuncu, 2018; Song et al., 2019; Thulasidasan et al., 2019;

Original: Charoenphakdee et al., 2019).
Revised: Charoenphakdee et al., 2019).

Original: We study this question, and provide three main contributions:
Revised: We study this question, and provide three main contributions:

Original: (a) we show that gradient clipping alone does not endow label noise robustness to even simple models.
Revised: (a) we show that gradient clipping alone does not endow label noise robustness to even simple models.

Original: Speciﬁcally, we show that under stochastic gradient descent with linear models, gradient clipping is related to optimising a “Huberised” loss (Lemma 1, 2).
Revised: Speciﬁcally, we show that under stochastic gradient descent with linear models, gradient clipping is related to optimising a “Huberised” loss (Lemma 1, 2).

Original: While such Huberised losses preserve classiﬁcation calibration (Lemma 3), they are not robust to label noise (Proposition 4).
Revised: While such Huberised losses preserve classiﬁcation calibration (Lemma 3), they are not robust to label noise (Proposition 4).

Original: (b) we propose composite loss-based gradient clipping , a variant that does have label noise robustness.
Revised: (b) we propose composite loss-based gradient clipping , a variant that does have label noise robustness.

Original: Speciﬁcally, for losses comprising a base loss composed with a link function, (e g., softmax crossentropy), we only clip the contribution of the base loss .
Revised: Speciﬁcally, for losses comprising a base loss composed with a link function, (e.g., softmax crossentropy), we only clip the contribution of the base loss .

Original: The resulting partially Huberised loss preserves classiﬁcation calibration (Lemma 6), while being robust to label noise (Proposition 7).
Revised: The resulting partially Huberised loss preserves classiﬁcation calibration (Lemma 6), while being robust to label noise (Proposition 7).

Original: (c) we empirically verify that on both synthetic and real-world datasets, partially Huberised versions of standard losses (e.g., softmax cross-entropy) perform well in the presence of label noise (§5).
Revised: (c) we empirically verify that on both synthetic and real-world datasets, partially Huberised versions of standard losses (e.g., softmax cross-entropy) perform well in the presence of label noise (§5).

Revised: [Figure]

Original: To illustrate the difference between standard and composite loss-based gradient clipping in (a) and (b), consider the pervasive softmax cross-entropy loss, viz.
Revised: To illustrate the difference between standard and composite loss-based gradient clipping in (a) and (b), consider the pervasive softmax cross-entropy loss, viz.

Original: the logistic loss for binary classiﬁcation.
Revised: the logistic loss for binary classiﬁcation.

Original: [Figure]

Original: In (a), we relate gradient clipping to loss “Huberisation”, which linearises the logistic loss when it exceeds a ﬁxed threshold.
Revised: In (a), we relate gradient clipping to loss “Huberisation”, which linearises the logistic loss when it exceeds a ﬁxed threshold.

Original: In (b), we introduce composite loss-based gradient clipping, which is equivalent to a “partial loss Huberisation” that linearises only the cross-entropy loss but leaves the sigmoid link untouched.
Revised: In (b), we introduce composite loss-based gradient clipping, which is equivalent to a “partial loss Huberisation” that linearises only the cross-entropy loss but leaves the sigmoid link untouched.

Original: Figure 1 illustrates the Huberised and partially Huberised logistic loss.
Revised: Figure 1 illustrates the Huberised and partially Huberised logistic loss.

Original:  B ACKGROUND AND RELATED WORK
Revised:  B ACKGROUND AND RELATED WORK

Original: We provide background on gradient clipping, loss functions for classiﬁcation, and label noise.
Revised: We provide background on gradient clipping, loss functions for classiﬁcation, and label noise.

Original: Gradient clipping .
Revised: Gradient clipping .

Original: Consider a supervised learning task over instances X and labels Y , where we have a family of models indexed by θ ∈ Θ , and the quality of a particular model is measured by a loss function (cid:96) θ : X × Y → R .
Revised: Consider a supervised learning task over instances X and labels Y , where we have a family of models indexed by θ ∈ Θ , and the quality of a particular model is measured by a loss function (cid:96) θ : X × Y → R .

Original: The gradient for a mini-batch{ ( x n , y n ) }
Revised: The gradient for a mini-batch

Original: Nn =1 is

Original: [Equation]
Revised: [Equation]

Original: One may instead compute the clipped gradient, which for user-speciﬁed threshold τ > 0 is
Revised: One may instead compute the clipped gradient, which for user-speciﬁed threshold τ > 0 is

Original: [Equation]
Revised: [Equation]

Original: Employing ¯ g τ ( θ ) for optimisation corresponds to clipped gradient descent .
Revised: Employing ¯ g τ ( θ ) for optimisation corresponds to clipped gradient descent .

Original: This is closely related to normalised gradient descent (NGD) (Hazan et al., 2015), wherein one employs ˜ g ( θ ) .
Revised: This is closely related to normalised gradient descent (NGD) (Hazan et al., 2015), wherein one employs ˜ g ( θ ) .

Original: = g ( θ) ||
Revised: = [Equation] ( θ

Revised: [Equation]

Original:  g ( θ ) || 2 .
Revised: || g ( θ ) || 2 .

Original: Hazan et al.
Revised: Hazan et al.

Original: (2015) showed that NGD can lead to convergence for a wider class of functions than standard gradient descent.
Revised: (2015) showed that NGD can lead to convergence for a wider class of functions than standard gradient descent.

Original: Levy (2016) showed that NGD can escape saddle points for non-convex problems.
Revised: Levy (2016) showed that NGD can escape saddle points for non-convex problems.

Original: Zhang et al.
Revised: Zhang et al.

Original: (2019) showed that gradient clipping can lead to accelerated convergence over gradient descent.
Revised: (2019) showed that gradient clipping can lead to accelerated convergence over gradient descent.

Original: Gradient clipping has also been explored for privacy (Abadi et al., 2016; Pichapati et al., 2019), motivated by it preventing any single instance from dominating parameter updates.
Revised: Gradient clipping has also been explored for privacy (Abadi et al., 2016; Pichapati et al., 2019), motivated by it preventing any single instance from dominating parameter updates.

Original: Loss functions .
Revised: Loss functions .

Original: In binary classiﬁcation, one observes samples from a distribution D over X × {± 1 } , and seeks a predictor f : X → R with low risk R ( f ) .
Revised: In binary classiﬁcation, one observes samples from a distribution D over X × {± 1 } , and seeks a predictor f : X → R with low risk R ( f ) .

Original: =
Revised: =

Original: E (cid:96) ( y, f ( x ))
Revised: E (cid:96) ( y, f ( x ))

Original: according to a loss which, in an overload of notation, we denote (cid:96) : {± 1 } × R → R + .
Revised: according to a loss which, in an overload of notation, we denote (cid:96) : {± 1 } × R → R + .

Original: For the zero-one loss (cid:96) 

Revised: For the zero-one loss (cid:96)

Original: ( y, f )
Revised: ( y, f )

Original: = (cid:74) y
Revised: = (cid:74) y

Original: · f < 0 (cid:75) , R ( f ) is known as the misclassiﬁcation risk.
Revised: · f < 0 (cid:75) , R ( f ) is known as the misclassiﬁcation risk.

Original: Rather than directly using (cid:96) 01 , for computational convenience one often employs a margin loss (cid:96) ( y, f )
Revised: Rather than directly using (cid:96) 01 , for computational convenience one often employs a margin loss (cid:96) ( y, f )

Original: =
Revised: =

Original: φ ( y · f ) , for convex φ : R → R + .
Revised: φ ( y · f ) , for convex φ : R → R + .

Original: We say that φ is classiﬁcation calibrated (Zhang, 2004; Bartlett et al., 2006) if driving the excess risk over the Bayes-optimal predictor for φ to zero also drives the excess risk for (cid:96) 01 to zero; that is, minimising the φ -risk is statistically consistent for classiﬁcation.
Revised: We say that φ is classiﬁcation calibrated (Zhang, 2004; Bartlett et al., 2006) if driving the excess risk over the Bayes-optimal predictor for φ to zero also drives the excess risk for (cid:96) 01 to zero; that is, minimising the φ -risk is statistically consistent for classiﬁcation.

Original: A canonical example is the hinge loss φ ( z ) =
Revised: A canonical example is the hinge loss φ ( z ) =

Original: [1 − z ] + .
Revised: [1 − z ] + .

Original: We call φ admissible if it is “well-behaved” in the sense of being bounded from below, strictly convex, continuously differentiable, non-increasing, and classiﬁcation calibrated.
Revised: We call φ admissible if it is “well-behaved” in the sense of being bounded from below, strictly convex, continuously differentiable, non-increasing, and classiﬁcation calibrated.

Original: We say that φ is proper composite (Reid & Williamson, 2010;
Revised: We say that φ is proper composite (Reid & Williamson, 2010;

Original: Nock & Nielsen, 2009) (or for brevity composite ) if its minimising scores can be interpreted as probabilities.
Revised: Nock & Nielsen, 2009) (or for brevity composite ) if its minimising scores can be interpreted as probabilities.

Original: “Composite” here refers to such losses comprising a base loss ϕ composed with an invertible link function F : R → [0 , 1] .
Revised: “Composite” here refers to such losses comprising a base loss ϕ composed with an invertible link function F : R → [0 , 1] .

Original: While φ accepts as input real-valued scores (e.g., the ﬁnal layer logits of a neural network), these are internally converted to probabilities via F .
Revised: While φ accepts as input real-valued scores (e.g., the ﬁnal layer logits of a neural network), these are internally converted to probabilities via F .

Original: A canonical example is the logistic loss φ ( z )
Revised: A canonical example is the logistic loss φ ( z )

Original: = − log F ( z ) with
Revised: = − log F ( z ) with

Original: F : z (cid:55)→ σ ( z ) for sigmoid σ ( z ) .
Revised: F : z (cid:55)→ σ ( z ) for sigmoid σ ( z ) .

Original: = (1+ e − z ) − 1 .
Revised: = (1+ e − z ) − 1 .

Original: The multiclass analogue is the softmax cross-entropy loss, wherein the sigmoid becomes a softmax.
Revised: The multiclass analogue is the softmax cross-entropy loss, wherein the sigmoid becomes a softmax.

Original: See Appendix B for a more technical discussion.
Revised: See Appendix B for a more technical discussion.

Original: Learning under label noise .
Revised: Learning under label noise .

Original: In classiﬁcation under label noise , one has samples from distribution ¯
Revised: In classiﬁcation under label noise , one has samples from distribution ¯

Original: D where P ¯ D ( y | x ) is a noisy version of P D ( y | x ) , e.g., all labels are corrupted with a ﬁxed constant probability.
Revised: D where P ¯ D ( y | x ) is a noisy version of P D ( y | x ) , e.g., all labels are corrupted with a ﬁxed constant probability.

Original: The goal remains to ensure low risk with respect to the clean D .
Revised: The goal remains to ensure low risk with respect to the clean D .

Original: This problem has a long history in statistics (Ekholm & Palmgren, 1982; Copas, 1988), and has emerged as a topic of recent interest in machine learning.
Revised: This problem has a long history in statistics (Ekholm & Palmgren, 1982; Copas, 1988), and has emerged as a topic of recent interest in machine learning.

Original: There has been particular interest in the problem of learning under symmetric label noise, wherein all instances have a constant probability of their labels being ﬂipped uniformly to any of the other classes.
Revised: There has been particular interest in the problem of learning under symmetric label noise, wherein all instances have a constant probability of their labels being ﬂipped uniformly to any of the other classes.

Original: Scott et al.
Revised: Scott et al.

Original: ; Katz-Samuels et al.
Revised: ; Katz-Samuels et al.

Original: (2019) proposed a framework for analysing the more general setting of class-conditional label noise.
Revised: (2019) proposed a framework for analysing the more general setting of class-conditional label noise.

Original: Takenouchi & Eguchi (2004); Natarajan et al. (2013); Cid-Sueiro et al.
Revised: Takenouchi & Eguchi (2004); Natarajan et al. (2013); Cid-Sueiro et al.

Original: ; Patrini et al.
Revised: ; Patrini et al.

Original: ; Natarajan et al.
Revised: ; Natarajan et al.

Original: (2018); van Rooyen & Williamson (2018) studied loss-correction for obtaining unbiased risk estimators.
Revised: (2018); van Rooyen & Williamson (2018) studied loss-correction for obtaining unbiased risk estimators.

Original: Recent works have explored ideas ranging from to model (dis)agreement (Malach &
Revised: Recent works have explored ideas ranging from to model (dis)agreement (Malach &

Original: Shalev-Shwartz, 2017; Han et al., 2018; Song et al., 2019) to abstention (Thulasidasan et al., 2019).
Revised: Shalev-Shwartz, 2017; Han et al., 2018; Song et al., 2019) to abstention (Thulasidasan et al., 2019).

Original:  G RADIENT CLIPPING AND H UBERISED LOSSES
Revised:  G RADIENT CLIPPING AND H UBERISED LOSSES

Original: We ﬁrst show that gradient clipping in general does not endow robustness to label noise, even in simple settings.
Revised: We ﬁrst show that gradient clipping in general does not endow robustness to label noise, even in simple settings.

Original: Speciﬁcally, we establish that stochastic gradient clipping with linear models is equivalent to modifying the underlying loss (Lemma 1).
Revised: Speciﬁcally, we establish that stochastic gradient clipping with linear models is equivalent to modifying the underlying loss (Lemma 1).

Original: This modiﬁed loss is closely related to a Huberised loss (Lemma 2), which is equivalent to loss-based gradient clipping ( L-gradient clipping ).
Revised: This modiﬁed loss is closely related to a Huberised loss (Lemma 2), which is equivalent to loss-based gradient clipping ( L-gradient clipping ).

Original: Unfortunately,
Revised: Unfortunately,

Original: Huberised losses, and thus L-gradient clipping, are not robust to label noise (Proposition 4).
Revised: Huberised losses, and thus L-gradient clipping, are not robust to label noise (Proposition 4).

Original: 3. F ROM GRADIENT TO L- GRADIENT CLIPPING
Revised: 3. F ROM GRADIENT TO L- GRADIENT CLIPPING

Original: Consider a binary classiﬁcation problem, with Y = {± 1 } .
Revised: Consider a binary classiﬁcation problem, with Y = {± 1 } .

Original: Suppose we use a scoring model s θ ( x ) for θ ∈ Θ with margin m θ ( x, y ) .
Revised: Suppose we use a scoring model s θ ( x ) for θ ∈ Θ with margin m θ ( x, y ) .

Original: =
Revised: =

Original: y · s θ ( x ) , and margin loss (cid:96) θ ( x, y ) .
Revised: y · s θ ( x ) , and margin loss (cid:96) θ ( x, y ) .

Original: =
Revised: =

Original: φ ( m θ ( x, y )) for admissible φ : R → R + .
Revised: φ ( m θ ( x, y )) for admissible φ : R → R + .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: Now suppose we perform stochastic gradient descent (i.e., we use N = 1 in (1)) and use a linear scorer s θ ( x ) = θ T x .
Revised: Now suppose we perform stochastic gradient descent (i.e., we use N = 1 in (1)) and use a linear scorer s θ ( x ) = θ T x .

Original: Here, gradient clipping of (3) is equivalent to modifying the loss as follows.
Revised: Here, gradient clipping of (3) is equivalent to modifying the loss as follows.

Original: Lemma 1.
Revised: Lemma 1.

Original: Pick any admissible margin loss φ , and τ > 0 .
Revised: Pick any admissible margin loss φ , and τ > 0 .

Original: Then, for loss (cid:96) θ
Revised: Then, for loss (cid:96) θ

Original: ( x, y ) .
Revised: ( x, y ) .

Original: =
Revised: =

Original: φ ( m θ ( x, y )) with linear scorer, the clipped loss gradient (2) is equivalent to the gradient of a modiﬁed loss ¯ (cid:96) : clip τ( ∇ (cid:96) θ ( x, y ))
Revised: φ ( m θ ( x, y )) with linear scorer, the clipped loss gradient (2) is equivalent to the gradient of a modiﬁed loss ¯ (cid:96) : clip τ

Original: =
Revised: [Equation]

Original: ∇ ¯ (cid:96) θ ( x, y ) , with ¯ (cid:96) θ ( x, y ) .
Revised: with ¯ (cid:96) θ ( x, y ) [Equation]

Original: =(cid:40)
Revised: =

Original: [Equation]
Revised: [Equation]

Original: The loss ¯ (cid:96) in (4) is intuitive: if the derivative of the original loss φ exceeds a certain effective clipping threshold , we replace φ with a linear loss function.
Revised: The loss ¯ (cid:96) in (4) is intuitive: if the derivative of the original loss φ exceeds a certain effective clipping threshold , we replace φ with a linear loss function.

Original: This effective threshold τ · ( (cid:107)∇ m θ
Revised: This effective threshold τ · ( (cid:107)∇ m θ

Original: [Equation]
Revised: [Equation]

Original: x, y ) [Equation] 2 ) − 1 takes into account the instance-dependent margin gradient, viz.[Equation]
Revised: x, y ) (cid:107) 2 ) − 1 takes into account the instance-dependent margin gradient, viz.

Original: linear

Original: [Equation]
Revised: [Equation]

Original: We will comment in §4.4 as to the properties of gradient clipping in more general scenarios.
Revised: We will comment in §4.4 as to the properties of gradient clipping in more general scenarios.

Original: For the moment, we note that Lemma 1 is closely related to the “Huber loss” (Huber, 1964),
Revised: For the moment, we note that Lemma 1 is closely related to the “Huber loss” (Huber, 1964),

Original: [Equation]
Revised: [Equation]

Revised: else

Revised: [Equation]

Revised: [Equation]

Original: This replaces the extremes of the square loss with the absolute loss.
Revised: This replaces the extremes of the square loss with the absolute loss.

Original: Both the absolute and Huber losses are widely employed in robust regression (Huber, 1981, Chapter 3), (Hampel et al., 1986, pg.
Revised: Both the absolute and Huber losses are widely employed in robust regression (Huber, 1981, Chapter 3), (Hampel et al., 1986, pg.

Original: Note that (4) slightly differs from (5) as the effective clipping threshold is instance-dependent.
Revised: Note that (4) slightly differs from (5) as the effective clipping threshold is instance-dependent.

Original: One may nonetheless arrive at Huber-style losses via a variant of gradient clipping; this connection will prove useful for our subsequent analysis.
Revised: One may nonetheless arrive at Huber-style losses via a variant of gradient clipping; this connection will prove useful for our subsequent analysis.

Original: Per (3), gradient clipping involves using
Revised: Per (3), gradient clipping involves using

Original: [Equation]
Revised: [Equation]

Original: wherein the clipping considers the margin gradient.
Revised: wherein the clipping considers the margin gradient.

Original: Consider now the following loss-based gradient clipping ( L-gradient clipping ), wherein we clip only the contribution arising from the loss:
Revised: Consider now the following loss-based gradient clipping ( L-gradient clipping ), wherein we clip only the contribution arising from the loss:

Original: [Equation]
Revised: [Equation]

Original: Compared to (6), we effectively treat the margin gradient norm (cid:107)∇ m θ ( x, y ) (cid:107) 2 as constant across instances, and focus on bounding the loss derivative.
Revised: Compared to (6), we effectively treat the margin gradient norm (cid:107)∇ m θ ( x, y ) (cid:107) 2 as constant across instances, and focus on bounding the loss derivative.

Original: The latter may be a signiﬁcant component in the gradient norm; e.g., for linear models, (cid:107)∇ m θ ( x, y ) (cid:107) 2 = (cid:107) x (cid:107) 2 is often bounded.
Revised: The latter may be a signiﬁcant component in the gradient norm; e.g., for linear models, (cid:107)∇ m θ ( x, y ) (cid:107) 2 = (cid:107) x (cid:107) 2 is often bounded.

Original: Observe further that for linear models with
Revised: Observe further that for linear models with

Original: [Equation]
Revised: [Equation]

Original:  2 ≡ R across instances, (7) is a rescaled version of the clipped gradient:
Revised: x (cid:107) 2 ≡ R across instances, (7) is a rescaled version of the clipped gradient:

Original: [Equation]
Revised: [Equation]

Original: Interestingly, L-gradient clipping equivalently uses the following “Huberised” version of the loss.
Revised: Interestingly, L-gradient clipping equivalently uses the following “Huberised” version of the loss.

Original: Lemma 2.
Revised: Lemma 2.

Original: Pick any admissible margin loss φ : R → R + with Fenchel conjugate φ ∗ , and τ > 0 .
Revised: Pick any admissible margin loss φ : R → R + with Fenchel conjugate φ ∗ , and τ > 0 .

Original: Then, the clipped gradient in (7) is equivalent to employing a Huberised loss function ¯ φ τ such that 
Revised: Then, the clipped gradient in (7) is equivalent to employing a Huberised loss function ¯ φ τ such that clip

Original: [Equation]
Revised: [Equation]

Original: Evidently, ¯ φ τ linearises φ once its derivative is sufﬁciently large, akin to the Huber loss in (5).
Revised: Evidently, ¯ φ τ linearises φ once its derivative is sufﬁciently large, akin to the Huber loss in (5).

Original: One may verify that for φ ( z ) =
Revised: One may verify that for φ ( z ) =

Original: (1 − z ) 2 , one arrives exactly at (5).
Revised: (1 − z ) 2 , one arrives exactly at (5).

Original: Example 3.1: For the logistic loss φ ( z ) =
Revised: Example 3.1: For the logistic loss φ ( z ) =

Original: log(1 + e − z ) , the Huberised loss for τ ∈ (0 , 1) is
Revised: log(1 + e − z ) , the Huberised loss for τ ∈ (0 , 1) is

Original: [Equation]
Revised: [Equation]

Original: Per Figure 1a, this linearises φ beyond a ﬁxed threshold.
Revised: Per Figure 1a, this linearises φ beyond a ﬁxed threshold.

Original: See Appendix C for further illustrations.
Revised: See Appendix C for further illustrations.

Original: The use of Huberised losses in classiﬁcation is not new (Zhang et al., 2002; Zhang & Johnson, 2003;
Revised: The use of Huberised losses in classiﬁcation is not new (Zhang et al., 2002; Zhang & Johnson, 2003;

Original: Rosset & Zhu, 2007).
Revised: Rosset & Zhu, 2007).

Original: However, we now provide a novel study of their label noise robustness.
Revised: However, we now provide a novel study of their label noise robustness.

Original: 3. C ONSISTENCY AND LABEL - NOISE ROBUSTNESS OF H UBERISED LOSSES
Revised: 3. C ONSISTENCY AND LABEL - NOISE ROBUSTNESS OF H UBERISED LOSSES

Original: Before studying the effect of label noise on Huberised losses, it is apposite to consider whether they are suitable for use even in the absence of noise.
Revised: Before studying the effect of label noise on Huberised losses, it is apposite to consider whether they are suitable for use even in the absence of noise.

Original: One way to formalise this is to ask whether the losses maintain classiﬁcation calibration , in the sense of §2.
Revised: One way to formalise this is to ask whether the losses maintain classiﬁcation calibration , in the sense of §2.

Original: This is a minimal requirement on a loss to be useful for classiﬁcation (Zhang, 2004; Bartlett et al., 2006).
Revised: This is a minimal requirement on a loss to be useful for classiﬁcation (Zhang, 2004; Bartlett et al., 2006).

Original: One may further ask whether the losses preserve class-probabilities, i.e., are proper composite in the sense of §2.
Revised: One may further ask whether the losses preserve class-probabilities, i.e., are proper composite in the sense of §2.

Original: It is desirable to preserve this key trait of losses such as the logistic loss.
Revised: It is desirable to preserve this key trait of losses such as the logistic loss.

Original: The following clariﬁes both points.
Revised: The following clariﬁes both points.

Original: Lemma 3.
Revised: Lemma 3.

Original: Pick any admissible margin loss φ and τ > 0 .
Revised: Pick any admissible margin loss φ and τ > 0 .

Original: Then, the Huberised loss ¯ φ τ in (8) is classiﬁcation calibrated.
Revised: Then, the Huberised loss ¯ φ τ in (8) is classiﬁcation calibrated.

Original: If φ is proper composite and τ ≥ − φ (cid:48) (0) , then ¯ φ τ is also proper composite.
Revised: If φ is proper composite and τ ≥ − φ (cid:48) (0) , then ¯ φ τ is also proper composite.

Original: L-gradient clipping is thus benign for classiﬁcation, generalising Rosset & Zhu (2007, Section 3.4), which was for the square-hinge loss.
Revised: L-gradient clipping is thus benign for classiﬁcation, generalising Rosset & Zhu (2007, Section 3.4), which was for the square-hinge loss.

Original: Interestingly, for composite φ and small τ , the proof reveals that ¯ φ τ has a non-invertible link function.
Revised: Interestingly, for composite φ and small τ , the proof reveals that ¯ φ τ has a non-invertible link function.

Original: Intuitively, such ¯ φ τ are effectively linear, and linear losses are unsuited for estimating probabilities (van Rooyen et al., 2015; Charoenphakdee et al., 2019).
Revised: Intuitively, such ¯ φ τ are effectively linear, and linear losses are unsuited for estimating probabilities (van Rooyen et al., 2015; Charoenphakdee et al., 2019).

Original: We now turn to our central object of inquiry: does gradient clipping endow robustness to label noise?
Revised: We now turn to our central object of inquiry: does gradient clipping endow robustness to label noise?

Original: To study this, we consider L-gradient clipping, which as noted above is a special case of gradient clipping under linear models with constant (cid:107)∇ m θ ( x, y ) (cid:107) 2 .
Revised: To study this, we consider L-gradient clipping, which as noted above is a special case of gradient clipping under linear models with constant (cid:107)∇ m θ ( x, y ) (cid:107) 2 .

Original: Since L-gradient clipping is in turn equivalent to using a Huberised loss, we may study the robustness properties of this loss.
Revised: Since L-gradient clipping is in turn equivalent to using a Huberised loss, we may study the robustness properties of this loss.

Original: Surprisingly, when our loss is convex (e.g., softmax cross-entropy), Huberised losses are not robust to even very simple forms of label noise.
Revised: Surprisingly, when our loss is convex (e.g., softmax cross-entropy), Huberised losses are not robust to even very simple forms of label noise.

Original: Essentially, since these losses are still convex, they can still be affected by errant outlier observations.
Revised: Essentially, since these losses are still convex, they can still be affected by errant outlier observations.

Original: Formally, using a result of Long & Servedio (2010) (see
Revised: Formally, using a result of Long & Servedio (2010) (see

Original: Appendix F.1) , we arrive at the following.
Revised: Appendix E.1), we arrive at the following.

Original: Proposition 4.
Revised: Proposition 4.

Original: Pick any admissible margin loss φ and τ > 0 .
Revised: Pick any admissible margin loss φ and τ > 0 .

Original: Then, ∃ a separable distribution for which the optimal linear classiﬁer under ¯ φ τ is equivalent to random guessing under symmetric noise.
Revised: Then, ∃ a separable distribution for which the optimal linear classiﬁer under ¯ φ τ is equivalent to random guessing under symmetric noise.

Original: To situate Proposition 4 in a broader context, we note that for regression problems, it is well known that the Huber loss is susceptible to “high leverage” outliers (Hampel et al., 1986, pg. 313), (Rousseeuw & Leroy, 1987, pg. 13), (Maronna et al., 2019, pg. 104), i.e., extremal instances which dominate the optimal solution.
Revised: To situate Proposition 4 in a broader context, we note that for regression problems, it is well known that the Huber loss is susceptible to “high leverage” outliers (Hampel et al., 1986, pg. 313), (Rousseeuw & Leroy, 1987, pg. 13), (Maronna et al., 2019, pg. 104), i.e., extremal instances which dominate the optimal solution.

Original: Proposition 4 complements these for the case of label noise in classiﬁcation.
Revised: Proposition 4 complements these for the case of label noise in classiﬁcation.

Original: Given that gradient clipping does not endow label noise robustness, how else might we proceed?
Revised: Given that gradient clipping does not endow label noise robustness, how else might we proceed?

Original: In a regression context, the outlier vulnerability of the Huber loss can be addressed by using a trimmed average of the loss (Rousseeuw & Leroy, 1987; Bhatia et al., 2015; Yang et al., 2018).
Revised: In a regression context, the outlier vulnerability of the Huber loss can be addressed by using a trimmed average of the loss (Rousseeuw & Leroy, 1987; Bhatia et al., 2015; Yang et al., 2018).

Original: Such ideas have been successfully explored for label noise problems (Shen & Sanghavi, 2019).
Revised: Such ideas have been successfully explored for label noise problems (Shen & Sanghavi, 2019).

Original: We will however demonstrate that a simple variant of clipping yields a loss that does possess label noise robustness.
Revised: We will however demonstrate that a simple variant of clipping yields a loss that does possess label noise robustness.

Original:  C OMPOSITE LOSS - BASED GRADIENT CLIPPING
Revised:  C OMPOSITE LOSS - BASED GRADIENT CLIPPING

Original: We now show that noise robustness can be achieved with CL-gradient clipping , a variant wherein for composite losses (e.g., softmax cross-entropy), we perform partial Huberisation of the base loss only .
Revised: We now show that noise robustness can be achieved with CL-gradient clipping , a variant wherein for composite losses (e.g., softmax cross-entropy), we perform partial Huberisation of the base loss only .

Original: 4. F ROM H UBERISATION TO PARTIAL H UBERISATION
Revised: 4. F ROM H UBERISATION TO PARTIAL H UBERISATION

Original: Consider a composite margin loss (cid:96) θ ( x, y ) .
Revised: Consider a composite margin loss (cid:96) θ ( x, y ) .

Original: =
Revised: =

Original: φ ( m θ ( x, y )) , where φ = ϕ ◦ F for some base loss ϕ and invertible link F : ¯ R → [0 , 1] ; e.g., the logistic loss has ϕ ( u ) =
Revised: φ ( m θ ( x, y )) , where φ = ϕ ◦ F for some base loss ϕ and invertible link F : ¯ R → [0 , 1] ; e.g., the logistic loss has ϕ ( u ) =

Original: − log u and F ( z ) =
Revised: − log u and F ( z ) =

Original: σ ( z ) .
Revised: σ ( z ) .

Original: We can interpret p θ ( x, y ) .
Revised: We can interpret p θ ( x, y ) .

Original: =
Revised: =

Original: F ( m θ ( x, y ))
Revised: F ( m θ ( x, y ))

Original: as a probability estimate; e.g., p θ ( x, 1) =
Revised: as a probability estimate; e.g., p θ ( x, 1) =

Original: σ ( m θ ( x, 1)) is the probability of x being positive.
Revised: σ ( m θ ( x, 1)) is the probability of x being positive.

Original: Now, rewriting (cid:96) θ ( x, y )
Revised: Now, rewriting (cid:96) θ ( x, y )

Original: =
Revised: =

Original: ϕ ( p θ ( x, y )) , we may express (3) as:
Revised: ϕ ( p θ ( x, y )) , we may express (3) as:

Original: [Equation]
Revised: [Equation]

Original: L-gradient clipping in (7) was deﬁned as only clipping φ (cid:48)
Revised: L-gradient clipping in (7) was deﬁned as only clipping φ (cid:48)

Original: = ( ϕ ◦ F ) (cid:48) above, which ensures that the resulting Huberised loss is Lipschitz.
Revised: = ( ϕ ◦ F ) (cid:48) above, which ensures that the resulting Huberised loss is Lipschitz.

Original: Typically, however, F is already Lipschitz; e.g., this is the case for the commonly used sigmoid or softmax link.
Revised: Typically, however, F is already Lipschitz; e.g., this is the case for the commonly used sigmoid or softmax link.

Original: This suggests the following composite loss-based gradient clipping ( CL-gradient clipping ), wherein we only clip the derivative for the base loss ϕ :
Revised: This suggests the following composite loss-based gradient clipping ( CL-gradient clipping ), wherein we only clip the derivative for the base loss ϕ :

Original: [Equation]
Revised: [Equation]

Original: As before, CL-gradient clipping corresponds to optimising a new, “partially Huberised” loss.
Revised: As before, CL-gradient clipping corresponds to optimising a new, “partially Huberised” loss.

Original: Lemma 5.
Revised: Lemma 5.

Original: Pick any admissible, composite margin loss φ = ϕ ◦ F , and τ > 0 .
Revised: Pick any admissible, composite margin loss φ = ϕ ◦ F , and τ > 0 .

Original: Then, the clipped gradient in (9) is equivalent to employing a partially Huberised loss ˜ φ τ = ˜
Revised: Then, the clipped gradient in (9) is equivalent to employing a partially Huberised loss ˜ φ τ = ˜

Original: ϕ
Revised: ϕ

Original: τ ◦ F , where
Revised: τ ◦ F , where

Original: [Equation]
Revised: [Equation]

Original: Compared to the Huberised loss (7), the partially Huberised loss only linearises the base loss ϕ , while retaining the link F .
Revised: Compared to the Huberised loss (7), the partially Huberised loss only linearises the base loss ϕ , while retaining the link F .

Original: Consequently, the composite loss ˜
Revised: Consequently, the composite loss ˜

Original: φ τ behaves like the link beyond a certain threshold, and will thus be bounded: indeed, ˜
Revised: φ τ behaves like the link beyond a certain threshold, and will thus be bounded: indeed, ˜

Original: φ τ ( z ) = − τ · F ( z ) − ϕ ∗ ( − τ ) if ϕ (cid:48)
Revised: φ τ ( z ) = − τ · F ( z ) − ϕ ∗ ( − τ ) if ϕ (cid:48)

Original: ( F ( z )) ≤ − τ .
Revised: ( F ( z )) ≤ − τ .

Original: Example 4.1: For the logistic loss, φ ( z )
Revised: Example 4.1: For the logistic loss, φ ( z )

Original: =
Revised: =

Original: ϕ ( F ( z )) with ϕ ( 
Revised: ϕ ( F ( z )) with ϕ ( u ) = − log u and F ( z ) = σ ( z ) ,

Original: [Equation]
Revised: [Equation]

Original: Note that partial Huberisation readily generalises to a multi-class setting.
Revised: Note that partial Huberisation readily generalises to a multi-class setting.

Original: Indeed, suppose we have softmax probability estimates p θ ( x, y )
Revised: Indeed, suppose we have softmax probability estimates p θ ( x, y )

Original: ∝ exp( m θ ( x, y )) .
Revised: ∝ exp( m θ ( x, y )) .

Original: Then, whereas the softmax cross-entropy employs (cid:96) θ ( x, y ) = − log p θ ( x, y ) , our partially Huberised softmax cross-entropy for τ > 1 is
Revised: Then, whereas the softmax cross-entropy employs (cid:96) θ ( x, y ) = − log p θ ( x, y ) , our partially Huberised softmax cross-entropy for τ > 1 is

Original: [Equation]
Revised: [Equation]

Original: 4. C ONSISTENCY AND LABEL NOISE ROBUSTNESS OF PARTIALLY H UBERISED LOSSES
Revised: 4. C ONSISTENCY AND LABEL NOISE ROBUSTNESS OF PARTIALLY H UBERISED LOSSES

Original: Following §3.2, we establish that CL-gradient clipping is always benign from a classiﬁcation perspective, and provided τ is sufﬁciently large, from a probability estimation perspective as well.
Revised: Following §3.2, we establish that CL-gradient clipping is always benign from a classiﬁcation perspective, and provided τ is sufﬁciently large, from a probability estimation perspective as well.

Original: As before, we do this by exploiting the equivalence of CL-gradient clipping to a partially Huberised loss.
Revised: As before, we do this by exploiting the equivalence of CL-gradient clipping to a partially Huberised loss.

Original: Lemma 6.
Revised: Lemma 6.

Original: Pick any admissible composite margin loss φ = ϕ ◦ F and τ > 0 .
Revised: Pick any admissible composite margin loss φ = ϕ ◦ F and τ > 0 .

Original: Then, the loss ˜ φ τ in (10) is classiﬁcation calibrated.
Revised: Then, the loss ˜ φ τ in (10) is classiﬁcation calibrated.

Original: If further τ ≥ − ϕ (cid:48) ( 1 / 2 ) , then ˜ ϕ τ is also proper composite.
Revised: If further τ ≥ − ϕ (cid:48) ( 1 / 2 ) , then ˜ ϕ τ is also proper composite.

Original: We now show that partially Huberised losses have an important advantage over Huberised losses: under symmetric label noise, the optimal solution on the clean distribution cannot be too far away from the optimal solution on the noisy distribution.
Revised: We now show that partially Huberised losses have an important advantage over Huberised losses: under symmetric label noise, the optimal solution on the clean distribution cannot be too far away from the optimal solution on the noisy distribution.

Original: This implies that label noise (such as that considered in Proposition 4) cannot have an excessively deleterious inﬂuence on the loss.
Revised: This implies that label noise (such as that considered in Proposition 4) cannot have an excessively deleterious inﬂuence on the loss.

Original: Proposition 7.
Revised: Proposition 7.

Original: Pick any proper loss ϕ and τ > 0 .
Revised: Pick any proper loss ϕ and τ > 0 .

Original: Let f ∗ be the risk minimiser of ˜ ϕ
Revised: Let f ∗ be the risk minimiser of ˜ ϕ

Original: τ on the clean distribution.
Revised: τ on the clean distribution.

Original: For any non-trivial level of symmetric label noise, let reg τ ( f ∗ ) denote the excess risk of f ∗ with respect to ˜ ϕ
Revised: For any non-trivial level of symmetric label noise, let reg τ ( f ∗ ) denote the excess risk of f ∗ with respect to ˜ ϕ

Original: τ on the noisy distribution.
Revised: τ on the noisy distribution.

Original: Then, there exists C > 0 such that reg τ ( f ∗ ) ≤ C.
Revised: Then, there exists C > 0 such that reg τ ( f ∗ ) ≤ C.

Original: Note that by van Rooyen et al.
Revised: Note that by van Rooyen et al.

Original: (2015, Proposition 4), it is impossible for the above bound to hold with
Revised: (2015, Proposition 4), it is impossible for the above bound to hold with

Original: C = 0 without using a linear loss.
Revised: C = 0 without using a linear loss.

Original: Nonetheless, by virtue of partially Huberised losses being partially linear, we are able to bound the degradation under label corruption.
Revised: Nonetheless, by virtue of partially Huberised losses being partially linear, we are able to bound the degradation under label corruption.

Original: The saturating behaviour of the partially Huberised loss also implies robustness to outliers in feature space; see Appendix E.
Revised: The saturating behaviour of the partially Huberised loss also implies robustness to outliers in feature space; see Appendix D.

Original: 4. R ELATION TO EXISTING WORK
Revised: 4. R ELATION TO EXISTING WORK

Original: The partially Huberised log loss in (10), (11) can be related to a family of losses studied in several works (Ding & Vishwanathan, 2010; Hung et al., 2018; Zhang & Sabuncu, 2018;
Revised: The partially Huberised log loss in (10), (11) can be related to a family of losses studied in several works (Ding & Vishwanathan, 2010; Hung et al., 2018; Zhang & Sabuncu, 2018;

Original:  Amid et al., 2019b;a): for α ∈ (0 , 1] , deﬁne the generalised cross-entropy ϕ α : [0 , 1] → ¯ R + by ϕ α ( u ) .
Revised: Sypherd et al., 2019; Amid et al., 2019b;a): for α ∈ (0 , 1] , deﬁne the generalised cross-entropy ϕ α : [0 , 1] → ¯ R + by ϕ α ( u ) .

Original: = (1 − u α ) /α.
Revised: = (1 − u α ) /α.

Original: See Appendix C.3 for an illustration of ϕ α .
Revised: See Appendix C.3 for an illustration of ϕ α .

Original: There are two similarities between (11) and ϕ α .
Revised: There are two similarities between (11) and ϕ α .

Original: First, both proposals interpolate between the log and linear losses: when α → 0 + , ϕ
Revised: First, both proposals interpolate between the log and linear losses: when α → 0 + , ϕ

Original: α approaches the log loss, and when α = 1 , ϕ
Revised: α approaches the log loss, and when α = 1 , ϕ

Original: α equals the linear loss.
Revised: α equals the linear loss.

Original: Second, both proposals modify the base loss, allowing the link F to be chosen independently.
Revised: Second, both proposals modify the base loss, allowing the link F to be chosen independently.

Original: In particular, one may use the heavy-tailed F of Ding & Vishwanathan (2010); Amid et al. (2019b) in conjunction with our partially Huberised loss.
Revised: In particular, one may use the heavy-tailed F of Ding & Vishwanathan (2010); Amid et al. (2019b) in conjunction with our partially Huberised loss.

Original: One difference between (11) and ϕ α is that the partially Huberised loss is exactly linear for a suitable region; consequently, it is guaranteed to be Lipschitz, unlike ϕ α .
Revised: One difference between (11) and ϕ α is that the partially Huberised loss is exactly linear for a suitable region; consequently, it is guaranteed to be Lipschitz, unlike ϕ α .

Original: This can be understood in terms of the loss gradients: for a class-probability estimate p θ ( x, y ) , let (cid:96) θ ( x, y ) .
Revised: This can be understood in terms of the loss gradients: for a class-probability estimate p θ ( x, y ) , let (cid:96) θ ( x, y ) .

Original: =
Revised: =

Original: ϕ
Revised: ϕ

Original: ( p θ ( x, y )) .
Revised: ( p θ ( x, y )) .

Original: For the generalised cross-entropy, ∇ (cid:96) θ ( x, y )
Revised: For the generalised cross-entropy, ∇ (cid:96) θ ( x, y )

Original: =
Revised: =

Original: − p
Revised: − p

Original: θ ( x, y )
Revised: θ ( x, y )

Original: α − 1 · ∇ p θ ( x, y ) , while for the partially Huberised loss, ∇ (cid:96) θ ( x, y )
Revised: α − 1 · ∇ p θ ( x, y ) , while for the partially Huberised loss, ∇ (cid:96) θ ( x, y )

Original: = − ( p θ ( x, y ) ∨ τ ) − 1 · ∇ p θ ( x, y ) .
Revised: = − ( p θ ( x, y ) ∨ τ ) − 1 · ∇ p θ ( x, y ) .

Original: Both gradients thus take into account whether a sample is “informative”, in the sense of being poorly-predicted ( p θ ( x, y ) ∼ 0 ).
Revised: Both gradients thus take into account whether a sample is “informative”, in the sense of being poorly-predicted ( p θ ( x, y ) ∼ 0 ).

Original: Further, to guard against such samples being the result of label noise, both ensure this inﬂuence is not overwhelming, but in different ways: the generalised cross-entropy softens the inﬂuence, while still allowing it to be unbounded as p θ ( x, y )
Revised: Further, to guard against such samples being the result of label noise, both ensure this inﬂuence is not overwhelming, but in different ways: the generalised cross-entropy softens the inﬂuence, while still allowing it to be unbounded as p θ ( x, y )

Original: → 0 .
Revised: → 0 .

Original: On the other hand, the partially Huberised loss enforces a hard cap of τ − 1 on the inﬂuence.
Revised: On the other hand, the partially Huberised loss enforces a hard cap of τ − 1 on the inﬂuence.

Original: This is to be contrast with a truncated loss also considered in Zhang & Sabuncu (2018), which enforces a hard cap on the loss , thus completely discarding poorly-predicted instances.
Revised: This is to be contrast with a truncated loss also considered in Zhang & Sabuncu (2018), which enforces a hard cap on the loss , thus completely discarding poorly-predicted instances.

Original:  For  a discussion ofother related work, including redescending M-estimators, see Appendix D.
Revised: As shown in Figure 1a, the partially Huberised loss in (10) exhibits saturating behaviour at the extremes. This by itself is hardly new in the context of robust statistics. For example, Tukey’s “biweight” loss replaces the Huber loss’ linear behaviour with hard saturation (Gross & Tukey, 1973), and is a member of the broader family of redescending M-estimators (Andrews et al., 1972; Shevlyakov et al., 2008).

Revised: In a similar spirit, Masnadi-Shirazi & Vasconcelos (2009) proposed the bounded Savage loss ϕ ( v ) =

Revised: ( σ (2 v ) − 1) 2 , while Yang et al. (2010); Menon & Williamson (2018) theoretically studied loss clipping.

Revised: Robust versions of logistic regression derived by variants of the

Revised: Huber loss have been explored, e.g., Pregibon (1982, Section 4.1), Copas (1988, Section 3.1), (Feng et al., 2014).

Revised: The distinction of the above proposals to our work is two-fold.

Revised: First, we explicitly consider the classiﬁcation setting, wherein we show consistency and robustness to label noise for the partially

Revised: Huberised loss.

Revised: Second, our proposal explicitly highlights the connection to gradient clipping.

Revised: Finally, while our focus has been on label noise, there are other notions of “robustness”.

Revised: Indeed, several works have proposed means of making gradient descent robust to heavy-tailed distributions

Revised: [Table]

Revised: [Table]

Revised: or outliers (Brownlees et al., 2015; Hsu & Sabato, 2016; Lugosi & Mendelson, 2016;

Revised: Lecué et al., 2018; Prasad et al., 2018; Holland & Ikeda, 2019), typically by passing the gradient through a robust operator (Tukey, 1960;

Revised: Hampel et al., 1986; Rousseeuw & Leroy, 1987; Lerasle & Oliveira, 2011;

Revised: Catoni, 2012; Minsker, 2015).

Revised: By contrast, we analyse the (simpler) gradient clipping approach, showing that while it is inadequate to cope with label noise, a simple modiﬁcation rectiﬁes this.

Revised: Since our proposal only modiﬁes the underlying loss, it may be combined with any of these methods.

Original: 4. D ISCUSSION AND IMPLICATIONS
Revised: 4. D ISCUSSION AND IMPLICATIONS

Original: Table 1 summarises our results, highlighting the perspective of gradient clipping as equivalently modifying the loss.
Revised: Table 1 summarises our results, highlighting the perspective of gradient clipping as equivalently modifying the loss.

Original: Before proceeding, we make some qualifying comments.
Revised: Before proceeding, we make some qualifying comments.

Original: First, our analysis has assumed symmetric label noise.
Revised: First, our analysis has assumed symmetric label noise.

Original: Often, one encounters asymmetric or instance-dependent noise (Menon et al., 2018).
Revised: Often, one encounters asymmetric or instance-dependent noise (Menon et al., 2018).

Original: While corresponding guarantees for the linear loss may be ported over to the partially Huberised loss, they require stronger distributional assumptions (Ghosh et al., 2015).
Revised: While corresponding guarantees for the linear loss may be ported over to the partially Huberised loss, they require stronger distributional assumptions (Ghosh et al., 2015).

Original: Second, Proposition 4 exhibits a speciﬁc distribution which defeats the Huberised loss under linear models.
Revised: Second, Proposition 4 exhibits a speciﬁc distribution which defeats the Huberised loss under linear models.

Original: In practice, distributions may be more benign (Patrini et al., 2016), and models are often nonlinear, meaning that Huberised losses (and thus gradient clipping) are thus unlikely to succumb as extremely to label noise as Proposition 4 suggests.
Revised: In practice, distributions may be more benign (Patrini et al., 2016), and models are often nonlinear, meaning that Huberised losses (and thus gradient clipping) are thus unlikely to succumb as extremely to label noise as Proposition 4 suggests.

Original: The aim of §4 is however to establish that a simple modiﬁcation of clipping avoids worst-case degradation, without adding signiﬁcant complexity.[Table]
Revised: The aim of §4 is however to establish that a simple modiﬁcation of clipping avoids worst-case degradation, without adding signiﬁcant complexity.

Original: [Table] [Table] [Figure] Third, for minibatch size N > 1 , the effect of clipping is not a simple loss modiﬁcation, since the loss gradients for each sample will be modiﬁed by the entire minibatch loss gradient norm (cid:107) g
Revised: Third, for minibatch size N > 1 , the effect of clipping is not a simple loss modiﬁcation, since the loss gradients for each sample will be modiﬁed by the entire minibatch loss gradient norm (cid:107) g

Original: ( θ ) (cid:107) 2 .
Revised: ( θ ) (cid:107) 2 .

Original: Since this minibatch is randomly drawn, one cannot mimic gradient clipping by a simple modiﬁcation of the loss function.
Revised: Since this minibatch is randomly drawn, one cannot mimic gradient clipping by a simple modiﬁcation of the loss function.

Original: However, the results for N = 1 sufﬁce to establish that gradient clipping cannot in general endow robustness.
Revised: However, the results for N = 1 sufﬁce to establish that gradient clipping cannot in general endow robustness.

Original: One may use our partially Huberised loss in conjunction with minibatch gradient clipping, to potentially obtain both robustness and optimisation beneﬁts.
Revised: One may use our partially Huberised loss in conjunction with minibatch gradient clipping, to potentially obtain both robustness and optimisation beneﬁts.

Original: Finally, partially Huberised losses such as (12) require setting a hyperparameter τ (e.g., by crossvalidation), similar to α in the generalised cross-entropy per §4.3.
Revised: Finally, partially Huberised losses such as (12) require setting a hyperparameter τ (e.g., by crossvalidation), similar to α in the generalised cross-entropy per §4.3.

Original: Intuitively, the optimal τ trades off the noise-robustness of the linear loss, and the gradient informativeness of the base loss (per the discussion in §4.3).
Revised: Intuitively, the optimal τ trades off the noise-robustness of the linear loss, and the gradient informativeness of the base loss (per the discussion in §4.3).

Original: Setting τ to be large tacitly assumes that one’s samples are largely noise-free.
Revised: Setting τ to be large tacitly assumes that one’s samples are largely noise-free.

Original:  E XPERIMENTAL ILLUSTRATION
Revised:  E XPERIMENTAL ILLUSTRATION

Original: We now present experiments illustrating that: (a) we may exhibit label noise scenarios that defeat a
Revised: We now present experiments illustrating that: (a) we may exhibit label noise scenarios that defeat a

Original: Huberised but not partially Huberised loss, conﬁrming Propositions 4, 7, and (b) partially Huberised versions of existing losses perform well on real-world datasets subject to label noise.
Revised: Huberised but not partially Huberised loss, conﬁrming Propositions 4, 7, and (b) partially Huberised versions of existing losses perform well on real-world datasets subject to label noise.

Original: Synthetic datasets .
Revised: Synthetic datasets .

Original: Our ﬁrst experiments involve two synthetic datasets, which control for confounding factors.
Revised: Our ﬁrst experiments involve two synthetic datasets, which control for confounding factors.

Original: We begin with a setting from Long & Servedio (2010), comprising a 2D linearly separable distribution.
Revised: We begin with a setting from Long & Servedio (2010), comprising a 2D linearly separable distribution.

Original: (See Appendix F.1 for an illustration.)
Revised: (See Appendix E.1 for an illustration.)

Original: We draw N = 1 , 000 random samples from this distribution, and ﬂip each label with ρ = 45% probability.
Revised: We draw N = 1 , 000 random samples from this distribution, and ﬂip each label with ρ = 45% probability.

Original: We train a linear classiﬁer to minimise one of several losses, and evaluate the classiﬁer’s accuracy on 500 clean test samples. 
Revised: We train a linear classiﬁer to minimise one of several losses, and evaluate the classiﬁer’s accuracy on 500 clean test samples. [Figure] cc [Figure]

Revised: [Figure]

Original: We compare the logistic loss with its Huberised and partially Huberised versions, using τ = 1 and τ = 2 respectively.
Revised: [Table] [Figure] [Table] [Figure]

Revised: [Table]

Revised: We compare the logistic loss with its Huberised and partially Huberised versions, using τ = 1 and τ = 2 respectively.

Original: Figure 2a presents the results over 500 independent trials.
Revised: Figure 2a presents the results over 500 independent trials.

Original: The logistic loss and its Huberised counterpart suffer signiﬁcantly under noise, while the partially Huberised loss often achieves perfect near-discrimination.
Revised: The logistic loss and its Huberised counterpart suffer signiﬁcantly under noise, while the partially Huberised loss often achieves perfect near-discrimination.

Original: This conﬁrms that in the worst case, L-gradient clipping may succumb to noise, while CL-gradient clipping performs well in the same scenario.
Revised: This conﬁrms that in the worst case, L-gradient clipping may succumb to noise, while CL-gradient clipping performs well in the same scenario.

Original: We next consider a 1D setting based on Ding (2013, Section 3.2.3), comprising 10 , 000 linearly separable “inliers” and 50 “outliers”.
Revised: We next consider a 1D setting based on Ding (2013, Section 3.2.3), comprising 10 , 000 linearly separable “inliers” and 50 “outliers”.

Original: Assuming the use of a linear model parameterised by scalar[Table] θ ∈ R , we plot the empirical risk of the samples with and without the outlier observations as θ is varied.
Revised: Assuming the use of a linear model parameterised by scalar θ ∈ R , we plot the empirical risk of the samples with and without the outlier observations as θ is varied.

Original: Figure 2b shows that the logistic loss and its Huberised variant are strongly affected by the outliers: their optimal solution goes from θ ∗ = + ∞ to θ ∗ = 0 .
Revised: Figure 2b shows that the logistic loss and its Huberised variant are strongly affected by the outliers: their optimal solution goes from θ ∗ = + ∞ to θ ∗ = 0 .

Original: However, the partially Huberised loss is largely immune to the outliers.
Revised: However, the partially Huberised loss is largely immune to the outliers.

Original: Appendix F contains additional synthetic experiments.
Revised: Appendix E contains additional details on this experiment.

Original: Real-world datasets .
Revised: Real-world datasets .

Original: We now demonstrate that partially Huberised losses perform well with deep neural networks trained on MNIST, CIFAR-10 and CIFAR-100 (Krizhevsky & Hinton, 2009).
Revised: We now demonstrate that partially Huberised losses perform well with deep neural networks trained on MNIST, CIFAR-10 and CIFAR-100 (Krizhevsky & Hinton, 2009).

Original: For
Revised: For

Original: MNIST, we train a LeNet (Lecun et al., 1998) using Adam with batch size N = 32 , and weight decay of 10 − 3 .
Revised: MNIST, we train a LeNet (Lecun et al., 1998) using Adam with batch size N = 32 , and weight decay of 10 − 3 .

Original: For CIFAR-10 and CIFAR-100, we train a ResNet-

Revised: For CIFAR-10 and CIFAR-100, we train a ResNet-

Original: (He et al., 2016) using SGD with momentum 0 .
Revised: (He et al., 2016) using SGD with momentum 0 .

Original: 1 , weight decay of 5 × 10 − 3 , batch normalisation, and N = 64 , 128 respectively.
Revised: 1 , weight decay of 5 × 10 − 3 , batch normalisation, and N = 64 , 128 respectively.

Original: For each dataset, we corrupt the training labels with symmetric noise at ﬂip probability ρ ∈ { 0 . 0 , 0 . 2 , 0 . 4 , 0 . 6 } .
Revised: For each dataset, we corrupt the training labels with symmetric noise at ﬂip probability ρ ∈ { 0 . 0 , 0 . 2 , 0 . 4 , 0 . 6 } .

Original: We compare the test set accuracy of various losses combined with a softmax link.
Revised: We compare the test set accuracy of various losses combined with a softmax link.

Original: Our baseline is the cross-entropy loss (CE).
Revised: Our baseline is the cross-entropy loss (CE).

Original: As representative noise-robust losses, we consider the linear or unhinged loss (van Rooyen et al., 2015; Ghosh et al., 2017), and the generalised cross-entropy (GCE) with α = 0 . 7 , following Zhang & Sabuncu (2018).
Revised: As representative noise-robust losses, we consider the linear or unhinged loss (van Rooyen et al., 2015; Ghosh et al., 2017), and the generalised cross-entropy (GCE) with α = 0 . 7 , following Zhang & Sabuncu (2018).

Original: We additionally assess global gradient clipping (with τ = 0 . 1 ) of the CE, which per §3 is akin to a Huberised loss.
Revised: We additionally assess global gradient clipping (with τ = 0 . 1 ) of the CE, which per §3 is akin to a Huberised loss.

Original: We apply our partial Huberisation of (10) to the CE (12) (“PHuber-CE”), and the GCE (“PHuberGCE”).
Revised: We apply our partial Huberisation of (10) to the CE (12) (“PHuber-CE”), and the GCE (“PHuberGCE”).

Original: The latter highlights that partial Huberisation is not tied to the cross-entropy, and is applicable even on top of existing noise-robust losses.
Revised: The latter highlights that partial Huberisation is not tied to the cross-entropy, and is applicable even on top of existing noise-robust losses.

Original: Recall that partial Huberisation offers a choice of tuning parameter τ , similar to the α parameter in GCE, and noise-rate estimates in loss-correction techniques more generally.
Revised: Recall that partial Huberisation offers a choice of tuning parameter τ , similar to the α parameter in GCE, and noise-rate estimates in loss-correction techniques more generally.

Original: For each dataset, we pick τ ∈ { 2 , 10 } (equivalently corresponding to probability thresholds 0.5 and 0.1 respectively)
Revised: For each dataset, we pick τ ∈ { 2 , 10 } (equivalently corresponding to probability thresholds 0.5 and 0.1 respectively)

Original: so as to maximise accuracy on a validation set of noisy samples with the maximal noise rate ρ = 0 .
Revised: so as to maximise accuracy on a validation set of noisy samples with the maximal noise rate ρ = 0 .

Original: 6 ; the chosen value of τ was then used for each noise level.
Revised: 6 ; the chosen value of τ was then used for each noise level.

Original: Tuning τ separately for each setting of the noise rate ρ can be expected to help performance, at the expense of increased computational cost.
Revised: Tuning τ separately for each setting of the noise rate ρ can be expected to help performance, at the expense of increased computational cost.

Original: Recall also that as τ → 1 , partial Huberisation mimics using the base loss, while as τ → + ∞ , partial Huberisation mimics using the linear loss; our hypothesis is that an intermediate τ can attain a suitable balance between noise robustness, and gradient informativeness.
Revised: Recall also that as τ → 1 , partial Huberisation mimics using the base loss, while as τ → + ∞ , partial Huberisation mimics using the linear loss; our hypothesis is that an intermediate τ can attain a suitable balance between noise robustness, and gradient informativeness.

Original: Table 2 shows that in the noise-free case ( ρ = 0 . 0 ), all methods perform comparably.
Revised: Table 2 shows that in the noise-free case ( ρ = 0 . 0 ), all methods perform comparably.

Original: However, when injecting noise, accuracy for the CE degrades dramatically.
Revised: However, when injecting noise, accuracy for the CE degrades dramatically.

Original: Further, gradient clipping sometimes offers improvements under high noise; however, the performance is far inferior to other losses, which is in keeping with their robustness guarantees.
Revised: Further, gradient clipping sometimes offers improvements under high noise; however, the performance is far inferior to other losses, which is in keeping with their robustness guarantees.

Original: Indeed, the linear loss, which is provably robust to symmetric noise, generally performs well even when ρ = 0 .
Revised: Indeed, the linear loss, which is provably robust to symmetric noise, generally performs well even when ρ = 0 .

Original: However, optimisation under this loss is more challenging, since the gradient does not account for instances’ importance (per §4.3).
Revised: However, optimisation under this loss is more challenging, since the gradient does not account for instances’ importance (per §4.3).

Original: This is particularly reﬂected on the CIFAR-100 dataset, where this loss suffers to learn even under no noise.
Revised: This is particularly reﬂected on the CIFAR-100 dataset, where this loss suffers to learn even under no noise.

Original: The GCE and partially Huberised losses do not suffer from this issue, even at high noise levels.
Revised: The GCE and partially Huberised losses do not suffer from this issue, even at high noise levels.

Original: Generally, the partially Huberised losses are competitive with or improve upon the counterpart losses they build upon.
Revised: Generally, the partially Huberised losses are competitive with or improve upon the counterpart losses they build upon.

Original: In particular, the partially Huberised CE performs much better than the CE under high noise, while the partially Huberised GCE slightly bumps up the GCE numbers on CIFAR-100.
Revised: In particular, the partially Huberised CE performs much better than the CE under high noise, while the partially Huberised GCE slightly bumps up the GCE numbers on CIFAR-100.

Original: This indicates that partially Huberised may be useful in combining with generic base losses to cope with noise.
Revised: This indicates that partially Huberised may be useful in combining with generic base losses to cope with noise.

Original: We reiterate here that our partially Huberised loss may be used in conjunction with other ideas, e.g., pruning (Zhang & Sabuncu, 2018), consensus (Malach & Shalev-Shwartz, 2017; Han et al., 2018), or abstention (Thulasidasan et al., 2019).
Revised: We reiterate here that our partially Huberised loss may be used in conjunction with other ideas, e.g., pruning (Zhang & Sabuncu, 2018), consensus (Malach & Shalev-Shwartz, 2017; Han et al., 2018), or abstention (Thulasidasan et al., 2019).

Original: We leave such exploration for future work.
Revised: We leave such exploration for future work.

Original:  C ONCLUSION AND FUTURE WORK
Revised:  C ONCLUSION AND FUTURE WORK

Original: We established that gradient clipping by itself does not sufﬁce to endow even simple models with label noise robustness; however, a simple variant resolves this issue.
Revised: We established that gradient clipping by itself does not sufﬁce to endow even simple models with label noise robustness; however, a simple variant resolves this issue.

Original: Experiments conﬁrm that our composite loss-based gradient clipping performs well on datasets corrupted with label noise.
Revised: Experiments conﬁrm that our composite loss-based gradient clipping performs well on datasets corrupted with label noise.

Original: One interesting direction for future work is to analyse the behaviour of gradient-clipping inspired losses for the more general problem of distributionally robust learning (Shaﬁeezadeh-Abadeh et al., 2015;
Revised: One interesting direction for future work is to analyse the behaviour of gradient-clipping inspired losses for the more general problem of distributionally robust learning (Shaﬁeezadeh-Abadeh et al., 2015;

Original: Namkoong & Duchi, 2016; 2017; Hu et al., 2018; Sinha et al., 2018; Duchi & Namkoong, 2019).
Revised: Namkoong & Duchi, 2016; 2017; Hu et al., 2018; Sinha et al.

Original: A P ROOFS OF RESULTS
Revised: A P ROOFS OF RESULTS

Original: Proof of Lemma 1.
Revised: Proof of Lemma 1.

Original: Following Equation
Revised: Following Equation

Original: [Equation]
Revised: [Equation]

Original: Similarly, for the linear loss function (cid:96) lin ( x, y ; θ ) .
Revised: Similarly, for the linear loss function (cid:96) lin ( x, y ; θ ) .

Original: =
Revised: =

Original: − m θ ( x, y )
Revised: − m θ ( x, y )

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: To compute the normalised gradient
Revised: To compute the normalised gradient

Original: [Equation]
Revised: [Equation]

Original: we need
Revised: we need

Original: [Equation]
Revised: [Equation]

Original: Thus,
Revised: Thus,

Original: [Equation]
Revised: [Equation]

Original: since φ (cid:48) ( z ) < 0 by assumption that φ is admissible
Revised: since φ (cid:48) ( z ) < 0 by assumption that φ is admissible

Original: [Equation]
Revised: [Equation]

Original: and thus decreasing.
Revised: and thus decreasing.

Original: Consequently, if N = 1 , 
Revised: Consequently, if N = 1 , ¯

Original: [Equation]
Revised: [Equation]

Original: clip
Revised: clip

Original: [Equation]
Revised: [Equation]

Original: else

Original: [Equation]

Original: Assuming a linear scorer s ( x ; θ )
Revised: Assuming a linear scorer s ( x ; θ )

Original: = θ T x , the score gradient ∇ s ( x ; θ ) =
Revised: = θ T x , the score gradient ∇ s ( x ; θ ) =

Original: x , which is independent of θ .
Revised: x , which is independent of θ .

Original: The clipped gradient thus corresponds to the gradient under a “Huberised” loss function
Revised: The clipped gradient thus corresponds to the gradient under a “Huberised” loss function

Original: [Equation]
Revised: [Equation]

Original: else

Original: [Equation]

Original: Proof of Lemma 2.
Revised: Proof of Lemma 2.

Original: Since φ is strictly convex and decreasing, it must be strictly decreasing.
Revised: Since φ is strictly convex and decreasing, it must be strictly decreasing.

Original: We have
Revised: We have

Original: [Equation]

Original: else

Original: [Equation]
Revised: [Equation]

Original: φ [Equation] z ) else since φ (cid:48) is strictly decreasing=
Revised: else since φ (cid:48) is strictly decreasing

Original: [Equation]
Revised: [Equation]

Original: z ) else, where ( φ (cid:48) ) − 1 exists since φ (cid:48) is strictly inceasing by deﬁnition of strict convexity.
Revised: else, where ( φ (cid:48) ) − 1 exists since φ (cid:48) is strictly inceasing by deﬁnition of strict convexity.

Original: Now, by deﬁnition, the Huberised loss is
Revised: Now, by deﬁnition, the Huberised loss is

Original: [Equation]
Revised: [Equation]

Original: ifφ
Revised: if

Original: [Equation]
Revised: [Equation]

Original: z
Revised: φ

Original: [Equation]
Revised: [Equation]

Original: τ φ ( z ) else, and so¯
Revised: z ) else, and so

Original: [Equation]
Revised: [Equation]

Original: if φ (cid:48) ( z ) ≤

Original: [Equation]

Original: τ φ (cid:48) ( z ) else, which exactly equals clip τ ( φ (cid:48) ( z )) .
Revised: else, which exactly equals clip τ ( φ (cid:48) ( z )) .

Original: We remark here that ¯ φ τ is continuous, since for any convex conjugate,
Revised: We remark here that ¯ φ τ is continuous, since for any convex conjugate,

Original: [Equation]
Revised: [Equation]

Original: Plugging in u = τ , at the intersection point z 0 .
Revised: Plugging in u = τ , at the intersection point z 0 .

Original: = ( φ ) (cid:48)− 1 ( − τ ) of the two pieces of the function,
Revised: = ( φ ) (cid:48)− 1 ( − τ ) of the two pieces of the function,

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma 3.
Revised: Proof of Lemma 3.

Original: For admissible φ , the Huberised loss ¯ φ τ of (8) is trivially convex, differentiable everywhere, and decreasing.
Revised: For admissible φ , the Huberised loss ¯ φ τ of (8) is trivially convex, differentiable everywhere, and decreasing.

Original: In particular, we must have ¯ φ (cid:48) τ (0) < 0 , and so ¯ φ τ must be classiﬁcation calibrated by Bartlett et al.
Revised: In particular, we must have ¯ φ (cid:48) τ (0) < 0 , and so ¯ φ τ must be classiﬁcation calibrated by Bartlett et al.

Original: (2006, Theorem 2).
Revised: (2006, Theorem 2).

Original: As an illustration, Figure 3 shows the minimiser of the conditional risk,
Revised: As an illustration, Figure 3 shows the minimiser of the conditional risk,

Original: [Equation]
Revised: [Equation]

Original: Note that this quantity must be non-negative if and only if η > 1 / 2 for a loss to be classiﬁcation calibrated.
Revised: Note that this quantity must be non-negative if and only if η > 1 / 2 for a loss to be classiﬁcation calibrated.

Original: This is easily veriﬁed to be true for the Huberised logistic loss, regardless of τ .
Revised: This is easily veriﬁed to be true for the Huberised logistic loss, regardless of τ .

Original: [Figure]
Revised: [Figure]

Original: Now suppose that φ is a proper composite loss.
Revised: Now suppose that φ is a proper composite loss.

Original: To show that ¯ φ τ is proper composite, it sufﬁces by Williamson et al.
Revised: To show that ¯ φ τ is proper composite, it sufﬁces by Williamson et al.

Original: (2016, Corollary 12) to show that ¯ φ (cid:48)
Revised: (2016, Corollary 12) to show that ¯ φ (cid:48)

Original: τ ( z ) ¯ φ (cid:48)
Revised: τ ( z [Equation] ¯ φ [Equation]

Original: τ ( − z ) is strictly monotone and continuous.
Revised: τ ( − z ) is strictly monotone and continuous.

Original: Continuity is immediate; for monotonicity, observe that by deﬁnition,
Revised: Continuity is immediate; for monotonicity, observe that by deﬁnition,

Original: [Equation]
Revised: [Equation]

Original:  else.
Revised: ) else.

Original: For brevity, let z 0 .
Revised: For brevity, let z 0 .

Original: = ( φ (cid:48) )
Revised: = ( φ (cid:48) )

Original: − 1 ( − τ ) , where we take z 0 =
Revised: − 1 ( − τ ) , where we take z 0 =

Original: −∞ if − τ / ∈ ran( φ (cid:48) ) .
Revised: −∞ if − τ / ∈ ran( φ (cid:48) ) .

Original: Then, ¯ φ (cid:48) τ ( z ) =
Revised: Then, ¯ φ (cid:48) τ ( z ) =

Original: (cid:26)
Revised: (cid:26)

Original: [Equation]
Revised: [Equation]

Original: else
Revised: else

Original: [Equation]
Revised: [Equation]

Original: else
Revised: else.

Original: [Equation]

Original: Thus, the quantity of interest is
Revised: Thus, the quantity of interest is

Original: [Equation]
Revised: [Equation]

Original: Since φ is strictly convex, φ (cid:48) is strictly decreasing and thus invertible.
Revised: Since φ is strictly convex, φ (cid:48) is strictly decreasing and thus invertible.

Original: Thus, the ratio ¯ φ (cid:48)
Revised: Thus, the ratio ¯ φ (cid:48)

Original: τ ( z ) ¯ φ (cid:48)
Revised: τ ( z ) ¯ φ (cid:48)

Original: τ ( − z ) is invertible, provided z 0 ≤ 0 , i.e., τ ≥ − φ (cid:48) (0) .
Revised: τ ( − z ) is invertible, provided z 0 ≤ 0 , i.e., τ ≥ − φ (cid:48) (0) .

Original: Consequently, ¯ φ τ is proper composite when τ ≥ − φ (cid:48) (0) .
Revised: Consequently, ¯ φ τ is proper composite when τ ≥ − φ (cid:48) (0) .

Original: To intuit the need for the restriction on τ , observe that by (Reid & Williamson, 2010,
Revised: To intuit the need for the restriction on τ , observe that by (Reid & Williamson, 2010,

Original: Corollary 12) the tentative link function for the loss is
Revised: Corollary 12) the tentative link function for the loss is

Original: [Equation]
Revised: [Equation]

Original: When τ < − φ (cid:48) (0) , the above is seen to be non-invertible .
Revised: When τ < − φ (cid:48) (0) , the above is seen to be non-invertible .

Original: See also Appendix C for an illustration of this link function for the logistic loss.
Revised: See also Appendix C for an illustration of this link function for the logistic loss.

Original: Proof of Proposition 4.
Revised: Proof of Proposition 4.

Original: In order to apply Long & Servedio (2010, Theorem 2), we simply need to check that the loss ¯ φ τ is a convex potential in the sense of Long & Servedio (2010, Deﬁnition 1).
Revised: In order to apply Long & Servedio (2010, Theorem 2), we simply need to check that the loss ¯ φ τ is a convex potential in the sense of Long & Servedio (2010, Deﬁnition 1).

Original: This requires that ¯ φ τ is convex, non-increasing, continuously differentiable, and asymptotes to zero (or equally, is bounded from below).
Revised: This requires that ¯ φ τ is convex, non-increasing, continuously differentiable, and asymptotes to zero (or equally, is bounded from below).

Original: Each of these is satisﬁed by assumption of φ being admissible.
Revised: Each of these is satisﬁed by assumption of φ being admissible.

Original: Proof of Lemma 5.
Revised: Proof of Lemma 5.

Original: By Lemma 2, we may write clip τ ( ϕ (cid:48) ( u )) as the derivative of a partially Huberised base loss ˜ ϕ
Revised: By Lemma 2, we may write clip τ ( ϕ (cid:48) ( u )) as the derivative of a partially Huberised base loss ˜ ϕ

Original: τ given by
Revised: τ given by

Original: [Equation]
Revised: [Equation]

Original: else
Revised: else

Original: This induces a composite margin loss ˜ φ τ . =
Revised: This induces a composite margin loss ˜ φ τ . =

Original: ˜ ϕ
Revised: ˜ ϕ

Original: τ ◦ F .
Revised: τ ◦ F .

Original: Now let p θ ( x, y )
Revised: Now let p θ ( x, y )

Original: =
Revised: =

Original: F ( m θ ( x, y )) , and deﬁne ˜ (cid:96) τ ( x, y ; θ ) .
Revised: F ( m θ ( x, y )) , and deﬁne ˜ (cid:96) τ ( x, y ; θ ) .

Original: =
Revised: =

Original: ˜
Revised: ˜

Original: φ τ ( m θ ( x, y )) .
Revised: φ τ ( m θ ( x, y )) .

Original: The gradient under this loss is
Revised: The gradient under this loss is

Original: [Equation]
Revised: [Equation]

Original: Thus, CL-gradient clipping is equivalent to using the loss ˜ (cid:96) τ .
Revised: Thus, CL-gradient clipping is equivalent to using the loss ˜ (cid:96) τ .

Original: Proof of Lemma 6.
Revised: Proof of Lemma 6.

Original: We proceed in a similar manner to Lemma 3: to show that the loss is proper, we must establish that ˜ φ (cid:48)
Revised: We proceed in a similar manner to Lemma 3: to show that the loss is proper, we must establish that ˜ φ (cid:48)

Original: τ ( z ) ˜
Revised: τ ( z ) ˜

Original: φ (cid:48)
Revised: φ (cid:48)

Revised: [Equation]

Original: τ ( − z ) is invertible.
Revised: ( − z ) is invertible.

Original: By Reid & Williamson (2010, Corollary 14), for the margin loss φ to be proper composite with link F , it must be true that F satisﬁes the symmetry condition
Revised: By Reid & Williamson (2010, Corollary 14), for the margin loss φ to be proper composite with link F , it must be true that F satisﬁes the symmetry condition

Original: F ( − z )
Revised: F ( − z )

Original: = 1 − F ( z ) .
Revised: = 1 − F ( z ) .

Original: We thus have
Revised: We thus have

Original: [Equation]
Revised: [Equation]

Original: Consequently,
Revised: Consequently,

Original: [Equation]
Revised: [Equation]

Original: Since F is invertible by assumption, the above is invertible if and only if
Revised: Since F is invertible by assumption, the above is invertible if and only if

Original: [Equation]
Revised: [Equation]

Original: invertible.
Revised: invertible.

Original: Observe that
Revised: Observe that

Original: [Equation]
Revised: [Equation]

Original: else
Revised: else

Original: [Equation]
Revised: [Equation]

Original: Let u 0 .
Revised: Let u 0 .

Original: =
Revised: =

Original: ( ϕ (cid:48) )
Revised: ( ϕ (cid:48) )

Original: − 1 ( − τ ) for brevity.
Revised: − 1 ( − τ ) for brevity.

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: This quantity is invertible provided u 0 ≤ 12 , i.e., τ ≥ − ϕ (cid:48) ( 1 / 2 ) .
Revised: This quantity is invertible provided u 0 ≤ 12 , i.e., τ ≥ − ϕ (cid:48) ( 1 / 2 ) .

Original: A subtlety, however, is that the above does not necessarily span the entire range of [0 , + ∞ ] ; consequently, ˜ ϕ
Revised: A subtlety, however, is that the above does not necessarily span the entire range of [0 , + ∞ ] ; consequently, ˜ ϕ

Original: τ itself is proper composite, with a link function of its own.
Revised: τ itself is proper composite, with a link function of its own.

Original: Even when τ is small, one may verify that the loss ˜ φ τ is nonetheless classiﬁcation calibrated: this is because for any η ∈ [0 , 1] , the minimiser z ∗ ( η ) of the conditional risk must satisfy the stationarity condition
Revised: Even when τ is small, one may verify that the loss ˜ φ τ is nonetheless classiﬁcation calibrated: this is because for any η ∈ [0 , 1] , the minimiser z ∗ ( η ) of the conditional risk must satisfy the stationarity condition

Original: [Equation]
Revised: [Equation]

Original: We thus need to ﬁnd a suitable z ∗ such that the left hand side equates to a given constant.
Revised: We thus need to ﬁnd a suitable z ∗ such that the left hand side equates to a given constant.

Original: Now, for any C (cid:54) = 1 there is a unique u such that
Revised: Now, for any C (cid:54) = 1 there is a unique u such that

Original: [Equation]
Revised: [Equation]

Original: = C .
Revised: = C .

Original: Thus, provided η (cid:54) = 12 , there is a unique z ∗ =
Revised: Thus, provided η (cid:54) = 12 , there is a unique z ∗ =

Original: F − 1 ( u ∗ ) such that
Revised: F − 1 ( u ∗ ) such that

Original: [Equation]
Revised: [Equation]

Original: One may verify that this z
Revised: One may verify that this z

Original: [Equation]
Revised: [Equation]

Original: > {+
+}
Revised: > {+
+}

Original: [Equation]

Original: ⇐⇒ 
Revised: ⇐⇒ η

Original: [Equation]
Revised: [Equation]

Original: 12 ; for example, see Figure 4, which visualises the risk minimiser for various values of τ .
Revised: 12 ; for example, see Figure 4, which visualises the risk minimiser for various values of τ .

Original: Thus, the sign of the minimising score conveys whether or not the positive class-probability is dominant; thus, the loss is classiﬁcation calibrated.
Revised: Thus, the sign of the minimising score conveys whether or not the positive class-probability is dominant; thus, the loss is classiﬁcation calibrated.

Original: [Figure]
Revised: [Figure]

Original: Proof of Proposition 7.
Revised: Proof of Proposition 7.

Original: Let R τ
Revised: Let R τ

Original: ( f ) denote the risk on the clean distribution of a predictor f with respect to the partially Huberised loss ˜ ϕ
Revised: ( f ) denote the risk on the clean distribution of a predictor f with respect to the partially Huberised loss ˜ ϕ

Original: τ with parameter τ .
Revised: τ with parameter τ .

Original: Similarly, let ¯
Revised: Similarly, let ¯

Original: R
Revised: R

Original: τ
Revised: τ

Original: ( f ) denote the risk on the noisy distribution.
Revised: ( f ) denote the risk on the noisy distribution.

Original: Following Ghosh et al.
Revised: Following Ghosh et al.

Original: ; van Rooyen et al. (2015), we have
Revised: ; van Rooyen et al. (2015), we have

Original: [Equation]
Revised: [Equation]

Original: That is, the risk on the noisy distribution equals a scaled version of the risk on the clean distribution, plus an additional term.
Revised: That is, the risk on the noisy distribution equals a scaled version of the risk on the clean distribution, plus an additional term.

Original: This term is a constant independent of f if and only if ˜ ϕ τ satisﬁes the symmetry condition of Ghosh et al.
Revised: This term is a constant independent of f if and only if ˜ ϕ τ satisﬁes the symmetry condition of Ghosh et al.

Original: (2015) , namely,
Revised: (2015) , namely,

Original: ˜ ϕ
Revised: ˜ ϕ

Original: τ ( u ) + ˜ ϕ
Revised: τ ( u ) + ˜ ϕ

Original: τ (1 − u ) =
Revised: τ (1 − u ) =

Original: C for some constant C .
Revised: C for some constant C .

Original: Even when the symmetry condition does not hold, one may nonetheless aim to bound this additional term as follows.
Revised: Even when the symmetry condition does not hold, one may nonetheless aim to bound this additional term as follows.

Original: For simplicity, we restrict attention here to ϕ being the log or cross-entropy loss ϕ
Revised: For simplicity, we restrict attention here to ϕ being the log or cross-entropy loss ϕ

Original: ( u ) =
Revised: ( u ) =

Original: − log u .
Revised: − log u .

Original: By deﬁnition, for any f ∈ (0 , 1) ,
Revised: By deﬁnition, for any f ∈ (0 , 1) ,

Original: [Equation]
Revised: [Equation]

Original: Evidently, all piecewise functions involved are bounded on the respective intervals.
Revised: Evidently, all piecewise functions involved are bounded on the respective intervals.

Original: For example,
Revised: For example,

Original: [Equation]
Revised: [Equation]

Original: By taking the maximum of each of the quantities on the right hand side – which are constants depending on τ – we may thus ﬁnd constants C 1 , C 2 such that
Revised: By taking the maximum of each of the quantities on the right hand side – which are constants depending on τ – we may thus ﬁnd constants C 1 , C 2 such that

Original: [Equation]
Revised: [Equation]

Original: Now let f ∗ denote the minimiser of the clean risk R τ ( f ) , and ¯ f ∗ the minimiser of the noisy risk ¯
Revised: Now let f ∗ denote the minimiser of the clean risk R τ ( f ) , and ¯ f ∗ the minimiser of the noisy risk ¯

Original: R τ ( f ) .
Revised: R τ ( f ) .

Original: Then, using each of the above inequalities,
Revised: Then, using each of the above inequalities,

Original: [Equation]
Revised: [Equation]

Original: where the last inequality is because R τ ( f ∗ ) ≤ R τ ( ¯ f ∗ ) by deﬁnition of f ∗ .
Revised: where the last inequality is because R τ ( f ∗ ) ≤ R τ ( ¯ f ∗ ) by deﬁnition of f ∗ .

Original: The claim follows.
Revised: The claim follows.

Original: B P ROPER AND PROPER COMPOSITE LOSSES
Revised: B P ROPER AND PROPER COMPOSITE LOSSES

Original: Beyond requiring classiﬁcation-calibration, it is often desirable to use classiﬁer outputs as valid probabilities.
Revised: Beyond requiring classiﬁcation-calibration, it is often desirable to use classiﬁer outputs as valid probabilities.

Original: Proper losses (Savage, 1971; Schervish, 1989; Buja et al., 2005) ϕ : {± 1 }× [0 , 1] → ¯ R + are the core losses of such class-probability estimation tasks, for which
Revised: Proper losses (Savage, 1971; Schervish, 1989; Buja et al., 2005) ϕ : {± 1 }× [0 , 1] → ¯ R + are the core losses of such class-probability estimation tasks, for which

Original: [Equation]
Revised: [Equation]

Original: Equation 13 stipulates that when using ϕ to distinguish positive and negative labels, it is optimal to predict the positive class-probability.
Revised: Equation 13 stipulates that when using ϕ to distinguish positive and negative labels, it is optimal to predict the positive class-probability.

Original: Typically, it is more useful to work with losses that accept real-valued scores, e.g. as output by the pre-activation of the ﬁnal layer of a neural network.
Revised: Typically, it is more useful to work with losses that accept real-valued scores, e.g. as output by the pre-activation of the ﬁnal layer of a neural network.

Original: To this end, proper composite losses (Reid &
Revised: To this end, proper composite losses (Reid &

Original: Williamson, 2010; Nock & Nielsen, 2009) are those (cid:96) : {± 1 }× R → R + are such that (cid:96) ◦ F − 1 is proper, for invertible link function F : ¯ R → [0 , 1] .
Revised: Williamson, 2010; Nock & Nielsen, 2009) are those (cid:96) : {± 1 }× R → R + are such that (cid:96) ◦ F − 1 is proper, for invertible link function F : ¯ R → [0 , 1] .

Original: Proper composite losses are inherently classiﬁcationcalibrated (Reid & Williamson, 2010, Theorem 16).
Revised: Proper composite losses are inherently classiﬁcationcalibrated (Reid & Williamson, 2010, Theorem 16).

Original: Canonical examples are the logistic loss (cid:96) ( y, v ) =
Revised: Canonical examples are the logistic loss (cid:96) ( y, v ) =

Original: log(1 + e − yv ) with F : v (cid:55)→ σ ( v ) , and exponential loss (cid:96) ( y, v ) =
Revised: log(1 + e − yv ) with F : v (cid:55)→ σ ( v ) , and exponential loss (cid:96) ( y, v ) =

Original: e − yv with F : v (cid:55)→ σ (2 · v ) .
Revised: e − yv with F : v (cid:55)→ σ (2 · v ) .

Original: When (cid:96) is differentiable, we have (Reid & Williamson, 2010, Corollary 12)
Revised: When (cid:96) is differentiable, we have (Reid & Williamson, 2010, Corollary 12)

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Given a proper loss ϕ
Revised: Given a proper loss ϕ

Original: and “symmetric” link F with F ( − v ) =
Revised: and “symmetric” link F with F ( − v ) =

Original: 1 − F ( v ) , the loss (cid:96) ( y, v ) = ϕ
Revised: 1 − F ( v ) , the loss (cid:96) ( y, v ) = ϕ

Original: ( y, F ( v )) deﬁnes a margin loss (Reid & Williamson, 2010, Corollary 14).
Revised: ( y, F ( v )) deﬁnes a margin loss (Reid & Williamson, 2010, Corollary 14).

Original: Proper composite losses may also be extended to multiclass settings in the natural way (Gneiting &
Revised: Proper composite losses may also be extended to multiclass settings in the natural way (Gneiting &

Original: Raftery, 2007; Williamson et al., 2016):
Revised: Raftery, 2007; Williamson et al., 2016):

Original: one now deﬁnes a proper loss ϕ :
Revised: one now deﬁnes a proper loss ϕ :

Original: [ K ] × ∆ K → R + , where
Revised: [ K ] × ∆ K → R + , where

Original: K is the number of classes and ∆ K denotes the K -simplex.
Revised: K is the number of classes and ∆ K denotes the K -simplex.

Original: A proper composite loss may be deﬁned using a link F : R K → ∆ K , such as the softmax operator F k ( v ) =
Revised: A proper composite loss may be deﬁned using a link F : R K → ∆ K , such as the softmax operator F k ( v ) =

Original: exp( v k ) (cid:80) k (cid:48) exp( v k (cid:48) ) .
Revised: exp( v k ) (cid:80) k (cid:48) exp( v k (cid:48) ) .

Original: Combined with the log-loss ϕ ( y, p ) =
Revised: Combined with the log-loss ϕ ( y, p ) =

Original: − log p y , this yields the standard softmax cross-entropy loss.
Revised: − log p y , this yields the standard softmax cross-entropy loss.

Original: C I LLUSTRATION OF LOSSES
Revised: C I LLUSTRATION OF LOSSES

Original: We illustrate the Huberised, partially Huberised, and generalised cross-entropy losses as their underlying tuning parameters are varied.
Revised: We illustrate the Huberised, partially Huberised, and generalised cross-entropy losses as their underlying tuning parameters are varied.

Original: Additionally, we illustrate the link functions that are implicit in each of the losses, which illustrates that they may be non-invertible if τ is too large.
Revised: Additionally, we illustrate the link functions that are implicit in each of the losses, which illustrates that they may be non-invertible if τ is too large.

Original: C. H UBERISED LOSS
Revised: C. H UBERISED LOSS

Original: Figure 5 illustrates the Huberised version of the logistic loss, and its derivative.
Revised: Figure 5 illustrates the Huberised version of the logistic loss, and its derivative.

Original: Following the proof of Lemma 3, for τ ∈ (0 , 1) and z 0 = − σ − 1 ( τ ) , the Huberised loss ¯ φ τ has an implicit link function (see Figure 6)
Revised: Following the proof of Lemma 3, for τ ∈ (0 , 1) and z 0 = − σ − 1 ( τ ) , the Huberised loss ¯ φ τ has an implicit link function (see Figure 6)

Original: [Equation]
Revised: [Equation]

Original: Compared to the standard sigmoid, the Huberised link saturates more slowly as τ is decreased.
Revised: Compared to the standard sigmoid, the Huberised link saturates more slowly as τ is decreased.

Original: Note that when τ ≤ 12 , the link function is not invertible everywhere: this results in the loss not being proper composite per our deﬁnition.
Revised: Note that when τ ≤ 12 , the link function is not invertible everywhere: this results in the loss not being proper composite per our deﬁnition.

Original: C. PARTIALLY H UBERISED LOSS
Revised: C. PARTIALLY H UBERISED LOSS

Original: Figure 7 illustrates the partially Huberised version of the logistic loss, as well as the base log-loss.
Revised: Figure 7 illustrates the partially Huberised version of the logistic loss, as well as the base log-loss.

Original: Following the proof of Lemma 6, the partially Huberised loss ˜ φ τ has implicit link function
Revised: Following the proof of Lemma 6, the partially Huberised loss ˜ φ τ has implicit link function

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: 9 illustrates the link function.
Revised: illustrates the link function.

Original: The logistic loss has ϕ ( u ) = − log u , and so ϕ (cid:48) ( 1 / 2 ) = − 2 .
Revised: The logistic loss has ϕ ( u ) = − log u , and so ϕ (cid:48) ( 1 / 2 ) = − 2 .

Original: For τ = 1 , the link function will be non-invertible everywhere, which is expected since the loss here is the linear loss, which is not suitable for class-probability estimation.
Revised: For τ = 1 , the link function will be non-invertible everywhere, which is expected since the loss here is the linear loss, which is not suitable for class-probability estimation.

Original: For τ ∈ (1 , 2) , the link function will be invertible for p / ∈ (cid:2)(cid:0) 1 − 1 τ (cid:1) , 1 τ (cid:3) .
Revised: For τ ∈ (1 , 2) , the link function will be invertible for p / ∈ (cid:2)(cid:0) 1 − 1 τ (cid:1) , 1 τ (cid:3) .

Original: Intuitively, the case τ ∈ (1 , 2) corresponds to the linear regions of the losses on positive and negative instances crossing over.
Revised: Intuitively, the case τ ∈ (1 , 2) corresponds to the linear regions of the losses on positive and negative instances crossing over.

Original: For τ ≥ 2 , the link function will always be invertible.
Revised: For τ ≥ 2 , the link function will always be invertible.

Original: It may be observed that partial Huberisation causes the link function to saturate at values [1 / (1 + τ ) , τ/ (1+ τ )] : this does not affect classiﬁcation calibration, but does imply that rescaling is necessary in order to intepret the output probabilities.
Revised: It may be observed that partial Huberisation causes the link function to saturate at values [1 / (1 + τ ) , τ/ (1+ τ )] : this does not affect classiﬁcation calibration, but does imply that rescaling is necessary in order to intepret the output probabilities.

Original: C. I LLUSTRATION OF GENERALISED CROSS - ENTROPY
Revised: C. I LLUSTRATION OF GENERALISED CROSS - ENTROPY

Original: Figure 8 illustrates the base ϕ α loss, and its composition with a sigmoid link function.
Revised: Figure 8 illustrates the base ϕ α loss, and its composition with a sigmoid link function.

Original: −−      v ¯ φ τ ( v ) τ
Revised: −−      v ¯ φ τ ( v ) τ

Original: = 14 τ =τ = 

Revised: = 14 τ =τ =

Original: τ =(a) Loss.
Revised: τ =(a) Loss.

Original: −−  −− 0 .− 0 .− 0 .− 

Revised: −−  −− 0 .− 0 .− 0 .−

Original: .v
Revised: .v

Original: ¯ φ 0 τ ( v ) τ
Revised: ¯ φ 0 τ ( v ) τ

Original: = 14 τ
Revised: = 14 τ

Original: =
Revised: =

Original: τ
Revised: τ

Original: = 34 τ =(b) Derivative.
Revised: = 34 τ =(b) Derivative.

Original: [Figure][Figure] [Figure] [Figure]
Revised: [Figure]

Original: D R ELATED WORK

Original: As shown in Figure 1a, the partially Huberised loss in (10) exhibits saturating behaviour at the extremes.

Original: This by itself is hardly new in the context of robust statistics.

Original: For example, Tukey’s “biweight” loss replaces the Huber loss’ linear behaviour with hard saturation (Gross & Tukey, 1973), and is a member of the broader family of redescending M-estimators (Andrews et al., 1972; Shevlyakov et al., 2008). In a similar spirit, Masnadi-Shirazi & Vasconcelos (2009) proposed the bounded Savage loss ϕ ( v ) = ( σ (2 v ) − 1) 2 , while Yang et al. (2010); Menon & Williamson (2018) theoretically studied loss clipping.
Revised: [Figure]

Original: Robust versions of logistic regression derived by variants of the

Original: Huber loss have been explored, e.g., Pregibon (1982, Section 4.1), Copas (1988, Section 3.1), (Feng et al., 2014). The distinction of the above proposals to our work is two-fold. First, we explicitly consider the classiﬁcation setting, wherein we show consistency and robustness to label noise for the partially Huberised loss.
Revised: [Figure]

Original: Second, our proposal explicitly highlights the connection to gradient clipping.

Original: Finally, while our focus has been on label noise, there are other notions of “robustness”.

Original: Indeed, several works have proposed means of making gradient descent robust to heavy-tailed distributions or outliers (Brownlees et al., 2015; Hsu & Sabato, 2016; Lugosi & Mendelson, 2016;

Original: Lecué et al., 2018; Prasad et al., 2018; Holland & Ikeda, 2019), typically by passing the gradient through a robust operator (Tukey, 1960; Hampel et al., 1986; Rousseeuw & Leroy, 1987; Lerasle & Oliveira, 2011; Catoni, 2012; Minsker, 2015).
Revised: [Figure]

Original: By contrast, we analyse the (simpler) gradient clipping approach, showing that while it is inadequate to cope with label noise, a simple modiﬁcation rectiﬁes this.

Original: Since our proposal only modiﬁes the underlying loss, it may be combined with any of these methods.

Original: E R OBUSTNESS TO INSTANCE OUTLIERS
Revised: D R OBUSTNESS TO INSTANCE OUTLIERS

Original: Following Ding (2013, Section 3.2.2), we consider robustness to an arbitrary corruption under a linear scoring model.
Revised: Following Ding (2013, Section 3.2.2), we consider robustness to an arbitrary corruption under a linear scoring model.

Original: For margin loss (cid:96) ( x, y ; θ ) .
Revised: For margin loss (cid:96) ( x, y ; θ ) .

Original: =
Revised: =

Original: φ ( y · θ T x ) with empirical risk minimiser ˆ θ N on a sample { ( x n , y n ) } Nn =1 , suppose that the sample is corrupted with an outlier ( x (cid:48) , y (cid:48) ) .
Revised: φ ( y · θ T x ) with empirical risk minimiser ˆ θ N on a sample { ( x n , y n ) } Nn =1 , suppose that the sample is corrupted with an outlier ( x (cid:48) , y (cid:48) ) .

Original: One would like to ensure that ˆ θ N is not swayed by making (cid:107) x (cid:48) (cid:107) 2 arbitrarily large: that is,[Equation]
Revised: One would like to ensure that ˆ θ N is not swayed by making (cid:107) x (cid:48) (cid:107) 2 arbitrarily large: that is,

Original: [Equation]
Revised: [Equation]

Original: or equivalently, lim (cid:107)
Revised: or equivalently, lim (cid:107)

Original: x (cid:48) (cid:107) 2 → + ∞ ∇ (cid:96) ( x (cid:48) , y (cid:48) ; ˆ θ N ) = 0 .
Revised: x (cid:48) (cid:107) 2 → + ∞ ∇ (cid:96) ( x (cid:48) , y (cid:48) ; ˆ θ N ) = 0 .

Original: Fortunately, the saturating behaviour of the partially Huberised loss affords this, as we now show.
Revised: Fortunately, the saturating behaviour of the partially Huberised loss affords this, as we now show.

Original: Proposition 8.
Revised: Proposition 8.

Original: Pick any convex, differentiable, proper composite margin loss φ , whose link F satisﬁes lim z →−∞ z · F (cid:48)
Revised: Pick any convex, differentiable, proper composite margin loss φ , whose link F satisﬁes lim z →−∞ z · F (cid:48)

Original: ( z ) = 0 .
Revised: ( z ) = 0 .

Original: For any τ > 0 , let ˜ (cid:96) τ
Revised: For any τ > 0 , let ˜ (cid:96) τ

Original: ( x, y ; θ ) .
Revised: ( x, y ; θ ) .

Original: =
Revised: =

Original: ˜
Revised: ˜

Original: φ τ ( y · θ T x ) be the τ -partially
Revised: φ τ ( y · θ T x ) be the τ -partially

Original: Huberised loss under a linear model.
Revised: Huberised loss under a linear model.

Original: Then, for any ( x, y ) and θ such that (cid:107) θ (cid:107) 2 < + ∞ and θ T x (cid:54) = 0 ,lim
Revised: Then, for any ( x, y ) and θ such that (cid:107) θ (cid:107) 2 < + ∞ and θ T x (cid:54) = 0 ,

Original: [Equation]
Revised: [Equation]

Original: Proof of Proposition 8.
Revised: Proof of Proposition 8.

Original: By deﬁnition,
Revised: By deﬁnition,

Original: [Equation]
Revised: [Equation]

Original: Thus,
Revised: Thus,

Original: [Equation]
Revised: [Equation]

Original: where ψ x,θ denotes the angle between x and θ .
Revised: where ψ x,θ denotes the angle between x and θ .

Original: If θ T x (cid:54) = 0 , then cos ψ x,θ (cid:54) = 0 , and so the third term is ﬁnite.
Revised: If θ T x (cid:54) = 0 , then cos ψ x,θ (cid:54) = 0 , and so the third term is ﬁnite.

Original: Thus, (cid:107)∇
Revised: Thus, (cid:107)∇

Original: [Equation]
Revised: [Equation]

Original:  ) , depending on the sign of y · θ T x .
Revised: z ) , depending on the sign of y · θ T x .

Original: By deﬁnition of ˜ φ τ , the derivative of the loss asymptotes to either F (cid:48) ( v ) , or φ(cid:48)
Revised: By deﬁnition of ˜ φ τ , the derivative of the loss asymptotes to either F (cid:48) ( v ) , or φ

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: by assumption, and lim z → + ∞ z · φ (cid:48) ( v ) = 0 since φ is convex, the claim is shown.
Revised: by assumption, and lim z → + ∞ z · φ (cid:48) ( v ) = 0 since φ is convex, the claim is shown.

Original: F A DDITIONAL SYNTHETIC EXPERIMENTS
Revised: E A DDITIONAL SYNTHETIC EXPERIMENTS

Original: We provide some more details regarding the synthetic data used in the body, as well as results on an additional two-dimensional synthetic dataset.
Revised: We provide some more details regarding the synthetic data used in the body.

Original: F. D ETAILS OF L ONG & S ERVEDIO (2010) DATASET
Revised: E. D ETAILS OF L ONG & S ERVEDIO (2010) DATASET

Original: The problem considered in Long & Servedio (2010) (see Figure 10) comprises a distribution concentrated on six atoms {± (1 , 0) , ± ( γ, 5 γ ) , ± ( γ, − γ ) } ⊂ R 2 for some γ > 0 ; we chose γ = 124 .
Revised: The problem considered in Long & Servedio (2010) (see Figure 10) comprises a distribution concentrated on six atoms {± (1 , 0) , ± ( γ, 5 γ ) , ± ( γ, − γ ) } ⊂ R 2 for some γ > 0 ; we chose γ = 124 .

Original: An instance ( x 1 , x 2 ) is labelled as y = (cid:74) x 1 ≥ 0 (cid:75) .
Revised: An instance ( x 1 , x 2 ) is labelled as y = (cid:74) x 1 ≥ 0 (cid:75) .

Original: The instances are weighted so that the ﬁrst four atoms have probability mass 18 , and the last two atoms mass 14 .
Revised: The instances are weighted so that the ﬁrst four atoms have probability mass 18 , and the last two atoms mass 14 .

Original: We modify this distribution slightly by treating the atoms as means of isotropic Gaussians, and treating the marginal distribution over instances to be a mixture of these Gaussians with mixing weights given by the corresponding probability masses of the atoms.
Revised: We modify this distribution slightly by treating the atoms as means of isotropic Gaussians, and treating the marginal distribution over instances to be a mixture of these Gaussians with mixing weights given by the corresponding probability masses of the atoms.

Original: F. D ETAILS OF OUTLIER DATASET
Revised: E. D ETAILS OF OUTLIER DATASET

Original: For the experiment involving outliers in feature space, the data comprises points on the line.
Revised: For the experiment involving outliers in feature space, the data comprises points on the line.

Original: Positively labelled samples are drawn from a unit variance Gaussian centered at (1 , 1) , with positively labeled outliers drawn from ( − 200 , 1) .
Revised: Positively labelled samples are drawn from a unit variance Gaussian centered at (1 , 1) , with positively labeled outliers drawn from ( − 200 , 1) .

Original: Negatively labelled samples comprise the negation of all points.
Revised: Negatively labelled samples comprise the negation of all points.

Original: We learn an unregularised linear classiﬁer from this data, which corresponds to a single scalar θ .
Revised: We learn an unregularised linear classiﬁer from this data, which corresponds to a single scalar θ .

Original: F. A DDITIONAL 2D SYNTHETIC DATASET

Original: We further illustrate the differences amongst methods on a 2D dataset inspired by Amid et al. (2019a).

Original: The data comprises 500 points, falling into two bands (Figure 11).

Original: The decision boundaries for various losses, when trained with a linear model using explicit quadratic features, are shown in

Original: Figure 12.

Original: We subject the data to 45% symmetric label noise.

Original: We see that the logistic and generalised cross-entropy losses see marked changes in their decision boundaries.

Original: By contrast, the partially

Original: Huberised loss maintains the correct classiﬁcation boundary.

Original: [Figure]
Revised: [Figure]

Original: [Figure]

Original: [Figure]