P EER L OSS F UNCTIONS : L EARNING FROM N OISY L ABELS WITHOUT K NOWING N OISE R ATES
Anonymous authors
Paper under double-blind review
A BSTRACT
Learning with noisy labels is a common problem in supervised learning.
Existing approaches require practitioners to specify noise rates , i.e., a set of parameters controlling the severity of label noises in the problem.
The speciﬁcations are either assumed to be given or estimated using additional approaches.
In this work, we introduce a technique to learn from noisy labels that does not require a priori speciﬁcation of the noise rates.
In particular, we introduce a new family of loss functions that we name as peer loss functions.
Our approach then uses a standard empirical risk minimization (ERM) framework with peer loss functions.
Peer loss functions associate each training sample with a certain form of “peer” samples, which evaluate a classiﬁer’ predictions jointly.
We show that, under mild conditions, performing ERM with peer loss functions on the noisy dataset leads to the optimal or a near optimal classiﬁer as if performing ERM over the clean training data, which we do not have access to.
To our best knowledge, this is the ﬁrst result on “learning with noisy labels without knowing noise rates” with theoretical guarantees.
We pair our results with an extensive set of experiments, where we compare with state-of-the-art techniques of learning with noisy labels.
Our results show that peer loss functions based method consistently outperforms the baseline benchmarks, as well as some recent new results.
Peer loss provides a way to simplify model development when facing potentially noisy training labels, and can be promoted as a robust candidate loss function in such situations.
I NTRODUCTION
The quality of supervised learning models depends on the training data { ( x n , y n ) }
Nn =1 .
In practice, label noise can arise due to a host of reasons.
For instance, the observed labels ˜
y n s may represent human observations of a ground truth label.
In this case, human annotators may observe the label imperfectly due to differing degrees of expertise or measurement error, see e.g., medical examples such as labeling MRI images from patients.
Many prior approaches to this problem in the machine learning literature aim to develop algorithms to learn models that are robust to label noise (Bylander, 1994; Cesa-Bianchi et al., 1999; 2011; Ben-David et al.
; Scott et al., 2013; Natarajan et al., 2013;
Scott, 2015).
Typical approaches require a priori knowledge of noise rates , i.e., a set of parameters that control the severity of label noise.
Working with unknown noise rates is difﬁcult in practice:
Often, one must estimate the noise rates from data, which may require additional data collection (Natarajan et al., 2013; Scott, 2015; Van Rooyen et al., 2015)
(e.g., be a redundant set of noisy labels for each sample point, or a set of ground truth labels for tuning these parameters) and may introduce estimation error that can affect the ﬁnal model in less predictable ways.
Our main goal is to provide an alternative that does not require the speciﬁcation of the noise rates, nor an additional estimation step for the noises.
This target solution might help when the practitioner does not have access to reliable estimates of the noise rates (e.g., when the training data has limited size for the estimation tasks, or when the training data is already collected in a form that makes the estimation hard to perform).
In this paper, we introduce a new family of loss functions, peer loss functions , to empirical risk minimization (ERM), for a broad class of learning with noisy labels problems.
Peer loss functions operate under different noise rates without requiring either a priori knowledge of the embedded noise rates, or an estimation procedure.
This family of loss functions builds on approaches developed in the peer prediction literature (Miller et al., 2005; Dasgupta & Ghosh, 2013; Shnayder et al., 2016), which studies how to elicit information from self-interested agents without veriﬁcation.
Typical approaches in the peer prediction literature design scoring functions to score each reported data using another noisy reference answer, without accessing ground truth information.
We borrow this idea and the associated scoring functions via making a connection through treating each classiﬁer’s prediction as an agent’s private information to be elicited and evaluated, and the noisy label as an imperfect reference from a “noisy label agent”.
The peer loss takes a form of evaluating classiﬁers’ prediction using noisy labels on both the targeted samples and a particular form of constructed “peer” samples.
The evaluation on the constructed peer sample encodes implicitly the information about the noises as well as the underlying true labels, which helps us offset the effects of label noises.
The peer sample evaluation returns us a favorable property that expected risk of peer loss turns to be an afﬁne transformation of the true risk of the classiﬁer deﬁned on the clean distribution.
In other words, peer loss is invariant to label noises when optimizing with it.
This effect helps us get rid of the estimation of noise rates.
The main contributions of this work are:
We propose a new family of loss functions that can easily adapt to existing ERM framework that i) is robust to asymmetric label noises with formal theoretical guarantees and ii) requires no prior knowledge or estimation of the noise rates ( no need for specifying noise rates ).
We believe having the second feature above is a non-trivial progress, and it features a promising solution to deploy in an unknown noisy training environment.
We present formal results showing that performing ERM with a peer loss function can recover an optimal, or a near optimal classiﬁer f ∗ as if performing ERM on the clean data (Theorem 2, 3, 4).
We also provide analysis for peer loss functions’ risk guarantees (Theorem 5 and 7).
We present extensive experimental results to validate the usefulness of peer loss (Section 5 and Appendix).
This result is encouraging as it is able to remove the long-standing requirement of learning error rates of noises (or estimating transition matrices as used in many relevant papers) before many of the existing methods can be applied.
We also provide preliminary results on how peer loss generalizes to multi-class classiﬁcation problems.
We will contribute to the community by publishing our codes and implementations.
1. R ELATED W ORK Learning from Noisy Labels
Our work ﬁts within a stream of research on learning with noisy labels.
A large stream of research on this topic works with the random classiﬁcation noise (RCN) model, where observed labels are ﬂipped independently with probability e ∈ [0 , 12 ] (Bylander, 1994;
Cesa-Bianchi et al., 1999; 2011; Ben-David et al.).
Recently, learning with asymmetric noisy data (or also referred as class-conditional random classiﬁcation noise (CCN)) for binary classiﬁcation problems has been rigorously studied in (Stempfel & Ralaivola, 2009; Scott et al., 2013; Natarajan et al., 2013; Scott, 2015; Van Rooyen et al., 2015; Menon et al., 2015).
For a more thorough survey of classical results on learning with noisy data, please refer to (Fr ´ enay & Verleysen, 2014).
Symmetric loss For RCN, where the noise parameters are symmetric, there exists works that show symmetric loss functions (Manwani & Sastry, 2013; Ghosh et al., 2015; 2017; Van Rooyen et al., 2015) are robust to the underlying noises, without specifying the noise rates.
It was also shown that under certain conditions, the proposed loss functions are able to handle asymmetric noises.
Our focus departs from this line of works, and we will exclusively focus on asymmetric noise setting, and study the possibility of an approach that can ignore the knowledge of noise rates.
Follow-up works (Du Plessis et al., 2013; van Rooyen et al., 2015; Menon et al., 2015; Charoenphakdee et al., 2019) have looked into leveraging symmetric conditions and 0-1 loss with asymmetric noises, and with more evaluation metrics, such as balanced error rate and AUROC.
In particular, experimental evidences are reported in (Charoenphakdee et al., 2019) on the importance of symmetricity when learning with noisy labels.
More recent works More recent developments include an importance re-weighting algorithm (Liu & Tao, 2016), a noisy deep neural network learning setting (Sukhbaatar & Fergus, 2014; Han et al., 2018; Song et al., 2019), and learning from massive noisy data for image classiﬁcation (Xiao et al., 2015), robust cross entropy loss for neural network (Zhang & Sabuncu, 2018), loss correction (Pa- trini et al., 2017), among many others.
Loss or sample correction has also been studied in the context of learning with unlabeled data with weak supervisions (Lu et al., 2018).
Most of above works either lacks theoretical guarantee of the proposed method against asymmetric noise rates ((Sukhbaatar &
Fergus, 2014; Zhang & Sabuncu, 2018)), or requires estimating the noise rate (or transition matrix between noisy and true labels, (Liu & Tao, 2016; Xiao et al., 2015; Patrini et al., 2017; Lu et al., 2018)).
A good number of the recent works can be viewed as derivatives or extention of the unbiased surrogate loss function idea introduced in (Natarajan et al., 2013), therefore they would naturally require the knowledge of the noise rates or transition matrix.
We do provide thorough comparisons between peer loss and the unbiased surrogate loss methods.
Mostly relevant to us is a recent work (Xu et al., 2019) that proposes an information theoretical loss (an idea adpated from an earlier theoretical contribution (Kong & Schoenebeck, 2018))
that is also robust to asymmetric noises rate.
We aimed for a simple-to-optimize loss function that can easily adapt to existing ERM solutions.
(Xu et al., 2019) involves estimating a joint distribution matrix between classiﬁers and noisy labels, and then invokes computing a certain information theoretical measure based on this matrix.
Therefore, its sample complexity requirement and the sensitivity to noises in this estimation are not entirely clear to us (not provided in the paper either).
We do provide calibration guarantees and generalization bounds.
We provide conditions when the loss functions are convex.
In general, we do think computationally peer loss functions are easy to optimize with, in comparing to information theoretical measures.
Experiments comparing with (Xu et al., 2019) are also given in Section 5.
Peer Prediction
Our work also builds on the literature for peer prediction (Prelec, 2004; Miller et al., 2005; Witkowski & Parkes, 2012; Radanovic & Faltings, 2013; Witkowski et al., 2013; Dasgupta & Ghosh, 2013; Shnayder et al., 2016; Liu & Chen, 2017).
The seminal work (Miller et al., 2005) established that strictly proper scoring rule (Gneiting & Raftery, 2007) could be adopted in the peer prediction setting for eliciting truthful reports from self-interested agents.
There have been a sequence of follow-up works that have been done to relax the assumptions imposed therein (Witkowski & Parkes, 2012; Radanovic & Faltings, 2013; Witkowski et al., 2013; Radanovic et al., 2016; Liu & Chen, 2017).
Most relevant to us is (Dasgupta & Ghosh, 2013; Shnayder et al., 2016) where a correlated agreement (CA) type of mechanism was proposed.
CA evaluates a report’s correlations with another reference agent - its speciﬁc form inspired our peer loss.
P RELIMINARIES
Notations and preliminaries: For positive integer n , denote by [ n ] := { 1 , 2 , ..., n } .
Suppose ( X, Y ) ∈
X × Y are drawn from a joint distribution D , with their marginal distributions denoted as P X , P Y respectively.
We assume X ⊆ R d , and Y = {− 1 , +1 } , that is we consider a binary classiﬁcation problem.
Denote by p := P ( Y = +1) ∈ (0 , 1) .
There are N training samples ( x 1 , y 1 ) , ..., ( x N , y N ) drawn i.i.d.
from D .
Instead of observing y n s, the learner can only collect a noisy set of training labels ˜
y n s, generated according to y n s and a certain error rate model, that is we observe a dataset { ( x n , ˜ y n ) } N n =1 .
We assume a uniform error model for all the training samples we collect, in that errors in ˜ y n s follow the same error rate model: denoting the random variable for noisy labels as ˜ Y and we denote
[Equation]
) such that 0 ≤ e +1 + e − 1 < 1 .
e − 1 + e +1 < 1 is not unlike the condition imposed in the existing learning literature (Natarajan et al., 2013), and it simply implies that the noisy labels are positively correlating with the true labels (informative about the true labels).
Label noises are conditional independent from the features, that is the error rate is uniform across x n s: P ( ˜ Y
=
y (cid:48) | Y = y ) =
P ( ˜ Y =
y (cid:48) | X, Y = y ) , ∀ y, y (cid:48) ∈ {− 1 , +1 } .
Denote the distribution of the noisy data ( X, ˜ Y ) as ˜ D .
f : X → R is a real-valued decision function, and its risk w.r.t.
the 0-1 loss is deﬁned as E ( X,Y ) ∼D
[ 1 ( f ( X ) , Y )] .
The Bayes optimal classiﬁer f ∗ is the one that minimizes the 0-1 risk: f ∗ = argmin
f E ( X,Y ) ∼D [ 1 ( f ( X ) , Y )] .
Denote this optimal risk as R ∗ .
Instead of minimizing the above 0-1 risk, the learner often uses a surrogate loss function (cid:96) :
R × {− 1 , +1 } → R + , and ﬁnd a f ∈ F that minimizes the following error: E ( X,Y )
∼D [ (cid:96) ( f ( X ) , Y )] .
Denote the following measures:
[Equation] [Equation]
)] , R (cid:96), D ( f ) =
E ( X,Y ) ∼D [ (cid:96) ( f ( X ) , Y )] .
When there is no confusion, we will also short-hand E ( X,Y )
∼D [ (cid:96) ( f ( X ) , Y )] as E D [ (cid:96) ( f ( X ) , Y )] .
Using D to denote a dataset collected from distribution D (correspondingly ˜ D := { ( x n , ˜ y n ) } Nn = for ˜ D ), the empirical risk measure for f is deﬁned as ˆ R (cid:96),D ( f )
=
| D | (cid:80) ( x,y ) ∈ D (cid:96) ( f ( x ) , y ) .
2. L EARNING WITH NOISY LABELS
Typical methods for learning with noisy labels include developing bias removal surrogates loss function methods to learn with noisy data (Natarajan et al., 2013).
For instance, Natarajan et al.
(2013) tackle this problem by deﬁning an “un-biased” surrogate loss functions over (cid:96) to help “remove” noise, when e − 1 + e +1 < 1 : ˜ (cid:96) ( t, y ) :
= (1 − e − y ) · (cid:96) ( t,y ) − e y · (cid:96) ( t, − y ) 1 − e − 1 − e + , ∀ t, y. ˜ (cid:96) is identiﬁed such that when a prediction is evaluated against a noisy label using this surrogate loss function, the prediction is as if evaluated against the ground-truth label using (cid:96) in expectation.
Hence the loss of the prediction is “unbiased”, that is ∀ prediction t , E ˜ Y | y
[˜ (cid:96) ( t, ˜ Y )]
= (cid:96) ( t, y )
[Lemma 1, (Natarajan et al., 2013)].
One important note to make is most, if not all, existing solutions require the knowledge of error rates e − 1 , e +1 .
Previous works either assumed the knowledge of it, or needed additional clean labels or redundant noisy labels to estimate them.
This becomes the bottleneck of applying these great techniques in practice.
Our work is also motivated by the desire to remove this limitation.
2. P EER P REDICTION : I NFORMATION E LICITATION WITHOUT V ERIFICATION
Peer prediction is a technique developed to truthfully elicit information when there is no ground truth veriﬁcation.
Suppose we are interested in eliciting private observations about a binary event y ∈ {− 1 , +1 } generated according to a random variable Y .
There are K agents indexed by [ K ] .
Each of them holds a noisy observation of y , denoted as y ( i ) ∈ {− 1 , +1 } , i ∈
[ K ] .
We would like to elicit the y ( i ) s, but they are completely private and we won’t observe y to evaluate agents’ reports.
Denote by r ( i ) the reported data from each agent i .
It is completely possible that r ( i ) (cid:54) =
y ( i ) if agents are not compensated properly for their information.
Results in peer prediction have proposed scoring or reward functions that evaluate an agent’s report using the reports of other peer agents.
For example, a peer prediction mechanism may reward agent i for her report r ( i ) using S ( r ( i ) , r ( j )) where r ( j ) is the report of a randomly selected reference agent j ∈ [ K ] \{ i } .
The scoring function S is designed so that truth-telling is a strict Bayesian Nash Equilibrium (implying other agents truthfully report their y ( j ) ), that is, ∀ i
E y ( j ) [ S ( y ( i ) , y ( j ))
| y
( i )]
> E y ( j ) [ S ( r ( i ) , y ( j ))
| y
( i )] , ∀ r
( i ) (cid:54) =
y ( i ) .
There is a rich literature on proposing and studying peer prediction scoring functions, but we will focus on the following knowledge-free peer prediction mechanism, which only require a minimal amount of prior knowledge of the data sources to implement.
Correlated Agreement (Shnayder et al., 2016; Dasgupta & Ghosh, 2013) (CA) is a recently established peer prediction mechanism for a multi-task setting 1 .
CA is also the core and the focus of our subsequent sections on developing peer prediction based loss functions.
This mechanism builds on a ∆ matrix that captures the stochastic correlation between the two sources of predictions y ( i ) and y ( j ) .
Denote the following mapping function: g (1) = − 1 , g (2) = +1 , ∆ ∈ R 2 × 2 is then deﬁned as a squared matrix with its entries deﬁned as follows:
[Equation]
The intuition of above ∆ matrix is that each ( i, j ) entry of ∆ captures the marginal correlation between the two predictions.
M ∈ R 2 × 2 is deﬁned as the sign matrix of ∆ : M :=
Sgn (∆) , where Sgn ( x ) = 1 , x > 0; Sgn ( x ) = 0 , o.w. Deﬁne the following score matrix
[Equation]
where g − 1 is the inverse function of g .
CA requires each agent i to perform multiple tasks: denote agent i ’s observations for the N tasks as y 1 ( i ) , ..., y N ( i ) .
Ultimately the scoring function S ( · ) for each task k that is shared between i, j is deﬁned as follows: randomly draw two other tasks k
p 1 , k p 2 ,
[Equation]
=
k
Note a key difference between the ﬁrst and second M S terms is that the second term is deﬁned for two independent peer tasks k
p 1 , k p 2 (as the reference answers).
It was established in (Shnayder et al., 2016) that CA is truthful and proper (Theorem 5.2, Shnayder et al.
2 ; in particular, if y ( j ) is categorical w.r.t.
y ( i ) : P ( y ( j ) =
y (cid:48) | y
( i ) =
y ) <
P ( y ( j ) =
y (cid:48) ) , ∀ i, j ∈ [ K ] , y (cid:48) (cid:54) =
y then S ( · ) is strictly truthful (Theorem 4.4, Shnayder et al.
L EARNING WITH NOISY DATA : A PEER PREDICTION APPROACH
In this section, we show that peer prediction scoring functions, when speciﬁed properly, will adopt
Bayes optimal classiﬁer as their maximizers (or minimizers for the corresponding loss form).
We ﬁrst state our problem of learning with noisy labels as a peer prediction problem.
The connection is made by ﬁrstly rephrasing the two data sources, the classiﬁers and the noisy labels, from agents’ perspective.
For a task y ∈ {− 1 , +1 } , say +1 for example, denote the noisy labels ˜
Y as r ( X ) , X ∼ P X | Y =1 .
In general, r ( X ) can be interpreted as the agent that observes ˜ y 1 , ..., ˜ y N for a set of randomly drawn feature vectors x 1 , ..., x N : ˜
y n ∼ r ( X ) .
Suppose the agent’s observations are deﬁned as follows (similar to the deﬁnition of e +1 , e − ): P ( r ( X ) = − 1 | Y = +1)
=
e +1 , P ( r ( X ) = +
| Y = − 1) =
e − 1 .
Denote another agent whose observations “mimic” the Bayes optimal classiﬁer f ∗ .
Again denote this optimal classiﬁer agent as r ∗ ( X ) :
= f ∗ ( X ) :
[Equation]
[Figure]
Suppose we would like to elicit predictions from the optimal classiﬁer agent r ∗ , while the reports from the noisy label agent r will serve as the reference reports.
Both r and r ∗ are randomly assigned a task x , and each of them observes a signal r ( x ) and r ∗ ( x ) respectively.
Denote the report from agent r ∗ as ˜ r ∗ .
A scoring function S : R × R → R is called to induce strictly truthfulness if the following fact holds: E X (cid:2) S (cid:0) r ∗ ( X ) , r ( X ) (cid:1)(cid:3) >
E X (cid:2) S (cid:0) ˜
r ∗ , r ( X ) (cid:1)(cid:3) , ∀ ˜ r ∗ (cid:54) =
r ∗ ( X ) .
Taking the negative of S ( · ) (changing a reward score one aims to maximize to a loss to minimize) we also have E X (cid:2) − S (cid:0) r ∗ ( X ) , r ( X ) (cid:1)(cid:3) <
E X (cid:2) − S (cid:0) ˜
r ∗ , r ( X ) (cid:1)(cid:3) , ∀ ˜ r ∗ (cid:54) =
r ∗ ( X ) , implying when taking − S ( · ) as the loss function, minimizing − S ( · ) w.r.t.
R will return us the Bayes optimal classiﬁer f ∗ .
Our idea can be summarized easily using Fig. 1.
When there is no ambiguity, we will shorthand r ( X ) , r ∗ ( X ) as r, r ∗ , with keeping in mind that r, r ∗ encode the randomness in X .
Suppose S ( · ) is able to elicit the Bayes optimal classiﬁer f ∗ (agent r ∗ ) using r , we have the following theorem formally:
Theorem 1.
f ∗ =
argmin
f E ( X, ˜ Y ) ∼ ˜ D (cid:2) − S ( f ( X ) , r ) (cid:3) .
This proof can be done via showing that any non-optimal Bayes classiﬁer corresponds to a misreporting strategy, thus establishing its non-optimality.
We emphasize that it is not super restrictive to have a strictly truthful peer prediction scoring function S .
We provide discussions in Appendix.
Theorem 1 provides a conceptual connection and can serve as an anchor point when connecting a peer prediction score function to the problem of learning with noisy labels.
So far we have not discussed about a speciﬁc form of how we construct a loss function using ideas from peer prediction, and have not mentioned the requirement of knowing the noise rates.
We will provide the detail about a particular peer loss in next section, and explain its independence of noise rates.
P EER L OSS F UNCTION
We now present peer loss, a family of loss functions inspired by a particular peer prediction mechanism, the correlated agreement (CA), as presented in Section 2.2.
We are going to show that peer loss is able to induce the minimizer of a concept class F , under a broad set of non-restrictive conditions.
In this Section, we do not restrict to Bayes optimal classiﬁers, nor do we impose any restrictions on the loss functions’ elicitation power.
4. P REPARATION : EXPLAINING CA IN OUR CLASSIFICATION PROBLEM
To give a gentle start, we repeat the setting of CA for our classiﬁcation problem.
∆ and scoring matrix First recall that ∆ ∈ R 2 × 2 is a squared matrix with entries deﬁned between r ∗ (the f ∗ ) and r (i.e., the noisy labels ˜
Y ):
[Equation]
Recall g ( · ) is simply a mapping function: g (1) = − 1 , g (2)
= +1 .
∆ characterizes the “marginal” correlations between the optimal classiﬁer’ prediction and the noisy label ˜ Y .
Then the following scoring matrix M ∈ R n × n , sign matrix of ∆ , M :=
Sgn (∆) is computed.
Example 1.
Consider a binary class label case: P ( Y = − 1)
= 0 .
4 , P ( Y = +1)
= 0 .
6 , the noises in the labels are e − 1 = 0 .
3 , e +1 = 0 .
4 and e ∗− 1 = 0 .
2 , e ∗ +1 = 0 .
Then we have ∆(1 , 1) = 0 .
036 , ∆(1 , 2) = − 0 . 036 , ∆(2 , 1) = − 0 .
036 , ∆(2 , 2) = 0 .
And:
[Equation]
[Equation]
[Equation]
[Equation]
Peer samples For each sample ( x i , ˜ y i ) , randomly draw another two samples ( x i p 1 , ˜ y
i p 1 ) , ( x i p 2 , ˜ y i p 2 ) such that i p 1 (cid:54) = i p 2 and i p 1 , i p 2 (cid:54) =
i. We will name ( x i p 1 , ˜ y i p 1 ) , ( x i p 2 , ˜ y i p 2 ) as i ’s peer samples.
The scoring function S ( · ) for each sample point x i is deﬁned as follows: note S ( f ( x i ) , ˜ y i ))
=
S ( f ( x i ) , r ( x i )) ,
[Equation]
Recall M S ( · ) is a sign score matrix deﬁned for ∆ (Eqn.
The ﬁrst term above evaluates the classiﬁer’s prediction on x i using noisy label ˜
y i , and the second “peer” term “punishes” the classiﬁer from overly agreeing with the noisy labels.
We will see this effect more clearly.
Deﬁne loss function ˜ (cid:96) ( · ) as the negative of S ( · ) :
[Equation]
According to Theorem 1, we already know that minimizing ˜ (cid:96) ( · ) is going to ﬁnd the optimal Bayes classiﬁer, if ˜ Y and f ∗ ( X ) are categorical, which is easily satisﬁed:
Lemma 1.
When e − 1 + e +1 < 1 and e ∗− 1 + e ∗ +1 < 1 , r and r ∗ ( ˜ Y and f ∗ ) are categorical.
e ∗− 1 + e ∗ +1 < 1 means that the optimal classiﬁer is at least informative ((Liu & Chen, 2017))
- if otherwise, we can ﬂip the classiﬁer’s output to obtain one.
4. P EER L OSS
Knowledge of Sgn (∆)
We need to know Sgn (∆) in order to specify M S and ˜ (cid:96) , which requires certain information about f ∗ and ˜ Y .
We show that for the cases that the literature is broadly interested in, Sgn (∆) is simply the identify matrix (same condition as stated in Lemma 1):
Lemma 2.
If e − 1 + e +1 < 1 , e ∗ − 1 + e ∗ +1 < 1 , then Sgn (∆) =
I 2 × 2 , i.e., the identity matrix.
Peer loss
When Sgn (∆) =
I 2 × 2 , M S ( y, y (cid:48) )
=
if y = y (cid:48) , and 0 otherwise.
˜ (cid:96) ( · ) deﬁned in Eqn.
(2) reduces to the following form:
[Equation]
To see this, for instance 1 − M S (cid:0) f ( x i ) =
+1 , ˜ y
i =
+1 (cid:1)
= 1 − M (2 , 2) =
1 − 1 =
= 1 ( f ( x i ) = − 1 , ˜ y
i = +1) .
Replacing 1 ( · ) with any generic loss (cid:96) ( · ) we deﬁne:
[Equation]
We name above loss as peer loss .
This strikingly simple form of (cid:96) peer ( f ( x i ) , ˜ y i ) implies that knowing e − 1 + e +
< 1 , e ∗− 1 + e ∗ +1 < 1 hold is all we need to specify (cid:96) peer .
Why do we not need the knowledge of noise rates explicitly?
Both of the terms 1 ( f ( x i ) , ˜ y i ) and 1 ( f ( x i p 1 ) , ˜ y
i p 2 ) encoded the knowledge of noise rates implicitly .
The carefully constructed form as presented in Eqn.
3 allows peer loss to be invariant against noises (Lemma 3, a property we will explain later).
For a preview, for example if we take expectation of 1 peer ( f ( x i )
=
+1 , ˜ y
i =
+1)
we will have E [ 1 peer ( f ( x i ) =
+1 , ˜ y
i = +1)]
= P ( f ( X ) = +1 , ˜ Y
= +1)
− P ( f ( X ) = +1) · P ( ˜ Y = +1) , the marginal correlation between f and ˜ Y , which is exactly capturing the entries of ∆ deﬁned between f and ˜ Y !
Using the sign of ∆ is all we need to recover this information measure in expectation.
In other words, both the joint and marginal distribution terms encode the noise rate information in an implicit way.
Later we will show this measure is invariant under label noises, which gives us the property of peer loss being invariant to label noises and the ability of dropping the requirement of knowing noise rates.
We will instantiate this argument formally with Lemma 3 and establish a link between the above measure and the true risk of a classiﬁer on the clean distribution.
The rest of presentation focuses on (cid:96) peer deﬁned in Eqn.
but we keep in mind that replacing (cid:96) with 1 in (cid:96) peer recovers 1 peer .
ERM with peer loss
[Equation]
Note again that the deﬁnition of (cid:96) peer does not require the knowledge of either e +1 , e − 1 or e ∗ +1 , e ∗− 1 .
4. P ROPERTY OF P EER L OSS
We now present a key property of peer loss, which shows that its risk over the noisy labels is simply an afﬁne transformation of its true risk on clean data.
We denote by E D [ (cid:96) peer ( f ( X ) , Y )]
the expected peer loss of f when ( X, Y ) , as well as its peer samples, are drawn i.i.d.
from distribution D .
Lemma 3.
E ˜ D [ (cid:96) peer ( f ( X ) , ˜ Y )] = (1 − e − 1 − e +1 ) · E D [ (cid:96) peer ( f ( X ) , Y )] .
The above Lemma states that peer loss is invariant to label noises in expectation.
We have also empirically observed this effect in our experiment.
Therefore minimizing it over noisy labels is equivalent to minimizing over the true distribution.
The Theorems below establish the connection between E D [ (cid:96) peer ( f ( X ) , Y )] , the expected peer loss over clean data, with the true risk: Denote ˜ f ∗ 1 peer = arg min f ∈F R 1 peer , ˜ D
( f ) .
With Lemma 3, we can easily prove the following:
Theorem 2.
[Optimality guarantee with equal prior]
When p = 0 .
5 , ˜ f ∗ 1 peer ∈ arg
min f ∈F R D
( f ) .
The above theorem states that for a class-balanced dataset with p = 0 . 5 , peer loss induces the same minimizer as the one that minimizes the 0-1 loss on the clean data.
Removing the constraint of F , i.e., ˜ f ∗ 1 peer =
arg min f R 1 peer , ˜ D ( f ) ⇒ ˜ f ∗ 1 peer = f ∗ .
In practice we can balance the dataset s.t. p → 0 . 5 .
When p (cid:54) = 0 .
5 , denote ∆ p
= P ( Y = +1)
− P ( Y = − 1) , we have the following theorem:
Theorem 3.
[Approximate optimality guarantee with unequal prior]
When p (cid:54) = 0 .
5 , suppose the following conditions hold: (1) e − 1 , e +1 < 0 .
5 ; (2) (1 − e ) · e − 1 + e · e +1 > e ; (3) (1 − e ) · e +1 + e · e − 1 > e , where e := 12 − (cid:15) | ∆ p | .
Then | R D ( ˜ f ∗ 1 peer ) − min f ∈F
R D ( f ) | ≤ 2 (cid:15) (¯ (cid:96) − (cid:96) ) , ∀ (cid:15) ≤
| ∆ p | / 2 , if (cid:96) is bounded with ¯ (cid:96), (cid:96) denoting its max and min.
Condition (1) is a well-adopted assumption in the literature of learning with noisy labels.
When e +1 , e − 1 > e , we have conditions (2) and (3) hold: (1 − e ) ·
e − 1 + e · e +1 > (1 − e ) · e + e · e = e, (1 − e ) · e +1 + e · e − 1 > (1 − e ) ·
e + e · e = e. When | ∆ p | is small, i.e., p is closer to 0 .
5 , this condition becomes weaker, as we will afford to have a small (cid:15) but also a small e .
Multi-class extension Our results in this section are largely generalizable to multi-class setting.
Suppose we have K classes of labels, denoting as { 1 , 2 , ..., K } .
We denote by Q a transition matrix that characterizes the relationships between noisy label ˜ Y and the true label Y .
The ( i, j ) entry of Q is deﬁned as Q ij = P
( ˜ Y =
j | Y = i ) .
We write Q ij = q ij .
For many classes of noise matrices, the
M ( · ) matrix is simply a diagonal matrix.
Consider the following case: suppose the noisy labels have uniform probability of ﬂipping to a wrong class, that is, we pose the following conditions: q ij = q
ik , for all j (cid:54) =
k (cid:54)
=
i .
This condition allows us to deﬁne K new quantities e i = q ij for all i (cid:54) =
j , and q ii = 1 − (cid:80) j (cid:54) =
i e j .
We show that M ( · ) is a diagonal matrix when (cid:80) Kj =1 e j < 1 , a similar condition as e − 1 + e +1 < 1 .
Adapting from our proof for Lemma 3, we also have (derivation provided in
Appendix) E ˜ D
[ 1 peer ( f ( X ) , ˜ Y )]
= (1 − (cid:80) Kj =1 e j ) · E D [ 1 peer ( f ( X ) , Y )] .
The above again will help us reach the conclusion that minimizing peer loss leads to the same minimizer on the clean data.
We provide experiment results for peer loss with multi-class labels in Section 5.
4. α - WEIGHTED PEER LOSS
We take a further look at the case with p (cid:54) = 0 .
Denote by R +1 ( f ) =
P ( f ( X ) = − 1 | y = +1) , R − 1 ( f ) =
P ( f ( X ) = +1 | y = − 1) .
It is easy to prove:
Lemma 4.
Minimizing E [ 1 peer ( f ( X ) , ˜ Y )] is equivalent to minimizing R − 1 ( f )
+ R
+
( f ) .
However, minimizing the true risk R D ( f ) is equivalent to minimizing p · R +
( f )+(1 − p ) ·
R − 1 ( f ) , a weighted sum of R +1 ( f ) and R − 1 ( f ) .
The above observation and the failure to reproduce the strong theoretical guarantee
when p (cid:54) = 0 .
5 motivated us to study a α -weighted version of peer loss, to make it robust to the case p (cid:54) = 0 .
We propose the following α -weighted peer loss via adding a weight α ≥ 0 to the second term, the peer term:
[Equation]
Denote 1 α -peer as (cid:96) α -peer when replacing (cid:96) with 1 , ˜ f
∗ 1 α -peer =
arg min f ∈F R 1 α -peer , ˜ D ( f ) as the optimal classiﬁer under 1 α -peer , and ∆ ˜ p
= P ( ˜ Y = +1) − P ( ˜ Y = − 1) .
Then we have:
Theorem 4.
Let α = 1 − (1 − e − 1 − e +1 ) · ∆ p ∆ ˜ p .
Then ˜ f ∗ 1 α -peer ∈ arg min f ∈F R D
( f ) .
Denote α ∗ := 1 − (1 − e − 1 − e +1 ) · ∆ p ∆ ˜ p .
Several remarks follow: (1) When p = 0 .
5 , we have α ∗ = 1 , we recover the earlier deﬁnition of (cid:96) peer .
(2) When e − 1 = e +1 , α ∗ = 0 , we recover (cid:96) for the clean learning setting.
(3) When the signs of P ( Y = 1) − P ( Y = − 1) and P ( ˜ Y = 1) − P ( ˜ Y = − 1) are the same, α ∗ < 1 .
Otherwise, α ∗ > 1 .
In other words, when the noise changes the relative quantitative relationship of P ( Y = 1) and P ( Y = − 1) , α ∗ > 1 and vice versa.
(4) Knowing α ∗ requires certain knowledge of e +1 , e − 1 when p (cid:54) = 0 .
Though we do not claim this knowledge, this result implies tuning α ∗ (using validation data) may improve the performance.
Theorem 2 and 4 imply that performing ERM with 1 α ∗ -peer : ˆ f ∗ 1 α ∗ -peer =
arg
min f ˆ R 1 α ∗ -peer , ˜ D ( f ) will lead to a classiﬁer converging to f ∗ :
Theorem 5.
With probability at least 1 − δ , R D ( ˆ f ∗ 1 α ∗ -peer
[Equation]
− R ∗ ≤
[Equation]
4. C ALIBRATION AND G ENERALIZATION
So far our results focused on minimizing 0-1 losses, which is hard in practice.
We provide evidences of (cid:96) peer ’s, and (cid:96) α -peer ’s in general, calibration and convexity with a generic and differentiable calibrated loss.
We consider a (cid:96) that is classiﬁcation calibrated, convex and L -Liptchitz.
Classiﬁcation calibration describes the property that the convergence to optimality using a loss function (cid:96) would also guarantee the convergence to optimality with 0-1 loss:
Deﬁnition 1.
(cid:96) is classiﬁcation calibrated if there ∃ a convex, invertible, nondecreasing transformation Ψ (cid:96) with Ψ (cid:96) (0)
=
s.t. Ψ (cid:96) ( R D ( ˜ f ) − R ∗ ) ≤ R (cid:96), D ( ˜ f )
− min f R (cid:96), D ( f ) .
Denote f ∗ (cid:96) ∈ arg
min f R (cid:96), D ( f ) .
Below we provide sufﬁcient conditions for (cid:96) α -peer to be calibrated.
Theorem 6.
(cid:96) α -peer is classiﬁcation calibrated when either of the following two conditions holds: (1) α
= 1 (i.e., (cid:96) α -peer = (cid:96) peer ), p = 0 .
5 , and f ∗ (cid:96) satisﬁes the following: E [ (cid:96) ( f ∗ (cid:96) ( X ) , − Y )] ≥ E [ (cid:96) ( f ( X ) , − Y )] , ∀ f. (2) α < 1 , max { e +1 , e − 1 } < 0 .
5 , and (cid:96) (cid:48)(cid:48)
( t, y ) =
(cid:96) (cid:48)(cid:48) ( t, − y ) .
(1) states that f ∗ (cid:96) not only achieves the smallest risk over ( X, Y ) but also performs the worst on the “opposite” distribution with ﬂipped labels ( X, − Y ) .
(2) (cid:96) (cid:48)(cid:48) ( t, y ) =
(cid:96) (cid:48)(cid:48) ( t, − y ) is satisﬁed by some common loss function, such as square losses and logistic losses, as noted in (Natarajan et al., 2013),
Under the calibration condition, and denote the corresponding calibration function for (cid:96) α -peer as Ψ (cid:96) α -peer .
Denote by
[Equation]
We have the following generalization bound:
Theorem 7.
The following generalization bound holds for (cid:96) α ∗ -peer with probability at least 1 − δ :
[Equation]
where (cid:60) ( F ) is Rademacher complexity of F .
Convexity In experiments, we use neural networks which are more robust to non-convex loss functions.
We provide sufﬁcient conditions for R (cid:96) α -peer , ˜ D ( f ) to be convex in Appendix (Lemma 8).
E XPERIMENTS
We implemented a two-layer ReLU Multi-Layer Perceptron (MLP) for classiﬁcation tasks on
UCI Benchmarks and applied our peer loss to update their parameters.
We show the robustness of peer loss with increasing rates of label noises on 10 real-world datasets.
We compare the performance of our peer loss based method with surrogate loss method (Natarajan et al., 2013) (unbiased loss correction with known error rates), symmetric loss method (Ghosh et al., 2015), DMI (Xu et al., 2019), C-SVM (Liu et al., 2003) and PAM (Khardon & Wachman, 2007), which are state-of-the-art methods for dealing with random binary-classiﬁcation noises, as well as a neural network solution with binary cross entropy loss (NN).
We use a cross-validation set to tune the parameters speciﬁc to the algorithms.
For surrogate loss, we use the true error rates e − 1 and e +1 instead of learning them on the validation set.
Thus, surrogate loss could be considered a favored and advantaged baseline method.
Accuracy of a classiﬁcation algorithm is deﬁned as the fraction of examples in the test set classiﬁed correctly with respect to the clean and true label.
For given noise rates e +
and e − 1 , labels of the training data are ﬂipped accordingly.
[Figure] [Figure]
( e −
[Figure]
= 0 .
[Figure]
e +1 = 0 .
[Figure]
e −
[Figure]
=
[Figure]
A subset of the experiment results are shown in Table 1.
A full table with all details can be found in Appendix.
Equalized Prior means that we pre-sample the dataset to guarantee p = 0 .
For this case we used (cid:96) peer without α (or rather α = 1 as in (cid:96) α -peer ).
For p (cid:54) = 0 .
5 , we use validation dataset (using noisy labels) to tune α .
Our method is competitive across all datasets and is even able to outperform the surrogate loss method with access to the true error rates in a number of datasets, as well as symmetric loss functions (which does not require the knowledge of noise rates when error rates are symmetric) and the recently proposed information theoretical loss (Xu et al., 2019).
Fig.shows that our method can prevent over-ﬁtting when facing noisy labels.
[Table] [Table]
Preliminary results on multi-class classiﬁcation We now provide some preliminary results on
CIFAR-10 in Table 2.
We followed the setup in (Xu et al., 2019) and used ResNet (He et al., 2016) as the underlying optimization solution.
However, different from (Xu et al., 2019) whose noise only exists between speciﬁc class pairs, our noise is universal.
For each class, we ﬂip the label to any other label with a probability of (cid:15)/ 9 , where (cid:15) is the error rate and 9 is the number of other classes.
We do show peer loss is competitive against Cross Entropy and DMI (Xu et al., 2019).
More results and complete details are available in the Appendix.
Conclusion This paper introduces peer loss, a family of loss functions that enables training a classiﬁer over noisy labels, but without using explicit knowledge of the noise rates of labels.
We provide both theoretical justiﬁcations and extensive experimental evidences.
I LLUSTRATION OF OUR IMPLEMENTATION OF PEER LOSS
[Figure]
O THER PEER PREDICTION FUNCTIONS
Other notable examples include quadratic and logarithmic scoring function, deﬁned as follows:
Example 2.
Quadratic scoring function:
[Equation]
Example 3.
Logarithmic scoring function:
[Equation]
We know the following is true:
Lemma 5 (Miller et al.
S deﬁned in Example 1 & 2 induce strict truthfulness when y ( i ) and y ( j ) are stochastically relevant .
with deﬁning stochastic relevance as follows:
Deﬁnition 2.
y ( i ) and y ( j ) are stochastically relevant if ∃ s ∈ {− 1 , +1 } s.t.
[Equation]
Similarly we conclude that when r and r ∗ are stochastic relevant, the correlated agreement scoring rule, quadratic scoring rule and logarithmic scoring rule are strictly truthful.
This stochastic relevance condition essentially states that the optimal classiﬁer is statistically different from the noisy data source r on some signals.
Stochastic relevance is further satisﬁed in the binary classiﬁcation setting when e ∗− 1 + e ∗ +1 (cid:54)
= 1 , under the assumption that e − 1 + e +1 < 1 , as similarly imposed in learning with noisy labels literature (Scott et al., 2013; Natarajan et al., 2013; Scott, 2015).
Lemma 6.
r and r ∗ are stochastically relevant if and only if e ∗− 1 + e ∗ +1 (cid:54)
= 1 .
Proof.
Since r ∗ can be written as a function of X and Y , due to conditional independence between r and X (conditional on Y ), by chain rule
[Equation]
Since
[Equation]
We have
[Equation] [Equation]
For the binary signal case
[Equation]
the condition for stochastic relevance writes as follows:
[Equation]
where the last step is a consequence of Eqn.(6).
P ROOF FOR T HEOREM
Proof.
It is equivalent to prove f ∗ =
argmax f E ( X, ˜ Y ) ∼ ˜ D (cid:2) S ( f ( X ) , r ) (cid:3) .
First S ( · ) is able to elicit the Bayes optimal classiﬁer f ∗ ( r ∗ ) using r implies that:
[Equation]
First note that the expected score of a classiﬁer over the data distribution further writes as follows:
[Equation]
Denote by f (cid:48) a sub-optimal classiﬁer that disagrees with f ∗ on set X + dis = { x | Y = +1 : f (cid:48)
( x ) (cid:54) =
f ∗ ( x ) } .
By sub-optimality of f (cid:48)
we know that (cid:15) :=
P X ( X ∈ X + dis ) > 0 , as a zero measure X + dis does not affect its optimality.
Construct the following reporting strategy that
[Equation]
Not hard to check that
[Equation]
Yet we have the following fact that
[Equation]
where the inequality is due to strict truthfulness of S and the fact that (cid:15) > 0 .
We similarly conclude that
[Equation]
Combine Eqn.
and (8) we conclude the proof.
P ROOF FOR L EMMA
Proof.
Being categorical means
[Equation]
which further implies
[Equation]
and
P
[Equation]
Consider the following fact
[Equation]
Since r ∗ can be written as a function of X and Y
[Equation]
due to conditional independence between r and
X (conditional on Y ) we have
[Equation]
Therefore
P
[Equation]
We also have
[Equation]
Then we have
[Equation]
∗−
[Equation]
>when 1 > e ∗ +1 + e ∗− 1 .
P ROOF FOR L EMMA
Proof.
Again recall that
[Equation]
Then we have
[Equation]
>when 1 − e +1 − e − 1 > 0 , 1 − e ∗ +1 − e ∗− 1 > 0 .
Interestingly this coincides with the condition imposed in (Natarajan et al., 2013).
Similarly we can prove that
[Equation]
The other entries for P ( r ∗ = − 1 , r = − 1) − P ( r ∗ = − 1) P ( r = − 1) and P ( r ∗ = − 1 , r = +1) − P ( r ∗ = − 1) P ( r = +1) are symmetric.
Therefore the sign matrix of above score matrix is exactly the diagonal matrix.
P ROOF FOR L EMMA
Proof.
We denote by X i p 1 , ˜ Y i p
the random variable corresponding to the peer samples x i p 1 , ˜ y i p 2 .
First we have
[Equation]
Consider the two terms on the RHS separately
[Equation]
And consider the second term:
[Equation]
j ) , ˜ Y k )]
+ E X (cid:2) e +1 · (cid:96) ( f ( X ) , − 1) + e − 1 · (cid:96) ( f ( X ) , +1) (cid:3)
Thus,
E
[Equation]
Multi-class extension Notice the following facts:
[Equation]
and
K (cid:88) k
=
P ( Y = k ) q jk =
P
( Y = j )(1 − (cid:88) k
(cid:54) =
j e k ) + (1 − P
( Y = j ))
e j
=
1 − (cid:88)
k e k ) P ( Y = j )
+ e j
P ( 1 ( f ( X ) =
˜ Y ))
=
K (cid:88) k
=
P ( Y = k )
K (cid:88)
j =
P ( f ( X ) =
j | Y = k ) q jk =
K (cid:88)
j =
K (cid:88) k
=
P ( f ( X ) =
j | Y = k ) P ( Y = k ) q jk =
K (cid:88)
j =
P ( f ( X ) =
j | Y =
j ) P ( Y = j )(1 − (cid:88) k
(cid:54) =
j e k ) +
K (cid:88)
j = (cid:88) k (cid:54) =
j
P ( f ( X ) =
j | Y = k ) P ( Y = k )
e j =
K (cid:88)
j =
P ( f ( X ) =
j | Y =
j ) P ( Y = j )(1 − (cid:88) k
(cid:54) =
j e k ) +
K (cid:88)
j = e j ( P ( f ( X ) =
j ) − P ( f ( X ) =
j | Y =
j ) P ( Y = j )) =(1 − (cid:88)
k e k )
K (cid:88)
j =
P ( f ( X ) =
j | Y =
j ) P ( Y = j )
+
K (cid:88)
j = e j P ( f ( X ) = j )
Now consider the following
[Equation]
=
[Equation]
=
[Equation] [Equation]
=
[Equation] [Equation]
k
[Equation]
(cid:33)
Therefore
E
[Equation]
( P
[Equation] [Equation]
) P
[Equation]
Y
[Equation] [Equation]
P ( f ( X ) =
[Equation]
) P ( Y = j ))
For clean labels we have
[Equation]
For the second term we have
[Equation]
=
[Equation]
Therefore
E
[Equation]
=
[Equation]
We ﬁnish the proof.
P ROOF FOR T HEOREM
Proof.
From Lemma 3 we know
E [ (cid:96) peer ( f ( X ) , ˜ Y )]
=(1 − e − 1 − e +1 ) ·
E [ (cid:96) peer ( f ( X ) , Y )]
=(1 − e − 1 − e +1 ) · (cid:18) E [ (cid:96) ( f ( X ) , Y )]
− E [ (cid:96) ( f ( X i p 1 ) , Y i p 2 )] (cid:19) =(1 − e − 1 − e +1 )
· E (cid:20) (cid:96) ( f ( X ) , Y )] − 0 .
5 · E X [ (cid:96) ( f ( X ) , − 1)] − 0 .
5 · E X [ (cid:96) ( f ( X ) , +1)] (cid:19)
When (cid:96) is the 0-1 loss we have (cid:96) ( f ( X ) , − 1)
+ (cid:96) ( f ( X ) , +1)
= 1 , ∀ x , and therefore
E [ (cid:96) peer ( f ( X ) , ˜ Y )]
= (1 − e − 1 − e +1 ) · (cid:18) E [ (cid:96) ( f ( X ) , Y )] − 1 (cid:19)
With above we proved ˜ f
∗ 1 peer ∈ arg
min f ∈F R D
( f ) .
P ROOF FOR T HEOREM
Proof.
Our proof is inspired by our argument for p = 0 .
We ask the following question: if it is possible to show that ˜ Y corresponds an error-ﬂipped distribution of another distribution ˆ Y whose marginals ˜ p
Y is close to or equal to 0.5.
Observe the following: randomly ﬂipping Y with probability e uniformly, we will have a new distribution of labels ˆ Y that satisﬁes:
[Equation]
Denote by (cid:15) the tolerance of ˜ p Y : (cid:15) =
| ˜ p Y − 0 .
5 | .
When e sets to be: 1 − 2 e = (cid:15) | ∆ p | , we have | ˜ p Y − 0 .
5 | = (cid:15) .
The next question we ask: is it possible to ﬁnd parameters ˆ e − 1 , ˆ e +1 :
P ( ˜ Y
[Equation]
| ˆ Y
[Equation]
=
[Equation]
=
ˆ
[Equation]
+
Note that
[Equation]
Similarly P ( ˜ Y = +
| Y = − 1) =
(1 − e ) · ˆ e − 1 + e · (1 − ˆ e +1 ) .
Jointly we need the following equations to hold:
[Equation]
Solving above equations we have
[Equation]
For a feasible solution to ˆ e − 1 , ˆ e +1 , the conditions need to satisfy that (1) ˆ e − 1 , ˆ e +1 ≥ 0 and (2) ˆ e − 1 + ˆ e +1 < 1 .
First of all, from (2) we have
[Equation]
Then a necessary condition for ˆ e − 1 + ˆ e +1 < 1 is
[Equation]
This condition holds as long as e − 1 , e +1 < 0 .
From ˆ e − 1 , ˆ e +1 ≥
we have
[Equation]
This above jointly proves that R (cid:96) α -peer , ˜ D ( f ) is equivalent to a peer loss deﬁned over the noisy distribution of ˆ y with error parameters ˆ e − 1 , e +1 .
Denote by f ∗F ∈ arg min f ∈F R D
( f ) .
From the optimality of ˜ f
∗ 1 peer we have
[Equation]
[Equation] [Equation]
Note ∀ f :
[Equation]
Notice that
[Equation]
Combining Eqn.
(10, 12, 13) we have
[Equation]
P ROOF FOR L EMMA
Proof.
E [ 1 peer
[Equation] [Equation]
Y
[Equation]
] =(
[Equation]
e
[Equation] [Equation]
e +1 ) · ( P ( f (
[Equation] [Equation]
1 , Y = +1)
[Equation]
P ( f ( X ) = +1 , Y = − 1) − P (
[Equation]
[Equation] [Equation]
[Equation] [Equation]
[Equation] [Equation]
( X ) =
[Equation]
1) P ( Y
[Equation]
1)) =(1 − e − 1 − e +1 ) · ( pR +1 + (1 − p )
R −− p · P ( f ( X ) = 1) − (1 − p ) · P ( f ( X ) = − 1)) =(1 − e − 1 − e +1 ) · ( pR +1 + (1 − p ) R −− p · (cid:0) pR
+1 + (1 − p )(1 − R − 1 ) (cid:1) − (1 − p ) · (cid:0) p (1 − R +1 )
+ (1 − p )
R − 1 ) (cid:1)
=
2(1 − e − 1 − e +1 ) · p (1 − p ) · ( R − 1 + R +1 − 1)
P ROOF FOR T HEOREM
Proof.
E [
[Equation]
[Equation] [Equation]
] =
E [ 1 ( f ( X ) , ˜ Y )] − α · E [ 1 ( f ( X i p 1 ) , ˜ Y i p 2 )]
=
E [ 1 peer ( f ( X ) , ˜ Y )]
+ (1 − α ) ·
E [ 1 ( f ( X i p 1 ) , ˜ Y i p 2 )] −= E [ 1 peer ( f ( X ) , ˜ Y )]
+ (1 − α ) · (cid:0) P ( f ( X ) = − 1)
· P ( ˜ Y = − 1) + P ( f ( X ) = +1) · P ( ˜ Y = +1) (cid:1) −= E [ 1 peer ( f ( X ) , ˜ Y )]
+ (1 − α ) · (cid:18) (cid:0) p · (1 − R +1 )
+ (1 − p ) ·
R − 1 (cid:1) · P ( ˜ Y = − 1) + (cid:0) pR +1 + (1 − p )(1 − R − 1 ) (cid:1) · P ( ˜ Y = +1) (cid:19)
−= E [ 1 peer ( f ( X ) , ˜ Y )]
+ (1 − α ) · ( P ( ˜ Y = +1) − P (
[Equation]
where C
[Equation]
C (cid:48) are constants:
[Equation]
Let
[Equation]
that
[Equation]
we obtain that
[Equation]
concluding our proof.
The last equation Eqn.(14) also implies the following proposition:
Proposition 8.
For any f, f (cid:48) , we have
[Equation]
P ROOF FOR T HEOREM
Proof.
∀ f , using Hoeffding’s inequality with probability at least 1 − δ
[Equation]
Note we also have the following:
[Equation]
Now we show
[Equation]
We conclude the proof.
P ROOF FOR T HEOREM
Proof.
We start with condition (1).
From Lemma 3,
E [ (cid:96) peer ( f ( X ) , ˜ Y )]
=(1 − e − 1 − e +1 ) · (cid:18) E [ (cid:96) ( f ( X ) , Y )] − 0 .
5 · E [ (cid:96) ( f ( X ) , − 1)] − 0 .
5 · E [ (cid:96) ( f ( X ) , +1)]
(cid:19)
The above further derives as
[Equation]
Denote by c :=
[Equation]
1 we have
[Equation]
Further by our conditions we know
[Equation]
Therefore we have proved
[Equation]
Since (cid:96) ( · ) is calibrated, and according to Proposition 8 and Theorem 2:
[Equation]
Therefore Ψ (cid:96) peer ( x ) = 1 c Ψ (cid:96) ( x 1 − e − 1 − e +1 ) .
It’s straight-forward to verify that Ψ (cid:96) peer ( x ) satisﬁes the conditions in Deﬁnition 1.
We conclude the proof.
Now we check condition (2).
Again, from previously, we know the following holds for a certain ˆ p y
=
p
y (1 − e y )
+ (1 − p y )
e − y where p +1 = p, p − 1 = 1 − p :
E [
[Equation]
α -peer ( f ( X ) , ˜ Y )]
=
E [ (cid:96) ( f ( X ) , ˜ Y
[Equation]
α · (cid:96)
[Equation]
f ( X )
[Equation]
˜ Y k )]
[Equation]
Let φ ( f ( X ) · Y ) :
= (cid:96) ( f ( X ) , Y ) , we have
[Equation]
[Equation] [Equation]
f
[Equation]
X ) · Y )]
We ﬁrst introduce a Theorem:
Theorem 9 (Theorem 6, (Bartlett et al., 2006)) .
Let ϕ be convex.
Then ϕ is classiﬁcation-calibrated if and only if it is differentiable at 0 and ϕ (cid:48) < 0 .
We now show that ϕ is convex:
[Equation]
when α < 1 .
The last inequality is due to the fact that (cid:96) is convex.
Secondly we show the ﬁrst derivative of ϕ is negative at 0 : ϕ (cid:48)
(0) < 0 :
[Equation]
Note that
[Equation]
Plug back to Eqn.
(15) we have
[Equation]
1 − e − y )
(cid:19) φ (cid:48) (0) (16)
Since (1 − αp y )(1 − 2 e y ) + α (1 − p y )
(1 − e − y ) > 0 and φ (cid:48) (0) <
(due to calibration property of (cid:96) , Theorem 6 of Bartlett et al.
, we proved that ϕ (cid:48) (0) < 0 .
Then based on Theorem 6 of
Bartlett et al.
(2006), we know (cid:96) (cid:96) α -peer is classiﬁcation calibrated.
P ROOF FOR T HEOREM
Proof.
We ﬁrst prove the following Rademacher complexity bound
Lemma 7.
Let (cid:60) ( F ) denote the Rademacher complexity of F .
L denote the Lipschitz constant of (cid:96) .
Then with probability at least
[Equation]
δ , max
[Equation]
Note we also have the following ∀ α :
[Equation]
[Table]
Then apply the calibration condition we have
[Equation]
(Lemma 7)
[Equation]
with probability at least 1 − δ .
P ROOF FOR L EMMA
Proof.
Due to the random sampling, via Hoeffding inequality we ﬁrst have there exists some ˆ p ˜ y n ∈ (0 , 1) , with probability at least 1 − δ ,
[Equation]
Deﬁne the following loss function:
[Equation]
Via Rademacher bound on the maximal deviation we have with probability at least 1 − δ
[Equation]
Since (cid:96) is L -Lipschitz, due to the linear combination, ˜ (cid:96) is (1+ α ) L -Lipschitz.
Based on the Lipschitz composition of Rademacher averages, we have
[Equation]
Therefore, via union bound
[Table]
we know with probability at least 1 − 2 δ :
[Equation]
[Table]
[Equation]
[Table]
[Equation]
In above R (cid:96) α -peer , ˜ D ( f ) = R ϕ, ˜ D ( f ) because (cid:96) α -peer and (cid:96) share the same expected risk by construction.
Plug in the fact that (cid:96) α -peer is linear in (cid:96) and an easy consequence that (cid:96)
[Equation]
, let δ :=
δ/ 2 , we conclude the proof.
P ROOF FOR L EMMA
Nonetheless, despite the fact that (cid:96) α -peer ( · ) is not convex in general, [Lemma 5, (Natarajan et al., 2013)] informs us that as long as ˆ R (cid:96) α -peer , ˜ D ( f ) is close to some convex function, mirror gradient type of algorithms will converge to a small neighborhood of the optimal point when performing
ERM with (cid:96) α -peer .
A natural candidate for this convex function is the expectation of ˆ R (cid:96) α -peer , ˜ D ( f ) as ˆ R (cid:96) α -peer , ˜ D ( f ) → R (cid:96) α -peer , ˜ D ( f ) when N → ∞ .
Lemma 8.
When α < 1 , max { e +1 , e − 1 } < 0 . 5 , and (cid:96) (cid:48)(cid:48) ( t, y ) = (cid:96) (cid:48)(cid:48) ( t, − y ) , R (cid:96) α -peer , ˜ D ( f ) is convex.
Proof.
This was proved in the proof for Theorem 6, when proving the classiﬁcation calibration property of (cid:96) α -peer under condition (2).
E XPERIMENT I MPLEMENTATION D ETAILS
We implemented neural networks (LeCun et al., 2015) for classiﬁcation on 10 UCI Benchmarks and applied our peer loss to update their parameters.
For surrogate loss, we use the true error rates e −and e +1 instead of learning them on the validation set.
Thus, surrogate loss could be considered a favored and advantaged baseline method.
On each benchmark, we use the same hyper-parameters for all neural network based methods.
For C-SVM, we ﬁx one of the weights to 1, and tune the other.
For PAM, we tune the margin.
R ESULTS
The full experiment results are shown in Table. ?? .
Equalized Prior indicates that in the corresponding experiments, we resample to make sure P ( Y = +1)
=
P ( Y = − 1) and we ﬁx α = 1 in these experiments.
Our method is competitive in all the datasets and even able to outperform the surrogate
[Figure]
loss method with access to the true error rates in most of them.
C-SVM is also robust when error rates are symmetric, and is competitive in 8 datasets.
From Figure.
[Figure]
we can see our peer loss can prevent over-ﬁtting, which is also part of the reason of its achieved high robustness across different datasets and error rates.
[Table]
Thyroid
[Table]