C ONFIDENCE S CORES M AKE I NSTANCE - DEPENDENT L ABEL - NOISE L EARNING P OSSIBLE
Anonymous authors
Paper under double-blind review
A BSTRACT
Learning with noisy labels has drawn a lot of attention.
In this area, most of recent works only consider class-conditional noise , where the label noise is independent of its input features.
This noise model may not be faithful to many real-world applications.
Instead, few pioneer works have studied instance-dependent noise , but these methods are limited to strong assumptions on noise models.
To alleviate this issue, we introduce conﬁdence-scored instance-dependent noise (CSIDN), where each instance-label pair is associated with a conﬁdence score .
The conﬁdence scores are sufﬁcient to estimate the noise functions of each instance with minimal assumptions.
Moreover, such scores can be easily and cheaply derived during the construction of the dataset through crowdsourcing or automatic annotation.
To handle CSIDN, we design a benchmark algorithm termed instance-level forward correction .
Empirical results on synthetic and real-world datasets demonstrate the utility of our proposed method.
I NTRODUCTION
The recent success of deep neural networks has increased the need for high-quality labeled data.
However, such a labelling process can be time-consuming and costly.
A compromise is to resort to weakly-supervised annotations, using crowdsourcing platforms or trained classiﬁers that annotate the data automatically.
These weakly-supervised annotations tend to be low-quality and noisy, which negatively affects the accuracy of high-capacity models due to memorization effects (Zhang et al., 2017).
Thus, learning with noisy labels has often drawn a lot of attention.
Early works on noisy labels studied random classiﬁcation noise (RCN) for binary classiﬁcation (Angluin & Laird, 1988; Kearns, 1993).
In the RCN model, each instance has its label ﬂipped with a ﬁxed noise rate ρ ∈ [0 , 12 ) .
A natural extension of RCN is class-conditional noise (CCN) for multiclass classiﬁcation (Stempfel & Ralaivola, 2009; Natarajan et al., 2013; Scott et al., 2013; Menon et al., 2015; van Rooyen & Williamson, 2015; Patrini et al., 2016)
(Appendix A).
In the CCN model, each instance from class i has a ﬁxed probability ρ i,j of being assigned to class j .
Thus, it is possible to encode some similarity information between classes.
For example, we can expect that the image of a “dog” is more likely to be erroneously labelled as “cat” than “boat”.
To handle the CCN model, a common method is the loss correction , which aims to correct the prediction or the loss of the classiﬁer using an estimated noise transition matrix (Patrini et al., 2017;
Sukhbaatar et al., 2015; Goldberger & Ben-Reuven, 2017; Ma et al., 2018).
Another common approach is the label correction , which aims to improve the label quality during training.
For example,
Reed et al.
(2015) introduced a bootstrapping scheme.
Similarly, Tanaka et al.
(2018) proposed to update the weights of a classiﬁer iteratively using noisy labels, and use the updated classiﬁer to yield more high-quality pseudo-labels for the training set.
Although these methods have theoretical guarantees, they are unable to cope with real-world noise, e.g., instance-dependent noise (IDN).
The IDN model considers a more general noise (Manwani & Sastry, 2013; Ghosh et al., 2014;
Menon et al., 2016; Cheng et al., 2017; Menon et al., 2018), where the probability that an instance is mislabeled depends on both its class and features.
Intuitively, this noise is quite realistic, as poorquality or ambiguous instances are more likely to be mislabeled in real-world datasets.
However, it is much more complex to formulate the IDN model, since the probability of a mislabeled instance is a function of not only the label space but also the input space that can be very high dimensional.
[Table]
As a result, several pioneer works have considered stronger assumptions on noise functions.
However, stronger assumptions tend to restrict the utility of these works (Table 1).
For instance, the boundary-consistent noise model considers stronger noise for samples closer to the decision boundary of the Bayesian optimal classiﬁer (Du & Cai, 2015; Menon et al., 2018).
However, such a model is restricted to binary and cannot estimate noise functions.
Cheng et al.
(2017) recently studied a particular case of the IDN model, where noise functions are upper-bounded.
Nonetheless, their method is limited to binary classiﬁcation and has only been tested on small datasets.
Instead of simplifying assumptions on noise functions, we propose to tackle the IDN model from the source, by considering conﬁdence scores to be available for the label of each instance.
We term this new setting conﬁdence-scored instance-dependent noise (CSIDN, Figure 1c).
The conﬁdence scores denote how likely an instance is to be correctly labeled.
Here, we make a simple assumption on realworld noisy data: the instance inﬂuences the “magnitude” of the noise, i.e., how likely the assigned label is to be wrong, but conditioned on an assigned label being erroneous, the transition probabilities to other classes are independent of the instance.
Based on this assumption and conﬁdence scores, we derive an instance-level forward correction algorithm.
In particular, this algorithm can fully estimate the transition probability for each instance, and subsequently train a robust classiﬁer with a loss-correction method similarly to Patrini et al.
It is noted that conﬁdence scores can be easily and cheaply derived during the construction of the dataset.
For example, in crowdsourcing platforms, simply counting how many annotators agree on a given instance can give a notion of how conﬁdent a label is.
Besides, many real-world datasets are automatically annotated using a trained classiﬁer, such as web-scraped datasets (Tong Xiao et al., 2015) and physiological features inferred from medical records (Agarwal et al., 2016).
In these cases, the class-probabilities of the labels assigned by the classiﬁer can be seen as conﬁdence scores, provided that the classiﬁer is well calibrated (Guo et al., 2017).
To sum up, we ﬁrst formulate instance-dependent noise in Section 2.1, and expose its robustness challenge in Section 2.2.
Then, we explain our motivation to use conﬁdence scores, and propose the conﬁdence-scored instance-dependent noise (CSIDN) model in Section 2.3.
Lastly, to handle this new noise model, we present the ﬁrst practical algorithm termed instance-level forward correction in Section 3, and validate the proposed algorithm through extensive experiments in Section 4.
T ACKLING INSTANCE - DEPENDENT NOISE FROM THE SOURCE
In this section, we present the IDN model along with the limitations of existing approaches, and introduce the CSIDN model as a tractable instance-dependent noise model.
2. N OISE MODELS : FROM CLASS - CONDITIONAL TO INSTANCE - DEPENDENT NOISE
We formulate the problem of learning with noisy labels in this section.
Let D be the distribution of a pair of random variables (
X, Y ) ∈
X × Y , where X ∈ R d , Y = { 1 , 2 , . . .
, K } and K is the number of classes.
In the classiﬁcation task with noisy labels, we hope to train a classiﬁer while having only access to samples from a noisy distribution ¯ D of random variables ( X, ¯ Y ) ∈
X × Y .
Given a point x sampled from X , ¯ Y is derived from the random variable Y via a noise transition matrix T ( x ) =
( T i,j ( x ))
Ki,j =1 ∈ [0 , 1] K × K :
[Equation]
[Figure]
Each noise function T i,j : X (cid:55)→ [0 , 1] is deﬁned as T i,j ( x ) =
P ( ¯ Y =
j | Y =
i, X = x ) .
In the class-conditional noise (CNN) model (Figure 1a), the transition matrix does not depend on the instance x
and the noise is entirely characterized by the K 2 constants T i,j .
However, in the instancedependent noise (IDN) model (Figure 1b), the transition matrix depends on the actual instance.
This tremendously complicates the problem, as the noise is now characterized by K 2 functions over the latent space X , which can be very high dimensional (e.g., d ∼ 10 4 - 10 6 for an object recognition dataset
2. C HALLENGES FROM INSTANCE - DEPENDENT NOISE
Limitation of existing CCN methods.
Due to the complexity of the IDN model, most recent works in learning with noisy labels have focused on the CCN model (Figure 1a), and the CCN model can be seen as a simpliﬁed IDN model (Figure 1b) free of feature information.
In addition to loss correction and label correction mentioned before, another method for the CCN model is sample selection , which aims to ﬁnd reliable samples during training, such as the smallloss approaches (Jiang et al., 2018; Han et al., 2018).
Inspired by the memorization in deep learning (Arpit et al., 2017), those methods ﬁrst run a standard classiﬁer on a noisy dataset, then select the small-loss samples for reliable training.
However, all approaches cannot handle the IDN model directly.
Speciﬁcally, loss correction considers the noise model to be characterized by a ﬁxed transition matrix, which does not include any instance-level information.
Meanwhile, label correction is vulnerable to the IDN model, since the classiﬁer will be much weaker on noisy regions and labels corrected by the current prediction would likely be erroneous.
Similarly, sample selection is easily affected by the IDN model.
For example, in the small-loss approaches, instance-dependent noise functions can leave partial regions of the input space clean and other regions very noisy (e.g., in an object recognition dataset, poor-quality pictures will tend to receive more noisy labels than high-quality ones).
Since clean regions will tend to receive smaller losses than noisy regions, the small-loss approaches will focus on clean regions and neglect harder noisy regions.
Then, since the distribution of clean regions will subsequently be different from the global distribution, this will introduce a covariate-shift (Shimodaira, 2000), which greatly degrades performances.
Moreover, it is hard to use importance reweighting (Sugiyama et al., 2007) for alleviate the issue, since importance reweighting would require estimating the clean posterior probability that is intractable for the IDN model.
To validate this fact, we generate a 3-class distribution of concentric circles ( cf.
Figure 2a), with ∀ ( x, y ) ∈
R 2 × { 1 , 2 , 3 } , P (¯ y (cid:54) =
y | x ) = 12 (cid:16) w · x (cid:107) w (cid:107)(cid:107) x (cid:107) + 1 (cid:17) with w = (0 , 1) ( cf.
Figure 2b).
We then train a network on the top R ( T
) small-loss instances at each epoch T based on the losses of
[Figure]
the previous epoch, with R ( T ) decreasing in T as described in Han et al.
Figure 2c shows the density of the top 50% small-loss instances selected after 10 epochs: since noisy regions are associated to higher losses, the network eventually tends to select instances from the clean region and neglect the noisy region, which leads to poor test accuracy because of the covariate-shift.
Limitation of pioneer IDN methods.
The main challenge of the IDN model is the wide range of possible noise functions included in its formulation.
Since each T i,j ( · ) is a function of the highdimensional input space X , it is challenging for a model to be ﬂexible enough to ﬁt any real-world noise function while being trainable on corrupted datasets, let alone derive theoretical results.
Instead, various recent works have considered stronger assumptions on noise functions.
For instance, boundary-consistent noise (BCN), ﬁrst introduced by (Du & Cai, 2015) and generalized in Menon et al.
(2018), considers stronger noise for samples closer to the decision boundary of the Bayesian optimal classiﬁer.
This is a reasonable model for noise from human annotators, since “harder” instances (i.e., instances closer to the decision boundary) are more likely to be corrupted.
Moreover, it is simple enough to derive some theoretical guarantees, as done in Menon et al.
Additionally, an extension of the BCN model was studied in Bootkrajang & Chaijaruwanich (2018), where the noise function is a Gaussian mixture of the distance to the Bayesian optimal boundary.
However, the BCN model and its extension are restricted to binary classiﬁcation, and their geometry-based assumption becomes difﬁcult to fathom for high-dimensional input spaces.
Furthermore, Cheng et al.
(2017) recently studied a particular case of the IDN model, where the probabilities that the true labels of samples ﬂip into corrupted ones have upper bounds.
They proposed a method based on distilled samples, where noisy labels agree with the optimal Bayesian classiﬁer on the clean distribution.
However, their method is limited to binary classiﬁcation and has only been tested on small UCI datasets.
Table 1 summarizes the characteristics of those approaches.
2. C ONFIDENCE - SCORED INSTANCE - DEPENDENT NOISE
Instead of simplifying assumptions on noise functions, we propose to tackle the IDN model from the source.
Namely, we consider that, for each instance, we have access to a measure of conﬁdence in the assigned label.
As most of noisy datasets arise from crowdsourcing or automatic annotation, such conﬁdence scores can be easily derived during the dataset construction, often with no extra cost.
This allows for a good approximation of noise functions with weaker assumptions.
Before introducing our proposed noise model conﬁdence-scored instance-dependent noise (CSIDN,
Figure 1c), we ﬁrst deﬁne what are the conﬁdence scores, and explain why the conﬁdence scores are available in real-world applications.
Deﬁnition of conﬁdence scores.
For any data point ( x, ¯ y ) sampled from the joint distribution ( X, ¯ Y ) , we deﬁne the conﬁdence score r x as follows.
[Equation]
Namely, the probability that the assigned label is correct.
Availability of conﬁdence scores.
Our rationale is that in tasks involving instance-dependent noise, the conﬁdence information can be easily derived with no extra cost.
Firstly, in crowdsourcing platforms, when multiple workers manually annotate datasets, an aggregation step is often took to aggregate answers of different workers for each instance (e.g., majority vote).
An estimation of r x could then be derived by taking the ratio of votes for the assigned label on the total number of workers.
Moreover, since this estimation would of course be less reliable as the number of workers decreases, an alternative could be to directly ask workers for self-reported conﬁdence scores of their responses (Cosmides & Tooby, 1996; Oyama et al., 2013).
Secondly, the conﬁdence information can also be available in automatic annotation via a softmax output layer of deep neural networks.
This layer outputs an estimation of the probability that each class is the true label: when a model outputs a given class with probability 0 .
9 , we expect the predicted class to be true 9 times out of 10 on average.
A model that estimates the accurate probability is well-calibrated.
Therefore, in the case of labels generated by a well-calibrated model, the softmax probability of the assigned label can be directly interpreted as a conﬁdence measure that the label is correct.
Even though Guo et al.
(2017) showed that recent deep neural networks are not usually well-calibrated (whereas early shallower networks were, as shown in Niculescu-Mizil & Caruana (2005)), model calibration can be achieved in a relatively straightforward way at the validation time, e.g., using temperature scaling (Section 4.2 in Guo et al.
CSIDN: a tractable instance-dependent noise model.
Recall the intrinsic difﬁculty of the IDN model: to fully characterize this noise, one would need to estimate K 2 functions T i,j ( · ) over the input space X .
This is of course intractable with a ﬁnite noisy dataset.
This is why pioneer solutions to the IDN model have been so far limited by very strong assumptions.
However, considering additional conﬁdence scores, one can wonder whether such information would make the IDN model tractable with less restrictive assumptions.
Hence, we introduce a new and tractable instance-dependent noise model: conﬁdence-scored instance-dependent noise (CSIDN, Figure 1c).
In this noise model, the training data takes the form S :
= { ( x i , ¯ y i , r x i ) , i = 1 , . .
. , N } , where { ( x i , ¯ y i ) }
i i.i.d.
∼ ¯ D and r x i
= P ( Y =
¯ y i
| ¯ Y =
¯ y i , X
= x i ) is the previously deﬁned conﬁdence scores in the assigned label of a given instance (Eq.
The conﬁdence information r x is decisive for robustness to instance-dependent noise, as it provides a proxy for the noise functions T i,j of the training data that are often intractable otherwise.
B ENCHMARK SOLUTION FOR HANDLING THE CSIDN MODEL
To tackle the CSIDN model, we propose a benchmark solution.
Inspired by forward correction (Patrini et al., 2017) for the CCN model, we want to correct each prediction P (¯ y | x ) with the noise transition matrix T ( x ) .
However, the transition matrix for the CSIDN model is instance-dependent, and has to be estimated for each instance x .
We term our solution instance-level forward correction .
3. E STIMATING INSTANCE - DEPENDENT TRANSITION MATRIX
Using the conﬁdence scores, we will ﬁrst estimate the diagonal terms ( T i,i ( · ))
Ki
=1 of the transition matrix, and then estimate the non-diagonal ones.
Diagonal terms.
The diagonal terms of the transition matrix correspond to the probabilities that assigned labels are equal to true labels.
However, the conﬁdence scores available are only relevant to the class corresponding to the observed label.
Therefore, we need to proceed differently whether the conﬁdence scores are available for the considered class or not.
First, note that for each sample ( x, ¯ y, r x ) ∈ S i :=
{ ( x, ¯ y, r x ) ∈ S | ¯ y
= i } , T i,i ( x ) can be derived for the most part from the conﬁdence scores alone:
[Equation]
where β i ( x ) =
P (¯ Y = i | X = x ) P
( Y = i
| X = x ) .
In practice, we use an iterative procedure to estimate in turn β i ( · ) and T i,i ( · ) (see Section 3.2 for details).
Then, for the rest of samples ( x, ¯ y, r x ) ∈ S \ S i , r x does not give any direct information on
T i,i ( · ) .
Hence, we simply set each function T i,i ( · ) as its empirical mean µ i estimated using samples from S i at the current epoch:
[Equation]
where | S | denotes the cardinality of S .
Non-diagonal terms.
For non-diagonal terms, we have:
[Equation]
where α i,j ( x ) = P ( ¯ Y =
j | ¯ Y (cid:54) =
i, Y
= i, X = x ) .
In Eq.
(4), α i,j ( x ) refers to the probability that an instance x with true label i has an observed label j , once we know that the observed label is different from the true one.
Then, a reasonable assumption is that ∀
i (cid:54) =
j, ∀ x ∈ X , α i,j ( x )
=
α i,j : once knowing that the observed label is erroneous, the class transitions are not inﬂuenced by the instance x .
In other words, the dependence in x of the noise function only impacts the “magnitude” of the noise and not the class transitions.
To illustrate this assumption, consider a crowdsourcing task of object recognition with adjacent classes which annotators can only differentiate with details that can be more or less visible depending on the instance.
For example, objects from a given class may have distinctive traits, but those can be more or less visible in the pictures.
When those traits are present, the annotators can conﬁdently predict the right class.
Otherwise, they will make errors towards adjacent classes.
In this case, the probability that the assigned label is wrong highly depends the instance (with distinctive traits being visible or not).
Nonetheless, once we know that the instance is corrupted, i.e., because those traits were not visible enough on the image, the transition probabilities to the adjacent classes are not inﬂuenced by the instance itself.
With the previous assumption, we obtain ∀ i (cid:54)
=
j, ∀ x ∈ X , T i,j ( x )
=
α i,j (1 − T i,i ( x )) with α i,j ∈ [0 , 1] .
This allows us to estimate the K ( K − 1) constants ( α i,j ) i (cid:54) =
j once, and derive the non-diagonal noise functions of T ( x ) directly from our estimates of the diagonal noise functions (Eq.
3. O VERALL ALGORITHM : I NSTANCE - LEVEL FORWARD CORRECTION
Estimating T i,i and β i .
To train a classiﬁer h with the instance-level forward correction method, we need to estimate both T i,i ( x ) and β i ( x ) = P (¯ Y = i | X = x ) P
( Y = i
| X = x ) from Eq.
(3), for all x ∈ S i .
Firstly, the noisy posterior P ( ¯ Y = i | X = x ) can be easily estimated by training a naive classiﬁer on the noisy dataset.
Secondly, the true posterior P ( Y = i | X = x ) can be estimated using the output of the classiﬁer h ( x ) =
ˆ P ( Y = i | X = x ) at the previous epoch.
Therefore, we iteratively update ˆ β and ˆ T with the following steps: 1) ∀ x ∈ X , initialize ˆ β i ( x ) =and train a naive classiﬁer h noisy on the noisy data ¯ D to obtain h noisy ( x ) =
ˆ P ( ¯ Y | X = x ) .
2) ∀ i ∈ [1 , K ] , for each sample ( x, ¯ y, r x ) ∈ S i , compute ˆ T
i,i ( x ) =
r x ˆ β
i ( x ) and train classiﬁer h for one epoch.
3) ∀ i ∈ [1 , K ] , for each sample ( x, ¯ y, r x ) ∈ S i , update ˆ β
i
( x ) =
h noisy ( x ) i h
( x ) i .
Then, we repeat steps 2) and 3) through training.
In this way, for every epoch, each function T i,i ( · ) is estimated for the samples from S i .
Lastly, for the rest of samples with noisy label j (cid:54) =
i , T i,i ( · ) is estimated at each epoch using Eq.
[Equation]
Computing α i,j .
The computation of α i,j boils down to approximating non-diagonal terms of the transition matrix in the CCN model.
As ∀ i (cid:54) =
j, ∀ x ∈ X , T i,j ( x )
=
α i,j (1 − T i,i ( x )) , we have:
[Equation]
A simple and reliable way is to use anchor points, i.e., points for which we can know the true class almost surely.
These points may be directly available when some training data has been curated, or they can be identiﬁed either theoretically as in Liu & Tao (2015) or heuristically as in Patrini et al.
Having S ∗ i := { ( x, ¯ y, r x ) ∈ S | P ( Y = i | X = x ) ≈ 1 } a set of class
i anchor points, we simply need compute:
[Equation]
Two noisy posteriors can be estimated using the same classiﬁer h noisy trained on the noisy distribution
h noisy ( x ) =
ˆ P ( ¯ Y | X = x ) aforementioned.
Thus, α i,j can be estimated as follows:
[Equation]
Summary of the training procedure.
Given samples S and K sets of anchor points ( S ∗ i )
Ki =1 , we want to train a classiﬁer h ( · ) equipped with a loss l .
For any loss l : y, ˆ y (cid:55)→ l
( y, ˆ y ) , we deﬁne the
T -corrected loss as l T : y, ˆ y (cid:55)→ l ( y, T ˆ y ) .
The overall procedure is in Algorithm 1 (Appendix B).
E XPERIMENTS
We compare our instance-level forward correction (ILFC) method with four representative baselines: forward correction (FC)
(Patrini et al., 2017), mean absolute error (MAE)
(Ghosh et al., 2017), L qnorm (LQ) (Zhang & Sabuncu, 2018) and co-teaching (CT) (Han et al., 2018).
Details are shown in
Appendix C. Note that the pioneer IDN methods cannot work for multi-class cases.
4. S YNTHETIC DATASET
Generation process.
We generate a synthetic dataset (Appendix D) consisting in three classes of concentric circles (Figure 5a).
We then apply the following instance-dependent noise to each label:
P ( ¯ Y (cid:54) =
Y | X = x ) =
ρ (cid:16) w · x (cid:107) w (cid:107)(cid:107) x (cid:107) + 1 (cid:17) / 2 with w = (0 , 1) and ρ controlling the mean noise rate.
If corrupted, each label is ﬂipped to another class uniformly.
Empirical results.
Figure 3 shows the test accuracy of different methods on the synthetic dataset.
Each experiment is repeated 5 times and we plot the conﬁdence intervals of each curve.
On low-level noise, all methods show good performances (Figure 3a).
On mild-level noise, both Co-teaching and
ILFC show good performances and outperform other baselines (Figure 3b).
On high-level noise, the performance of all the baselines collapse, whereas ILFC constantly maintains good performances (Figures 3c and 3d).
More experiments are shown in Appendix E.
[Figure]
[Figure]
4. R EAL - WORLD DATASET
Generation process.
In order to corrupt labels from clean datasets such as SVHN and CIFAR10, we adopt the following procedure: (1) train a classiﬁer h : x (cid:55)→ σ ( g ( x )) on a small subset of the clean dataset; (2) using a small validation set, calibrate the classiﬁer by selecting the temperature t that maximizes the expected calibration error as in Guo et al.
(2017); (3) for each instance x , set: ¯
y = argmax i h t ( x ) i and r x = max i h t ( x ) i .
With this process, we attempt to emulate the construction of a real-world dataset (Appendix F).
Empirical results.
Figures 4a and 4b show the test accuracy on SVHN with 25% and 45% instance-dependent noise, respectively.
We can clearly observe that, on both low-level and highlevel noise, ILFC shows good performances with a fast convergence rate, and outperforms other baselines.
Figures 4c and 4d show the test accuracy on CIFAR10 with 25% and 45% instancedependent noise, respectively.
On low-level noise, all methods show good performances.
However, on high-level noise, ILFC shows a fast convergence rate and outperforms other baselines.
C ONCLUSION
In this paper, we give an overview of label-noise learning from class-conditional noise (easier) to instance-dependent noise (harder).
We explain why existing approaches cannot handle instancedependent noise well, and try to address this challenge via conﬁdence scores.
Thus, we formally propose the conﬁdence-scored instance-dependent noise (CSIDN) model.
To tackle the CSIDN model, we design a practical algorithm termed instance-level forward correction (ILFC).
Our ILFC method robustly outperforms existing methods, especially in the case of high-level noise.
In future works, we would like to extend label correction and sample selection approaches with the conﬁdence scores from the CSIDN model.
A R ELATED WORKS
Besides the works aforementioned, we survey other approaches to learning with noisy labels.
Robust losses.
Various approaches propose to use a provably robust loss function in the learning process.
In the case of class-dependent label noise, Natarajan et al.
(2013) constructed an unbiased estimator of any loss function under the noisy distribution.
Masnadi-shirazi & Vasconcelos (2009) introduced a robust non-convex loss.
Recently, works on symmetric losses showed that such loss offer theoretical robustness results to various types of noise (Ghosh et al., 2017; Charoenphakdee et al., 2019).
Motivated by the robustness to noise of the mean absolute error loss (MAE) shown in
Ghosh et al.
(2017), Zhang & Sabuncu (2018) introduced generalized cross entropy loss that allows for a trade-off between the efﬁcient learning properties of the CCE loss and the noise-robustness of
MAE.
Shen & Sanghavi (2019) introduced a trimmed loss with an iterative minimization process that allows for theoretical guarantees in the simpler setting of generalized linear models.
Annotator-level modelling.
Another recent line of related works attempts to model labels and worker’s quality directly during the crowdsourcing annotation process, in order to produce more accurate labels efﬁciently.
Branson et al.
(2017) modeled the annotators’ skill and instances difﬁculty while incrementally training a computer vision model during the annotation process, effectively reducing the time burden of the annotation process as well as the error rate in the assigned labels.
Guan et al.
(2018) modeled each annotator individually in order to better aggregate labels based on each worker’s skill and area of expertise.
Khetan et al.
(2018) introduced a method that allows to learn each workers’ skill even when each example is only annotated once, by jointly modelling the assigned labels and the workers during the annotation process.
Explicit/implicit regularizers.
Recently, several other regularization techniques have shown good robustness in weakly-supervised settings.
Temporal Ensembling (TE) (Laine & Aila, 2017) method labels some additional unlabeled instances using a consensus of predictions from models from previous epochs and with different regularizations and input augmentation conditions.
Mean-teacher (MT) (Tarvainen & Valpola, 2017) instead uses predictions from a model obtained by averaging the weights of a set of models similar to TE, as using the prediction from a unique model is more efﬁcient when a large amount of unlabeled data is available.
Virtual Adversarial Training (Miyato et al., 2018) regularizes the network using a measure of local smoothness of the conditional label distribution given the input, deﬁned as the robustness of the prediction to local adversarial perturbations in the input space.
Introduced in Zhang et al.
(2018), mixup trains a neural network on convex combinations of instance pairs and their respective labels, and has been shown to reduce the memorization of corrupted labels.
B A LGORITHM
We present Algorithm 1 here, which can be referred to Section 3.2 in details.
C B ASELINES
Forward correction.
Introduced in Patrini et al.
(2017), forward correction estimates a ﬁxed transition matrix T before training, and trains a classiﬁer with the corrected loss l T : ( y, ˆ y )
(cid:55)→ l ( y, T ˆ y ) .
Mean absolute error loss.
Due to its symmetric property, the Mean Absolute Error (MAE) has been theoretically justiﬁed to be robust to label noise under assumptions (Ghosh et al., 2017).
However, this loss is more difﬁcult to train, especially on complex datasets.
L q norm.
Introduced in Zhang & Sabuncu (2018), L q norm attempts to bring the best of both worlds between the CCE and the MAE loss: the CCE is easy to train, while the MAE is robust to label noise.
The authors therefore deﬁne this loss using the negative box-cox transformation:
[Equation]
so that the L q tends to the CCE when q → 0 and to the MAE when q = 1 .
In the following experiments, we set q = 0 .
7 , suggested by authors.
Co-teaching (Han et al., 2018).
Co-teaching algorithm is a small-loss approach where two classiﬁers are trained in parallel.
At each epoch, each classiﬁer selects the instances with the smallest loss, and feed them to the other network as a training set for the next iteration.
This recent work has proved to be a leading benchmark in the ﬁeld of noisy labels.
D S YNTHETIC DATASET
Figure 5 shows three synthetic datasets, which cover clean, IDN and CSIDN models.
[Figure]
In the IDN setting (b), each point x has a probability P ( ¯ Y (cid:54) =
Y | x )
= ρ (cid:16) w · x (cid:107) w (cid:107)(cid:107) x (cid:107) + 1 (cid:17) / 2 with w = (0 , 1) of being corrupted, where ρ is a parameter controlling the mean noise rate.
Therefore the noise is the strongest towards the direction (0 , 1) and the weakest in the direction (0 , − 1) .
If corrupted, the label is ﬂipped to another class uniformly.
The CSIDN setting (c) is similar to the
IDN setting, but each point is associated with measure of the conﬁdence in the assigned label.
A lower conﬁdence is represented by a lower opacity in
E D ECISION BOUNDARIES
Figure 6 shows the decision boundaries of our approach versus the ones of a benchmark model, for different levels of noise.
With high levels of noise, a model that does not include any instance-level modelling will degenerate around the most noisy region of the input space.
On the other hand, our model successfully accounts for the high noise in this region and is able to keep consistent predictions.
F E XAMPLES OF REAL - WORLD DATASETS
For example, the method would be similar to constructing a dataset with images scraped from the web, and automatically labelling them from neighbouring text ﬁelds using a classiﬁer such as a recurrent neural network.
Then, a small subset of curated images could be used at the beginning of the process to calibrate the classiﬁer, in order to make the predictions of the softmax output faithful to the conﬁdence in each label.
This way, we could construct a very large dataset for a very low-cost that, while involving some instance-dependent noise, would be equipped with conﬁdence information and therefore could be tackled with our proposed algorithm.
[Figure]