P UMPOUT : A M ETA A PPROACH FOR R OBUSTLY T RAINING D EEP N EURAL N ETWORKS WITH N OISY L ABELS
Anonymous authors
Paper under double-blind review
A BSTRACT
It is challenging to train deep neural networks robustly on the industrial-level data, since labels of such data are heavily noisy, and their label generation processes are normally agnostic.
To handle these issues, by using the memorization effects of deep neural networks, we may train deep neural networks on the whole dataset only the ﬁrst few iterations.
Then, we may employ early stopping or the small-loss trick to train them on selected instances.
However, in such training procedures, deep neural networks inevitably memorize some noisy labels, which will degrade their generalization.
In this paper, we propose a meta algorithm called Pumpout to overcome the problem of memorizing noisy labels.
By using scaled stochastic gradient ascent , Pumpout actively squeezes out the negative effects of noisy labels from the training model, instead of passively forgetting these effects.
We leverage
Pumpout to upgrade two representative methods: MentorNet and Backward Correction.
Empirical results on benchmark vision and text datasets demonstrate that Pumpout can signiﬁcantly improve the robustness of representative methods.
I NTRODUCTION
Learning from the industrial-level data is quite demanding, since labels of such data are heavily noisy (i.e., 50% of noisy labels), and their label generation processes are usually agnostic (Xiao et al., 2015; Jiang et al., 2018).
Essentially, noisy labels of such data are corrupted from groundtruth labels without any prior assumptions (i.e., class-conditional noise (Natarajan et al., 2013)), which degrades the robustness of learning models.
It is noted that industrial-level data is frequently emerging in our daily life, such as social-network data (Cha & Cho, 2012), E-commerce data (Xiao et al., 2015) and crowdsourcing data (Welinder et al., 2010; Han et al., 2018a).
Due to the large data volume, industrial-level data can be well handled by deep neural networks (Xiao et al., 2015).
Thus, the key issue is how to train deep neural networks robustly on noisy labels of such data, since deep neural networks have the high capacity to ﬁt noisy labels eventually (Zhang et al., 2017).
To handle noisy labels, one common direction focuses on estimating the noise transition matrix (Goldberger & Ben-Reuven, 2017; Han et al., 2018b).
For instance, Patrini et al.
ﬁrst leveraged a two-step solution to estimate the noise transition matrix.
Based on this estimated matrix, they conducted backward loss correction, which is used for training deep neural networks robustly.
However, the noise transition matrix is not easy to be estimated accurately, especially when the number of classes is large.
Motivated by the memorization effects of deep neural networks (Arpit et al., 2017), one emerging direction focuses on training only on selected instances (Jiang et al., 2018;
Ren et al., 2018; Han et al., 2018c), which does not require any prior assumptions on noisy labels.
Speciﬁcally, deep learning models are known to learn easy instances ﬁrst, then gradually adapt to hard instances when training epochs become large (Arpit et al., 2017).
Therefore, in the ﬁrst few iterations, we may train deep neural networks on the whole dataset, and let them sufﬁciently learn clean instances in the noisy dataset.
Then we may later conduct early stopping (Goodfellow et al., 2016), which tries to stop the training on noisy instances; or we may employ the small-loss trick (Jiang et al., 2018; Han et al., 2018c), which tries to perform the training selectively on clean (small-loss) instances.
However, when noisy labels indeed exist, no matter using early stopping or small-loss trick, deep networks inevitably memorize some noisy labels (Arpit et al., 2017), which will lead to the poor generalization performance (Zhang et al., 2017).
In this paper, we design a meta algorithm called Pumpout , which allows us to overcome the issue of memorizing noisy labels.
The main idea of Pumpout is to actively squeeze out the negative effects of noisy labels from the training model, instead of passively forgetting these effects by further training.
Speciﬁcally, on clean labels, Pumpout conducts stochastic gradient descent typically; while on noisy labels, Pumpout conducts scaled stochastic gradient ascent , instead of stopping gradient computation as usual.
This aggressive policy can erase the negative effects of noisy labels actively and effectively.
We leverage Pumpout to upgrade two representative but orthogonal approaches in the area of “deep learning with noisy labels”:
MentorNet (Jiang et al., 2018) and Backward Correction (Patrini et al., 2017; van Rooyen & Williamson, 2018).
We conducted experiments on benchmark vision and text datasets, namely simulated noisy MNIST , CIFAR-10 and NEWS datasets.
Empirical results demonstrated that, under both extremely noisy labels (i.e., 45% and 50% of noisy labels) and low-level noisy labels (i.e., 20% of noisy labels), the robustness of two upgraded approaches (by Pumpout) is obviously superior than that of original approaches.
P UMPOUT M EETS N OISY S UPERVISION
Meta algorithm.
The original idea of Pumpout is to actively squeeze out the negative effects of noisy labels from the training model, instead of passively forgetting these effects.
Intuitively, take “heart-broken” story as a supportive example.
When you are disappointed in your former love, the best way to recover is to embrace the new love, which actively squeezes out the hurt from your mind, instead of repeatedly self-blaming.
In the design of meta algorithm, we should consider how our meta algorithm can simultaneously beneﬁt multiple orthogonal approaches in the area of deep learning with noisy labels, such as training on selected instances (Jiang et al., 2018), estimating the noise transition matrix (Patrini et al., 2017) and designing regularization (Miyato et al., 2016).
For this purpose, we generalize noisy labels into “not-ﬁtting” labels, and generalize clean labels into “ﬁtting” labels (details of the ﬁtting condition will be discussed in Q1 below).
In the high level, the meta algorithm Pumpout is to train deep neural networks by stochastic gradient descent on “ﬁtting” labels, and train deep neural networks by scaled stochastic gradient ascent on “not-ﬁtting” labels.
In the low level, the proposed Algorithm 1 is named Pumpout.
Speciﬁcally, we maintain deep neural network f (with parameter w f ).
When a single point { x i , y i } is sequentially selected from noisy set D (step 3), we ﬁrst check whether { x i , y i } is ﬁtting the discriminative condition or not.
If yes, we conduct stochastic gradient descent typically (step 4); otherwise, we conduct scaled ( γ ) stochastic gradient ascent (step 5), which erases the negative effects of “not-ﬁtting” labels. These “not-ﬁtting” labels hinder us to train a robust model.
The abstract algorithm arises three important questions.
Algorithm
//noisy end end end
Three important questions.
Q1.
What is the ﬁtting condition?
Q2.
Why do we need gradient ascent on non-ﬁtting data, in addition to gradient descent on ﬁtting data?
Q3. Why do we need to scale the stochastic gradient ascent on non-ﬁtting data?
To answer the ﬁrst question , we need to emphasize a view that orthogonal approaches require different ﬁtting conditions.
Intuitively, if a single point { x i , y i } satisﬁes a discriminative ﬁtting condition, it means that our training model will regard this data point as a useful knowledge, and ﬁtting on this point will beneﬁt training the robust model.
Conversely, if a single point { x i , y i } does not satisfy the discriminative ﬁtting condition, it means that, our training model will regard this data point as useless knowledge, and want to erase the negative effects of this point actively.
To instantiate the ﬁtting condition, we provide two concrete cases in Algorithm 2 and Algorithm 3, respectively.
The above answer motivates our second question : why cannot we only conduct stochastic gradient descent on ﬁtting data points (step 4).
In other words, can we remove scaled stochastic gradient ascent (step 5) in Algorithm 1?
In this case (removing step 5), our algorithm degenerates to training only on selected instances.
However, once some of the selected instances are found to be false positives 1 , our training model will ﬁt on them, and thus the negative effects will inevitably occur (i.e., degrading the generalization (test accuracy)).
Instead of passively forgetting these negative effects (i.e., further training over many epochs), we hope to actively squeeze out the negative effects from the training model by using scaled stochastic gradient ascent (step 5).
Lastly, the third question closely connects with the second one.
Namely, why do we need scaled instead of ordinary stochastic gradient ascent?
The answer can be intuitively explained.
Assume that we view stochastic gradient ascent as correction to “not-ﬁtting” labels, and view 0 ≤ γ ≤as a scale parameter.
When γ = 1 , our Pumpout will squeeze out the negative effects with full fast rate; while when γ = 0 , our Pumpout will not squeeze out any negative effects.
Both cases are not optimal, since we empirically ﬁnd that the best performance is usually chosen when 0 < γ < 1 by using the validation set (Section 4).
For the ﬁrst case, the fast squeezing rate will negatively affect the convergence of our algorithm.
For the second case, no squeezing rate will inevitably let deep neural networks memorize some “not-ﬁtting” labels, which degrades their generalization (Zhang et al., 2017; Neyshabur et al., 2017).
P UMPOUT B ENEFITS S TATE - OF - THE -A RT A LGORITHMS
In this section, we employ the idea of Pumpout to upgrade MentorNet and Backward Correction as follows.
First, we brieﬂy introduce the background of MentorNet and Backward Correction. Then, by using Pumpout, we propose upgraded MentorNet (Pumpout SL ) and upgraded Backward Correction (Pumpout BC ). Lastly, we explain the relations between MentorNet (Backward Correction) and Pumpout SL (Pumpout BC ).
MentorNet. To handle noisy labels, an emerging direction focuses on training only on selected instances (Jiang et al., 2018; Ren et al., 2018; Han et al., 2018c), which is free of estimating the noise transition matrix, and also free of the class-conditional noise assumption (Natarajan et al., 2013).
These works try to select clean instances out of the noisy ones, and then use them to update the network.
Among those works, a representative method is MentorNet (Jiang et al., 2018), which employs the small-loss trick.
Speciﬁcally, MentorNet pre-trains an extra network, and then uses the extra network for selecting small-loss instances as clean instances to guide the training. However, the idea of MentorNet is similar to the self-training approach (Chapelle et al., 2009), thus MentorNet inherits the same drawback of accumulated error caused by the sample-selection bias. Pumpout SL . Algorithm 2 represents the upgraded MentorNet using Pumpout approach (denoted as Pumpout SL ), where MentorNet uses the small-loss trick.
Speciﬁcally, we maintain deep neural network f (with parameter w f ).
When a mini-batch ¯ D is formed (step 3), we ﬁrst let f select a small proportion of instances in this mini-batch ¯ D s that have small training losses (step 4).
The number of instances is controlled by R ( T ) , and f only samples R ( T ) percentage of instances out of the minibatch.
More importantly, we let f select a proportion of instances in this mini-batch ¯ D b that have big training losses (step 5).
The number of instances is controlled by 1 − R ( T ) , and f only samples 1 − R ( T ) percentage of instances out of the mini-batch.
Then, we conduct stochastic gradient descent on small-loss instances ¯ D s (step 6); while we conduct scaled stochastic gradient ascent on big-loss instances
¯ D b (step 7), which actively erases the negative effects of big-loss instances.
The update of
R ( T ) (step 8) follows Han et al.
(2018c), in which extensive discussion has been conducted.
Algorithm 2 Pumpout SL .
The ﬁtting condition is whether a point belongs to small-loss instances.
noisy
Relations between MentorNet and Pumpout SL .
If we remove step 5 and step 7 in Algorithm 2, Pumpout SL algorithm will be reduced to the core version of MentorNet, namely self-paced
MentorNet.
It means that Pumpout SL algorithm is more aggressive than MentorNet in essence.
Namely, Pumpout SL conducts not only stochastic gradient descent on small-loss instances (like MentorNet), but also scaled stochastic gradient ascent on big-loss instances.
3. U PGRADE B ACKWARD C ORRECTION
Backward Correction and its non-negative version.
To handle noisy labels, the other popular direction focuses on estimating the noise transition matrix (Goldberger & Ben-Reuven, 2017; Patrini et al., 2017; Han et al., 2018b).
Among those works, a representative method is Backward Correction.
Speciﬁcally, Patrini et al.
(2017) leveraged a two-step solution to estimate the noise transition matrix heuristically.
Then they employed the estimated matrix to correct the original loss, and robustly train a deep neural network based on the new loss function.
Theorem 1 (Backward Correction, Theorem 1 in (Patrini et al., 2017)) Suppose that the noise transition matrix T is non-singular, where T ij =
Pr(˜ y
=
j | y
= i ) given that noisy label ˜
y
=
j is ﬂipped from clean label y
= i .
Given loss (cid:96) and network parameter w f , Backward Correction is deﬁned as
[Equation]
Then, corrected loss is unbiased, namely,
[Equation]
Remark 1 Backward Correction operates on the loss vector directly.
It is unbiased.
LHS of Eq.
(2) draws from noisy labels, and RHS of Eq.
(2) draws from clean labels.
Note that the corrected loss is differentiable, but not always non-negative (van Rooyen & Williamson, 2018).
If the model being trained is ﬂexible, such as a deep neural network, the backward loss correction will lead to negative risks, and the hazardous aspect is to yield an over-ﬁt issue.
Motivated by Kiryo et al.
(2017), we should conduct a non-negative correction again
Theorem 2 (Non-negative Backward Correction) Suppose that the noise transition matrix T is nonsingular, where T ij = Pr(˜ y
=
j | y
= i ) given that noisy label ˜
y
=
j is ﬂipped from clean label y
= i .
Given loss (cid:96) and network parameter w f , Non-negative Backward Correction is deﬁned as
[Equation]
where 1 k × 1 .
Then, the corrected loss (cid:96) ← m ( x , y ; w f ) is non-negative.
Remark 2 (cid:96) ← m ( x , y ; w f ) is a non-negative scalar.
Our key claim is to overcome the over-ﬁt issue by non-negative correction.
However, the above non-negative correction is passive, since max operator means stopping gradient computation on negative-risk instances.
This correction may not achieve the optimal performance.
Namely, when 1 (cid:62) T − 1 (cid:96) ( x , y ; w f ) ≥ 0 , we conduct stochastic gradient descent; otherwise, we do not perform the operation of stochastic gradient.
To propose an aggressive non-negative correction, we reverse the gradient computation at negative-risk instances.
Speciﬁcally, we use the Pumpout approach to improve Non-negative Backward Correction.
Namely, when 1 (cid:62) T − 1 (cid:96) ( x , y ; w f ) ≥ 0 , we conduct stochastic gradient descent typically; when 1 (cid:62) T − 1 (cid:96) ( x , y ; w f ) ≤ 0 , we conduct scaled stochastic gradient ascent.
This brings our Algorithm 3.
Pumpout BC .
Algorithm 3 represents the upgraded Backward Correction using the Pumpout approach (denoted as Pumpout BC ), where Backward Correction is deﬁned in Theorem 1.
If the model being trained is ﬂexible (i.e., a deep neural network), Backward Correction will lead to negative risks (Patrini et al., 2017), which subsequently yields an over-ﬁt issue.
To mitigate this issue, we maintain deep neural network f (with parameter w f ).
When a single point { x i , y i } is sequentially selected from the j -th mini-batch ¯ D (step 5), we ﬁrst compute the temporary gradient g t at this point (step 6).
If Backward Correction produces a positive risk at this point, namely 1 (cid:62) T − 1 (cid:96) ( x i , y i ; w f ) ≥ β ≥ 0 (deﬁnitions of T and (cid:96) are in Theorem 1), we accumulate gradient G a by the gradient descent (step 7); otherwise, we accumulate gradient G a by the scaled gradient ascent (step 8), and this step erases the negative effects of negative-risk instances. Lastly, we average the accumulated gradient (step 9) and update parameter w f by stochastic optimization (step 10). Algorithm 3 Pumpout BC .
The ﬁtting condition is whether a point satisﬁes 1 (cid:62) T − 1 (cid:96) ( x i , y i ; w f ) ≥ β .
Select
//temp
gradient endelse
Relations between Non-negative Backward Correction and Pumpout BC .
If we remove line 8 in
Algorithm 3, Pumpout BC algorithm will be reduced to Non-negative Backward Correction.
It means Pumpout BC algorithm is an aggressive version of Non-negative Backward Correction. Namely,
Pumpout BC conducts not only stochastic gradient descent on nonnegative-risk instances, but also scaled stochastic gradient ascent on negative-risk instances to erase their negative effects.
E XPERIMENTS
Datasets.
We verify the effectiveness of our Pumpout approach on three benchmark datasets, including two vision datasets and one text dataset.
MNIST , CIFAR10 and NEWS are used here (Table 1), as these data sets are popularly used for evaluation of noisy labels in the literature (Reed et al., 2015; Goldberger & Ben-Reuven, 2017; Patrini et al., 2017; Kiryo et al., 2017).
[Table]
Since all datasets are clean, following (Reed et al., 2015; Patrini et al., 2017), we need to corrupt these datasets manually by the noise transition matrix T , where T ij =
Pr(˜ y
=
j | y
= i ) given that noisy ˜
y is ﬂipped from clean y .
Assume that the matrix T has two representative structures (Figure 1): , where you may make mistake only within very similar classes the adjunct positions
Symmetry Van Rooyen et al
Their precise is in Appendix A.
(a) Pair ( (cid:15) = 45% ). (b) Symmetry ( (cid:15) = 50% ).
This paper ﬁrst veriﬁes whether Pumpout can signiﬁcantly improve the robustness of representative methods on extremely noisy supervision, the noise rate (cid:15) is chosen from { 0 . 45 , 0 . 5 } .
Intuitively, this means almost half of the instances have noisy labels.
Note that, the noise rate > 50% for pair ﬂipping means over half of the training data have wrong labels that cannot be learned without additional assumptions.
In addition to extremely noisy settings, we also verify whether Pumpout can signiﬁcantly improve the robustness of representative methods on low-level noisy supervision, where (cid:15) is set to 0 .
Note that pair case is much harder than symmetry case.
In Figure 1(a), the true class only has 10% more correct instances over wrong ones.
However, the true has 37 .
5% more correct instances in Figure 1(b).
Meanwhile, similarly to (Reed et al., 2015; Goldberger &
Ben-Reuven, 2017; Jiang et al., 2018), we did not make any implicit assumption behind Pumpout.
Baselines.
To verify the efﬁcacy of Pumpout, we compare two orthogonal approaches in deep learning with noisy labels.
The ﬁrst set (SET1) comparison is to check whether Pumpout can improve the robustness of MentorNet.
(i) MentorNet (Jiang et al., 2018).
(ii) Pumpout SL (Algorithm 2).
The second set (SET2) comparison is to check whether Pumpout can improve the robustness of
Backward Correction.
(i) Backward Correction (Patrini et al., 2017) (denoted as “BC”, Theorem 1).
(ii).
Non-negative backward correction (denoted as “nnBC”, Theorem 2).
(iii) Pumpout BC (Algorithm 3).
As a simple baseline, we also compare with the standard deep neural network that directly learns on the noisy training set (denoted as “Standard”).
Note that, the choice of two baselines is to justify whether Pumpout can beneﬁt representative state-of-the-art algorithms.
The readers are encouraged to upgrade other methods, such as
For the fair comparison, we implement all methods with default parameters by PyTorch, and conduct all the experiments on a NVIDIA K80 GPU.
Standard CNN is used with Leaky ReLU activation function (Maas et al., 2013); ResNet is used with ReLU activation function; and MLP is used with
Softsign activation function (Glorot & Bengio, 2010).
The detailed architectures are in Appendix B. Namely, we used the 9-layer CNN (Miyato et al., 2016; Laine & Aila, 2017) for MNIST , ResNet32 (He et al., 2016) for CIFAR-10 , and 3-layer MLP (Kiryo et al., 2017) for NEWS , since the network structures we used here are standard test bed for weakly-supervised learning.
For all datasets, Adam optimizer (momentum=0.9) with an initial learning rate of 0.001, the batch size is set to 128 and runs for 200 epoch.
Besides, dropout and batch-normalization are also used.
Note that, the focus of our paper is to explore the efﬁcacy of Pumpout.
Therefore, we use Adam optimizer in all experiments for fair comparison without using data augmentation trick (Zhang & Sabuncu, 2018; Ma et al., 2018).
Experimental setup.
For SET1 , the most important parameter of our Pumpout SL and MentorNet is R ( T ) .
Here, we assume the noise level (cid:15) is known and set R ( T )
= 1 − τ · min ( T/T k , 1) with
T k = 10 and τ = (cid:15) .
If (cid:15) is not known in advanced, (cid:15) can be inferred using validation sets (Liu &
Tao, 2016).
The choices of R ( T ) and τ follows Han et al.
(2018c).
Note that R ( T ) only depends on the memorization effect of deep networks but not any speciﬁc datasets.
For SET2 , the most important parameters of our Pumpout BC and nnBC are β and γ respectively.
Speciﬁcally, the degree of tolerance is controlled by β ( β ≥ 0 ), and the scale of gradient ascent is controlled by γ ( 0 ≤ γ ≤ 1 ).
The choices of β and γ follows Kiryo et al.
Namely, for β , we directly set it to zero and γ is chosen among { 0 , 0 . 001 , 0 . 005 , 0 . 01 , 0 . 05 , 0 . 1 , 0 . 5 , 1 } via a validation set.
This paper provides two upgraded approaches to train deep neural networks robustly under noisy labels.
Thus, our goal is to classify the clean instances as accurately as possible, and the measurement for both SET1 and SET2 is the test accuracy, i.e., test accuracy =
(# of correct predictions) /
(# of test dataset) .
Besides, for SET1 , we also use the label precision in each mini-batch, i.e., label precision =
(# of clean labels) /
(# of all selected labels) .
Speciﬁcally, we sample R ( T ) of small-loss instances in each mini-batch, and then calculate the ratio of clean labels in the small-loss instances.
Intuitively, higher label precision means less noisy instances in the mini-batch after sample selection; and the algorithm with higher label precision is also more robust to the label noise.
All experiments are repeated ﬁve times.
In each ﬁgure, the error bar for standard deviation is highlighted as a shade.
Before delving into Section 4.1 and 4.2, there are two important points to be emphasized.
First, the memorization effects of deep networks (Arpit et al., 2017) means that standard deep networks will ﬁt clean instances ﬁrst, then overﬁt noisy instances gradually.
These effects will inevitably degrade the generalization performance (i.e., test accuracy).
Second, Pumpout SL may not suffer from or greatly alleviate the accumulated error in MentorNet, since Pumpout SL actively squeezes out the negative effects of noisy labels from the training model, instead of passively forgetting these effects.
4. R ESULTS OF P UMPOUT SL AND M ENTOR N ET
MNIST .
In Figure 2, we show test accuracy (top) and label precision (bottom) vs number of epochs on MINIST dataset.
In all three plots, we can clearly see the memorization effects of deep networks (Arpit et al., 2017), i.e., test accuracy of Standard ﬁrst reaches a very high level and then gradually decreases.
Thus, a good robust training method should stop or alleviate the decreasing process.
On this point, our Pumpout SL almost stops the decreasing process in the easier Symmetric50% and Symmetric-20% cases.
Meanwhile, compared to MentorNet, our Pumpout SL alleviates the decreasing process in the hardest Pair-45% case.
Thus, Pumpout SL consistently achieves the higher accuracy over MentorNet.
To explain such good performance, we plot label precision (bottom).
Compared to Standard, we can clearly see that both Pumpout SL and MentorNet can successfully pick clean instances out.
However, our Pumpout SL achieves the higher label precision on not only the easier Symmetric-50% and
Symmetric-20% cases, but also the hardest Pair-45% case.
This shows our approach is better at ﬁnding clean instances due to the usage of scaled stochastic gradient ascent.
CIFAR-10 .
Figure 3 shows test accuracy and label precision vs number of epochs on CIFARdataset.
Again, on test accuracy, we can see Pumpout SL strongly stops the memorization effects of deep networks.
More importantly, on the easier Symmetric-50% and Symmetric-20% cases, it works better and better along with the training epochs.
On label precision, while Standard fails to ﬁnd clean instances, both Pumpout SL and MentorNet can do this.
However, due to the usage of scaled stochastic gradient ascent, Pumpout SL is stronger and ﬁnd more clean instances. [Figure] [Figure] NEWS .
Figure 4 shows test accuracy and label precision vs number of epochs on NEWS dataset.
On test accuracy, we can see Pumpout SL stops the memorization effects of deep networks to some degree.
Especially on the harder Pair-45% and Symmetric-50% cases, Pumpout SL obviously achieves the higher accuracy over MentorNet along with the training epochs. On label precision, while Standard fails to ﬁnd clean instances again, Pumpout SL can achieve this especially on the hardest case due to the usage of scaled stochastic gradient ascent. [Figure] 4. R ESULTS OF P UMPOUT BC AND NN BC
MNIST .
Figure 5 shows test accuracy vs number of epochs on MNIST dataset.
In all three plots, we can see the memorization effects of deep networks, i [Figure] e., test accuracy of Standard ﬁrst reaches a very high level and then gradually decreases.
However, our Pumpout BC fully stops the decreasing process in the hardest Pair-45% case.
Meanwhile, in the easier Symmetric-50% and Symmetric-20% cases, our Pumpout BC works better and better along with the training epochs though it ﬂuctuates. Moreover, our Pumpout BC ﬁnally achieves the higher accuracy over both BC and nnBC.
[Figure]
CIFAR-10 .
Figure 6 shows test accuracy vs number of epochs on CIFAR-10 dataset.
Again, in all three plots, we can see the memorization effects of deep networks.
However, in the hardest Pair-45% case and the easiest Symmetry-20% case, our Pumpout BC overcomes this issue and works better and better along with the training epochs though it ﬂuctuates slightly.
Speciﬁcally, in the hardest case, our Pumpout BC obviously achieves the higher accuracy over both BC and nnBC.
Meanwhile, in the Symmetric-50% case, our Pumpout BC becomes comparable with other methods. NEWS . Figure 7 shows test accuracy vs number of epochs on NEWS dataset.
In all three plots, we can see the memorization effects of deep networks again.
However, our Pumpout BC fully stops the decreasing process in two harder Pair-45% and Symmetry-50% cases, and alleviates the decreasing
[Figure]
process in one easier Symmetry-20% case.
Meanwhile, in the hardest Pair-45% case, our Pumpout BC works better and better along with the training epochs.
In this hardest case, our Pumpout BC ﬁnally achieves the higher accuracy over both BC and nnBC, although its accuracy falls behind BC in the ﬁrst 90 epochs and nnBC in the ﬁrst 50 epochs.
Besides, in two symmetry cases, our Pumpout BC obviously achieves the higher accuracy over both BC and nnBC
[Figure]
C ONCLUSION
This paper presents a meta algorithm called Pumpout, which signiﬁcantly improves the robustness of state-of-the-art methods under noisy labels.
Our key idea is to squeeze out the negative effects of noisy labels actively from the training model, instead of passively forgetting these effects.
The realization of Pumpout is to train deep neural networks by stochastic gradient descent on “ﬁtting” labels; while train deep neural networks by scaled stochastic gradient ascent on “not-ﬁtting” labels.
To demonstrate the efﬁcacy of Pumpout, based on MentorNet and Backward Correction, we design two upgraded versions called Pumpout SL and Pumpout BC , respectively.
The experimental results show that, both updated approaches can train deep models more robustly over previous ones.
In future, we can extend our work in the following aspects.
First, we can leverage Pumpout approach to train deep models under another weak supervision, e.g., complementary labels (Ishida et al., 2017).
Second, we should investigate the theoretical guarantees for Pumpout approach.
Third, we should adapt our Pumpout approach to several much harder noise cases, for example, instance-dependent noise datasets (i.e., Open Images (Veit et al., 2017) or Clothing1M (Xiao et al., 2015)).
A D EFINITION OF N OISE
The deﬁnition of transition matrix T is as follow, where (cid:15) is the noise rate and n is the number of the classes.
Pair ﬂipping: T =   1 − (cid:15) (cid:15) . . .
1 − (cid:15) (cid:15) ... 1 − (cid:15) (cid:15) (cid:15) . . .
1 − (cid:15) 
 ,
Symmetry ﬂipping: T =   1 − (cid:15) (cid:15)n −. . .
(cid:15)n −(cid:15)n − 1 (cid:15)n −1 − (cid:15) (cid:15)n −. . .
(cid:15)n − 1 .
(cid:15)n −. . .
(cid:15)n −1 − (cid:15) (cid:15)n − 1 (cid:15)n −(cid:15)n −. . .
(cid:15)n −1 − (cid:15) 
 .
B N ETWORK S TRUCTURES
For MNIST , 32 × 32 gray image, the structure is as follows.
We also summarize it into Table 2.
(1*28*28)-[C(3*3,128)]*2-maxpool(2*2,2)-dropout(0.25)-[C(3*3,256)]*3-maxpool(2*2,2)- dropout(0.25)-C(3*3,512)-C(3*3,256)-C(3*3,128)-avgpool(1*1)-128-10, where the input is a 28*28 image, C(3*3,128) means 128 channels of 3*3 convolutions followed by LReLU (negative slop=0.01), maxpool(2*2,2) means max pooling (kernel size=2, stride=2), avgpool(2*2) means average pooling (kernel size=2), [.]*n means n such layers, etc.
Batch normalization was applied before LReLU activations.
[Table]
For CIFAR-10 , 32 × 32 RGB image, the structure is ResNet-32.
For NEWS , the structure is 3-layer.
We also summarize it into Table 3.
[Table]