S EARCHING TO E XPLOIT M EMORIZATION E FFECT IN L EARNING FROM NOISY L ABELS
Anonymous authors
Paper under double-blind review
A BSTRACT
Sample-selection approaches, which attempt to pick up clean instances from the noisy training data set, have become one promising direction to robust learning from noisy labels.
These methods all build on the memorization effect, which means deep networks learn easy patterns ﬁrst and then gradually over-ﬁt the training data set.
In this paper, we show how to properly select instances so that the training process can beneﬁt the most from the memorization effect is a hard problem.
Speciﬁcally, memorization can heavily depend on many factors, e.g., data set and network architecture.
Nonetheless, there still exists general patterns of how memorization can occur.
These facts motivate us to exploit memorization by automated machine learning (AutoML) techniques.
First, we design an expressive but compact search space based on observed general patterns.
Then, we propose to use the natural gradient-based search algorithm to efﬁciently search through space.
Finally, extensive experiments on both synthetic data sets and benchmark data sets demonstrate that the proposed method can not only be much efﬁcient than existing AutoML algorithms but can also achieve much better performance than the state-of-the-art approaches for learning from noisy labels.
I NTRODUCTION
Learning with deep neural networks has enjoyed huge empirical success in recent years across a wide variety of tasks, from image processing to speech recognition, and from language modeling to recommender system (Goodfellow et al., 2016).
However, their success highly counts on the availability of well-annotated and big data, which is barely available for real-world applications.
Instead, what we are facing with in practice are large data sets which are collected from crowdsourcing platforms or crawled from the Internet, thus containing many noisy labels (Li et al., 2017b; Patrini et al., 2017).
Besides, due to the vast learning capacity of deep networks, they will eventually over-ﬁt on these noisy labels, leading to poor predicting performance, which can be worse than that obtained from simple models (Zhang et al., 2016; Arpit et al., 2017).
To reduce negative effects from noisy labels, many methods have been proposed (Sukhbaatar et al., 2015;
Reed et al., 2015; Patrini et al., 2017; Ghosh et al., 2017; Malach & Shalev-Shwartz, 2017) .
Recently, a promising direction is training networks only on selected instances that are more likely to be clean (Jiang et al., 2018; Han et al., 2018b; Ma et al., 2018;
Yu et al., 2019; Wang et al., 2019).
Intuitively, as the training data becomes less noisy, better performance can be obtained.
Among those works, the representative methods are MentorNet (Jiang et al., 2018) and Co-teaching (Han et al., 2018b; Yu et al., 2019), they take small-loss samples in each mini-batch as clean instances.
Speciﬁcally, MentorNet pre-trains an extra network, and then uses the extra network for selecting clean instances to guide the training.
When the clean validation data is not available, MentorNet has to use a predeﬁned curriculum (Bengio et al., 2009).
Co-teaching is an improvement over MentorNet, it simultaneously maintains two networks which have identical architectures during the training process.
And in each mini-batch of data, each network is updated using the other network’s small-loss instances.
To the success of these sample-selection methods, the memorization effect of deep networks (Zhang et al., 2016; Arpit et al., 2017) is the crux.
Memorization happens widely in various architectures of deep network, e.g., multilayer perceptron (MLP) and convolutional neural network (CNN).
Specifically, it means that deep networks tend to learn easy and correct patterns ﬁrst and then over-ﬁt on (possibly noisy) training data set (see Fig.1(a)-(b)).
Thus, when learning with noisy labels, while the validation loss will ﬁrst increase and then signiﬁcantly decrease, the training loss will continuously get smaller with more training epochs.
Due to such effect, sample-selection methods can learn correct patterns at early stage and then use the obtained discriminative ability to ﬁlter out corrupted instances in subsequent training epochs (Jiang et al., 2018; Han et al., 2018b; Chen et al., 2019).
While the memorization effect is critical to the success of sample-selection methods, however, how to properly exploit it is not addressed in the literature. And trivial attempts can easily lead to even worse performance than standard deep networks (Han et al., 2018b).
Some recent endeavors seek to evade from this problems by integrating with other auxiliary information, e.g., a small clean subset is used in (Ren et al., 2018), and knowledge graphs are utilized in (Li et al., 2017b).
In this paper, motivated by the success of automated machine learning (AutoML) on designing data-dependent models (Hutter et al., 2018), and the fact that memorization heavily depends on many factors (Zhang et al., 2016; Arpit et al., 2017), we propose to exploit memorization effects automatically using AutoML techniques.
Contributions are summarized as follows:
• First, to have an in-depth understanding of why it is difﬁcult to tune sample-selection methods with good performance. We examine behaviors of memorization effect from multiple perspectives.
We ﬁnd that, while there exist general patterns in how memorization occurs with the training process (see Fig.1(a)-(b)), it is hard to quantize to which extend such effect can happen (see Fig.1(b)-(f)).
Especially, memorization can be affected by many factors, e.g., data sets, network architectures, and the choice of the optimizers.
It is exactly such complex dependency make the design of proper sample-selection rules a hard problem, which motivates us to solve the problem by AutoML techniques.
• To make good use of AutoML techniques, we then derive an expressive search space for exploiting memorization, which is from the above observations,
i
e., the curvature of how many instances need to be sampled during iterating should be similar with the inverse of the learning curve on the validation set.
Such a space is not too huge since it has only a few variables, thus allows subsequent algorithms converging fast to promising candidates.
• Then, to design an efﬁcient algorithm, we show the failure of gradient-based methods and the inefﬁciency of derivative-free methods.
These motivate us to take a probabilistic view of the search problem and adopt natural gradient descent (Amari, 1998; Pascanu & Bengio, 2013) for optimization.
The designed algorithm can effectively address above problems and is signiﬁcantly faster than other popular search algorithms.
Finally, we conduct extensive experiments on both synthetic, benchmark, and real data sets, under various settings using different network architectures.
These experiments demonstrate that the proposed method can not only be much more efﬁcient than existing AutoML algorithms, but also can achieve much better performance than the state-of-the-art sample-selection approaches designed by humans.
Besides, we further visualize and explain the searched functions, which can also help design better rules to control memorization effects in the future.
R ELATED WORK 2. L EARNING FROM N OISY L ABELS
The mainstream research focuses on class-conditional noise (CCN) (Angluin & Laird, 1988), where the label corruption is independent of features.
Generally, recent methods for handling CCN model can be classiﬁed into three categories.
The ﬁrst one is based on the estimation of transition matrix, which tries to capture how correct labels ﬂip into wrong ones (Sukhbaatar et al., 2015;
Reed et al., 2015; Patrini et al., 2017; Ghosh et al., 2017).
These methods then use the estimated matrix to correct gradients or loss during training.
However, they are fragile to heavy noise and unable to handle many classes (Han et al., 2018b).
The second type is the regularization approach (Miyato et al., 2016;
Laine & Aila, 2017; Tarvainen & Valpola, 2017).
Although regularization approach can achieve a satisfying performance, it is still an incomplete approach since (Jiang et al., 2018) shows that it can only delay the overﬁtting progress rather than avoid it, i.e. given enough training time, it can still ﬁt the noisy data completely.
Thus, it requires much domain knowledge to determine the appropriate number of training epochs in order to prevent overﬁtting.
The last one is sample-selection approach, which attempts to reduce negative effects from noisy labels by selecting clean instances during training.
The recent state-of-the-art method is also built on sample-selection approach (Jiang et al., 2018; Han et al., 2018b; Malach & Shalev-Shwartz, 2017; Yu et al., 2019).
Active learning (Settles, 1994) is a closely related method, which iteratively selects unlabeled samples with high high-conﬁdent predictions into the training data set.
Thus, to do active learning, we need to obtain a classiﬁer of which the performance is good enough.
As a result, active learning is not applicable for directly learning from noisy labels here.
A promising criteria to select “clean instances” is to pick up instances that has relatively small losses in each mini-batch (Jiang et al., 2018; Han et al., 2018b).
The fundamental property behind these methods is the memorization effect of deep networks (Zhang et al., 2016; Arpit et al., 2017), which means deep networks can learn simple patterns ﬁrst and then start to over-ﬁt.
Such effect helps classiﬁers set up discriminate ability in the early stage, then make clean instances more likely to have smaller loss that those noisy ones.
The general framework of sample-selection approach is in Alg.1.
Speciﬁcally, some small-loss instances ¯ D f are selected from the mini-batch ¯ D in step 5.
These “clean” instances are then used to update network parameters in step 6.
The R ( t ) in step 8, which controls how many instances to be kept in each epoch, is the most important hyper-parameter as it explicitly exploits the memorization effect (Han et al., 2018b; Jiang et al., 2018; Yu et al., 2019). do
9: end for However, it is hard to exactly determine how much proportion of small-loss samples should be selected in each epoch (Jiang et al., 2018;
Ren et al., 2018).
As will be discussed in Sec.3.1, due to various practical usages issues, to which extend memorization effect can happen is hard to quantize.
Thus, performance obtained from existing solutions are far from desired, and we are motivated to solve this issue by AutoML.
2. A UTOMATED M ACHINE L EARNING (A UTO ML)
Automated machine learning (AutoML) (Hutter et al., 2018) has recently exhibited its power in easing the usage of and designing better machine learning models.
Basically, AutoML can be regarded as a black-box optimization problem where we need to efﬁciently and effectively search for hyperparameters or designs for the underlying learning models evaluated by the validation set.
Regarding the success of AutoML, there are two important perspectives (Feurer et al., 2015; Zoph & Le, 2017; Xie & Yuille, 2017; Bender et al., 2018):
• Search space : First, it needs to be general enough, which means it should cover existing models as special cases.
This also helps experts better understand limitations of existing models and thus facilitate future researches.
However, the space cannot be too general, otherwise searching in such a space will be too expensive.
• Search algorithm : Optimization problems in AutoML are usually black-box.
Unlike convex optimization, there is no universal and efﬁcient optimization tools.
Once the search space is determined, domain knowledge should also be explored in the design of search algorithm so that good candidates in the space can be identiﬁed efﬁciently.
Search space is domain-speciﬁc and needs to specially designed for every AutoML problem. There are two types of search algorithms popularly used.
The ﬁrst one is derivative-free optimization methods, it is usually used for searching in a general search space, e.g., reinforcement learning (Zoph & Le, 2017; Baker et al., 2017), genetic programming (Escalante et al., 2009; Xie & Yuille, 2017), and Bayes optimization (Feurer et al., 2015; Snoek et al., 2012).
More recently, gradientbased methods, which alternatively update parameters and hyper-parameters, have been developed as more efﬁcient replacements for derivative-free optimization methods on some AutoML problems, e.g., neural network architecture search (Liu et al., 2019; Akimoto et al., 2019; Xie et al., 2018). However, existing AutoML techniques cannot be directly used in exploiting memorization here.
First, we need to carefully deﬁne a domain-speciﬁc space. Besides, we will also show existing algorithms are neither not applicable nor too slow. This motivates us to propose a new algorithm based on natural gradient.
T HE P ROPOSED M ETHOD
Here, we ﬁrst give a closer look at why it is difﬁcult to exploit the memorization effect (Sec.3.1).
This also helps us identify key observations on how memorization can happen.
This observation subsequently enables us to design an expressive but compact search space (Sec.3.2), and motivates us to use natural gradient method (Amari, 1998; Ollivier et al., 2017) that can generate gradients in the parameterized space for efﬁcient optimization (Sec.3.3).
3. D IFFICULTIES IN E XPLOITING M EMORIZATION
As in Sec.2.2, an important challenge in designing search spaces is to balance the size (or dimension) and the expressive ability of the search space.
An overly constrained search space may not contain candidates that have a satisfying performance, whereas too large search spaces will be difﬁcult to effectively search.
All these require us to have an in-depth understanding of the difﬁcult in designing of R ( T ) , which is at step 8 of Alg.1 and is used to exploit the memorization effect.
Thus, we are motivated to look at factors which can affect the memorization of deep networks, and to seek patterns from the resultant inﬂuences.
Speciﬁcally, we examine memorization when data sets, architectures, or optimizers are changed.
Results are in Fig.1.
From these ﬁgures, we can observe:
• Foot-stone of space design : There exists a general pattern among all cases, i.e., all models’ test accuracy will ﬁrst increase, then decrease.
[Figure]
Fig.(c-f) show memorization not only heavily depends on data sets, and also many factors (details of experiment are in Appendix A.1).
This general pattern is consistent with that in the literature (Zhang et al., 2016; Arpit et al., 2017;
Tanaka et al., 2018; Han et al., 2018b).
It is also the key domain knowledge for designing an expressive and compact search space, which makes the subsequent search possible.
However, the more important observation is:
• Need of AutoML :
Curvature can be signiﬁcantly affected by these factors.
When the peak will appear (i e., stop learning from simple patterns and start to over-ﬁt), and to which extend the performance will drop from peak (i e., over-ﬁt on noisy
labels) are
all hard to quantize This observation shows a great variety in appearances of the memorization effect, which further poses a signiﬁcant need for automated exploiting of such effect.
Besides, it is hard to know in advance what learning curve will exactly look like in advance and thus impossible to manually design before learning is performed.
3. S EARCH S PACE
Recall that in step 8 of Alg.1, R ( t ) controls how many instances are kept in each mini-batch, and we want to exploit the memorization effect through the variation of R ( t ) .
Based on the ﬁrst empirical observation in Sec.3.1, our design of R ( t ) ﬁrst should satisfy:
• Curvature : R ( t ) should be similar with the inverse of the learning curve.
In other words, R ( t ) should ﬁrst drop, then (possibly) rise.
The reason behind this constraint is as follows: Since the learning curve represents the model’s accuracy, when it rises, we should drop more large-loss samples as the large loss is more likely the result of noisy labels than model’s misclassiﬁcation.
And when the learning curve falls, we should drop less to help the model learn more.
• Range : R ( t ) ∈ [0 , 1] for t ∈ { 1 , . .
, T } with R (1) = 1 .
Since R ( t ) denotes the proportion of selected instances, it is naturally in [0 , 1] .
Besides, at the beginning, dropping small-loss samples will be same as dropping samples randomly, and we need to pass sufﬁcient number of instances such that the model can learn from patterns.
functions deﬁnition decreasing power (pow-d) 1 / (1+ bt ) a exponential (exp-d)
[Figure]
increasing power
[Figure]
[Figure]
As R ( t ) itself can be seen as a function which takes t as input and outputs a scalar, we can use some simple f i as basis functions to approximate the complicate R ( t ) .
We list some simple monotonously decreasing or increasing functions in Fig.2 and express R ( t ) as a linear combination of them:
[Equation]
where the α i is the hyper-parameters controlling each term (for example a and b in Fig.2).
Let the clean validation set be G val , F be Alg.1 with network parameter w and sampling rule α
( R ( t ) is parameterized by Eqa.(1)).
M ( w , G val ) measures the validation performance with the model parameter w .
Thus, memorization effect can be automatically exploited by solving the following problem
[Equation]
Then, the optimal curvature is derived from R ( t ) =
(cid:80) ki
=1 f i ( t ; α ∗ i ) .
Remark 3.1.
In Co-teaching algorithm (Han et al., 2018b), R ( t ) is determined as follows
[Equation]
which contains three hyper-parameters: τ , c and t k .
Thus, we can formulate a search space as: τ ∈ (0 , 1) , c ∈ (0 , + ∞ ) and t k ∈ { 1 , · · · , T } .
The expressivity of such R ( t ) is not enough as Eqa.(1) here (since Eqa.(3) is covered as a special case).
3. S EARCH A LGORITHM
Here, we ﬁrst discuss problems of using gradient-based methods here (Sec.3.3.1).
These problems motivate us design an efﬁcient search algorithm based on natural gradient algorithm (Sec.3.3.2).
3.3. I SSUES OF E XISTING A LGORITHMS
Gradient-based methods need the chain rule to obtain the gradient w.r.t hyper-parameters from the gradient w.r.t network weights, i.e., ∇ α M = ∂ M / ∂ w · ∂ w / ∂ α .
This condition does not hold for our problem since our hyper-parameters control how many samples will be used to update the weights.
Thus, it is hard to compute ∇ α M here.
Besides, the learned parameters w cannot be shared among different α .
In previous methods, hyper-parameters are not coupled with the training process, e.g., network architectures (Liu et al., 2019; Akimoto et al., 2019) and regularization (Luketina et al., 2016).
However, R ( t ) here will heavily inﬂuence the training process, which is also shown in our
Fig.1.
Thus, gradient-based methods cannot be applied here.
3.3. P ROPOSED A LGORITHM
The analysis above demonstrates that only derivative-free methods are applicable to our problem, which can be slow.
Here, we discover that natural gradient (NG) (Amari, 1998; Pascanu & Bengio, 2013; Ollivier et al., 2017), can be used.
The most interesting point here is that NG can still beneﬁt from gradient descent, but without the computation of ∂ M / ∂ α or sharing of w .
The basic idea of NG is summarized as follows: instead of directly optimizing w.r.t α , we consider a random distribution p θ ( α ) over α parametrized by θ , and maximize the expected value of our validation performance M w.r.t θ , i.e.,
[Equation]
To optimize J ( θ ) w.r.t θ , NG updates θ by
[Equation]
where ρ is the step-size, H ( θ m ) is the Fisher information matrix at θ m , and ∇ θ J ( θ )
= (cid:90)
[Equation]
By sampling candidate R ( t ) s from the given distribution p θ m in each iteration, Eqa.(6) just needs validation’s performance (without computation of ∂ M / ∂ α ).
Besides, such gradient has proved to give the steepest ascend direction in the probabilistic space (Theorem 1 in (Angluin & Laird, 1988)).
The last question is how to choose p θ , which may signiﬁcantly inﬂuence the algorithm’s convergence behavior.
Fortunately, NG exhibits strong robustness against different choices over p θ .
The reason is that Fisher matrix, which encodes the second order approximation to the KullbackLeibler divergence, can accurately capture curvatures introduced by various p θ .
Thus, NG descent is parameterization-invariant, has good generalization ability, and moreover can be regarded as a second-order method in the space of θ (Ollivier et al., 2017).
The proposed search algorithm is in
Alg.2 (details are in Appendix A.2).
As will be shown in experiments, all these properties make NG an ideal search algorithm here, and can be faster than other popular AutoML approaches.
E XPERIMENTS
We implement all our experiments using PyTorch 0.4.1 on a GTX 1080 Ti GPU.
4. E XPERIMENTS ON SYNTHETIC DATA
In this section, we demonstrate the superiority of the proposed search space and search algorithm on the synthetic data.
The ground-truth R ( t ) is shown in Fig.3(a), which is an example curvature satisfying two requirements in Sec.3.2.
4.1. S EARCH SPACE COMPARISON
The goal here is to approximate R ( t ) in Fig.3(a).
Let the estimated function be ¯ R ( t ) where t
= 1 , · · · , 200 .
The target is to minimize RMSE, i.e., f ( ¯ R ) =
[ 1 / 200 (cid:80) 200 t =1 ( R ( t ) − ¯ R ( t )) 2 ] 1 / 2 .
Three different search spaces are compared: 1).
Full space : ¯ R 1 ( t ) = { e 1 , · · · , e 200 } , i.e., there is no constraint in the space, and estimation for R ( t ) at each time stamp is performed independently; 2).
Co-teaching ’s space in Eqa.(3), i.e., R 2 ( t ) = 1 − τ min(( t / t k ) c , 1) , and { τ, c, t k } needs to be estimated; and 3).
The proposed space in Eqa.(1), which encodes our observations in Sec.3.1.
Random search (Bergstra & Bengio, 2012) is performed in all three spaces.
[Figure]
algorithm.
Results are in Fig.3(b).
We can see that no constraints on the search space will lead to disastrous performance and extremely slow convergence rate.
And compare our space with previous Co-teaching’s space, we can see that our space can approximate the target better due to its larger degree of freedom.
4.1. S EARCH ALGORITHM COMPARISON
We ﬁrst test our proposed method’s efﬁciency over other hyper-parameter optimization (HPO) algorithms on a synthetic problem as follows:
Given a pre-deﬁned R ( t ) as target, we try to search for a R ( t ) that has the lowest squared loss to the target.
We compare our proposed method with random search (Bergstra & Bengio, 2012) and Bayesian optimization (Kandasamy et al., 2019), which are two popular methods for hyper-parameter optimization.
Results are in Fig.3(c).
The results demonstrate that our proposed natural gradient approach has the fastest convergence rate.
And it can ﬁnd the best R ( t ) with the least RMSE to the target.
4. E XPERIMENTS ON BENCHMARK DATA SETS
We verify the efﬁciency of our approach on three benchmark data sets, i.e., MNIST, CIFAR-10 and
CIFAR-100 (details are in Appendix A.3).
These data sets are popularly used for the evaluation of learning with noisy labels in the literature (Zhang et al., 2016; Arpit et al., 2017; Jiang et al., 2018;
Han et al., 2018b).
Following (Patrini et al., 2017; Han et al., 2018b; Chen et al., 2019), we corrupt these data sets manually by two types of noise.
(1) Symmetry ﬂipping (with 20% and 50% noise level) and (2) Pair ﬂipping (with 45% noise level). All these noise patterns correspond to real-world scenarios.
For example, on the macro-level, class cat ﬂipping to the class dog makes sense, while class dog ﬂipping to class cat also makes sense. Such ﬂipping yields a noise pattern called symmetric-ﬂip (Patrini et al., 2017). On the micro-level, for dogs, class Norfolk terrier ﬂipping to class Norwich terrier makes sense, while class Norfolk terrier ﬂipping to class Australian terrier not. This ﬂipping yields a noise pattern called pair-ﬂip (Han et al., 2018a), which depicts the ﬁne-grained classiﬁcation case.
We set the network architecture as the same in (Yu et al., 2019)
(Appendix A.4).
To measure the performance, same as (Patrini et al., 2017; Han et al., 2018b; Chen et al., 2019), we use the test accuracy, i.e., test accuracy =
(# of correct predictions ) /
(# of test dataset ) .
Speciﬁcally, the full learning curve, i.e., testing accuracy v.s. epochs, is reported. Ideally, if a method is robust to noisy labels, its performance will increase with more training epochs. Thus, if the learning curve of one method quick falls down after reaching the maximum, then it means the method is not robust intrinsically (Zhang et al., 2016; Arpit et al., 2017). [Figure]
4.2. C OMPARISON ON L EARNING P ERFORMANCE
To show better accuracy can be achieved by automatically exploiting the memorization effect, we compared the proposed method with 1).
MentorNet (Jiang et al., 2018); 2).
Co-teaching (Han et al., 2018b); 3).
Co-teaching+ (Yu et al., 2019) (enhancing Co-teaching by disagreements on predictions); 4).
Decoupling (Malach & Shalev-Shwartz, 2017); 5).
F-correction (Patrini et al., 2017); 6).
As a simple baseline, we also compare with the standard deep network that directly trains on noisy datasets (abbreviated as Standard ).
As an example usage, the proposed method is combined with Co-teaching, i.e., Co-teaching is run with search R ( t ) .
Fig.4 shows the comparison with various human-designed methods.
We can see that the proposed method signiﬁcantly outperforms existing methods by a large margin, especially on the more noisy cases (i.e., symmetric-50% and pair-45%).
Besides, the proposed method not only beats Co-teaching due to better exploiting of the memorization effect, but also wins Co-teaching+, which further ﬁlter small-loss instances in Co-teaching by checking disagreements on predictions of labels.
These demonstrates the importance and beneﬁts of searching proper R ( t ) .
4.2. C ASE S TUDY ON S AMPLING R ATE
To understand why the proposed method can obtain much higher testing accuracy, we plot the searched R ( t ) in Fig.5.
We can see that all methods ﬁnally drop more large-loss instances than actual noise level.
The reason is very intuitive, a large-loss instance usually also has larger gradient, and it can have much more inﬂuences than several clean small-loss instances if its label is wrong.
Thus, we may want to drop more samples eventually.
However, this is not an easy task, as in Tab.
of (Han et al., 2018b), simply making τ larger sometimes leads to decrease in testing accuracy; and in the third row of Fig.5, we may also need to keep more samples during the training.
[Figure]
-10 and CIFAR-100.
4.2. C OMPARISON WITH HPO METHODS
Finally, in this section, we compare the proposed natural gradient (NG) algorithm with 1).
random search (Bergstra & Bengio, 2012) and 2).
Hyperband (Li et al., 2017a).
Note that Bayesian optimization is slower than Hyperband , thus not compared ( Hyperband cannot be used in Sec.4.1.
due to no inner loops).
Besides, reinforcement learning (RL) (Zoph & Le, 2017) is not compared as the searching problem is not a multi-step one.
Genetic programming (Xie & Yuille, 2017) is not considered neither, as the search space is a continuous one.
Fig.6 compares the proposed method with random search and Hyperband.
From the ﬁgure, we can see that natural gradient converges faster than other HPO methods in this problem.
Our proposed method can also ﬁnd better R ( t ) s under different data sets and noise settings.
[Figure]
4. E XPERIMENTS ON R EAL F ACE D ATA S ET
Finally, we test the validity of our approach on real data sets.
We conduct experiments on the application of deep face recognition (Parkhi et al., 2015).
Following (Wang et al., 2019), we use VggFace2R (Cao et al., 2018) data set, which is a noisy data set collected from Google image search, for training, then tune hyper-parameters and report testing accuracy on four small and clean image data sets, i.e., CALFW (Zheng et al., 2017), CPLFW (Zheng & Deng, 2018), AgeDB (Moschoglou et al., 2017), and CFP (Sengupta et al., 2016) (details are in Appendix A.3).
We compare the performance of our proposed method with those methods used in Sec.4.2.1 and
Co-mining (Wang et al., 2019).
Testing accuracy is reported in Tab.1.
We can see that, our proposed method consistently achieves the best performance.
Thus, our method is not only useful with synthetic noise but also work well on real applications.
[Table]
C ONCLUSION
In this paper, motivated by the main difﬁculty that to what extent the memorization effect of deep networks can happen, we propose to exploit memorization by automated machine learning (AutoML) techniques.
This is done by ﬁrst designing an expressive but compact search space, which is based on observed general patterns for memorization, and designing a natural gradient-based search algorithm, which overcomes the problem of non-differential and failure of parameter-sharing.
Extensive experiments on both synthetic data sets and benchmark data sets demonstrate that the proposed method can not only be much efﬁcient than existing AutoML algorithms, but also achieve much better performance than the state-of-the-art sample-selection approach.
A I MPLEMENTATION D ETAILS A. D ETAILS FOR F IG .
(a-b) We use the same MNIST/CIFAR-10/100 datasets and their details are in Appendix A.3.
The network model is model 1 and its details are given in Appendix A.4. The number of training epochs is 200 and the batch size is set to 128. The initial learning rate for the Adam optimizer is 0.001 and it is linearly decayed to zero from the 80th epoch to ﬁnal.
(c) Details of each model is listed in Appendix A.4. Note that model 2 on CIFAR-10 has the output dimension of 10 instead of 100.
The number of training epochs, batch size, optimizer and learning rate schedule follows (a) and (b). (d) The learning rate schedule for Adam follows (a) and (b).
The initial learning rate for the SGD optimizer is 0.1, and decay to 0.01 and 0.001 from the 500th and 750th epoch, respectively.
And for the RMSProp optimizer, the learning rate is set constantly to 0.01.
To demonstrate the memorization effects for SGD optimizer, the number of training epochs is 1000 instead of 200.
The model is model 1 and the batch size is 128.
(e) The model, number of training epochs, optimizer and learning rate schedule is the same as (a) and (b).
In a word, we only change the batch size.
(f) We only change the initial learning rate for the learning rate schedule.
Also, to demonstrate the memorization effects for small learning rate cases, the number of training epochs is 1000 instead of 200.
And the model, batch size, optimizer all follow (a) and (b).
A. N ATURAL G RADIENT
Since our search space for α is bounded, we use Beta distribution in natural gradients, i.e. p θ ( λ ) =
(cid:81)
ki
=1 Beta ( α i ; θ i ) .
Each Beta distribution has two parameters x, y and the probability density function (PDF) is given by
[Equation]
where Γ( x ) is the gamma function.
The Fisher information matrix [Equation]
is estimated by samples since no explicit formula is available.
And the gradient in Fisher ∇ θ log p θ ( α ) is different for different parameters x and y , as
[Equation]
where ψ ( x ) denotes the digamma function.
For synthetic data, we run for 10 iterations and in each iteration, we sample 10 samples and evaluate its performance.
The learning rate for natural gradient is set to 80.
For benchmark and real noisy data sets, all search algorithms will run for 5 iterations in total.
In random search and natural gradient, we sample 10 samples for CIFAR-10 and 20 samples for other four data sets to evaluate its performance in each iteration.
The learning rate for natural gradient is set to 20, 500, 80, 30 for MNIST, CIFAR-10, CIFAR-100 and VggFace2-R data set respectively.
To obtain a better estimation of the Fisher information matrix, in spite of those samples for evaluation, we also sample another 10000 samples to compute the Fisher in each iteration.
A. D ATA SETS
We obtain MNIST, CIFAR-10, CIFAR-100 from PyTorch’s torchvision package, and VggFace2-R,
CALFW, CPLFW, AgeDB, CFP from its original source.
Some statistics about those datasets are given below.
[Table]
A. N ETWORK S TRUCTURE
For VggFace2-R, we use ResNet-50 as in (Wang et al., 2019).
Adam optimizer (Kingma & Ba, 2014) (momentum=0.9) is used with an initial learning rate of 0.001, and the batch size is set toand we run 200 epochs.
The learning rate is linearly decayed to zero from 80 to 200 epochs. [Table] [Table] BN,
[Table]
[Table]
LReLU
[Table]
[Table]
[Table]
, LReLU
[Table]
BN, ReLU
[Table]
pool, stride
[Table]
pool
Dropout, p=0.
[Table]
[Table]
B M ORE R ESULTS B. P ERFORMANCE C OMPARISON WITH E ARLY S TOPPING
To make a better comparison
[Table]
we apply early-stop trick to all those methods in Sec.4.2 and report their best accuracy instead of the ﬁnal accuracy.
The accuracy is averaged in 5 runs and we also report the standard deviation.
Results are shown below:
[Table]
[Table]
[Table]
B. P ERFORMANCE C OMPARISON WITH S IMPLE D ECAY F UNCTION
Here, we compare the performance obtained (under early stop setup in Sec.
B.1) from
• the proposed space in Eqa.(1); and • that from a space spanned by a simple decay function (
i
[Table]
e., power in Fig.2)
Natural gradient proposed in Sec.3.3.2 is used for optimization. Results are shown in Fig.7. We can see that performance obtained from the proposed method is much better than that from a simple decay function. This again demonstrates the needs of approximating R ( T ) by a linear combination of some basis functions
[Table]
[Table]