D
YNAMIC
I
NSTANCE
H ARDNESS
Anonymous authors
Paper under double-blind review
A BSTRACT
We introduce dynamic instance hardness (DIH) to facilitate the training of machine learning models.
DIH is a property of each training sample and is computed as the running mean of the sample’s instantaneous hardness as measured over the training history.
We use DIH to evaluate how well a model retains knowledge about each training sample over time.
We ﬁnd that for deep neural nets (DNNs), the
DIH of a sample in relatively early training stages reﬂects its DIH in later stages and as a result, DIH can be effectively used to reduce the set of training samples in future epochs.
Speciﬁcally, during each epoch, only samples with high DIH are trained (since they are historically hard) while samples with low DIH can be safely ignored.
DIH is updated each epoch only for the selected samples, so it does not require additional computation.
Hence, using DIH during training leads to an appreciable speedup.
Also, since the model is focused on the historically more challenging samples, resultant models are more accurate.
The above, when formulated as an algorithm, can be seen as a form of curriculum learning, so we call our framework DIH curriculum learning (or DIHCL).
The advantages of DIHCL, compared to other curriculum learning approaches, are: (1) DIHCL does not require additional inference steps over the data not selected by DIHCL in each epoch, (2) the dynamic instance hardness, compared to static instance hardness (e.g., instantaneous loss), is more stable as it integrates information over the entire training history up to the present time.
Making certain mathematical assumptions, we formulate the problem of DIHCL as ﬁnding a curriculum that maximizes an unknown multi-set function f ( · ) from its partial observations, and derive an approximation bound for a DIH-produced curriculum relative to the optimal curriculum.
Empirically, DIHCL-trained DNNs signiﬁcantly outperform random mini-batch SGD and other recently developed curriculum learning methods in terms of efﬁciency, early-stage convergence, and ﬁnal performance, and this is shown in training several state-of-the-art DNNs on 11 modern datasets.
I NTRODUCTION
We study the dynamics of training a machine learning model, and in particular, the difﬁculty a model has over time (i.e., training epochs) in learning each sample from a training set.
To this end, we introduce the concept of “dynamic instance hardness” (DIH) and propose several metrics to measure
DIH, all of which share the same form as a running mean over different instantaneous sample hardness measures.
Let a t ( i ) be a measure of instantaneous (i.e., at time t ) hardness of a sample, where i is a sample index and t is a time iteration index (typically a count of mini-batches that have been processed).
In previous work, a t ( i ) has been called the “instance hardness” (Smith et al., 2014) corresponding to 1 − p w ( y i | x i ) , i.e., the complement of the posterior probability of label y
i given input x i for the i th sample under model w .
We introduce three different notions of instantaneous instance hardness in this work: (A) the loss (cid:96) ( y i , F ( x i ; w t )) , where (cid:96) ( · , · ) is the loss function and F ( · ; w ) is the model with parameters w , (B) the loss change | (cid:96) ( y i , F ( x i ; w t ))
− (cid:96) ( y i , F ( x i ; w t − 1 ))
| between two consecutive time steps, and (C) the prediction ﬂip 1 [ˆ y ti
=
ˆ y t − 1 i ] , where ˆ y ti is the prediction of sample i in step t , e.g., argmax j F ( x i ; w t )
[ j ] for classiﬁcation.
Our (A) corresponds closely to the “instance hardness” of Smith et al.
However, our (B) and (C) require information from previous time steps.
Nevertheless, we consider (A), (B), and (C) all variations of instantaneous instance hardness since they use information from only a very local time window around training iteration t .
Dynamics is achieved when we compute a running average over instantaneous instance hardness, computed recursively as follows:
[Equation]
where γ ∈ [0 , 1] is a discount factor, S t ⊆ V , and V =
[ n ] is the set of all n training sample indices.
S t is the set of sample selected for training at time t by some method (e.g., a DIH-based curriculum learning (DIHCL) algorithm
we introduce and study below) or simply a random batch.
In general,
S t should be large early during training, but as r t ( i ) decreases to small values for many samples, choosing signiﬁcantly smaller S t is possible to result in faster training and more accurate models.
We ﬁnd that r t ( i ) can vary dramatically between different samples since very early stage (with small t ).
One can think of this as some samples being more memorable and are retained more easily, while other samples are harder to learn and retain.
In addition, the predictions of the hard samples are less stable under changes in optimization parameters (such as the learning rate).
More importantly, once a sample’s r t ( i ) is established (i.e., once t is sufﬁciently but not unreasonably large) each sample tends to maintain its DIH properties.
That is, a sample’s DIH value converges relatively quickly to its ﬁnal relative position amongst all of the samples DIH values.
For example, if a sample’s DIH becomes small (i.e., meaning the sample is easily learned), it stays small relative to the other samples, or if it becomes large DIH (i.e., the sample is difﬁcult to learn), it stays there.
I.e., once r t ( i ) for a sample has converged, its DIH status is retained throughout the remainder training.
We can therefore accurately identify categories of sample hardness relatively early in the course of training.
This suggests a natural curriculum learning strategy where S t corresponds mostly to those samples that are hard according to r t − 1 ( i ) .
In other words, the model concentrates on that which it ﬁnds difﬁcult.
This is similar to strategies that improve human learning, such as the Leitner system for spaced repetition (Leitner, 1970).
This is also analogous to boosting (Schapire, 1990) — in boosting, however, we average the instantaneous sample performance of multiple weak learners at the current time, while in DIHCL we average the instantaneous sample performance of one strong learner over the training history.
As mentioned above, instance hardness has been studied before (Smith et al., 2014; Prudencio et al., 2015; Smith & Martinez, 2016) where it corresponds to the complement posterior probability.
More recently, instance hardness has also been studied as an average over training steps in Toneva et al.
(2019) where the mean of prediction ﬂips over the entire training history is computed.
We note that
Toneva et al.
(2019) is a special case of DIH in Eq.
(1) with γ = 1 / t +1 and t = T , where T is the total number of training steps.
Our study generalizes Toneva et al.
(2019) to the running dynamics computed during training.
This therefore leads to a novel curriculum learning strategy and also steps towards a better theoretical understanding of curriculum learning.
Also, in Toneva et al.
(2019), a small neural net is trained beforehand to determine the hard samples, and this is then used to train large neural nets.
In our approach, we take the average over time of a t ( i ) , which requires no additional model or inference steps and hence is computationally trivial.
Another observation we ﬁnd is that r t ( i ) , for any sample, tends to monotonically decrease with t for any i .
This means, not surprisingly, that during training samples become easier in terms of small
DIH (i.e., they are better learned).
This also means that easy samples stay easy throughout training, and hard samples also become easier the more we train on them.
If we also make (admittedly) a mathematical leap, and assume that r t ( i ) is generated by the marginal gain of an unknown diminishing returns function f ( · ) that measures the quality of any curriculum, we can formulate DIHCL as an online learning problem that maximizes the unknown f ( · ) by observing its partial observation r t ( i ) over time for each i .
Here, f is deﬁned over an integer lattice and has a diminishing returns property, although the function is accessible only via the gains of every element.
This formulation provides a setting where the quality of the learnt curriculum is provably approximately good.
As will be shown below, DIHCL performs optimization in a greedy manner.
At each time step t ,
DIHCL selects a subset S t of samples using r t ( i ) where the hard samples have higher probabilities of being selected relative to the easy samples.
The model is then updated based only on the selected subset S t rather than V , which requires performing inference (e.g., a forward pass of a DNN) only on S t .
This therefore leads to a speedup to the extent that | S t | (cid:28) | V | .
The inference produces new instantaneous instance hardness a t ( i ) that is then used to update r t
+1 ( i ) as in Equation 1.
To encourage exploration, improve stability, and get an initial estimate of r t ( i ) for all i ∈ V , during the ﬁrst few epochs, DIHCL sweeps through the entire training set.
We provide several options for
DIH-weighted subset sampling, which introduces different types of randomness in the selection since randomness is essential in optimizing non-convex problems.
Under certain additional mathematical assumptions, we also give theoretical bounds on the curriculum achieved by DIHCL compared to the optimal curriculum.
We empirically evaluate several variants of DIHCL and compare them against random mini-batch SGD as well as against recent curriculum learning algorithms, and test on 11 datasets including CIFAR10, CIFAR100, STL10, SVHN, Fashion-MNIST, Kuzushiji-MNIST,
Food-101, Birdsnap, FGVC Aircraft, Stanford Cars and ImageNet.
DIHCL shows an advantage over other baselines in terms both of time/sample efﬁciency and test set accuracy.
1. R ELATED W ORK
Early curriculum learning (CL)
(Khan et al., 2011; Basu & Christensen, 2013; Spitkovsky et al., 2009)
work shows that feeding an optimized sequence of training sets (i.e., a curriculum), that can be designed by a human expert (Bengio et al., 2009), into the training algorithms can improve the models’ performance.
Self-paced learning (SPL) (Kumar et al., 2010; Tang et al., 2012a; Supancic III &
Ramanan, 2013; Tang et al., 2012b) chooses the curriculum adaptive to some instance hardness (e.g., per-sample loss) during training.
SPL selects samples with smaller losses, and gradually increases the subset size over time to cover all the training data.
Self-paced curriculum learning (Jiang et al., 2015) combines the human expert in CL and loss-adaptation in SPL.
SPL with diversity (SPLD) (Jiang et al., 2014) adds a negative group sparse regularization term to SPL and increases its weight to increase selection diversity.
Machine teaching (Khan et al., 2011; Zhu, 2015; Patil et al., 2014) aims to ﬁnd the optimal and smallest training subset leading to similar performance as training on all the data.
Minimax curriculum learning (MCL) (Zhou & Bilmes, 2018) argues that the diversity of samples is more critical in early learning since it encourages exploration, while difﬁculty becomes more useful later.
It also uses a form of instantaneous instance hardness (the loss) but is not dynamic like DIH, and formulates optimization as a minimax problem.
Compared to the above methods, DIHCL has the following advantages: (1) DIHCL improves the efﬁciency of CL since extra inference on the entire training set per step is not required; and (2) DIHCL uses DIH as the metric for hardness which is a more stable measure than instantaneous hardness.
Our paper is also related to Zhang et al.
(2017), which refers to overﬁtting in noisy data.
Our observations suggest that the learning of simple patterns (Arpit et al., 2017) happen mainly amongst the easy memorable early during in training (additional discussion is given in the Appendix and Figure 5).
Our paper is also distinct from catastrophic forgetting (Kirkpatrick et al., 2017), which considers sequential learning of multiple tasks, where later learned tasks make the model forget what has been learned from earlier tasks.
In our work, we consider single task learning and show that easy samples remain easy.
If we make certain additional mathematical assumptions (as we do in our theoretical discussion below), our work is related to online submodular function optimization.
Speciﬁc forms have been studied including maximization (Streeter & Golovin, 2009; Chen & Krause, 2013), maximization in the bandit setting with noisy feedback (Chen et al., 2017), and continuous submodular function maximization (Chen et al., 2018b;a).
The work most related to ours, perhaps, is the work on instance hardness (Smith et al., 2014; Prudencio et al., 2015; Smith & Martinez, 2016), where the hardness of a sample corresponds to the complement posterior probability, as discussed above.
Also, a special case of DIH was studied in Toneva et al.
(2019): they compute DIH after training completes, and show that removing the easy samples (those having the smallest DIH over training set) leads to less degradation on generalization performance than removing random samples.
By contrast, our study of DIH focuses on its dynamics during training.
D YNAMIC I NSTANCE H ARDNESS
We start out by conducting an empirical study of DIH in DNN training.
We train a WideResNet with depth of 28 and width factor 10 on CIFAR10 dataset, and apply a modiﬁed cosine annealing learning rate schedule (Loshchilov & Hutter, 2017) for multiple episodes of increasing length ( 300 epochs in total) and target learning rate decaying.
We contend that a cyclic learning rate suits our study because: (1) it includes the most commonly used monotone decreasing schedule since the learning rate in each cycle is decreasing; (2) compared to monotone decreasing schedule, it can uncover the properties of DIH in more scenarios such as increasing learning rate and different decaying speeds of the learning rate.
In the study, we test two type of instantaneous instance hardness, where a t ( i ) is either prediction ﬂips or loss (i.e., cases (A) and (C) in the previous section).
We visualize r t ( i ) for
[Figure]
all i ∈ [50000] training samples, but divide [50000] them into three groups according to r t ( i ) (with a t ( i ) as prediction ﬂips), and we do this at epoch either 10, 40, or 210.
For example, at epoch 40, the 10,000 samples with the largest r 40 ( i ) comprise the ﬁrst group, the 10,000 samples ones with the smallest r 40 ( i ) comprise the next group, and the remaining 30,000 samples comprise the ﬁnal group.
In Figure 1, we plot the dynamics for the average prediction ﬂips over each group (left plot) and the mean/standard deviation of loss in each group (right plot).
[Figure]
We observe that samples with small r t ( i ) are learned quickly in the early epochs and their losses remain small and predictions almost unchanged thereafter, indicating they are easy over time.
By contrast, the samples with large r t ( i ) have large variance, i.e., their losses oscillate rapidly between small and large values, and their predictions frequently change, indicating difﬁculty.
The quickly identiﬁed easy samples are never unlearnt, and do not suffer from any large loss later in the training.
The hard samples are also quickly identiﬁed, and remain difﬁcult to learn even after training for many epochs.
On average, the dynamics on the hard samples is more consistent with the learning rate schedule, which implies that doing well on these samples can only be achieved at a shared sharp local minima.
This suggests that the model generalizes better in the regions containing the easy samples than those containing the hard ones.
Similar to human learning, a natural resolution is to learn the hard samples more frequently and reduce the learning on the easy, already learnt, ones.
That is, based on r t ( i ) (even during the early stages when t is not large), it is prudent to apply additional training effort on hard samples and begin to ignore already learnt easy samples.
Further empirical veriﬁcation can be found in experiments and Figure 4 in the Appendix.
2. P ROBLEM F ORMULATION
Figure 2 shows that all three types (base on (A), (B), and (C)) of DIH metrics decrease during training for both easy and hard samples.
This indicates that as learning continues, samples become less informative as more training takes place.
If we make additional mathematical assumptions, we can model the curriculum learning procedure as a scheme that maximizes an unknown diminishing return function f ( · ) via only observing its marginal gains r t ( i ) .
A curriculum is a sequence of selected subsets, where each subset is a mini-batch of data points, i.e. ( S 1 , S 2 , . . .
, S T ) .
We deﬁne a function f : Z V ≥
→ R ≥ 0 on the non-negative integer lattice
Z V > 0 .
Each point z ∈ Z V ≥ 0 is an non-negative integer valued vector of length | V | where z ( i ) counts how many times sample i has been selected in the T training data subsets.
Any subset S ⊆ V has a characteristic vector e S where e S ( i )
= 1 if i ∈ S
and otherwise s S ( i )
=
if i / ∈ S .
We also deﬁne S 1: t (cid:44) (cid:80) t(cid:96)
=1 e S t as a multi-set input to function f and S 1: t ∈ Z V ≥ 0 .
Ideally, our goal becomes ﬁnding the best curriculum ( S 1 , S 2 , . .
, S T ) , i.e., one that maximizes f ( S 1: T ) , as in:
[Equation]
where k t is a limit on the size of the set of samples selected at time t .
However, f ( · ) can be an arbitrary unknown function in practice.
It is also intractable to estimate since it measures the utility of all possible training sequences (in exponential number), so it is inaccessible for optimization.
As a surrogate, information about f ( · ) might be available at each step in the form of the DIH values r t ( i ) .
That is, if we make the mathematical assumption that there is some function f such that r t ( i ) =
f ( i | S 1: t − 1 ) (cid:44) f
( e i + S 1: t − 1 ) − f ( S 1: t − 1 ) , then r t ( i ) may be used in f ’s stead
and we can optimize the unknown f ( · ) only based on its partial observation r t ( i ) .
In such case, DIHCL can be seen as a form of online optimization problem whose goal is to ﬁnd a curriculum that maximizes f : at every step t , we select a subset of samples (e.g., a mini-batch) S t ⊆ V of size | S t | =
k t to train the model, observing only marginal gains r t ( i ) of f ( · ) for each i .
We can therefore deﬁne the following objective:
[Equation]
(cid:88)
[Equation]
(cid:88)
i
[Equation]
t f ( i |
[Equation] [Equation]
For simplicity, we slightly overload the notation of S 1: T for function g ( · ) so that we retain information about the subset selected at every time step (i.e., we can extract S t for 1 ≤ t ≤ T from
S 1: T ).
In practice, we update r t ( i ) only for samples in S t since it is a byproduct of training (i.e., the information needed to update r t ( i ) requires no more computation than what needed to train the model on S t ).
Hence, for i / ∈ S t , r t ( i ) =
r t − 1 ( i ) .
Although our solution is an approximate solution to the ideal but intractable optimization in Eq.
(2), we give its approximation bound in Corollary 1 to the global optimal solution of Eq.
(2), and the bound is tight (the best we can get) up to a constant factor.
DIH C URRICULUM L EARNING
We arrive at a curriculum learning strategy that increases the probability learning on hard samples, and reduces learning on easy ones.
However, directly solving Eq.
(3) requires costly inference over all the n training samples before selecting every subset S t , as most previous curriculum learning methods do.
3. A “F REE ” C URRICULUM
Algorithm
1: input: { ( x i , y i ) } ni =1 , π ( · ; η ) , η 1: T , (cid:96) ( · , · ) , F ( · ; w ) ;
T, T 0 ; γ, γ k ∈ [0 , 1] 2: initialize: w , η 1 , k 1 = n , r
( i ) =
1 ∀ i ∈ [ n ] 3: for t ∈ { 1 , · · · , T } do 4: if t ≤ T 0 then 5:
S t ← [ n ] ; 6: else 7:
Let S t = argmax S : | S | =
k t (cid:80) i ∈
S r t − 1 ( i ) ; 8: end if 9:
Apply optimization π ( · ; η ) and record F ( x i ; w t − 1 ) for i ∈ S t ; w t ← w t − 1 + π (cid:32) ∇ w t − 1 (cid:88)
i ∈
S t (cid:96) ( y i , F ( x i ; w t − 1 )); η t (cid:33) 10:
Compute normalized a t ( i ) for i ∈ S t using Eq.
Update dynamic instance hardness r t +1 ( i ) using Eq.
(1); 12: k t +1 ← γ k × k t ; 13: end for
Most optimization algorithms require inference on the training samples before updating the model parameters, which generates the predictions and losses for the samples used for training.
In step t , after the model gets trained on
S t , the feedback a t ( i ) for i ∈ S t is already available.
However, for i / ∈ S t , extra inference is inevitable if the curriculum design requires instantaneous instance hardness on the remaining samples to select next subset
S t +1 .
By contrast, DIHCL relies on r t ( i ) which is a running mean of a t ( i ) , and it only updates r t ( i ) for i ∈ S t and keeps r t ( i ) for i / ∈ S t unchanged, thereby saves extra computation.
At step t of DIHCL, we select subset S t ⊆
[ n ] with large r t − 1 ( i ) and then update the model by training on S t .
We then update r t ( i ) via Equation (1).
Since the learning rate can change over different steps, and large learning rates means greater model change, we normalize a t ( i ) by the learning rate η t − 1 1 .
Speciﬁcally, we use one of the following depending on if we’re in case (A), (B), or (C):
[Equation]
t ( i ) ←
[Equation]
DIHCL is given in Algorithm 1, where { ( x i , y i ) } ni =1 is the training data, π ( · ; η ) is an optimization method such as SGD, η 1: T are the learning rates of steps 1 to T and γ k is the reduction factor for subset sizes k t .
DIHCL trains on more samples early on to produce an initial more accurate estimate of r t ( i ) .
This is indicated by T 0 , the number of warm start epochs over the whole training set at the start.
After this, we start by selecting larger subsets each step and gradually reduce k t down to the most difﬁcult samples as training proceeds.
A simple method to further reduce training time in the earlier stages is to extract a small subset of S t by encouraging the diversity of the selected samples.
We gradually reduce the diversity preference as training switching to the exploitation stage (reduce λ t by 0 ≤ γ λ ≤ 1 for every step t ).
Inspired by
MCL (Zhou & Bilmes, 2018), after line 7, we reduce S t to a subset of size k (cid:48) t
=
γ k
[Equation]
k t ( 0 < γ k (cid:48) ≤ 1 ) by (approximately) solving the following submodular maximization.
[Equation]
The function G : 2 S t → R + is may be any submodular function (Fujishige, 2005), and hence we can exploit fast greedy algorithms (Nemhauser et al., 1978; Minoux, 1978; Mirzasoleiman et al., 2015) to solve Eq.
(5) with an approximation guarantee.
3. A PPROXIMATION B OUND UNDER F URTHER M ATHEMATICAL A SSUMPTIONS
If in addition to the assumption that there exists some function f : Z V ≥
→ R ≥ 0 such that r t ( i ) =
f ( i | S 1: t − 1 ) (cid:44) f
( e i + S 1: t − 1 ) − f ( S 1: t − 1 ) , we also assume that f has certain properties, then an approximation bound is achievable.
The diminishing return (DR) property for f can be deﬁned if, ∀ 0 ≤
[Equation]
≤
[Equation]
Recall that e i is a one-hot vector with all zeros except for a single one at the i th position.
We also assume f is normalized and monotone, i.e., f (0)
=
and f ( x ) ≤ f ( y ) , ∀ 0 ≤ x ≤ y .
W.l.o.g.
we assume the max singleton gain is bounded by 1 ( max i f
( e i ) ≤ 1 ).
With such an assumption, we see that r t ( i ) is monotonically decreasing with increasing t .
That is, r t ( i ) monotonically decreasing is a necessary, but not sufﬁcient, condition for the DR property on f to hold.
Empirically we observe, in
Figure 2 that r t ( i ) is indeed decreasing, meaning this evidence does not rule out there being a DR function governing r t ( i ) ’s behavior.
On the other hand, this of course does not guarantee the DR property.
Nevertheless, if it is the case that r t ( i ) is produced as above from some DR function, it enables the following analysis to proceed.
Under the above assumptions, we may derive bounds of DIHCL-Greedy(Alg.
when k t = k ∀ t
∈
[ T ] .
For simplicity, assume n mod k =
and let m (cid:44) n k .
We ﬁrst show the bound on function g of observed gains, and then connect it to the unknown function f .
Theorem 1.
For f : Z V ≥
→ R ≥ 0 on ground set V with DR property, compared to any solution S ∗ 1: T ,
S 1: T , the solution of DIHCL-Greedy , achieves
[Equation]
Where C f,m (cid:44) m min
A 1: m g ( A 1: m ) such that {+(cid:83) mi =
+} A i = V, and | A i | = k .
Corollary 1.
[Equation]
The proofs are given the Appendix.
The C f,m term in the bound reﬂects our loss during the warm start phase, where we cannot estimate the gain of each sample unless we select each sample at least once, which is independent of T and vanishes in the long run.
The 1 − e − 1 comes from the DR property and our greedy procedure.
For the 1 /k factor and the k/n factor of the bound on g , we give hard cases in the Appendix so our bound is tight to constant factors.
These factors result from our assumption about the function f , which may have arbitrary interactions among data points.
In practice, similar data points tend to have similar DIH, and we can incorporate such information by adding an additional term of submodular function G to the DIH value to model data point interactions.
3. P RACTICAL M ETHODS FOR DIH- WEIGHTED S AMPLING IN DIHCL
In line 7 of Alg. 1, we select S t with the highest r t − 1 ( i ) values.
In practice, we ﬁnd adding randomness to the selection procedure gives better performance as (1) exploration on samples with small r t ( i ) is necessary for accurate estimate to r t ( i ) , and (2) randomness of training samples is essential to achieve a good quality solution w for non-convex models such as DNNs.
Instead of choosing greedily the top k t samples, we perform random sampling with probability p t,i ∝ h ( r t − 1 ( i )) , where h ( · ) is a monotone non-decreasing function, and we still prefer data points with high DIH.
An ideal choice of h ( · ) should balance between the exploration of data with poorly estimated DIH and exploitation of data with well estimated DIH.
We propose the following three sampling methods to replace line 7 of Alg. 1, and give extensive evaluations in the experiment section.
DIHCL- Rand: Let h ( r t ( i ))
=
r t ( i ) .
We sample data points weighted by their DIH values.
DIHCL-
Exp: We trade-off exploration and exploitation similarly to Exp3 (Auer et al., 2003), which samples based on the softmax value and reweigh the observation by the selection probability to encourage exploration:
[Equation]
DIHCL-
Beta: We utilize the idea of Thompson sampling (Thompson, 1933) and use a Beta distribution prior to balance exploration and exploitation, i.e., h ( r t ( i )) ∼ Beta ( r t ( i ) , c − r t ( i )) , where c is a sufﬁciently large constant that c ≥ r t ( i ) , e.g., c = 1 when a t ( i ) is prediction ﬂip.
The
Beta distribution encourages exploration when the difference between r t ( i ) and c − r t ( i ) is small.
[Table]
E MPIRICAL E XPERIMENTAL E VALUATION
We train different DNNs by using variants of DIHCL, and compare them with three baselines, vanilla random mini-batch SGD, self-paced learning (SPL) (Kumar et al., 2010), and minimax curriculum learning (MCL) (Zhou & Bilmes, 2018) on 11 image classiﬁcation datasets (without pre-training), i.e., (1) WideResNet-28-10 (Zagoruyko & Komodakis, 2016) on CIFAR10 and CIFAR100 (Krizhevsky & Hinton, 2009); (2) ResNeXt50-32x4d (Xie et al., 2017) on Food-101 (Bossard et al., 2014), FGVC
Aircraft (Aircraft) (Maji et al., 2013), Stanford Cars (Krause et al., 2013), and Birdsnap (Berg et al., 2014); (3) ResNet
He et al.
(2016) on ImageNet (Deng et al., 2009); (4) WideResNet-16-8 on
Fashion-MNIST (FMNIST) (Xiao et al., 2017) and Kuzushiji-MNIST (KMNIST)
(Clanuwat et al., 2018)
; (5) PreActResNet34 He et al.
(2016) on STL10 (Coates et al., 2011)
and SVHN (Netzer et al., 2011).
We use mini-batch SGD with momentum of 0 .
9 and cyclic cosine annealing learning rate schedule (Loshchilov & Hutter, 2017) (multiple episodes with starting/target learning rate decayed by a multiplicative factor 0 . 85 ).
We use T 0 = 5 , γ = 0 .
95 , γ k = 0 .
85 for all DIHCL variants, and gradually reduce k from n to 0 .
2 n .
On each dataset, we apply each method to train the same model for the same number of epochs, but each method may select different amount of samples per epoch.
More details about the datasets and settings can be found in the Appendix.
For DIHCL variants that further reduce S t by solving Eq.
(5), we use λ 1 = 1 . 0 , γ λ = 0 .
8 , γ k (cid:48)
= 0 .
4 and employ the “facility location” submodular function (Cornuéjols et al., 1977) G ( S ) =
(cid:80) j ∈ S t max i ∈ S ω
i,j where ω i,j represents the similarity between sample x i and x j .
We utilize a Gaussian kernel for similarity using neural net features (e.g., the inputs to the last fully connected layer in our experiments)
z ( x ) for each x , i.e., ω i,j = exp (cid:0) − (cid:107) z ( x i ) − z ( x j ) (cid:107) 2 / 2 σ 2 (cid:1) , where σ is the mean value of all the k ( k − 1) / 2 pairwise distances.
In Figure 3, we show how the test set accuracy changes when increasing the number of training batches in each curriculum learning method on 3 datasets.
The results for other 8 datasets can be found in the Appendix, together with the wall-clock time for (1) the entire training and (2) the submodular maximization part in DIHCL with diversity and MCL.
The ﬁnal test accuracy achieved by each method is reported in Table 1.
DIHCL and its variants show signiﬁcantly faster and smoother gains on test accuracy than baselines during training especially at earlier stages.
They also achieve higher ﬁnal accuracy and show improvements in sample efﬁciency (meaning they reach their best performance sooner, after less computation has taken place).
MCL can reach similar performance as DIHCL on some datasets but it shows less stability and requires more computation for submodular maximization.
We also observe a similar instability of SPL.
The reason is that, compared to the methods that use
DIH, both MCL and SPL deploy instantaneous instance hardness (i.e., current loss) as the score to select samples, a measure that is more sensitive to randomness and perturbation that occurs during training.
Compared to MCL and DIHCL, SPL and the random mini-batch curriculum method requires more epochs to reach their best accuracy, since they spend training effort on the easier and memorable samples but lack sufﬁcient repeated-learning of the forgettable ones.
Although every variant of
DIHCL achieves the best accuracy among all the evaluated methods on some datasets, DIHCL-
Exp using loss and DIHCL-Beta using prediction ﬂip, as the instantaneous hardness, exhibit advantages over the other DIHCL variants.
One possible explanation is that the running mean computed on the loss and prediction ﬂips are more stable along the training trajectory as shown in Figure 2, or perhaps they are more in line with our assumption in Section 3.2 about the diminishing return property of f ( · ) .[-A P ROOFS f : Z V ≥ 
 → R ≥ 0 on ground set V is deﬁned over an integer lattice.-]
[-For f : Z V ≥ 
-]
[-i, A 1: t ) to return the subsequence of A 1: t from A 1 so the last set that element i is present, i.e., ζ ( i, A 1: t ) = argmax A 1: t (cid:48) t (cid:48) 
-]
C E XPERIMENTS ( CONT .)
[-= 
-]
[-A 2 ⊆ V , denote A 
-]
[-33, we get 2 g ( ˆ S 1: T | S 1: T − 1 ) ≥ 
-]
[-For f : Z V ≥ 
-]
[-= 
-]
We use cosine annealing learning rate schedule for [-completeness. Theorem 3. For f : Z V ≥ 
-]multiple episodes.
[-= 
-]
[-We get the following regret bound: [Equation] Where, C = 8 / log(1 + σ − 2 ) , β T = 2 B 2 + 
 ρ T ln 3 ( T/δ ) .-]The switching epoch between each two consecutive episode for different datasets are listed below.
• CIFAR10, CIFAR100: (5 , 10 , 15 , 20 , 30 , 40 , 60 , 90 , 140 , 210 , 300) ; • Food-101, Birdsnap, FGVCaircraft, StanfordCars: (10 , 20 , 30 , 40 , 60 , 90 , 150 , 240 , 400) ; • ImageNet: (5 , 10 , 15 , 20 , 30 , 45 , 75 , 120 , 200) ; • STL10: (20 , 40 , 60 , 80 , 120 , 160 , 240 , 360 , 560 , 840 , 1200) ; • SVHN: (5 , 10 , 15 , 20 , 30 , 40 , 60 , 90 , 140 , 210 , 300) ; • KMNIST, FMNIST: (5 , 10 , 15 , 20 , 30 , 40 , 60 , 90 , 140 , 210 , 300) ;
We report how the test accuracy changes with the number of training batches for each method, and the wall-clock time for all the 11 datasets in Figure 9-
[Table]
[Table]
#Class     #Epoch T
BatchSize     lr_start 4 × 10 −4 × 10 −4 × 10 −4 × 10 −4 × 10 −lr_target 1 × 10 −1 × 10 −1 × 10 −1 × 10 −1 × 10 −