Original: I N Y OUR P ACE : L EARNING THE R IGHT E XAMPLE AT THE R IGHT T IME
Revised: I N Y OUR P ACE : L EARNING THE R IGHT E XAMPLE AT THE R IGHT T IME

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Training neural networks is traditionally done by sequentially providing random mini-batches sampled uniformly from the entire dataset.
Revised: Training neural networks is traditionally done by sequentially providing random mini-batches sampled uniformly from the entire dataset.

Original: In our work, we show that sampling mini-batches non-uniformly can both enhance the speed of learning and improve the ﬁnal accuracy of the trained network.
Revised: In our work, we show that sampling mini-batches non-uniformly can both enhance the speed of learning and improve the ﬁnal accuracy of the trained network.

Original: Speciﬁcally, we decompose the problem using the principles of curriculum learning: ﬁrst, we sort the data by some difﬁculty measure; second, we sample mini-batches with a gradually increasing level of difﬁculty.
Revised: Speciﬁcally, we decompose the problem using the principles of curriculum learning: ﬁrst, we sort the data by some difﬁculty measure; second, we sample mini-batches with a gradually increasing level of difﬁculty.

Original: We focus on CNNs trained on image recognition.
Revised: We focus on CNNs trained on image recognition.

Original: Initially, we deﬁne the difﬁculty of a training image using transfer learning from some competitive ”teacher” network trained on the Imagenet database, showing improvement in learning speed and ﬁnal performance for both small and competitive networks, using the CIFAR-10 and the CIFAR-100 datasets.
Revised: Initially, we deﬁne the difﬁculty of a training image using transfer learning from some competitive ”teacher” network trained on the Imagenet database, showing improvement in learning speed and ﬁnal performance for both small and competitive networks, using the CIFAR-10 and the CIFAR-100 datasets.

Original: We then suggest a bootstrap alternative to evaluate the difﬁculty of points using the same network without relying on a ”teacher” network, thus increasing the applicability of our suggested method.
Revised: We then suggest a bootstrap alternative to evaluate the difﬁculty of points using the same network without relying on a ”teacher” network, thus increasing the applicability of our suggested method.

Original: We compare this approach to a related version of Self-Paced
Revised: We compare this approach to a related version of Self-Paced

Original: Learning, showing that our method beneﬁts learning while SPL impairs it.
Revised: Learning, showing that our method beneﬁts learning while SPL impairs it.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Teaching complex tasks to humans and animals can be difﬁcult.
Revised: Teaching complex tasks to humans and animals can be difﬁcult.

Original: Often such tasks cannot be grasped by the learners, or ”students”, immediately, and need to be broken down into simpler problems.
Revised: Often such tasks cannot be grasped by the learners, or ”students”, immediately, and need to be broken down into simpler problems.

Original: Therefore, in order to teach complex tasks, teachers are often required to create a curriculum.
Revised: Therefore, in order to teach complex tasks, teachers are often required to create a curriculum.

Original: The curriculum imposes some order on the learning task; it introduces different concepts at different times, hence exploiting previously learned concepts in order to ease the abstraction of new ones.
Revised: The curriculum imposes some order on the learning task; it introduces different concepts at different times, hence exploiting previously learned concepts in order to ease the abstraction of new ones.

Original: Imposing a curriculum in order to speed up learning is widely used in the context of human learning, and also routinely used in animal training (Skinner, 1958; Pavlov, 2010; Krueger & Dayan, 2009).
Revised: Imposing a curriculum in order to speed up learning is widely used in the context of human learning, and also routinely used in animal training (Skinner, 1958; Pavlov, 2010; Krueger & Dayan, 2009).

Original: In many traditional machine learning approaches, known as supervised learning, a target function is estimated by using a set of labeled examples.
Revised: In many traditional machine learning approaches, known as supervised learning, a target function is estimated by using a set of labeled examples.

Original: The examples can be thought of as given by a teacher while the learning algorithm can be thought of as a student.
Revised: The examples can be thought of as given by a teacher while the learning algorithm can be thought of as a student.

Original: The ﬁeld of curriculum learning (CL), which is motivated by the idea of a curriculum in human learning, attempts at imposing some structure on the labeled data.
Revised: The ﬁeld of curriculum learning (CL), which is motivated by the idea of a curriculum in human learning, attempts at imposing some structure on the labeled data.

Original: Such structure essentially relies on a notion of ”easy” and ”hard” examples and utilizes this distinction in order to teach the student how to generalize easier concepts before harder ones.
Revised: Such structure essentially relies on a notion of ”easy” and ”hard” examples and utilizes this distinction in order to teach the student how to generalize easier concepts before harder ones.

Original: Empirically, the use of CL has been shown to accelerate and improve the learning process (e.g. Selfridge et al., 1985; Bengio et al., 2009).
Revised: Empirically, the use of CL has been shown to accelerate and improve the learning process (e.g. Selfridge et al., 1985; Bengio et al., 2009).

Original: In this work, we aim at extending the understanding of CL in the context of deep neural learning.
Revised: In this work, we aim at extending the understanding of CL in the context of deep neural learning.

Original: More speciﬁcally, we wish to understand to what extent curriculum can improve the accuracy and convergence rate of deep neural networks.
Revised: More speciﬁcally, we wish to understand to what extent curriculum can improve the accuracy and convergence rate of deep neural networks.

Original: The main challenge in making CL practical is, arguably, ﬁnding a way to construct a good curriculum for a newly unseen database.
Revised: The main challenge in making CL practical is, arguably, ﬁnding a way to construct a good curriculum for a newly unseen database.

Original: In order to do so, we investigate two ideas motivated by transfer learning and bootstrapping respectively.
Revised: In order to do so, we investigate two ideas motivated by transfer learning and bootstrapping respectively.

Original: When establishing a curriculum for human students, teachers need to arrange the material in a way that will present simple concepts before harder ones, so that the abstraction of simple ideas can help the student grasp more complex ones (Hunkins & Ornstein, 2016).
Revised: When establishing a curriculum for human students, teachers need to arrange the material in a way that will present simple concepts before harder ones, so that the abstraction of simple ideas can help the student grasp more complex ones (Hunkins & Ornstein, 2016).

Original: However, sorting the concepts by difﬁculty is not sufﬁcient.
Revised: However, sorting the concepts by difﬁculty is not sufﬁcient.

Original: The teacher also needs to attend to the pace by which the material is presented – going over the simple ideas too fast may lead to more confusion than beneﬁt, while moving along too slowly may lead to boredom and unproductive learning (Hunkins & Ornstein, 2016).
Revised: The teacher also needs to attend to the pace by which the material is presented – going over the simple ideas too fast may lead to more confusion than beneﬁt, while moving along too slowly may lead to boredom and unproductive learning (Hunkins & Ornstein, 2016).

Original: These principles can also be beneﬁcial when the learner is a neural network.
Revised: These principles can also be beneﬁcial when the learner is a neural network.

Revised: Speciﬁcally, formalizing and generalizing what was implicitly done in Weinshall et al.

Original: Speciﬁcally, we decompose the problem of CL and deﬁne two separate - but closely related - functions.
Revised: (2018), we decompose the problem of CL and deﬁne two separate - but closely related - functions.

Original: The ﬁrst function, termed scoring function , determines the ”hardness” or ”complexity” of each example in the data.
Revised: The ﬁrst function, termed scoring function , determines the ”hardness” or ”complexity” of each example in the data.

Original: The scoring function enables us to sort the data by concept difﬁculty, allowing us to present to the network the easier (and presumably simpler) examples ﬁrst.
Revised: The scoring function enables us to sort the data by concept difﬁculty, allowing us to present to the network the easier (and presumably simpler) examples ﬁrst.

Original: The underlying assumption is that generalization from the easier examples can simplify the learning of harder examples in the data.
Revised: The underlying assumption is that generalization from the easier examples can simplify the learning of harder examples in the data.

Original: The second function, termed pacing function , determines the pace by which data is presented to the network.
Revised: The second function, termed pacing function , determines the pace by which data is presented to the network.

Original: The pace depends on both the data itself and the learner.
Revised: The pace depends on both the data itself and the learner.

Original: In our work, we analyze several scoring and pacing functions, investigating their inter-dependency and presenting ways to combine them in order to achieve faster learning and better generalization.
Revised: In our work, we analyze several scoring and pacing functions, investigating their inter-dependency and presenting ways to combine them in order to achieve faster learning and better generalization.

Original: The main challenge is, arguably, how to obtain an effective scoring function without additional human supervision.
Revised: The main challenge is, arguably, how to obtain an effective scoring function without additional human supervision.

Original: To this end we investigate two approaches, each providing a different estimator for the ideal scoring function : (i) Knowledge transfer.
Revised: To this end we investigate two approaches, each providing a different estimator for the ideal scoring function : (i) Knowledge transfer.

Original: The ﬁrst scoring function is based on transfer learning from networks trained on the large and versatile Imagenet dataset (Deng et al., 2009).
Revised: The ﬁrst scoring function is based on transfer learning from networks trained on the large and versatile Imagenet dataset (Deng et al., 2009; Weinshall et al., 2018).

Original: (ii) Bootstrapping.
Revised: (ii) Bootstrapping.

Original: The second scoring function is based on self-tutoring - we train the network once without curriculum, then use the resulting classiﬁer to rank the training data in order to train the same network again from scratch.
Revised: The second scoring function is based on self-tutoring - we train the network once without curriculum, then use the resulting classiﬁer to rank the training data in order to train the same network again from scratch.

Original: Both scoring functions are shown in Section 3 to speed up learning and improve the generalization of neural networks.
Revised: Both scoring functions are shown in Sectionto speed up learning and improve the generalization of neural networks.

Original: A third scoring function is motivated by the closely related work on Self-Paced Learning (SPL) by Kumar et al. SPL is an iterative procedure where a weight is assigned to  each training point in a manner inversely proportional to the difﬁculty of the point, with higher weights assigned to easier points (Jiang et al., 2014; 2015).
Revised: In many approaches, including Self-Paced Learning (SPL), Active-Learning and hard example mining (Kumar et al., 2010; Schein & Ungar, 2007; Shrivastava et al., 2016), the mini-batches which presented to the learner model are as sampled dynamically, based at each time point on the current hypothesis of the model.

Original: However, there is a crucial difference between the two approaches: while in CL we typically measure the difﬁculty ofa point by its loss with respect to the desired ﬁnal hypothesis, SPL measures the loss with respect to  the current hypothesis.
Revised: While in some contexts these approaches are beneﬁcial (Chang et al., 2017; Zhang et al., 2017), they are based on the knowledge of the student at a speciﬁc time point. While a student can report what is easy/hard for it right now, it might be oblivious to some aspects of the bigger problem at hand, ignoring concepts which if learned early, could prove helpful in a later time.

Original: Importantly, Weinshall et al.
Revised: In the context of linear regression loss, Weinshall et al.

Original: (2018) showed, in the context of the linear regression loss, that  while it is beneﬁcial to prefer points with lower loss with respect to the target hypothesis as suggested by CL, it is on the other hand beneﬁcial to prefer points with higher loss with respect to the current hypothesis in agreement with hard data mining (Shrivastava et al., 2016) and boosting, contrary to SPL.
Revised: (2018) showed that such distinction indeed holds: while it is beneﬁcial to prefer points with lower loss with respect to the target hypothesis as suggested by CL, it is on the other hand beneﬁcial to prefer points with higher loss with respect to the current hypothesis in agreement with hard data mining (Shrivastava et al., 2016) and boosting, contrary to SPL.

Original: To clarify this somewhat confusing point, we have implemented a simpliﬁed version of the procedure described above, where the scoring function is based on the loss of the training points with respect to the current hypothesis,  in ascending order.
Revised: To examine this somewhat confusing point, we have implemented a simpliﬁed version of the procedure described above, where the scoring function is based on the loss of the training points with respect to the current hypothesis, both in ascending and descending orders.

Original: As predicted by the theory, this variant ofthe SPLheuristics is not beneﬁcial, and it leads to deterioration in performance. Thus, when giving preference to easier points and where difﬁculty is measured by the loss function, it is crucial to measure the loss with respect to the target hypothesis and not with respect to the current hypothesis .
Revised: These variants of SPL and hard example mining respectively learn slower and reach lower ﬁnal accuracy when compared to self-taught, throughout all of our experiments.

Original: We have also investigated three pacing functions .
Revised: We have also investigated three pacing functions .

Original: (i) Fixed exponential pacing presents the learner initially with a small percentage of the data, increasing the amount exponentially every ﬁxed number of learning iterations.
Revised: (i) Fixed exponential pacing presents the learner initially with a small percentage of the data, increasing the amount exponentially every ﬁxed number of learning iterations.

Original: (ii) Varied exponential pacing allows the number of iterations in each step to vary as well.
Revised: (ii) Varied exponential pacing allows the number of iterations in each step to vary as well.

Original: (iii) Single-step pacing is a simpliﬁed version of the ﬁrst protocol, where mini-batches are initially sampled from a ﬁxed fraction of the data that includes the easiest examples, after which mini-batches are sampled from the whole data as usual.
Revised: (iii) Single-step pacing is a simpliﬁed version of the ﬁrst protocol, where mini-batches are initially sampled from a ﬁxed fraction of the data that includes the easiest examples, after which mini-batches are sampled from the whole data as usual.

Original: We show that the three functions have comparable performance, and analyze the complexity of their use.
Revised: We show that the three functions have comparable performance, and analyze the complexity of their use.

Original: Previous work.
Revised: Previous work.

Original: While remaining in the fringes of machine learning, there has been some recent work on CL and its applications.
Revised: While remaining in the fringes of machine learning, there has been some recent work on CL and its applications.

Original: Bengio et al.
Revised: Bengio et al.

Original: (2009) introduced the idea of CL for machine learning algorithms, showing simple examples where CL beneﬁts learning.
Revised: (2009) introduced the idea of CL for machine learning algorithms, showing simple examples where CL beneﬁts learning.

Original: Weinshall et al.
Revised: Weinshall et al.

Original: (2018) proved that CL boosts the speed of convergence in the convex case of linear regression.
Revised: (2018) proved that CL boosts the speed of convergence in the convex case of linear regression.

Original: Otherwise most prior art is empirical, and almost always ranking by difﬁculty (i.e., the scoring function deﬁned above) is provided by the user based on prior knowledge (in other words, supervision) as in Jesson et al.
Revised: Otherwise most prior art is empirical, and almost always ranking by difﬁculty (i.e., the scoring function deﬁned above) is provided by the user based on prior knowledge (in other words, supervision) as in Jesson et al.

Original: In  related areas such as robotics, it is common practice to introduce easier concepts to a learner followed by more complex concepts in ascending order of difﬁculty, where difﬁculty is judged by the human operator (Murphy et al., 2008).
Revised: In a closely related line of works, a pair of teacher and student networks are trained simultaneously, where mini-batches for the student network are sampled dynamically by the teacher, based on the student’s output in each time point (Jiang et al., 2018; Fan et al., 2018).

Original: In the evaluation of computer programs (Zaremba & Sutskever, 2014) or machine translation (Amodei et al., 2016), problem-dependent automatic criteria for the evaluation of task difﬁculty have been used.
Revised: As opposed to our method, these works base the curriculum on the current hypothesis of the students, and achieve better performance for corrupted (Jiang et al., 2018) or smaller (Fan et al., 2018) datasets, instead of improved generalization on the original dataset.

Original: Our contribution , with respect to this previous work, is to provide a formal deﬁnition of CL algorithms by way of 2 functions for scoring and pacing, analyze and comparatively evaluate these functions, and show how CL can beneﬁt learning in CNNs even without human supervision about the ranking of examples by difﬁculty and in a problem-free manner.
Revised: Our contribution , with respect to this previous work, is to provide a formal deﬁnition of CL algorithms by way of 2 functions for scoring and pacing, analyze and comparatively evaluate these functions, and show how CL can beneﬁt learning in CNNs even without human supervision about the ranking of examples by difﬁculty and in a problem-free manner.

Original:  C URRICULUM L EARNING
Revised:  C URRICULUM L EARNING

Original: Curriculum learning deals with the question of how to use prior knowledge about the difﬁculty of the training examples , in order to sample each mini-batch non-uniformly and thus boost the rate of learning and the accuracy of the ﬁnal classiﬁer.
Revised: Curriculum learning deals with the question of how to use prior knowledge about the difﬁculty of the training examples , in order to sample each mini-batch non-uniformly and thus boost the rate of learning and the accuracy of the ﬁnal classiﬁer.

Original: The paradigm of CL is based on the intuition that it helps the learning process when the learner is presented with simple concepts ﬁrst.
Revised: The paradigm of CL is based on the intuition that it helps the learning process when the learner is presented with simple concepts ﬁrst.

Original: 2. N OTATIONS AND D EFINITIONS
Revised: 2. N OTATIONS AND D EFINITIONS

Original: Let X = { ( x i , y i ) } Ni =1 denote the data, where x
Revised: Let X = { ( x i , y i ) } N i =1 denote the data, where x

Original: i ∈ R
Revised: i ∈ R

Original: d denotes a single data point and y i ∈
Revised: d denotes a single data point and y i ∈

Original: [ K ] its corresponding label.
Revised: [ K ] its corresponding label.

Original: Let F θ : R d → [ K ] denote the target classiﬁer (or learner), and mini-batch B ⊆
Revised: Let F θ : R d → [ K ] denote the target classiﬁer (or learner), and mini-batch B ⊆

Original: X denote a subset of X .
Revised: X denote a subset of X .

Original: In the most common training procedure, which is a robust variant of
Revised: In the most common training procedure, which is a robust variant of

Original: Stochastic Gradient Descent (SGD), F θ is trained sequentially when given as input a sequence of mini-batches [ B 1 , ..., B M ] (Shalev-Shwartz & Ben-David, 2014).
Revised: Stochastic Gradient Descent (SGD), F θ is trained sequentially when given as input a sequence of mini-batches [ B 1 , ..., B M ] (Shalev-Shwartz & Ben-David, 2014).

Original: The common approach – denoted vanilla in the following sections – samples each mini-batch B
Revised: The common approach – denoted vanilla in the following sections – samples each mini-batch B

Original: i uniformly from X .
Revised: i uniformly from X .

Original: Both in the common approach and in our work, the size of each mini-batch remains constant, to be considered as a hyper-parameter deﬁning the learner.
Revised: Both in the common approach and in our work, the size of each mini-batch remains constant, to be considered as a hyper-parameter deﬁning the learner.

Original: We measure the difﬁculty of point x
Revised: We measure the difﬁculty of point x

Original: i by its minimal loss with respect to the set of optimal hypotheses under consideration.
Revised: i by its minimal loss with respect to the set of optimal hypotheses under consideration.

Original: We deﬁne a scoring function (or a ” difﬁculty ” function) to be any function f : X → R , and say that example ( x i , y i ) is more ”difﬁcult” than example ( x j , y j ) if f ( x i , y i ) >
Revised: We deﬁne a scoring function (or a ” difﬁculty ” function) to be any function f : X → R , and say that example ( x i , y i ) is more ”difﬁcult” than example ( x j , y j ) if f ( x i , y i ) >

Original: f ( x j , y j ) .
Revised: f ( x j , y j ) .

Original: Choosing f is the main challenge of CL, as it encodes the prior knowledge of the teacher.
Revised: Choosing f is the main challenge of CL, as it encodes the prior knowledge of the teacher.

Original: We deﬁne a pacing function to be a function g F θ : [ M ] → [ N ] , which may depend on the learner
Revised: We deﬁne a pacing function to be a function g F θ : [ M ] → [ N ] , which may depend on the learner

Original: F θ .
Revised: F θ .

Original: The pacing function is used to determine a sequence of subsets X (cid:48) 1 , ..., X (cid:48)
Revised: The pacing function is used to determine a sequence of subsets X (cid:48) 1 , ..., X (cid:48)

Original: M ⊆ X , of size | X (cid:48)
Revised: M ⊆ X , of size | X (cid:48)

Original: i | =
Revised: i | =

Original: g F θ ( i ) , from which { B i } Mi =1 are sampled uniformly.
Revised: g F θ ( i ) , from which { B i } Mi =1 are sampled uniformly.

Original: In CL the i -th subset X (cid:48)
Revised: In CL the i -th subset X (cid:48)

Original: i includes the ﬁrst g F θ ( i ) elements of the training data when sorted by the scoring function f in an ascending order.
Revised: i includes the ﬁrst g F θ ( i ) elements of the training data when sorted by the scoring function f in an ascending order.

Original: Although the choice of the subset can be encoded in the distribution from which each B i is sampled, adding a pacing function simpliﬁes the exposition and analysis.
Revised: Although the choice of the subset can be encoded in the distribution from which each B i is sampled, adding a pacing function simpliﬁes the exposition and analysis.

Original: 2. C URRICULUM L EARNING M ETHOD
Revised: 2. C URRICULUM L EARNING M ETHOD

Original: Together, each scoring function f and pacing function g F θ deﬁne a curriculum.
Revised: Together, each scoring function f and pacing function g F θ deﬁne a curriculum.

Original: Any learning algorithm which uses the ensuing sequence [ B i ] Mi
Revised: Any learning algorithm which uses the ensuing sequence [ B i ] Mi

Original: =1 is a curriculum learning algorithm .
Revised: =1 is a curriculum learning algorithm .

Original: We note that in order to avoid bias when picking a subset of the N examples for some N , it is important to keep the sample balanced with the same number of examples from each class as in the training set.
Revised: We note that in order to avoid bias when picking a subset of the N examples for some N , it is important to keep the sample balanced with the same number of examples from each class as in the training set.

Original: Pseudo-code for the CL algorithm is given in Alg.
Revised: Pseudo-code for the CL algorithm is given in Alg.

Original: In order to narrow down the speciﬁc effects of using a scoring function based on ascending difﬁculty level, we examine two control conditions.
Revised: In order to narrow down the speciﬁc effects of using a scoring function based on ascending difﬁculty level, we examine two control conditions.

Original: Speciﬁcally, we deﬁne 2 additional scoring functions and corresponding algorithms: (i) The anti-curriculum algorithm uses the scoring function f (cid:48)
Revised: Speciﬁcally, we deﬁne 2 additional scoring functions and corresponding algorithms: (i) The anti-curriculum algorithm uses the scoring function f (cid:48)

Original: = − f , where the training examples are sorted in a descending order ; that results in presenting the harderAlgorithm examples before the easier ones.
Revised: = − f , where the training examples are sorted in a descending order ; that results in presenting the harder examples before the easier ones.

Original: (ii)
Revised: (ii)

Original: The random-curriculum algorithm (henceforth denoted random) uses a scoring function where the training examples are randomly sorted.
Revised: The random-curriculum algorithm (henceforth denoted random) uses a scoring function where the training examples are randomly sorted.

Original: 2. S CORING AND P ACING F UNCTIONS
Revised: 2. S CORING AND P ACING F UNCTIONS

Original: We evaluate two scoring functions : (i) Transfer scoring function , computed as follows:
Revised: We evaluate two scoring functions : (i) Transfer scoring function , computed as follows:

Original: First, take the pre-trained Inception network (Szegedy et al., 2016) and run each training image through it, using the activation levels of its penultimate layer as a feature vector (Caruana, 1995).
Revised: First, take the pre-trained Inception network (Szegedy et al., 2016) and run each training image through it, using the activation levels of its penultimate layer as a feature vector (Caruana, 1995).

Original: Second, use these features to train a classiﬁer and use its conﬁdence score as the scoring function for each image 1 .
Revised: Second, use these features to train a classiﬁer and use its conﬁdence score as the scoring function for each image 1 .

Original: (ii)
Revised: (ii)

Original: Self-taught scoring function , computed as follows:
Revised: Self-taught scoring function , computed as follows:

Original: First, train the network using uniformly sampled mini-batches (the vanilla method).
Revised: First, train the network using uniformly sampled mini-batches (the vanilla method).

Original: Second, compute this network’s conﬁdence score for each image to deﬁne a scoring function 2 .
Revised: Second, compute this network’s conﬁdence score for each image to deﬁne a scoring function 2 .

Original: Although the pacing function can be any function g F θ : [ M ] → [ N ] , we limit ourselves to monotonic increasing functions so that the likelihood of the easier examples can only decrease.
Revised: Although the pacing function can be any function g F θ : [ M ] → [ N ] , we limit ourselves to monotonic increasing functions so that the likelihood of the easier examples can only decrease.

Original: For simplicity, g F θ is limited to staircase functions.
Revised: For simplicity, g F θ is limited to staircase functions.

Original: Thus each pacing function is deﬁned by the following hyper-parameters, where step denotes all the learning iterations during which g F θ remains constant: step length - the number of iterations in each step ; increase - an exponential factor used to increase the size of the data used for sampling mini-batches in each step ; starting percent - the fraction of the data in the initial step .
Revised: Thus each pacing function is deﬁned by the following hyper-parameters, where step denotes all the learning iterations during which g F θ remains constant: step length - the number of iterations in each step ; increase - an exponential factor used to increase the size of the data used for sampling mini-batches in each step ; starting percent - the fraction of the data in the initial step .

Original: An illustration of these parameters can be seen in Fig.
Revised: An illustration of these parameters can be seen in Fig.

Original: We evaluate three pacing functions : (i) Fixed exponential pacing has a ﬁxed step length , and exponentially increasing data size in each step .
Revised: We evaluate three pacing functions : (i) Fixed exponential pacing has a ﬁxed step length , and exponentially increasing data size in each step .

Original: Formally, the pacing function is given by:
Revised: Formally, the pacing function is given by:

Original: [Equation]
Revised: [Equation]

Original:  (ii) Varied exponential pacing , which allows step length to vary as well 3 :
Revised: [Table] [Table] (ii) Varied exponential pacing , which allows step length to vary as well 3 :

Original: [Equation]
Revised: [Equation]

Original: The total number of steps can be calculated from starting percent and increase :
Revised: The total number of steps can be calculated from starting percent and increase :

Original: [Equation]
Revised: [Equation]

Revised: (iii) Single

Original: (iii) Single step pacing , which is a simpliﬁcation of the staircase function into a step function:
Revised: step pacing , which is a simpliﬁcation of the staircase function into a step function:

Original: [Equation]
Revised: [Equation]

Original: This function has only 2 hyper-parameters hence it is simpler to use than 
Revised: This function has only 2 hyper-parameters, hence it is simpler to use than the previous two.

Original: the

Original: previous two

Original:  E MPIRICAL E VALUATION
Revised:  E MPIRICAL E VALUATION

Original: Methodology.
Revised: Methodology.

Original: All the code used in this work will be published upon acceptance.
Revised: All the code used in this work will be published upon acceptance.

Original: We deﬁneempirical cases: Case 1 replicates the experimental design described in (Weinshall et al., 2018), by using the same dataset and network architecture.
Revised: We deﬁneempirical cases: Case 1 replicates the experimental design described in (Weinshall et al., 2018), by using the same dataset and network architecture.

Original: The dataset is the ”small mammals” superclass of CIFAR-100 (Krizhevsky & Hinton, 2009), containing a subset of 3000 images from CIFAR100, divided into 5 classes of small mammals (hamster, mouse, rabbit, shrew, squirrel).
Revised: The dataset is the ”small mammals” superclass of CIFAR-100 (Krizhevsky & Hinton, 2009), containing a subset of 3000 images from CIFAR100, divided into 5 classes of small mammals (hamster, mouse, rabbit, shrew, squirrel).

Original: Each class contains 500 training images and 100 test images.
Revised: Each class contains 500 training images and 100 test images.

Original: The neural network is a moderate size handcrafted convolutional network, whose architecture details can be found in Appendix C. Cases 2 and 3 adopt the same architecture used above while being applied to the entire CIFAR-10 and CIFAR100 datasets, where the network’s output layer is adjusted to size 10 and 100 respectively.
Revised: The neural network is a moderate size handcrafted convolutional network, whose architecture details can be found in Appendix C. Cases 2 and 3 adopt the same architecture used above while being applied to the entire CIFAR-10 and CIFAR100 datasets, where the network’s output layer is adjusted to size 10 and 100 respectively.

Original: Caseuses a public-domain VGG-based architecture 4 , which achieves competitive results (Simonyan &
Revised: Caseuses a public-domain VGG-based architecture 4 , which achieves competitive results (Simonyan &

Original: Zisserman, 2014; Liu & Deng, 2015), to classify the CIFAR-100 dataset.
Revised: Zisserman, 2014; Liu & Deng, 2015), to classify the CIFAR-100 dataset.

Original: Hyper-parameter tuning.
Revised: Hyper-parameter tuning.

Original: As in all empirical studies involving deep learning, the results are quite sensitive to the values of the hyper-parameters, hence parameter tuning is required.
Revised: As in all empirical studies involving deep learning, the results are quite sensitive to the values of the hyper-parameters, hence parameter tuning is required.

Original: Issues related to how a fair comparison between the different conditions is achieved are discussed in Appendix B.
Revised: Issues related to how a fair comparison between the different conditions is achieved are discussed in Appendix B.

Original: In practice, in order to reduce the computation time of parameter tuning, we varied only the ﬁrst 2 step length instances in the varied exponential pacing condition.
Revised: In practice, in order to reduce the computation time of parameter tuning, we varied only the ﬁrst 2 step length instances in the varied exponential pacing condition.

Original: Accordingly, ﬁxed exponential pacing , varied exponential pacing and single step pacing deﬁne 3, 5 and 2 new hyper-parameters respectively, referred to henceforth as the pacing hyper-parameters.
Revised: Accordingly, ﬁxed exponential pacing , varied exponential pacing and single step pacing deﬁne 3, 5 and 2 new hyper-parameters respectively, referred to henceforth as the pacing hyper-parameters.

Original: In the CL framework, the use of a pacing function affects the optimal values of other hyperparameters, in particular, the learning rate.
Revised: In the CL framework, the use of a pacing function affects the optimal values of other hyperparameters, in particular, the learning rate.

Original: Speciﬁcally, since it signiﬁcantly reduces the size of the data-set from which each mini-batch is sampled, this has the concomitant effect of increasing the effective learning rate.
Revised: Speciﬁcally, since it signiﬁcantly reduces the size of the data-set from which each mini-batch is sampled, this has the concomitant effect of increasing the effective learning rate.

Original: As a result, when using the ﬁxed exponential or the single step pacing functions , the learning rate must be tuned separately for every test condition.
Revised: As a result, when using the ﬁxed exponential or the single step pacing functions , the learning rate must be tuned separately for every test condition.

Original: As traditionally done (e.g Simonyan & Zisserman, 2014; Szegedy et al., 2016; He et al., 2016), we set an initial learning rate and decrease it exponentially every ﬁxed number of iterations.
Revised: As traditionally done (e.g Simonyan & Zisserman, 2014; Szegedy et al., 2016; He et al., 2016), we set an initial learning rate and decrease it exponentially every ﬁxed number of iterations.

Original: This method gives rise tolearning rate hyper-parameters which require tuning: (i) the initial learning rate; (ii) the factor by which the learning rate is decreased; (iii) the length of each step with constant learning rate 5 .
Revised: This method gives rise tolearning rate hyper-parameters which require tuning: (i) the initial learning rate; (ii) the factor by which the learning rate is decreased; (iii) the length of each step with constant learning rate 5 .

Original: When varied exponential pacing is used, varying step length has the opposite concomitant effect on the learning rate, as it determines the number of mini-batch samples in each step .
Revised: When varied exponential pacing is used, varying step length has the opposite concomitant effect on the learning rate, as it determines the number of mini-batch samples in each step .

Original: Effective tuning of this parameter can make the additional tuning of parameters affecting the learning rate redundant.
Revised: Effective tuning of this parameter can make the additional tuning of parameters affecting the learning rate redundant.

Original: In practice, in order to reach the improvement achieved by the ﬁxed exponential pacing , we decrease the corresponding learning rate parameters used in the vanilla condition by some small factor 6 .
Revised: In practice, in order to reach the improvement achieved by the ﬁxed exponential pacing , we decrease the corresponding learning rate parameters used in the vanilla condition by some small factor 6 .

Original: 3. R ESULTS : CL B ENEFITS L EARNING
Revised: 3. R ESULTS : CL B ENEFITS L EARNING

Original: Case 1: A moderate size network is trained to distinguish 5 classes from CIFAR-100, which are members of the same super-class as deﬁned in the original dataset.
Revised: Case 1: A moderate size network is trained to distinguish 5 classes from CIFAR-100, which are members of the same super-class as deﬁned in the original dataset.

Original: Results are shown in Fig.
Revised: Results are shown in Fig.

Original: Curriculum learning is clearly and signiﬁcantly beneﬁcial - learning starts faster, and converges to a better solution.
Revised: Curriculum learning is clearly and signiﬁcantly beneﬁcial - learning starts faster, and converges to a better solution.

Original: We observe that the performance of CL with a random scoring function is similar to vanilla , indicating that the main reason for the improvement achieved by CL is due to its beneﬁcial transfer scoring function .
Revised: We observe that the performance of CL with a random scoring function is similar to vanilla , indicating that the main reason for the improvement achieved by CL is due to its beneﬁcial transfer scoring function .

Original: In fact, although tuned separately, the learning rate hyper-parameters for both the random and the curriculum test conditions are very similar, conﬁrming that the improved performance is due to the use of an effective transfer scoring function .
Revised: In fact, although tuned separately, the learning rate hyper-parameters for both the random and the curriculum test conditions are very similar, conﬁrming that the improved performance is due to the use of an effective transfer scoring function .

Original: To check the robustness of these results, we repeated the same empirical evaluation using different super-classes of CIFAR-100, with similar results (see Appendix A).
Revised: To check the robustness of these results, we repeated the same empirical evaluation using different super-classes of CIFAR-100, with similar results (see Appendix A).

Original: Interestingly, we note that the observed advantage of CL is more signiﬁcant when the task is more difﬁcult (i.e. lower vanilla test accuracy).
Revised: Interestingly, we note that the observed advantage of CL is more signiﬁcant when the task is more difﬁcult (i.e. lower vanilla test accuracy).

Original: The reason may be that in easier problems there is a sufﬁcient number of easy examples in each mini-batch even without CL.
Revised: The reason may be that in easier problems there is a sufﬁcient number of easy examples in each mini-batch even without CL.

Original: Although the results reported here are based on transfer from the
Revised: Although the results reported here are based on transfer from the

Original: Inception network, we are able to obtain the same results using scoring functions based on transfer learning from other large networks, including VGG-16 and Resnet, as shown in Appendix A.
Revised: Inception network, we are able to obtain the same results using scoring functions based on transfer learning from other large networks, including VGG-16 and Resnet, as shown in Appendix A.

Original: Cases 2 and 3: Similar empirical evaluation as in case 1 , using the same moderate size network to classify two benchmark datasets.
Revised: Cases 2 and 3: Similar empirical evaluation as in case 1 , using the same moderate size network to classify two benchmark datasets.

Original: The results are shown in Fig.
Revised: The results are shown in Fig.

Original: Like before, the test accuracy in the curriculum test condition increases faster and achieves better ﬁnal performance in both cases, as compared to the vanilla test condition.
Revised: Like before, the test accuracy in the curriculum test condition increases faster and achieves better ﬁnal performance in both cases, as compared to the vanilla test condition.

Original: The beneﬁcial effect of CL is larger when classifying the
Revised: The beneﬁcial effect of CL is larger when classifying the

Original: CIFAR-100 dataset, which is a harder dataset.
Revised: CIFAR-100 dataset, which is a harder dataset.

Original: [Figure]
Revised: [Figure]

Original: Case 4: Similar empirical evaluation as in case 1 , using a competitive public-domain architecture.
Revised: Case 4: Similar empirical evaluation as in case 1 , using a public-

Original: Speciﬁcally, we use the Inception-based transfer scoring function to train a VGG-based network  (Liu & Deng, 2015) to classify the CIFAR-100 dataset.
Revised: Speciﬁcally, we use the Inception-based transfer scoring function to train a VGG-based network [Figure] (Liu & Deng, 2015) to classify the CIFAR-100 dataset.

Original: Differently from the previous cases, here we use the varied exponential pacing function with a slightly reduced learning rate, as it has the fewest hyper-parameters to tune, an important factor when training such a big network.
Revised: Differently from the previous cases, here we use the varied exponential pacing function with a slightly reduced learning rate, as it has the fewest hyper-parameters to tune, an important factor when training such a big network.

Original: Results are shown in Fig.
Revised: Results are shown in Fig.

Original: 4a (with no data augmentation), showing the same qualitative results as in the previous cases;
Revised: 4a (with no data augmentation), showing the same qualitative results as in the previous cases;

Original: CL gives a smaller beneﬁt, but the beneﬁt is still signiﬁcant.
Revised: CL gives a smaller beneﬁt, but the beneﬁt is still signiﬁcant.

Original: 3. S ELF -T AUGHT C URRICULUM L EARNING VS . S ELF -P ACED L EARNING
Revised: Case 5: Similar empirical evaluation as in case 1 , using the same moderate size network to distinguish 7 classes of cats from the ImageNet dataset 7 . The results are shown in Fig. Again, the test accuracy in the curriculum test condition increases faster and achieves better ﬁnal performance in the curriculum case, as compared to the vanilla test condition. [Figure]

Original: Curriculum learning is closely related to the idea of Self-Paced Learning (SPL), an iterative procedure where higher weights are given to training examples that have lower cost with respect to the current hypothesis.
Revised: Curriculum learning is closely related to the idea of Self-Paced Learning (SPL), an iterative procedure where higher weights are given to training examples that have lower cost with respect to the current hypothesis.

Original: In fact, SPL may appear similar, or closely related, to the idea of self-taught learning.
Revised: In fact, SPL may appear similar, or closely related, to the idea of self-taught learning.

Original: The main difference between the methods is that self-paced learning determines the scoring function according to the loss with respect to the current hypothesis (or network), while the self-taught scoring function is based on the loss with respect to the ﬁnal hypothesis of a trained [Figure]
Revised: The main difference between the methods is that self-paced learning determines the scoring function according to the loss with respect to the current hypothesis (or network), while the self-taught scoring function is based on the loss with respect to the ﬁnal hypothesis of a trained network.

Original: network. In accordance, we deﬁne the self-paced scoring function , where each point is scored by its loss with respect to the current network.
Revised: In accordance, we deﬁne the self-paced scoring function , where each point is scored by its loss with respect to the current network.

Original: Note that when using CL to optimize the linear regression loss (see introduction), self-taught curriculum and self-paced learning are discordant.
Revised: Note that when using CL to optimize the linear regression loss (see introduction), self-taught curriculum and self-paced learning are discordant.

Original: To compare the self-taught scoring function and the self-paced scoring function , we investigate their effect on CL in the context of empirical case 1 .
Revised: To compare the self-taught scoring function and the self-paced scoring function , we investigate their effect on CL in the context of empirical case 1 .

Original: Results are shown in Fig. 4b.
Revised: Results are shown in Fig. 4b.

Original: As expected, we see that CL using the self-taught scoring function improves the test accuracy throughout the entire learning session.
Revised: As expected, we see that CL using the self-taught scoring function improves the test accuracy throughout the entire learning session.

Original: On the other hand, CL training using the self-paced scoring function decreases the test accuracy throughout.
Revised: On the other hand, CL training using the self-paced scoring function decreases the test accuracy throughout.

Original: This decrease is more prominent at the beginning of the learning, where most of the beneﬁcial effects of the curriculum are observed, suggesting that the self-paced scoring function can signiﬁcantly delay learning.
Revised: This decrease is more prominent at the beginning of the learning, where most of the beneﬁcial effects of the curriculum are observed, suggesting that the self-paced scoring function can signiﬁcantly delay learning.

Original: 3. T HE S CORING F UNCTION : A NALYSIS AND E MPIRICAL E VALUATION
Revised: 3. T HE S CORING F UNCTION : A NALYSIS AND E MPIRICAL E VALUATION

Original: In order to analyze the effects of transfer based scoring functions , we turn to analyze the gradients of the network’s weights w.r.t the empirical loss.
Revised: In order to analyze the effects of transfer based scoring functions , we turn to analyze the gradients of the network’s weights w.r.t the empirical loss.

Original: We evaluate the gradients using a pre-trained vanilla network in the context of case 1 .
Revised: We evaluate the gradients using a pre-trained vanilla network in the context of case 1 .

Original: First, for each method and each scoring function , we collect the subset of points used to sample the ﬁrst mini-batch according to the pacing function g F θ (1) 7 .
Revised: First, for each method and each scoring function , we collect the subset of points used to sample the ﬁrst mini-batch according to the pacing function g F θ (1) 8 .

Original: For comparison, we also consider the set of all training points, which are used to compute the exact gradient of the empirical loss in batch learning using GD.
Revised: For comparison, we also consider the set of all training points, which are used to compute the exact gradient of the empirical loss in batch learning using GD.

Original: We then compute the corresponding set of gradients for the training points in each of these subsets of training points, treating each layer’s parameters as a single vector, and subsequently estimate the gradients’ mean and total variance 8 , used to evaluate the coherence of the gradients in the ﬁrst mini-batch of each scoring function .
Revised: We then compute the corresponding set of gradients for the training points in each of these subsets of training points, treating each layer’s parameters as a single vector, and subsequently estimate the gradients’ mean and total variance 9 , used to evaluate the coherence of the gradients in the ﬁrst mini-batch of each scoring function .

Original: The
Revised: The

Original: Euclidean distance between the mean gradient in the different conditions is used to estimate the similarity between the different scoring functions , based on the average preferred gradient.
Revised: Euclidean distance between the mean gradient in the different conditions is used to estimate the similarity between the different scoring functions , based on the average preferred gradient.

Original: We can now compare the set of gradients thus deﬁned using three transfer scoring functions , which differ in the parent network used for scoring the points: ’VGG-16’, ’Resnet’, and ’Inception’.
Revised: We can now compare the set of gradients thus deﬁned using three transfer scoring functions , which differ in the parent network used for scoring the points: ’VGG-16’, ’Resnet’, and ’Inception’.

Original: We include in the comparison the gradients of the random scoring function denoted ’Random’, and the gradients of the whole batch of training data denoted ’All’.
Revised: We include in the comparison the gradients of the random scoring function denoted ’Random’, and the gradients of the whole batch of training data denoted ’All’.

Original: Results are shown in Fig.
Revised: Results are shown in Fig.

Revised: [Figure]

Original: We see in Fig.
Revised: We see in Fig.

Original: 5a - blue bars - that the average gradient vectors, computed based on the 3 transfer scoring functions , are quite similar to each other.
Revised: 6a - blue bars - that the average gradient vectors, computed based on the 3 transfer scoring functions , are quite similar to each other.

Original: This suggests that they are pointing towards nearby local minima in parameters space.
Revised: This suggests that they are pointing towards nearby local minima in parameters space.

Original: We also see - green bar - that the average gradient vector computed using a random subset of examples resembles the exact empirical gradient computed using all the training data.
Revised: We also see - green bar - that the average gradient vector computed using a random subset of examples resembles the exact empirical gradient computed using all the training data.

Original: This suggests that a random subset provides a reasonable estimate of the true empirical gradient.
Revised: This suggests that a random subset provides a reasonable estimate of the true empirical gradient.

Original: The picture changes completely when we compute - red bars - the distance between the average gradient corresponding to one of the 3 transfer scoring functions , and[Figure] the average random gradient or the empirical gradient.
Revised: The picture changes completely when we compute - red bars - the distance between the average gradient corresponding to one of the 3 transfer scoring functions , and the average random gradient or the empirical gradient.

Original: Now the distances are rather large, which suggests that CL by transfer indeed stirs the weights towards different local minima in parameterspace as compared tovanilla training.
Revised: Now the distances are rather large, which suggests that CL by transfer indeed stirs the weights towards different local minima in parameter to training.

Original: We see in Fig.
Revised: We see in Fig.

Original: 5b that the total variance for the 3 transfer scoring functions is much smaller than the total variance of some random subset of the whole training set.
Revised: 6b that the total variance for the 3 transfer scoring functions is much smaller than the total variance of some random subset of the whole training set.

Original: This intuitive result demonstrates the difference between training with easier examples and training with random examples, and may – at least partially – explain the need for a different learning rate when training with easier examples.
Revised: This intuitive result demonstrates the difference between training with easier examples and training with random examples, and may – at least partially – explain the need for a different learning rate when training with easier examples.

Original: 3. A LTERNATIVE P ACING F UNCTIONS
Revised: 3. A LTERNATIVE P ACING F UNCTIONS

Original: Single step pacing.
Revised: Single step pacing.

Original: Curriculum learning can be costly, and it affects the entire learning protocol via the pacing function .
Revised: Curriculum learning can be costly, and it affects the entire learning protocol via the pacing function .

Original: At the same time, we note that the main effect of the procedure takes place at the beginning of training.
Revised: At the same time, we note that the main effect of the procedure takes place at the beginning of training.

Original: This empirical observation may be due, in part, to the fact that the proposed scoring function f is based on transfer from another network trained on a different dataset, which only approximates the unknown ideal scoring function .
Revised: This empirical observation may be due, in part, to the fact that the proposed scoring function f is based on transfer from another network trained on a different dataset, which only approximates the unknown ideal scoring function .

Original: Possibly, since the scoring function is based on one local minimum in a complex optimization landscape which contains many local minima, the score given by f is more reliable for low scoring (easy) examples than high scoring (difﬁcult) examples, that may be in the vicinity of a different local minimum.
Revised: Possibly, since the scoring function is based on one local minimum in a complex optimization landscape which contains many local minima, the score given by f is more reliable for low scoring (easy) examples than high scoring (difﬁcult) examples, that may be in the vicinity of a different local minimum.

Original: [Figure]
Revised: [Figure]

Original: Once again we evaluate case 1 , using the transfer scoring function and the single step pacing function .
Revised: Once again we evaluate case 1 , using the transfer scoring function and the single step pacing function .

Original: We see improvement in the test accuracy in the curriculum test condition which resembles the improvement achieved using the exponential pacing .
Revised: We see improvement in the test accuracy in the curriculum test condition which resembles the improvement achieved using the exponential pacing .

Original: Results are shown in Fig.
Revised: Results are shown in Fig.

Original: 6a.
Revised: 7a.

Original: It is important to note that this pacing function ignores most of the prior knowledge provided by the scoring function , as it only uses a small percent of the easiest examples, and yet it achieves competitive results.
Revised: It is important to note that this pacing function ignores most of the prior knowledge provided by the scoring function , as it only uses a small percent of the easiest examples, and yet it achieves competitive results.

Original: Thus we see that in our empirical setup, most of the power of CL lies at the beginning of training.
Revised: Thus we see that in our empirical setup, most of the power of CL lies at the beginning of training.

Original: Varied exponential pacing.
Revised: Varied exponential pacing.

Original: This pacing function allows us to run a CL procedure without the need for further tuning of learning rate.
Revised: This pacing function allows us to run a CL procedure without the need for further tuning of learning rate.

Original: Once again we evaluate case 1 , ﬁxing the learning rate parameters to be the same as in the vanilla test condition, while tuning the remaining hyper-parameters as described in Section 2.3 using a grid search with cross-validation.
Revised: Once again we evaluate case 1 , ﬁxing the learning rate parameters to be the same as in the vanilla test condition, while tuning the remaining hyper-parameters as described in Section 2.3 using a grid search with cross-validation.

Original: We see improvement in the accuracy throughout the entire learning session, although smaller than the one observed with ﬁxed exponential pacing .
Revised: We see improvement in the accuracy throughout the entire learning session, although smaller than the one observed with ﬁxed exponential pacing .

Original: However, decreasing the learning rate of the vanilla by a small fraction and then tuning the curriculum parameters achieves results which are very similar to the ﬁxed exponential pacing , suggesting that this method can almost completely nullify the indirect manipulation of the learning rate in the ﬁxed exponential pacing function.
Revised: However, decreasing the learning rate of the vanilla by a small fraction and then tuning the curriculum parameters achieves results which are very similar to the ﬁxed exponential pacing , suggesting that this method can almost completely nullify the indirect manipulation of the learning rate in the ﬁxed exponential pacing function.

Original: These results are shown in Fig.
Revised: These results are shown in Fig.

Original: 6b.
Revised: 7b.

Original: 3. S UMMARY OF R ESULTS
Revised: 3. S UMMARY OF R ESULTS

Original: [Figure]

Original: Fig.
Revised: Fig.

Original: 7 summarizes the main results presented in the paper, including: curriculum with an Inceptionbased scoring function for (i) ﬁxed exponential pacing (denoted curriculum ), (ii) varied exponential pacing, and (iii) single step pacing.
Revised: 8 summarizes the main results presented in the paper, including: curriculum with an Inceptionbased scoring function for (i) ﬁxed exponential pacing (denoted curriculum ), (ii) varied exponential pacing, and (iii) single step pacing.

Original: It also shows curriculum with ﬁxed exponential pacing for (iv) self-paced scoring, and (v) self-taught scoring.
Revised: It also shows curriculum with ﬁxed exponential pacing for (iv) self-paced scoring, and (v) self-taught scoring.

Original: In addition, we plot the control conditions of vanilla , anti -curriculum, and random .
Revised: In addition, we plot the control conditions of vanilla , anti -curriculum, and random .

Original: In Fig.
Revised: In Fig.

Original: 7a we see the learning curves of the above conditions, with inset bars that depict the ﬁnal accuracy of each condition, and error bars that represent the standard error after 50 repetitions.
Revised: 8a we see the learning curves of the above conditions, with inset bars that depict the ﬁnal accuracy of each condition, and error bars that represent the standard error after 50 repetitions.

Original: All the curriculum conditions seem to improve the learning accuracy throughout the entire learning session while converging to similar performance, excluding the selfpaced scoring function which impairs learning. 
Revised: All the curriculum conditions seem to improve the learning accuracy throughout the entire learning session while converging to similar performance, excluding the selfpaced scoring function which impairs learning. [Figure]

Original: The learning curves shown in Fig.
Revised: The learning curves shown in Fig.

Original: 7a were obtained by searching for the parameters that maximize the ﬁnal accuracy.
Revised: 8a were obtained by searching for the parameters that maximize the ﬁnal accuracy.

Original: This procedure only takes into account a few data points, which makes it less robust.
Revised: This procedure only takes into account a few data points, which makes it less robust.

Original: In Fig.
Revised: In Fig.

Original: 7b we plot the bars of the ﬁnal accuracy of the learning curves obtained by searching for the parameters that maximize the Area Under the Learning Curve.
Revised: 8b we plot the bars of the ﬁnal accuracy of the learning curves obtained by searching for the parameters that maximize the Area Under the Learning Curve.

Original: AUC is positively correlated with high ﬁnal performance while being more robust.
Revised: AUC is positively correlated with high ﬁnal performance while being more robust.

Original: Comparing the different conditions using this maximization criterion gives similar qualitative results - the performance in all the curriculum conditions is still signiﬁcantly higher than the control conditions.
Revised: Comparing the different conditions using this maximization criterion gives similar qualitative results - the performance in all the curriculum conditions is still signiﬁcantly higher than the control conditions.

Original: However, now the curriculum based on the Inception-based scoring function with ﬁxed exponential pacing achieves performance that is signiﬁcantly higher than the other curriculum methods, in evidence that it is more robust.
Revised: However, now the curriculum based on the Inception-based scoring function with ﬁxed exponential pacing achieves performance that is signiﬁcantly higher than the other curriculum methods, in evidence that it is more robust.

Original:  S UMMARY AND D ISCUSSION
Revised:  S UMMARY AND D ISCUSSION

Original: Above we formally deﬁned a curriculum learning algorithm, decomposing it into two separate problems: (i) How to determine the difﬁculty of the training data (via the scoring function )?
Revised: Above we formally deﬁned a curriculum learning algorithm, decomposing it into two separate problems: (i) How to determine the difﬁculty of the training data (via the scoring function )?

Original: (ii) At which pace should the learner be shown the more advanced data (via the pacing function )?
Revised: (ii) At which pace should the learner be shown the more advanced data (via the pacing function )?

Original: We deﬁned a scoring function based on transfer learning from a large network, showing that it can both speed up the rate of learning and improve the ﬁnal accuracy.
Revised: We deﬁned a scoring function based on transfer learning from a large network, showing that it can both speed up the rate of learning and improve the ﬁnal accuracy.

Original: This was shown using a number of test cases, including particularly challenging subsets ofthe CIFAR 100 dataset, and the entire CIFAR-10 and CIFAR-100 datasets.
Revised: This was shown using a number of test cases, including particularly challenging subsets of CIFAR 100 and ImageNet datasets, and the entire CIFAR-10 and CIFAR-100 datasets.

Original: We used both a relatively small hand-crafted CNN, and a large public-domain completive VGG-based network.
Revised: We used both a relatively small hand-crafted CNN, and a large public-domain completive VGG-based network.

Original: We observed that most of the beneﬁcial effect of CL was achieved at the beginning of the learning and that the beneﬁts were more signiﬁcant when using harder datasets.
Revised: We observed that most of the beneﬁcial effect of CL was achieved at the beginning of the learning and that the beneﬁts were more signiﬁcant when using harder datasets.

Revised: During our experiments, we saw that the quality of the teacher network also impacts curriculum learning by transfer.

Revised: In order for the teacher network to differentiate between easier and harder examples, it should have reasonable generalization accuracy.

Revised: A teacher with low performance will classify all points as hard, while a ”too good” teacher will classify all points as easy, results in a less efﬁcient curriculum.

Original: Based on these observations, we investigated two alternative pacing functions that achieved CL with less overhead as compared to training without a curriculum.
Revised: Based on these observations, we investigated two alternative pacing functions that achieved CL with less overhead as compared to training without a curriculum.

Original: In addition to the transfer scoring function , we introduced the self-taught scoring function .
Revised: In addition to the transfer scoring function , we introduced the self-taught scoring function .

Original: This function does not rely on transfer from a large network, and can therefore, presumably, better scale up to larger datasets.
Revised: This function does not rely on transfer from a large network, and can therefore, presumably, better scale up to larger datasets.

Original: Self-Taught scoring is closely related to Self Paced Learning, yet it boils down to essentially the opposite scoring heuristics, since the self-taught scoring function relies on the ﬁnal hypothesis of a pre-trained network while SPL relies on the current hypothesis.
Revised: Self-Taught scoring is closely related to Self Paced Learning, yet it boils down to essentially the opposite scoring heuristics, since the self-taught scoring function relies on the ﬁnal hypothesis of a pre-trained network while SPL relies on the current hypothesis.

Original: In agreement with the theory reviewed in the introduction, we showed that the self-paced scoring function impaired the learning, while the self-taught scoring function enhanced it.
Revised: In agreement with the theory reviewed in the introduction, we showed that the self-paced scoring function impaired the learning, while the self-taught scoring function enhanced it.

Original: In other words, when choosing easier points to guide the learning, it is important to measure difﬁculty with respect to the ﬁnal hypothesis, not the current hypothesis.
Revised: In other words, when choosing easier points to guide the learning, it is important to measure difﬁculty with respect to the ﬁnal hypothesis, not the current hypothesis.

Original: A PPENDIX A A DDITIONAL E MPIRICAL R ESULTS
Revised: A PPENDIX A A DDITIONAL E MPIRICAL R ESULTS

Original: CL with other CIFAR-100 super-classes.
Revised: CL with other CIFAR-100 super-classes.

Original: In Section 3 we present results when learning to discriminate the ”small mammals” superclass of CIFAR-100.
Revised: In Section 3 we present results when learning to discriminate the ”small mammals” superclass of CIFAR-100.

Original: Similar results can be obtained for other super-classes of CIFAR-100, including the super-classes of ”people”, ”insects” and ”aquatic mammals”.
Revised: Similar results can be obtained for other super-classes of CIFAR-100, including the super-classes of ”people”, ”insects” and ”aquatic mammals”.

Original: CL trained on these different super-classes shows the same qualitative results.
Revised: CL trained on these different super-classes shows the same qualitative results.

Original: We note once again that CL is more effective in the harder tasks, namely, the super-classes containing classes that are harder to discriminate (as seen by lower vanilla accuracy).
Revised: We note once again that CL is more effective in the harder tasks, namely, the super-classes containing classes that are harder to discriminate (as seen by lower vanilla accuracy).

Original: As an example, Fig. 8 shows results using the ”aquatic mammals” dataset which greatly resembles the results we’ve seen when discriminating the ”small mammals” dataset (cf. Fig.7)
Revised: As an example, Fig. 9 shows results using the ”aquatic mammals” dataset, which greatly resembles the results we’ve seen when discriminating the ”small mammals” dataset (cf. Fig.8).

Original: Transfer based scoring function.
Revised: Transfer based scoring function.

Original: In the experiments described in Section 3, when using the transfer scoring function deﬁned in Section 2.3, we use the pre-trained Inception network available from https://github.com/Hvass-Labs/TensorFlow-Tutorials.
Revised: In the experiments described in Section 3, when using the transfer scoring function deﬁned in Section 2.3, we use the pre-trained Inception network available from https://github.com/Hvass-Labs/TensorFlow-Tutorials.

Original: We ﬁrst normalized each training image to the range [ − 1 , 1] , resized it, and ran it through the Inception network.
Revised: We ﬁrst normalized each training image to the range [ − 1 , 1] , resized it, and ran it through the Inception network.

Original: We then used the penultimate layer’s activations as features for each training image, resulting in 2048 features per image.
Revised: We then used the penultimate layer’s activations as features for each training image, resulting in 2048 features per image.

Original: Using these features, we trained a Radial Basis Kernel (RBF)
Revised: Using these features, we trained a Radial Basis Kernel (RBF)

Original: SVM
Revised: SVM

Original: (Scholkopf et al., 1997)
Revised: (Scholkopf et al., 1997)

Original: and used its conﬁdence score to determine the difﬁculty of each image.
Revised: and used its conﬁdence score to determine the difﬁculty of each image.

Original: The conﬁdence score of the
Revised: The conﬁdence score of the

Original: SVM was provided by sklearn.svm.libsvm.predict proba from Python’s Sklearn library and is based on cross-validation.
Revised: SVM was provided by sklearn.svm.libsvm.predict proba from Python’s Sklearn library and is based on cross-validation.

Original: Choosing Inception as the teacher and RBF SVM as the classiﬁer was a reasonable arbitrary choice – the same qualitative results are obtained when using other large networks trained on ImageNet as teachers, and other classiﬁers to establish a conﬁdence score.
Revised: Choosing Inception as the teacher and RBF SVM as the classiﬁer was a reasonable arbitrary choice – the same qualitative results are obtained when using other large networks trained on ImageNet as teachers, and other classiﬁers to establish a conﬁdence score.

Original: Speciﬁcally, we repeated the experiments with a transfer scoring function based on the pre-trained VGG-16 and Resnet networks, which are also trained on Imagenet.
Revised: Speciﬁcally, we repeated the experiments with a transfer scoring function based on the pre-trained VGG-16 and Resnet networks, which are also trained on Imagenet.

Original: The curriculum method using the transfer scoring function and ﬁxed exponential pacing function are shown in Fig. 9a, demonstrating the same qualitative results.
Revised: The curriculum method using the transfer scoring function and ﬁxed exponential pacing function are shown in Fig. 10a, demonstrating the same qualitative results.

Original: Similarly, we used a linear SVM instead of the RBF kernel SVM with similar results, as shown in Fig. 9b.
Revised: Similarly, we used a linear SVM instead of the RBF kernel SVM with similar results, as shown in Fig. 10b.

Original: We note that the STE error bars are relatively large for the control conditions described above because we only repeated these conditions 5 times each, instead of 50 in the main experiments.
Revised: We note that the STE error bars are relatively large for the control conditions described above because we only repeated these conditions 5 times each, instead of 50 in the main experiments.

Original: [Figure]
Revised: [Figure]

Original: B E XTENDED D ISCUSSION
Revised: B E XTENDED D ISCUSSION

Original: Self-taught bootstrapping In principle, the self-taught scoring function can be used repeatedly to boost the performance of the network indeﬁnitely: after training the network using a curriculum, we can use its conﬁdence score to deﬁne a new scoring function and retrain the network from scratch.
Revised: Self-taught bootstrapping In principle, the self-taught scoring function can be used repeatedly to boost the performance of the network indeﬁnitely: after training the network using a curriculum, we can use its conﬁdence score to deﬁne a new scoring function and retrain the network from scratch.

Original: However, scoring functions created by repeating this procedure tend to accumulate errors: once an example is misclassiﬁed as being easy, this example will be shown more often in subsequent iterations, making it more likely to be considered easy.
Revised: However, scoring functions created by repeating this procedure tend to accumulate errors: once an example is misclassiﬁed as being easy, this example will be shown more often in subsequent iterations, making it more likely to be considered easy.

Original: In practice, we did not observe any beneﬁt to repeated bootstrapping, and even observed the impairment after a large number of repetitions.
Revised: In practice, we did not observe any beneﬁt to repeated bootstrapping, and even observed the impairment after a large number of repetitions.

Original: F AIR COMPARISON IN PARAMETER TUNING
Revised: F AIR COMPARISON IN PARAMETER TUNING

Original: When using the moderate size hand-crafted network ( cases 1, 2 and 3 ), learning rate tuning is done for the vanilla case as well.
Revised: When using the moderate size hand-crafted network ( cases 1, 2 and 3 ), learning rate tuning is done for the vanilla case as well.

Original: In these cases, for the curriculum , anti-curriculum and random test conditions, we perform a coarse grid search for the pacing hyper-parameters as well as the learning rate hyper-parameters, with an identical range of values for all conditions.
Revised: In these cases, for the curriculum , anti-curriculum and random test conditions, we perform a coarse grid search for the pacing hyper-parameters as well as the learning rate hyper-parameters, with an identical range of values for all conditions.

Original: For the vanilla condition, there are no pacing hyper-parameters.
Revised: For the vanilla condition, there are no pacing hyper-parameters.

Original: Therefore, we expand and reﬁne the range of learning rate hyper-parameters in the grid search, such that the total number of parameter combinations for each condition is approximately the same.
Revised: Therefore, we expand and reﬁne the range of learning rate hyper-parameters in the grid search, such that the total number of parameter combinations for each condition is approximately the same.

Original: When using a public domain competitive network ( case 4 ), the published learning rate scheduling is used.
Revised: When using a public domain competitive network ( case 4 ), the published learning rate scheduling is used.

Original: Therefore we employ the varied exponential pacing function without additional learning rate tuning and perform a coarse grid search on the pacing hyper-parameters.
Revised: Therefore we employ the varied exponential pacing function without additional learning rate tuning and perform a coarse grid search on the pacing hyper-parameters.

Original: To ensure a fair comparison, we repeat the experiment with the vanilla condition the same number of times as in the total number of experiments done during grid search, choosing the best results.
Revised: To ensure a fair comparison, we repeat the experiment with the vanilla condition the same number of times as in the total number of experiments done during grid search, choosing the best results.

Original: The exact range of values that are used for each parameter is given below in Appendix C. All prototypical results were conﬁrmed with cross-validation, showing similar qualitative behavior as when using the coarse grid search.
Revised: The exact range of values that are used for each parameter is given below in Appendix C. All prototypical results were conﬁrmed with cross-validation, showing similar qualitative behavior as when using the coarse grid search.

Original: L EARNING R ATE T UNING
Revised: L EARNING R ATE T UNING

Original: To control for the possibility that the results we report are an artifact of the way the learning rate is being scheduled, which is indeed the method in common use, we test other learning rate scheduling methods, and speciﬁcally the method proposed by Smith (2017) which dynamically changes the learning rate, increasing and decreasing it periodically in a cyclic manner.
Revised: To control for the possibility that the results we report are an artifact of the way the learning rate is being scheduled, which is indeed the method in common use, we test other learning rate scheduling methods, and speciﬁcally the method proposed by Smith (2017) which dynamically changes the learning rate, increasing and decreasing it periodically in a cyclic manner.

Original: We have implemented and tested this method using cases 2 and 3 .
Revised: We have implemented and tested this method using cases 2 and 3 .

Original: The ﬁnal results of both the vanilla and curriculum conditions have improved, suggesting that this method is superior to the na ¨ ıve exponential decrease with grid search.
Revised: The ﬁnal results of both the vanilla and curriculum conditions have improved, suggesting that this method is superior to the na¨ıve exponential decrease with grid search.

Original: Still, the main qualitative advantage of the CL algorithm holds now as well - CL improves the training accuracy during all stages of learning.
Revised: Still, the main qualitative advantage of the CL algorithm holds now as well - CL improves the training accuracy during all stages of learning.

Original: As before, the improvement is more signiﬁcant when the training dataset is harder.
Revised: As before, the improvement is more signiﬁcant when the training dataset is harder.

Original: Results for case 3 (CIFAR-100) are shown in Fig.
Revised: Results for case 3 (CIFAR-100) are shown in Fig.

Original: C M ETHODOLOGY , ADDITIONAL DETAILS
Revised: C M ETHODOLOGY , ADDITIONAL DETAILS

Original: Exponential Pacing Throughout this work, we use pacing functions that increase the data size each step exponentially.
Revised: Exponential Pacing Throughout this work, we use pacing functions that increase the data size each step exponentially.

Original: This is done in line with the customary change of learning rate in an exponential manner.
Revised: This is done in line with the customary change of learning rate in an exponential manner.

Original: Architecture Details
Revised: Architecture Details

Original: The moderate-size neural network we used for cases 1,2,3 , is a convolutional neural network, containing 8 convolutional layers with 32, 32, 64, 64, 128, 128, 256, 256 ﬁlters respectively.
Revised: The moderate-size neural network we used for cases 1,2,3 , is a convolutional neural network, containing 8 convolutional layers with 32, 32, 64, 64, 128, 128, 256, 256 ﬁlters respectively.

Original: The ﬁrst 6 layers have ﬁlters of size 3 × 3 , and the last 2 layers have ﬁlters of size 2 × 2 .
Revised: The ﬁrst 6 layers have ﬁlters of size 3 × 3 , and the last 2 layers have ﬁlters of size 2 × 2 .

Original: Every second layer there is a 2 × 2 max-pooling layer and a 0 . 25 dropout layer.
Revised: Every second layer there is a 2 × 2 max-pooling layer and a 0 . 25 dropout layer.

Original: After the convolutional layers, the units are ﬂattened, and there is a fully-connected layer with 512 units followed by 0 .
Revised: After the convolutional layers, the units are ﬂattened, and there is a fully-connected layer with 512 units followed by 0 .

Original: 5 dropout layer.
Revised: 5 dropout layer.

Original: Batch size was 100.
Revised: Batch size was 100.

Original: The output layer is a fully connected layer with 5 output units, followed by a softmax layer.
Revised: The output layer is a fully connected layer with 5 output units, followed by a softmax layer.

Original: We trained the network using the SGD optimizer, with cross-entropy loss.
Revised: We trained the network using the SGD optimizer, with cross-entropy loss.

Original: All the code will be published upon acceptance.
Revised: All the code will be published upon acceptance.

Original: Grid-search hyper parameters When using grid search, identical ranges of values are used for the curriculum, anti-curriculum and random test conditions.
Revised: Grid-search hyper parameters When using grid search, identical ranges of values are used for the curriculum, anti-curriculum and random test conditions.

Original: Since vanilla contains fewer parameters to tune – as it has no pacing parameters – we used a ﬁner and broader search range.
Revised: Since vanilla contains fewer parameters to tune – as it has no pacing parameters – we used a ﬁner and broader search range.

Original: The range of parameters was similar between different scoring functions and pacing functions and was determined by the architecture and dataset.
Revised: The range of parameters was similar between different scoring functions and pacing functions and was determined by the architecture and dataset.

Original: The range of parameters for case 1 : (i) initial learning rate: 0 .
Revised: The range of parameters for case 1 : (i) initial learning rate: 0 .

Original: 1 ∼ 0 .
Revised: 1 ∼ 0 .

Original: 01 ; (ii) learning rate exponential decrease 2 ∼ 1 . 1 ; (iii) learning rate step size 200 ∼ 800 ; (iv) step size 20 ∼ 400 , for both varied and ﬁxed; (v) increase 1 . 1 ∼ 3 ; (vi) starting percent 4% ∼ 15% (note that 4% is in the size of a single mini-batch).
Revised: 01 ; (ii) learning rate exponential decrease 2 ∼ 1 . 1 ; (iii) learning rate step size 200 ∼ 800 ; (iv) step size 20 ∼ 400 , for both varied and ﬁxed; (v) increase 1 . 1 ∼ 3 ; (vi) starting percent 4% ∼ 15% (note that 4% is in the size of a single mini-batch).

Original: For cases 2, 3 the ranges is wider since the dataset is larger: (i) initial learning rate: 0 .
Revised: For cases 2, 3 the ranges is wider since the dataset is larger: (i) initial learning rate: 0 .

Original: 2 ∼ 0 .
Revised: 2 ∼ 0 .

Revised: 05 ; (ii) learning rate exponential decrease 2 ∼ 1 . 1 ; (iii) learning rate step size 200 ∼ 800 ; (iv) step size 100 ∼ 2000 , for both varied and ﬁxed; (v) increase 1 . 1 ∼ 3 ; (vi) starting percent 0 .

Revised: 4% ∼ 15% .

Original: 05 ; (ii) learning rate exponential decrease 2 ∼  1 . 1  ; (iii) learning rate step size
Revised: For cases 4 , the learning rate parameters are left as publicly determined, while the initial learning rate has been decreased by 10% from 0 . 1 to 0 . The pacing parameter ranges are: (i) step size 50 ∼ 2500 , for both varied and ﬁxed; (ii) increase 1 . 1 ∼ 2 ; (iii) starting percent 2% ∼ 20% .

Revised: ImageNet Dataset Details In case 5 , we used a subset of the ImageNet dataset ILSVRC 2012.

Revised: We used 7 classes of cats, which obtained by picking all the hyponyms of the cat synset that appeared in the dataset.

Revised: The 7 cat classes were: ’Egyptian cat’, ’Persian cat’, ’cougar, puma, catamount, mountain lion, painter, panther, Felis concolor’, ’tiger cat’, ’Siamese cat, Siamese’, ’tabby, tabby cat’, ’lynx, catamount’.

Revised: All images were resized to size 56 × 56 for faster performance.

Revised: All classes contained 1300 train images and 50 test images.