Original: An Analytical Theory of Curriculum Learning in Teacher-Student Networks
Revised: An Analytical Theory of Curriculum Learning in Teacher-Student Networks

Original: Anonymous Author(s)

Original: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: In animals and humans, curriculum learning—presenting data in a curated order—is critical to rapid learning and effective pedagogy.
Revised: In animals and humans, curriculum learning—presenting data in a curated order— is critical to rapid learning and effective pedagogy.

Original: A long history of experimentshas demonstrated the impact of curricula in a variety of animals but, despiteits ubiquitous presence, a theoretical understanding of the phenomenon is stilllacking.
Revised: A long history of experiments has demonstrated the impact of curricula in a variety of animals but, despite its ubiquitous presence, a theoretical understanding of the phenomenon is still lacking.

Original: Surprisingly, in contrast to animal learning, curricula strategies are notwidely used in machine learning and recent simulation studies reach the conclusionthat curricula are moderately effective or even ineffective in most cases.
Revised: Surprisingly, in contrast to animal learning, curricula strategies are not widely used in machine learning and recent simulation studies reach the conclusion that curricula are moderately effective or even ineffective in most cases.

Original: Thisstark difference in the importance of curriculum raises a fundamental theoreticalquestion: when and why does curriculum learning help?
Revised: This stark difference in the importance of curriculum raises a fundamental theoretical question: when and why does curriculum learning help?

Original: In this work, we analyse aprototypical neural network model of curriculum learning in the high-dimensional limit, employing statistical physics methods.
Revised: In this work, we analyse a prototypical neural network model of curriculum learning in the high-dimensional limit, employing statistical physics methods.

Original: We study a task in which a sparseset of informative features are embedded amidst a large set of noisy features.
Revised: We study a task in which a sparse set of informative features are embedded amidst a large set of noisy features.

Original: We analytically derive average learning trajectories for simple neural networkson this task, which establish a clear speed beneﬁt for curriculum learning in the online setting.
Revised: We analytically derive average learning trajectories for simple neural networks on this task, which establish a clear speed beneﬁt for curriculum learning in the online setting.

Original: However, when training experiences can be stored and replayed(for instance, during sleep), the advantage of curriculum in standard neural networksdisappears, in line with observations from the deep learning literature.
Revised: However, when training experiences can be stored and replayed the advantage of curriculum in standard neural networks disappears, in line with observations from the deep learning literature.

Original: Inspiredby synaptic consolidation techniques developed to combat catastrophic forgetting,we propose curriculum-aware algorithms that consolidate synapses at curriculumchange points and investigate whether this can boost the beneﬁts of curricula.
Revised: Inspired by synaptic consolidation techniques developed to combat catastrophic forgetting, we propose curriculum-aware algorithms that consolidate synapses at curriculum change points and investigate whether this can boost the beneﬁts of curricula.

Original: We derive generalisation performance as a function of consolidation strength(implemented as an L 2 regularisation/elastic coupling connecting learning phases),and show that curriculum-aware algorithms can yield a large improvement intest performance.
Revised: We derive generalisation performance as a function of consolidation strength (implemented as an L 2 regularisation/elastic coupling connecting learning phases), and show that curriculum-aware algorithms can yield a large improvement in test performance.

Original: Our reduced analytical descriptions help reconcile apparentlyconﬂicting empirical results, trace regimes where curriculum learning yields thelargest gains, and provide experimentally-accessible predictions for the impact oftask parameters on curriculum beneﬁts.
Revised: Our reduced analytical descriptions help reconcile apparently conﬂicting empirical results, trace regimes where curriculum learning yields the largest gains, and provide experimentally-accessible predictions for the impact of task parameters on curriculum beneﬁts.

Original: More broadly, our results suggest thatfully exploiting a curriculum may require explicit consolidation at curriculumboundaries.
Revised: More broadly, our results suggest that fully exploiting a curriculum may require explicit adjustments in the loss.

Original:  Introduction
Revised:  Introduction

Original: Presenting learning materials in a meaningful order according to a curriculum greatly helps learningin animals and humans [1, 2, 3, 4], and is considered an essential aspect of good pedagogy [5].
Revised: Presenting learning materials in a meaningful order according to a curriculum greatly helps learning in animals and humans [1, 2, 3, 4] and is considered an essential aspect of good pedagogy [5].

Original: For example, humans have been shown to learn visual discriminations faster when presented withexamples that exaggerate the relevant difference between classes, a phenomenon known as “fading”[6, 7, 8].
Revised: For example, humans have been shown to learn visual discriminations faster when presented with examples that exaggerate the relevant difference between classes, a phenomenon known as “fading” [6, 7, 8].

Original: Beyond humans, curricula in the form of “shaping” or “staircase” procedures are a nearuniversal feature of task designs in animal studies, without which training often fails entirely.
Revised: Beyond humans, curricula in the form of “shaping” or “staircase” procedures are a nearuniversal feature of task designs in animal studies, without which training often fails entirely.

Original: For instance, the International Brain Laboratory task, a standardised perceptual decision-making trainingparadigm in mice, involves six stages of increasing difﬁculty before reaching ﬁnal performance [9].
Revised: For instance, the International Brain Laboratory task, a standardised perceptual decision-making training paradigm in mice, involves six stages of increasing difﬁculty before reaching ﬁnal performance [9].

Original: Building from this intuition, a seminal series of papers proposed a similar curriculum learningapproach for machine learning (ML) [10, 11, 12].
Revised: Building from this intuition, a seminal series of papers proposed a similar curriculum learning approach for machine learning (ML) [10, 11, 12].

Original: In striking contrast to the clear beneﬁts ofcurriculum in biological systems, however, curriculum learning has generally yielded equivocalbeneﬁts in artiﬁcial systems.
Revised: In striking contrast to the clear beneﬁts of curriculum in biological systems, however, curriculum learning has generally yielded equivocal beneﬁts in artiﬁcial systems.

Original: Experiments in a variety of domains [13, 14] have found usually modestspeed and generalisation improvements from curricula.
Revised: Experiments in a variety of domains [13, 14] have found usually modest speed and generalisation improvements from curricula.

Original: Recent extensive empirical analyses havefound minimal beneﬁts on standard datasets [15].
Revised: Recent extensive empirical analyses have found minimal beneﬁts on standard datasets [15].

Original: Indeed, a common intuition in deep learningpractice holds that training distributions should ideally be as close as possible to testing distributions,a notion which runs counter to curriculum.
Revised: Indeed, a common intuition in deep learning practice holds that training distributions should ideally be as close as possible to testing distributions, a notion which runs counter to curriculum.

Original: Perhaps the only areas where curricula are actively usedare in large language models [16] and certain reinforcement learning settings [17].
Revised: Perhaps the only areas where curricula are actively used are in large language models [16] and certain reinforcement learning settings [17].

Original: This gap between the effect of curriculum in biological and artiﬁcial learning systems poses a puzzlefor theory.
Revised: This gap between the effect of curriculum in biological and artiﬁcial learning systems poses a puzzle for theory.

Original: When and why is curriculum learning useful?
Revised: When and why is curriculum learning useful?

Original: What properties of a task determine theextent of possible beneﬁts?
Revised: What properties of a task determine the extent of possible beneﬁts?

Original: What ordering of learning material is most beneﬁcial?
Revised: What ordering of learning material is most beneﬁcial?

Original: And can newlearning algorithms better exploit curricula?
Revised: And can new learning algorithms better exploit curricula?

Original: Compared to the empirical investigations of curriculumlearning, theoretical results on curriculum learning remain sparse.
Revised: Compared to the empirical investigations of curriculum learning, theoretical results on curriculum learning remain sparse.

Original: Most notably, [18, 19] show thatcurriculum can lead to faster learning in a simple setting, but the effects of curriculum on asymptoticgeneralisation and the dependence on task structure remain unclear.
Revised: Most notably, [18, 19] show that curriculum can lead to faster learning in a simple setting, but the effects of curriculum on asymptotic generalisation and the dependence on task structure remain unclear.

Original: A hint that indeed curriculumlearning might lead to statistically different minima comes from a connection between constraintsatisfaction problems and physics results on ﬂow networks [20], but to our knowledge no direct resulthas been reported in the modern theoretical ML literature.
Revised: A hint that indeed curriculum learning might lead to statistically different minima comes from a connection between constraintsatisfaction problems and physics results on ﬂow networks [20], but to our knowledge no direct result has been reported in the modern theoretical ML literature.

Original: In this work we study the impact of curriculum using the analytically tractable teacher-studentframework and the tools of statistical physics [21, 22, 23, 24].
Revised: In this work we study the impact of curriculum using the analytically tractable teacher-student framework and the tools of statistical physics [21, 22, 23, 24].

Original: High-dimensional teacher-studentmodels are a popular approach for systematically studying learning behaviour in neural networks[25, 26, 22], and have recently been leveraged to analyse a variety of phenomena [27, 28, 29, 30, 31,32].
Revised: High-dimensional teacher-student models are a popular approach for systematically studying learning behaviour in neural networks [25, 26, 22], and have recently been leveraged to analyse a variety of phenomena [27, 28, 29, 30, 31, 32].

Original: Using a simple model to build structured data [12], we examine the impact of ordering examplesby increasing difﬁculty (curriculum), decreasing difﬁculty (anti-curriculum), or standard shufﬂedtraining.
Revised: Using a simple model to build structured data [12], we examine the impact of ordering examples by increasing difﬁculty (curriculum), decreasing difﬁculty (anti-curriculum), or standard shufﬂed training.

Original: We derive exact expressions for the online learning dynamics and the performance of batchlearning.
Revised: We derive exact expressions for the online learning dynamics and the performance of batch learning.

Original: However, in the latter, curriculum confers no beneﬁt under standard training.
Revised: However, in the latter, curriculum confers no beneﬁt under standard training in our model setting.

Original: Motivatedby theories of synaptic consolidation and elastic weight consolidation [33, 34], we introduce elasticpenalties (Gaussian priors) that regularise training toward solutions obtained in prior curriculumphases.
Revised: Motivated by theories of synaptic consolidation and elastic weight consolidation [33, 34], we introduce elastic penalties (Gaussian priors) that regularise training toward solutions obtained in prior curriculum phases, instantiating a long-term memory effect.

Original: With these priors, curriculum yields beneﬁts both in the online 3 and in the batch 4 settings.
Revised: With these priors, curriculum yields beneﬁts both in the online 3 and in the batch 4 settings.

Revised: Further related work.

Revised: The ﬁrst empirical investigation of curriculum learning appeared in[35], consisting in a visual discrimination task for dogs under curriculum and no-curriculum paradigms.

Revised: Later behavioural studies proved curricula to be beneﬁcial independent of the animal (dogs, mice, rats, pigeons, humans) and the data modality (visual, auditory, or tactile stimuli) [36, 1, 2, 37, 38, 6].

Revised: However, these experimental observations were not observed in standard artiﬁcial neural networks (ANNs).

Original: Model deﬁnition and overview of approach
Revised: Several ideas in the connectionist community were proposed in order to show curriculum effects in the learning dynamics of ANNs [39, 40, 10, 11]. While these studies were able to match previous experimental data, they also required substantial changes in the architecture of the ANN and/or in the learning rule. Except for very few instances [16, 17], standard ML practice tends to avoid taking curricula into account. An obvious obstacle is the fact that most datasets do not provide meta-data about sample difﬁculties.

Revised: An interesting line of research pointed out the possible relevance of implicit curricula, based on the observation that neural networks tend to consistently learn the samples in a certain order [41].

Revised: Thus, a possible way of addressing the lack of difﬁculty labels would be to use the natural learning order as indicative of the various difﬁculties of the training samples.

Revised: However, a recent work [15], which compared several heuristics for curriculum learning —including implicit curricula— in a variety of settings, showed limited beneﬁts with this strategy.

Revised: The picture that emerges from the literature seems contradictory: on the one hand, curricula appear fundamental to biological learning; on the other hand, curricula appear largely irrelevant in many machine learning settings.

Revised: The core motivation behind our work is to reconcile these views and contribute to a theoretical understanding of curriculum learning.

Revised: [Figure]

Revised: [Figure]

Revised: R e l e v a n t

Revised: Irr e l e v a n t

Original: [Figure]
Revised: [Figure]

Original: Parameters: α 1 = 1 , α 2 = 1 , ∆ 1 = 0 , ∆ 2 = 1 , γ = 10 − 5 , η = 3 .
Revised: Since the teacher network is sparse, its only on a subset relevant We consider curricula which order examples by difﬁculty, here taken to be the variance in the irrelevant feature dimensions to increasing  Model deﬁnition and overview of approach

Original: In the following, we revisit a prototypical model of curriculum learning from [12] that ﬁnds correspondence to the fading literature [6] as highlighted in Sec. 4.
Revised: In the following, we revisit a prototypical model of curriculum learning from [12] that ﬁnds correspondence to the fading literature [6] as highlighted in Sec. 5.

Original: Our setting is summarised in Fig.
Revised: Our setting is summarised in Fig.

Original: The model entails a simple teacher-student setup, where teacher and student are each shallow 1-layerneural networks of size N (also known as perceptrons).
Revised: The model entails a simple teacher-student setup, where teacher and student are each shallow 1-layer neural networks of size N (also known as perceptrons).

Original: The learning task for the student is a binaryclassiﬁcation problem, with dataset D
Revised: The learning task for the student is a binary classiﬁcation problem, with dataset D

Original: = { ( y µ ,xxx µ ) } Mµ =1 , where the ground-truth labels are producedby the teacher network y µ = sign WWW T · xxx µ .
Revised: = { ( y µ ,xxx µ ) } Mµ =1 , where the ground-truth labels are produced by the teacher network y µ = sign WWW T · xxx µ .

Revised: The student learns via empirical risk minimisation of an

Revised: L 2 regularised convex loss.

Original: A key feature of this model is that the teacher networkis sparse, with only a fraction ρ < 1 of ∼ N (0 , 1) non-zero components.
Revised: A key feature of this model is that the teacher network is sparse, with only a fraction ρ < 1 of ∼ N (0 , 1) non-zero components.

Original: Therefore, in order toachieve a good test accuracy, the student has to guess which components should be set to zero andalign the relevant weights in the correct direction.
Revised: Therefore, in order to achieve a good test accuracy, the student has to guess which components should be set to zero and align the relevant weights in the correct direction.

Original: A large range of 0 < ρ < 1 could give rise to thephenomenology we seek to analyse.
Revised: A large range of 0 < ρ < 1 could give rise to the phenomenology we seek to analyse.

Original: In the remainder of the paper we will focus on the case ρ = 0 .
Revised: In the remainder of the paper we will focus on the case ρ = 0 .

Original: We model the variable degree of difﬁculty in the samples by decomposing each input vector asxxx µ = [ xxx µr ,xxx µi ] ∈ R N , where xxx µr ∈ R ρN denotes the relevant components of the input, and xxx µi ∈
Revised: We model the variable degree of difﬁculty in the samples by decomposing each input vector as xxx µ = [ xxx µr ,xxx µ i ] ∈ R N , where xxx µr ∈ R ρN denotes the relevant components of the input, and xxx µ i ∈

Original: R (1 − ρ ) N the irrelevant ones.
Revised: R (1 − ρ ) N the irrelevant ones.

Original: Note that, crucially, the sparse teacher network is completely blindto the irrelevant part of the input: y µ = sign (cid:80) ρNj =1 W T,j x µr,j .
Revised: Note that, crucially, the sparse teacher network is completely blind to the irrelevant part of the input: y µ = sign (cid:80) ρNj =1 W T,j x µr,j .

Original: While x µr,j i.i.d.
Revised: While x µr,j i.i.d.

Original: N (0 , 1) , ∀ µ , 1 weconsider the variance for the irrelevant components to be sample-dependent x µi,j ∼ N (0 , ∆ µ ) .
Revised: N (0 , 1) , ∀ µ , 1 we consider the variance for the irrelevant components to be sample-dependent x µi,j ∼ N (0 , ∆ µ ) .

Original: Notethat a smaller variance in the irrelevant part induces a higher SNR in the student learning problem.
Revised: A smaller variance in the irrelevant part induces a higher SNR in the student learning problem.

Original: The dataset is partitioned according to difﬁculty levels given by the variances of the irrelevant inputs.
Revised: The dataset is partitioned according to difﬁculty levels given by the variances of the irrelevant inputs.

Original: For simplicity we consider only two partitions in most of our analysis, but generalisations to moredifﬁculty levels follow straightforwardly.
Revised: For simplicity we consider only two partitions in most of our analysis, but generalisations to multiple difﬁculty levels follow straightforwardly.

Original: We  have a dataset with M =
Revised: We thus have a dataset with M =

Original: ( α 1 + α 2 ) N total samples,in which the irrelevant inputs of the ﬁrst α 1 N samples  have variance ∆ 1 , and the remaining α 2 Nsamples have variance  ∆ 2 > ∆ 1 .
Revised: ( α 1 + α 2 ) N = αN samples in total. In the ﬁrst α 1 N samples the irrelevant inputs have variance ∆ 1 , while for the remaining α 2 N samples the variance is ∆ 2 > ∆ 1 .

Original: In the curriculum learning condition we present the easy examplesﬁrst, while in the anti-curriculum condition we present the hard examples ﬁrst.
Revised: In the curriculum learning condition we present the easy examples ﬁrst, while in the anti-curriculum condition we present the hard examples ﬁrst.

Original: Standard learningpresents examples shufﬂed in random order.
Revised: Standard learning presents examples shufﬂed in random order.

Original:  Online dynamical solution in the large input limit
Revised:  Online dynamical solution in the large input limit

Original: We start by focusing on the same online learning setting explored in [12].
Revised: We start by focusing on the same online learning setting explored in [12].

Original: We consider a 1-layerstudent network with sigmoidal activation function, σ ( · ) = erf ( · / √ 2) , that learns to minimise a meansquare error loss with L 2 regularisation of intensity γ , using gradient descent.
Revised: We consider a 1-layer student network with sigmoidal activation function, σ ( · ) = erf ( · / √ 2) , that learns to minimise a mean square error loss with L 2 regularisation of intensity γ , using gradient descent.

Original: This yields the updates
Revised: This yields the updates

Original: [Equation]
Revised: [Equation]

Original: The dynamics of the model can be analysed in the high-dimensional limit N → ∞ .
Revised: The dynamics of the model can be analysed in the high-dimensional limit N, M → ∞ with α =

Original: Generalisingthe results of [26, 35] on the online stochastic gradient descent dynamics in single-layer regressionproblems, we obtain a precise description of the performance at all times, as a function of severalorder parameters: the squared norm of the relevant and irrelevant part of the student weights 
Revised: M/N = O (1) . Generalising the results of [26, 42] on the online stochastic gradient descent dynamics in single-layer regression problems, we obtain a precise description of the performance at all times, as a function of several order parameters: the squared norm of the relevant and irrelevant part of the student weights Q r = 1 N WWW r · WWW r and Q i = 1 N WWW

Original: Q r = 1 N WWW r · WWW r and Q i = 1 N WWW i · WWW i , respectively; the overlap of the relevant weights of thestudent and teacher R =
Revised: i · WWW i , respectively; the overlap of the relevant weights of the student and teacher R =

Original: 1 N WWW r · WWW T ; and the squared norm of the teacher vector T = 1 N WWW T · WWW T .
Revised: 1 N WWW r · WWW T ; and the squared norm of the teacher vector T = 1 N WWW T · WWW T .

Original: In particular, given Q r , Q i , R and T , the test loss (i.e. average loss on a new example) on a datasetwith variance ∆ in the irrelevant inputs is given by
Revised: In particular, given Q r , Q i , R and T , the test loss (i.e. average loss on a new example) on a dataset with variance ∆ in the irrelevant inputs is given by

Original: [Equation]
Revised: [Equation]

Original: the accuracy by
Revised: the accuracy by

Original: [Equation]
Revised: [Equation]

Original: If the dataset contains a random mixture of different difﬁculty levels ∆ 1 , ∆ 2 , . .
Revised: If the dataset contains a random mixture of different difﬁculty levels ∆ 1 , ∆ 2 , . .

Original: . , the loss and accuracy can be obtained by taking a weighted average over the partitions.
Revised: . , the loss and accuracy can be obtained by taking a weighted average over the partitions.

Original: To understand how test performance changes through learning, we study the evolution of the order parameters.
Revised: To understand how test performance changes through learning, we study the evolution of the order parameters.

Original: Combining their deﬁnition with the deﬁnition of the dynamics (1) and the fact that the

Original: 2.0 1 +0.

Original: (a) Curriculum learning.

Original: 2.0 1 +0.

Original: (b) Anti-curriculum learning. 2.0 1 +0. [Figure] random variables concentrate in the high-dimension as N → ∞ , we obtain an analytic form for theupdates: Q r ← f Q r (cid:0) Q r , Q i , R, T (cid:1) , Q i ← f Q i (cid:0) Q r , Q i , R, T (cid:1) , R ← f R (cid:0) Q r , Q i , R, T (cid:1) ; where f Q r ,f Q i and f R are long but explicit expressions that are reported in the supplementary material (SM).
Revised: Combining their deﬁnition with the deﬁnition of the dynamics (1) and the fact that the random variables concentrate in the high-dimension as N → ∞ , we obtain an analytic form for the updates: Q r ← f Q r (cid:0) Q r , Q i , R, T (cid:1) , Q i ← f Q i (cid:0) Q r , Q i , R, T (cid:1) , R ← f R (cid:0) Q r , Q i , R, T (cid:1) ; where f Q r , f Q i and f R are long but explicit expressions that are reported in the supplementary material (SM).

Original: Dynamical advantages of curriculum.
Revised: Dynamical advantages of curriculum.

Original: With these theoretical results in hand, we  now characterisethe performance of curricula in the online setting.
Revised: With these theoretical results in hand, we can now characterise the performance of curricula in the online setting.

Revised: We obtain a description of the learning trajectories for each learning protocol, yielding the evolution of training and test accuracies, and of other observables such as the norm of the student and its overlap with the teacher.

Original: The dynamical equations have two key advantagesrelative to simulating models in this setting.
Revised: Solving the dynamical equations gives two key advantages relative to simulating models in this setting.

Original: First, they are free of ﬁnite size effects and stochasticﬂuctuations.
Revised: First, they are free of ﬁnite size effects and stochastic ﬂuctuations.

Original: And second, their evaluation is very fast, enabling systematic exploration of theparameter space of the problem, along with ﬁne-grained optimisation over hyper-parameters such aslearning rate, weight decay and scaling in the initialisation.
Revised: And second, their evaluation is very fast (up to 6 orders of magnitude in simulation time reduction see SM ?? ) , enabling systematic exploration of the parameter space of the problem, along with ﬁne-grained optimisation over hyper-parameters such as learning rate, weight decay and scaling in the initialisation.

Original: Optimising ﬁnal test accuracy separately for each curriculum strategy, we ﬁnd that curriculum learningis the optimal strategy, followed by baseline (no-curriculum) and lastly anti-curriculum.
Revised: Optimising ﬁnal test accuracy separately for each curriculum strategy, we ﬁnd that curriculum learning is the optimal strategy, followed by baseline (no-curriculum) and lastly anti-curriculum.

Original: In Fig.
Revised: In Fig.

Original: 1cwe show typical learning trajectories for a dataset with equal numbers of easy and hard samples.
Revised: 1c we show typical learning trajectories for a dataset with equal numbers of easy and hard samples.

Original: Theresults of the simulations (solid lines) are well-described by our theoretical equations (dashed lines),and show that the curriculum strategy leads to better performance throughout training.
Revised: The results of the simulations (solid lines) are well-described by our theoretical equations (dashed lines), and show that the curriculum strategy leads to better performance throughout training.

Original: Fig.
Revised: Fig.

Original: 1c showsthe evolution during training of the test accuracy computed on the whole dataset.
Revised: 1c shows the evolution during training of the test accuracy computed on the whole dataset.

Original: Next we systematically trace the effect of curriculum for a range of total dataset sizes ( α 1 + α 2 ) andnumber of easy examples α 1 in the phase diagram in Fig.
Revised: Next, we systematically trace the effect of curriculum for a range of total dataset sizes ( α 1 + α 2 ) and number of easy examples α 1 in the phase diagram in Fig.

Original: This diagram shows on the left (centre)the accuracy on hard instances reached at the end of training by curriculum learning (anti-curriculumlearning respectively) normalised by the accuracy reached by the standard strategy.
Revised: This diagram shows in panels (a) and (b) the accuracies on hard instances reached at the end of training, by curriculum learning and anti-curriculum learning respectively, normalised by the accuracy reached by the standard strategy.

Original: The two heatmapsshow that curriculum learning always outperforms standard learning and that, on the other hand,anti-curriculum learning outperforms standard learning only in part of the diagram.
Revised: The two heatmaps show that curriculum learning always outperforms standard learning and that, on the other hand, anti-curriculum learning outperforms standard learning only in part of the diagram.

Original: Comparingthe two strategies,  Fig.
Revised: Comparing the two strategies, in Fig.

Original: 2 (right), we can observe that there is a region for small α and α 1 whereanti-curriculum learning is the best strategy, while in the majority of the situations curriculum learningis the best strategy.
Revised: 2 (c), we can observe that there is a region for small α and αwhere anti-curriculum learning is the best strategy, while in the majority of the situations curriculum 2.0 1 +0. (a) Curriculum learning. 2.0 1 +0.

Revised: (b) Anti-curriculum learning.

Original:  Interestingly, there is a sizeable region of the diagram in which both curriculumand anti-curriculum help, possibly explaining why both have been recommended in prior work[12, 14, 36, 37, 38].
Revised: 2.0 1 +0. [Figure] learning is best. Interestingly, there is a sizeable region of the diagram in which both curriculum and anti-curriculum help, possibly explaining why both have been recommended in prior work [12, 14, 43, 44, 45].

Revised: A possible intuition behind this counter-intuitive phenomenon highlighted by our analysis is that, in some settings, the large amount of noise contained in the hard data will always be too disruptive for effective learning.

Revised: Thus, leaving the easy (cleaner) data for last could allow the model to better exploit it.

Original: Further, we ﬁnd that our setting, in which a small task-relevant signal is embedded in large taskirrelevant variation, is critical to the beneﬁt of curriculum.
Revised: Further, we ﬁnd that our setting, in which a small task-relevant signal is embedded in large taskirrelevant variation, is critical to the beneﬁt of curriculum.

Original: Fig.
Revised: Fig.

Original: 5 shows performance as a function ofsparsity ρ  .
Revised: 4 shows performance as a function of sparsity ρ , additional details are deferred in the SM ?? .

Original: Non-sparse tasks do not beneﬁt.
Revised: Non-sparse tasks do not beneﬁt.

Original: Hence curriculum aids tasks with many irrelevant factorsof variation.
Revised: Hence curriculum aids tasks with many irrelevant factors of variation.

Original: Interestingly, the literature from human psychology shows precisely this: no curriculumbeneﬁts for low-D tasks or tasks with no variation in irrelevant dimensions [6].
Revised: Interestingly, the literature from human psychology shows precisely this: no curriculum beneﬁts for low-dimensional tasks or tasks with no variation in irrelevant dimensions [6].

Original: Our results also highlight the intricate dependence of curriculum on parameters of the learning setup.
Revised: Our results also highlight the intricate dependence of curriculum on parameters of the learning setup.

Original: If not all parameters are correctly optimised, we can observe more complex scenarios.
Revised: If not all parameters are correctly optimised, we can observe more complex scenarios.

Original: For instance,anti-curriculum learning is always the best strategy starting from a large initialisation, as Fig.
Revised: For instance, the initialisation condition for the norm of the weights of the student plays an important role.

Original: 3 showsfor weights of order 1.
Revised: We explore this dependence by changing the variance of the normal distribution from which the initial weights are sampled from.

Original: In this case, curriculum learning shows an advantage only in the ﬁrst phase
Revised: We observe that anti-curriculum learning becomes the best strategy when the variance is large, as shown in Fig.

Original: 2.0 1 +0. (a) Curriculum learning. 2.0 1 +0.
Revised: 3 for weights of order 1.

Original: [Figure] 1.0    a cc u r a c y % curriculumanti-curriculumno curricula [Figure] when easy examples are shown, which is consistent with the results of [19].
Revised: In this case, curriculum learning shows an advantage only in the ﬁrst phase when easy examples are shown, which is consistent with the results of [19].

Original: However, in the nextphase when hard examples are shown, the curriculum strategy does not extract enough informationand it is outperformed by the other two strategies.
Revised: However, in the next phase when hard examples are shown, the curriculum strategy does not extract enough information and it is outperformed by the other two strategies.

Original: The fact that curriculum or anti-curriculum canlook better depending on other learning parameters like initialisation might help explain the confusionin the literature over the best protocol [12, 14, 36, 37, 38].
Revised: The fact that curriculum or anti-curriculum can look better depending on the parameter setting might help explain the confusion in the literature over the best protocol [12, 14, 43, 44, 45].

Original: At least in this model, better performancefrom anti-curriculum is a signature of a sub-optimal choice of the parameters.
Revised: At least in this model, better performance from anti-curriculum is a signature of a sub-optimal choice of the parameters.

Original: To summarise ourﬁndings in this online learning setting, curriculum mainly offers a dynamical advantage : it speedslearning, with only minimal impact on asymptotic performance.
Revised: To summarise our ﬁndings in this online learning setting, curriculum mainly offers a dynamical advantage : it speeds up learning but has minimal impact on asymptotic performance.

Original:  Batch learning solution
Revised:  Batch learning solution

Original: The previous section discussed the online case where each example is used once and then discarded.
Revised: The previous section discussed the online case where each example is used once and then discarded.

Original: However, in common machine learning practice, neural networks typically revisit each sample repeatedly until convergence.
Revised: However, in common machine learning practice, neural networks typically revisit each sample repeatedly until convergence.

Original: Therefore an important question is: can curricula lead to a generalisationimprovement when trained on the same dataset until convergence?
Revised: Therefore an important question is: can curricula lead to a generalisation improvement when trained on the same dataset until convergence?

Original: We investigate this question by considering a student that learns from slices of a dataset in distinctoptimisation phases, where in each phase the student optimises a L 2 -regularised logistic loss. 
Revised: We investigate this question by considering a student that learns from slices of a dataset in distinct optimisation phases, where in each phase the student optimises a L 2 -regularised logistic loss. Without 2.0 1 +0. (a) Curriculum learning.

Revised: 2.0 1 +0.

Original: Withoutfurther modiﬁcation, curriculum can have no effect in this setting: due to the convex nature of theteacher-student setup [22], the network is bound to converge to a minimum uniquely determinedby the ﬁnal slice of data, with no memory of the progress made at intermediate steps.
Revised: [Figure] 1.0    a cc u r a c y % curriculumanti-curriculumno curricula [Figure] further modiﬁcation, curriculum can have no effect in this setting: due to the convex nature of the teacher-student setup [22], the network is bound to converge to a minimum uniquely determined by the ﬁnal slice of data, with no memory of the progress made at intermediate steps.

Original: This simpleobservation may help explain empirical observations on real data, such as [15], which ﬁnd no beneﬁtof curriculum in standard settings (though we note that curriculum could still inﬂuence non-convexproblems [12]).
Revised: This simple observation may help explain empirical observations on real data, such as [15], which ﬁnd no beneﬁt of curriculum in standard settings. In fact, in principle curriculum could still inﬂuence non-convex problems [12] but empirical results in the ML ﬁeld are not showing clear signals of memory retention.

Revised: A possible explanation of this is that relying on dynamical memory effects requires careful tuning of the learning rate and of the number of training epochs, while typical choices for these hyperparameters could lead to memory loss and performance inconsistencies.

Original: This observation raises the theoretical question of how  curriculum learning couldinduce a non-vanishing effect  in batch learning settings.
Revised: These observations raise the theoretical question of how to better implement curriculum learning to induce a non-vanishing effect also in batch learning settings.

Original: To instantiate a  memory effect in our model, we propose biasing the optimisation landscape via a Gaussian prior, centred around the optimiser of the previous learning phase.
Revised: To instantiate a long-term memory effect in our model, we propose biasing the optimisation landscape via a Gaussian prior, centred around the optimiser of the previous learning phase.

Original: The additional term inthe loss acts as an elastic coupling between the successive phases, and the associated intensity γ 12 isthen an additional hyper-parameter of the model.
Revised: The additional term in the loss acts as an elastic coupling between the successive phases, and the associated intensity γis then an additional hyper-parameter of the model.

Original: This scheme is similar to regularisation methodsproposed against catastrophic interference in continual learning, such as Synaptic Intelligence [39].
Revised: This scheme is similar to regularisation methods proposed against catastrophic interference in continual learning, such as Synaptic Intelligence [46].

Revised: Changing the loss according to the curriculum prescription effectively makes the learning algorithm aware of the different levels of difﬁculty in the dataset.

Original: Tools from statistical physics can be used to analytically compute test performance under this scheme.
Revised: Tools from statistical physics can be used to analytically compute test performance under this scheme.

Original: In order to simplify the presentation, we ﬁrst consider just two learning phases.
Revised: In order to simplify the presentation, we ﬁrst consider just two learning phases.

Original: It is natural to framethis setting as a 2-level problem, involving two systems with independent copies of the networkweights WWW 1 and WWW 2 .
Revised: It is natural to frame this setting as a 2-level problem, involving two systems with independent copies of the network weights WWW 1 and WWW 2 .

Original: In a typical statistical physics approach, we associate a Boltzmann-Gibbsmeasure to the systems, with an energy function determined by the regularised logistic loss L γ .
Revised: In a typical statistical physics approach, we associate a Boltzmann-Gibbs measure to the systems, with an energy function determined by the regularised logistic loss L γ .

Original: Whilethe statistical properties of the ﬁrst system can be determined self-consistently, the added elasticinteraction creates a dependence of the second measure on the conﬁgurations of the ﬁrst system.In 3 log0. a cc u r a c y h a r d s a m p l e
Revised: While the statistical properties of the ﬁrst system can be determined self-consistently, the added elastic interaction creates a dependence of the second measure on the conﬁgurations of the ﬁrst system.

Original: s % curriculum anti-curriculum random splitting no-curriculum (a) Learning with curricula 0.

Original: [Figure]

Original:  mathematical terms, the coupled system is represented by the following partition function:
Revised: In mathematical terms, the coupled system is represented by the following partition function:

Original: [Equation]
Revised: [Equation]

Original: where D 1 , D 2 denote the two dataset slices.
Revised: where D 1 , D 2 denote the two dataset slices.

Original: This object represents the normalisation of the BoltzmannGibbs measure, and allows one to extract relevant information on the asymptotic behaviour of ourmodel.
Revised: This object represents the normalisation of the BoltzmannGibbs measure, and allows one to extract relevant information on the asymptotic behaviour of our model.

Original: The optimisations entailed in each learning phase can be described in the “low noise” limitof β 1 , β 2 → ∞ , where the measures focus on the minimisers of the respective losses.
Revised: The optimisations entailed in each learning phase can be described in the “low noise” limit of β 1 , β 2 → ∞ , where the measures focus on the minimisers of the respective losses.

Original: In order tostudy a self-averaging quantity that does not depend on a speciﬁc realisation of the dataset, we aim tocompute the associated average free-energy: 
Revised: In order to study a self-averaging quantity that does not depend on a speciﬁc realisation of the dataset, we aim to compute the associated average free-energy: [Equation]

Original: [Equation]
Revised: [Equation]

Revised: 3 log0.

Revised: a cc u r a c y h a r d s a m p l e

Revised: s % curriculum anti-curriculum random splitting no-curriculum (a) Learning with curricula 0.

Revised: (b) Curriculum vs no-curriculum

Original: This quantity can be seen as a special case of the so-called Franz-Parisi potential computation [40, 41],and the entailed double average can be evaluated through the replica method.
Revised: This quantity can be seen as a special case of the so-called Franz-Parisi potential computation [47, 48], and the entailed double average can be evaluated through the replica method.

Original: Refer to SM for details.
Revised: Refer to SM for details.

Original: Similar to the online case, in high-dimensions the free-entropy concentrates on a deterministicfunction that depends on several order parameters that capture the geometrical distribution of teacherand student conﬁgurations.
Revised: Similar to the online case, in high-dimensions the free-entropy concentrates on a deterministic function that depends on several order parameters that capture the geometrical distribution of teacher and student conﬁgurations.

Original: In addition to those already introduced in Sec. 3, we also have δQ , whichis linked to the variance of the student norm.
Revised: In addition to those already introduced in Sec. 3, we also have δQ , which is linked to the variance of the student norm.

Original: Moreover, for each order parameter we also need tointroduce a conjugate parameter, denoted in the following with the hat symbol.
Revised: Moreover, for each order parameter we also need to introduce a conjugate parameter, denoted in the following with the hat symbol.

Original: The ﬁnal expressionfor the free-energy reads:
Revised: The ﬁnal expression for the free-energy reads:

Original: [Equation]
Revised: [Equation]

Original: (5) where g S and g E are two scalar functions, often called entropic and energetic channels, that encodethe dependence of the optimisation problem on the Gaussian prior and the logistic loss respectively.
Revised: where g S and g E are two scalar functions, often called entropic and energetic channels, that encode the dependence of the optimisation problem on the Gaussian prior and the logistic loss respectively.

Original: The extremum condition for the free-energy yields a system of ﬁxed-point equations that convergeto an asymptotic prediction for the order parameters, comparable with the results of numericalsimulations on large instances, Fig. 4.
Revised: The extremum condition for the free-energy yields a system of ﬁxed-point equations that converge to an asymptotic prediction for the order parameters, comparable with the results of numerical simulations on large instances, Fig. 4.

Original: At convergence, the order parameters can be inserted again in
Revised: At convergence, the order parameters can be inserted again in

Original: Eq.
Revised: Eq.

Original: 2 to obtain an estimate of the test accuracy.
Revised: 2 to obtain an estimate of the test accuracy.

Original: Note that this formalism is not limited to two phases,but can be extended to the case of a discrete number of sequential stages.
Revised: Note that this formalism is not limited to two phases, but can be extended to the case of a discrete number of sequential stages.

Original: The importance of sparsity.
Revised: The importance of sparsity.

Original: Sparsity is a key ingredient in determining the impact of curriculumstrategies.
Revised: Sparsity is a key ingredient in determining the impact of curriculum strategies.

Original: It naturally introduces a notion of relevant and irrelevant input components, and deﬁnes asecondary learning goal, i.e. identifying what part of the presented data should be disregarded bythe model.
Revised: It naturally introduces a notion of relevant and irrelevant inputs, and deﬁnes a secondary learning goal: identifying what part of the presented data should be disregarded by the model.

Original: Curriculum learning can be extremely helpful in this identiﬁcation process, since theeasy samples are more transparent to this structure.
Revised: Curriculum learning can aid this identiﬁcation process, since the easy samples are more transparent to this structure.

Original: This is also observed in human experiments [6].
Revised: This is also observed in human experiments [6].

Original: However, the relative difﬁculty of the problem of inferring the support of the teacher and the problemof aligning with its non-zero components depends on the degree of sparsity ρ , so the effectiveness ofcurriculum can vary with it.
Revised: However, the relative difﬁculty of the problem of inferring the support of the teacher and the problem of aligning with its non-zero components depends on the degree of sparsity ρ , so the effectiveness of curriculum can vary with it.

Original: In the right panel of Fig. 4, we explore the interplay between the sparsity of the teacher ρ and thefraction of easy samples in the dataset α 1 , comparing curriculum with the no-curriculum baseline.
Revised: In the right panel of Fig. 4, we explore the interplay between the sparsity of the teacher ρ and the fraction of easy samples in the dataset α 1 , comparing curriculum with the no-curriculum baseline.

Original: The phase diagram highlights the variability in the impact of the curriculum ordering:
Revised: The phase diagram highlights the variability in the impact of the curriculum ordering:

Original: • Curriculum is most effective at low values of ρ and close to the diagonal, where the fraction 
Revised: • Curriculum is most effective at low values of ρ and close to the diagonal, where the fraction of easy

Original: of easy examples in the dataset is comparable to the fraction of relevant dimensions.
Revised: examples in the dataset is comparable to the fraction of relevant dimensions.

Original: • When ρ > 0 . 5 , the possible gain from ordering the samples according to difﬁculty is counterbalanced by the instrinsic cost of splitting the information content into two blocks, thus curriculum can become detrimental.
Revised: • When ρ > 0 . 5 , the possible gain from ordering the samples according to difﬁculty is counterbalanced by the instrinsic cost of splitting the information content into two blocks, thus curriculum can become detrimental.

Original: • When α 1 is too small compared to ρ (above diagonal), the ﬁrst stage in the curriculum strategy can only help in the support identiﬁcation problem, but will not allow a good estimation of the direction of the teacher.
Revised: • When α 1 is too small compared to ρ (above diagonal), the ﬁrst stage in the curriculum strategy can only help in the support identiﬁcation problem, but will not allow a good estimation of the direction of the teacher.

Original: Because of the elastic prior, the second stage cannot improve too much over it and the effect of curriculum is small.
Revised: Because of the elastic prior, the second stage cannot improve too much over it and the effect of curriculum is small.

Original: • When α is larger than the sparsity (below diagonal), the easy examples contain sufﬁcient information for solving both the support and the teacher estimation problems, and this information is also exploited by the baseline.
Revised: • When α is larger than the sparsity (below diagonal), the easy examples contain sufﬁcient information for solving both the support and the teacher estimation problems, and this information is also exploited by the baseline.

Original: Thus the improvement of curriculum becomes negligible.
Revised: Thus the improvement of curriculum becomes negligible.

Original: We refer to the SM for an in-depth comparison with anti-curriculum.
Revised: We refer to the SM for an in-depth comparison with anti-curriculum

Original: 2.0 1 +0.

Original: [Figure]
Revised: [Figure]

Revised: In contrast to the online case, performance beneﬁts are greater and curriculum is strictly better than anti-curriculum.

Revised: Both γ 1 = γ 2 and γ 12 are optimised point-wise, in order to yield the best test accuracy.

Revised: Parameters: ρ = 0 .

Revised: 50 , ∆ 1 = 0 , ∆ 2 = 1 .

Original: Asymptotic advantages of curriculum.
Revised: Asymptotic advantages of curriculum.

Original: Contrary to the  online SGD case, if the fraction of relevantdirections is small, batch learning with elastic coupling notably improves test accuracy of bothcurriculum and anti-curriculum above the baseline.
Revised: Contrary to the case of online SGD, if the fraction of relevant directions is small, batch learning with elastic coupling notably improves test accuracy of both curriculum and anti-curriculum above the baseline.

Original: This conﬁrms the utility of curriculum strategieswhen the signal is partially "hidden in clutter" [42].
Revised: This conﬁrms the utility of curriculum strategies when the signal is partially "hidden in clutter" [49].

Original: Fig.
Revised: Fig.

Original: 5 shows similar phase diagrams to Fig.
Revised: 5 shows similar phase diagrams to Fig.

Original: 2but for the batch setting.
Revised: 2 but for the batch setting.

Original: At each point in the phase diagram the regularisation level γ 1 = γ 2 and thecoupling γ 12 are optimised to yield the best accuracy.
Revised: At each point in the phase diagram the regularisation level γ 1 = γ 2 and the coupling γ 12 are optimised to yield the best accuracy.

Original: In batch learning the performance order appearsto be nearly always preserved: curriculum followed by anti-curriculum followed by baseline.a cc u r a c y r a t i o      1. 11.8 difficulty 1. [Figure]
Revised: We ﬁnd that the performance order is nearly always preserved: curriculum followed by anti-curriculum followed by baseline.

Original: difﬁculty. a cc u r a c y r a t i o 0. [Figure] [Figure]
Revised: In the SM we see similar improvements by applying the elastic coupling strategy both in the online setting and on real data.

Original: 1 =0. [Figure] [Figure] [Figure]
Revised: In summary, in the batch setting, splitting the learning process in stages might not be advantageous per se.

Original: old world from daemons of the new world. The distinguishing feature (horn length) is diluted among many irrelevant features (colour, eye size, mouth size). Highlighting the relevant feature to participants leads to better and faster learning.
Revised: However, our observations show that if the loss is modiﬁed to reduce memory loss between the learning stages, curriculum learning strategies can offer a measurable asymptotic advantage .

Original:  Connection with experimental literature
Revised:  Connection with experimental literature

Original: Recent work has suggested that curriculum learning could provide an important window into thelearning algorithms at work in biology [44].
Revised: Recent work has suggested that curriculum learning could provide an important window into the learning algorithms at work in biology [51].

Original: Our analysis makes several predictions for curriculumeffects.
Revised: Our analysis makes several predictions for curriculum effects.

Original: In this section we assess these predictions based on connections to extant experiments andpropose future experimental tests.
Revised: In this section we assess these predictions based on connections to extant experiments and propose future experimental tests.

Original: First, we ﬁnd that a curriculum strategy yields a speed up in learning in all the tested settings (see
Revised: First, we ﬁnd that a curriculum strategy yields a speed up in learning in all the tested settings (see

Original: Fig. 1c).
Revised: Fig. 1c).

Original: This acceleration is broadly consistent with the ﬁndings from cognitive science [1, 2, 6].
Revised: This acceleration is broadly consistent with the ﬁndings from cognitive science [1, 2, 6].

Original: Bycontrast, our results show that the speed improvement does not necessarily translate into a sizeablegeneralisation error improvement, and the performance achieved at the end of training can evendeteriorate when learning hyperparameters are not fully optimised (c.f. Fig. 3).
Revised: By contrast, our results show that the speed improvement does not necessarily translate into a sizeable a cc u r a c y r a t i o      1. [Figure] 8 difficulty 1. [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: a

Revised: [Figure]

Revised: the old world from daemons of the new world.

Revised: The distinguishing feature (horn length) is diluted among many irrelevant features (colour, eye size, mouth size).

Revised: Highlighting the relevant feature to participants leads to better and faster learning.

Revised: generalisation error improvement, and the performance achieved at the end of training can even deteriorate when learning hyperparameters are not fully optimised (c.f. Fig. 3).

Original: Deterioration due tocurricula has generally not been reported in the psychology literature, though it has been observed in
Revised: Deterioration due to curricula has generally not been reported in the psychology literature, though it has been observed in

Original: ML [15].
Revised: ML [15].

Original: This fact may suggest that animals naturally learn with near-optimal hyperparameters suchthat curricula generally confer beneﬁts.
Revised: This fact may suggest that animals naturally learn with near-optimal hyperparameters such that curricula generally confer beneﬁts.

Original: A more speciﬁc observation concerns the performance on different difﬁculties after learning.
Revised: A more speciﬁc observation concerns the performance on different difﬁculties after learning.

Original: Asreported in [43], human and rodent subjects trained in an auditory task using curricula showed thegreatest improvement for intermediate level of difﬁculties as depicted in Fig.
Revised: As reported in [50], human and rodent subjects trained in an auditory task using curricula showed the greatest improvement for intermediate levels of difﬁculty as depicted in Fig.

Original: 6a bottom panel.
Revised: 6a bottom panel.

Original: Thesame conclusion can be drawn from the experiment of [7, 8], where, surprisingly, subjects trainedwith curricula to classify medical images showed poor performance in hard tasks compared to thecontrol group.
Revised: The same conclusion can be drawn from the experiment of [7, 8], where, surprisingly, subjects trained with curricula to classify medical images showed poor performance in hard tasks compared to the control group.

Original: To address this phenomenon, we calculate accuracy as a function of difﬁculty inthe model in Fig.
Revised: To address this phenomenon, we calculate accuracy as a function of difﬁculty in the model in Fig.

Original: 6a top panel.
Revised: 6a top panel.

Original: Consistent with these experiments, we ﬁnd regimes where the gapbetween curriculum learning and the baseline is non-monotonic, with the largest performance gain forintermediate difﬁculties.
Revised: Consistent with these experiments, we ﬁnd regimes where the gap between curriculum learning and the baseline is non-monotonic, with the largest performance gain for intermediate difﬁculties.

Original: Contrary to [7, 8], however, we do not observe negative effects of curriculumfor high difﬁculties.
Revised: Contrary to [7, 8], however, we do not observe negative effects of curriculum for high difﬁculties.

Original: Further experiments that more systematically manipulate training and transferdifﬁculties could provide a stronger test of these predictions.
Revised: Further experiments that more systematically manipulate training and transfer difﬁculties could provide a stronger test of these predictions.

Original: A key ingredient in our model is the role of sparsity, such that a small signal is embedded amidst manyirrelevant features.
Revised: A key ingredient in our model is the role of sparsity, such that a small signal is embedded amidst many irrelevant features.

Original: Experimentally, the importance of having many factors of variation to obtaininga curriculum effect has been documented in the “fading” experiments of [6].
Revised: Experimentally, the importance of having many factors of variation to obtaining a curriculum effect has been documented in the “fading” experiments of [6].

Original: Human subjectswere trained on classiﬁcation tasks involving stimuli with one task-relevant feature dimension anda variable number of task-irrelevant feature dimensions.
Revised: Human subjects were trained on classiﬁcation tasks involving stimuli with one task-relevant feature dimension and a variable number of task-irrelevant feature dimensions.

Original: Example cartoon “daemon” stimuli aredepicted in Fig.
Revised: Example cartoon “daemon” stimuli are depicted in Fig.

Original: 6c, where for instance horn height might be the distinguishing feature while colour, [Figure] y d a t a s e t s i z
Revised: 6c, where for instance horn height might be the distinguishing feature while colour, eye size, and mouth size might constitute task-irrelevant features.

Original: e

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: eye size, and mouth size might constitute task-irrelevant features.

Original: Without any irrelevant factors ofvariation ( ρ = 1 ), they report no curriculum beneﬁt.
Revised: Without any irrelevant factors of variation ( ρ = 1 ), they report no curriculum beneﬁt.

Original: By contrast when 75% of features are irrelevant( ρ = . 25 ), they record a strong curriculum effect, as shown in Fig.
Revised: By contrast when 75% of features are irrelevant ( ρ = . 25 ), they record a strong curriculum effect, as shown in Fig.

Original: 6b bottom.
Revised: 6b bottom.

Original: This qualitative trendis also observed in our model (Fig.
Revised: This qualitative trend is also observed in our model (Fig.

Original: 6b top).
Revised: 6b top).

Original: While these experiments tested only two sparsity levels,further experiments could sample this dimension more extensively and test for interactions with thefraction of easy and hard examples.
Revised: While these experiments tested only two sparsity levels, further experiments could sample this dimension more extensively and test for interactions with the fraction of easy and hard examples.

Original: We note that while the connectionist literature has addressed theeffect of curriculum in several settings [45, 46, 10, 11], we found that easy-to-hard effects appeareven in a simple setup without need for complex networks and/or dynamics.
Revised: We note that while the connectionist literature has addressed the effect of curriculum in several settings [39, 40, 10, 11], we found that easy-to-hard effects appear even in a simple setup without need for complex networks and/or dynamics.

Original: Finally, our results may shed light on self-generated curricula during human development [47, 48].
Revised: Finally, our results may shed light on self-generated curricula during human development [52, 53].

Original: Children undergo aspurt of vocabulary development that coincides with their ability to grasp andcentre objects in the visual ﬁeld [48].
Revised: Children undergo a vocabulary spurt that coincides with their ability to grasp and centre objects in the [Figure] a [Figure]

Revised: y d a t a s e t s i

Revised: z

Original: Quantitative estimates of the amount of clutter (irrelevantobjects) in self-generated views decrease due to this grasping ability, yielding a self-generatedcurriculum [42, 49].
Revised: e [Figure] [Figure] [Figure]

Revised: visual ﬁeld [53].

Revised: Quantitative estimates of the amount of clutter (irrelevant objects) in self-generated views decrease due to this grasping ability, yielding a self-generated curriculum [49, 54].

Original: Our model similarly predicts that reducing clutter should improve learning speedand performance.
Revised: Our model similarly predicts that reducing clutter should improve learning speed and performance.

Original:  To verify this  in a richer visual setting, weapply our curriculum-aware algorithmon real data. We construct a simple cluttered object recognition task from the CIFAR10 dataset [50]by patching two images together into a 32 × 64 input image (Fig. 7a).
Revised: Real-World Demonstration. To verify this prediction in a richer visual setting, we construct a simple cluttered object classiﬁcation task from the CIFAR10 dataset [55] by patching two images together into a 32 × 64 input image (Fig. 7a).

Original: The network has to learnthat the classiﬁcation depends only on the left image, while the  right image is a distractor that isirrelevant to the classiﬁcation.
Revised: The task is to produce the class label of the image on the left. The right image is a distractor that is irrelevant to the classiﬁcation.

Original: Easy and hard instances are obtained by reducing the brightness of thedistractor.
Revised: To vary difﬁculty, we scale the contrast of the irrelevant image (Fig.

Original:  We train a single-layer network with the cross-entropy loss and the curriculum protocolwith Gaussian prior betweenthe two stages. Weights are optimised using SGD and momentum, withan annealed learning rate.
Revised: 7a-d). We train a single-layer network with the cross-entropy loss and the curriculum protocol with Gaussian prior between two curriculum stages, implemented in Pytorch Lightning to ensure that training parameters accord with standard practice.

Original: All training parameters are optimised, and full details are presented inthe SM.
Revised: We optimised hyperparameters in each curriculum phase separately.

Original:  Fig.
Revised: We trained all combinations of ﬁve elastic penalties log spaced between 1 e − 3 and 1 e 2 , and weight decay parameters { 0 , . We then compute the best performing model for ﬁve random seeds and take the mean over seeds. Further dataset, model and experimental details are given in Appendix ?? . As shown in Fig.

Original: 7 shows a robust curriculum advantage in this setting, suggesting a possible functionalbeneﬁt for children’s self-generated curriculum.
Revised: 7b, curriculum improves performance, particularly when easy examples make up a large proportion of the dataset, conﬁrming that curricula that reduce clutter can beneﬁt learning.

Original:  Conclusions
Revised:  Conclusions

Original: We analysed a model of curriculum learning introduced by [12] and amenable of analytical treatment.
Revised: We analysed a model of curriculum learning introduced by [12] and amenable of analytical treatment.

Original: This simple setting sheds light on results observed in the cognitive science and machine learningliterature, and the theoretical tractability allows for exploration of a wide range of parameters thatwould be costly to obtain through experiments.
Revised: This simple setting sheds light on results observed in the cognitive science and machine learning literature, and the theoretical tractability allows for exploration of a wide range of parameters that would be costly to obtain through experiments.

Original: Future work will need to move beyond models withsimple loss landscapes to address the impact of curricula in complex tasks like reinforcement learning.
Revised: Future work will need to move beyond models with simple loss landscapes to address the impact of curricula in complex tasks like reinforcement learning.

Original: Nevertheless, the model recapitulates a variety of observations in the literature [43, 51, 52], revealingthat easy-to-hard effects can appear when a sparse signal is embedded in many irrelevant dimensionsof variation.
Revised: Nevertheless, the model recapitulates a variety of observations in the literature [50, 56, 57], revealing that easy-to-hard effects can appear when a sparse signal is embedded in many irrelevant dimensions of variation.

Original: We ﬁnd that  curriculum-aware algorithms can better exploit curricula, offering a potentialroute for improved practical algorithms.
Revised: We ﬁnd that making the algorithm curriculum-aware by modifying the loss can better exploit curricula, offering a potential route for improved practical algorithms.

Original: On the psychology side, some of our predictions can help indesigning new experiments.

Original: The beneﬁt of anti-curriculum learning for intermediate sparsity, is acounter-intuitive result testable in animal experiments.

Original: Checklist

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope? [Yes]
Revised: Other curriculum-aware approaches are possible such as adapting the learning algorithm [58] or the architecture [10].

Original: (b) Did you describe the limitations of your work?

Original: [Yes]

Original: (c) Did you discuss any potential negative societal impacts of your work?

Original: [No] The modelis theoretical in nature and we do not foresee any negative societal impact.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?

Original: [Yes]

Original: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] Refer to

Original: sections "model deﬁnition". (b) Did you include complete proofs of all theoretical results? [N/A] In the SM we provide
Revised: On the psychology side, our predictions can help in designing new experiments, for instance testing the counter-intuitive beneﬁt of anti-curriculum learning for intermediate sparsity.

Original: details about the derivation.

Original: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [No] The architecture used is very simple to simulate and we do not provide code.
Revised: Acknowledgments and Disclosure of Funding

Original: (b) Did you specify all the training details (e g., data splits, hyperparameters, how they were chosen)?
Revised: We thank Miguel Ruiz-Garcia and Ronald Dekker for important discussions.

Original: [Yes] All the ﬁgures resulting from the experiments contain details to

Original: produce them.

Original: Full parameters for the experiments on real data are given in the SM.

Original: (c) Did you report error bars (e

Original: g., with respect to the random seed after running experiments multiple times)?

Original: [Yes]

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes] We report estimated total amountof compute and type of compute in the SM for the experiment on real data ( ≈

Original: GPU hours, ≈ 1110 kg CO2 eq).

Original: The data for each theory-based ﬁgure can be obtained

Original: in a few hours of laptop simulation.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators? [Yes] We use and cite the CIFAR10 dataset.
Revised: L.S. acknowledges funding from the ERC European Union Horizon 2020 Research and Innovation Program Grant

Original: (b) Did you mention the license of the assets? [No] It is widely known that CIFAR10 is anMIT License dataset. (c) Did you include any new assets either in the supplemental material or as a URL? [No]
Revised: Agreement 714608-SIiLe.

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating? [N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable
Revised: S.S.M. & A.S. were supported by a Wellcome and Royal Society Henry

Original: information or offensive content? [N/A] If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if
Revised: Dale Fellowship (216386/Z/19/Z) and Sainsbury Wellcome Centre Core Grant (219627/Z/19/Z,

Original: applicable? [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: GAT3755).

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
Revised: A.S. is a CIFAR Azrieli Global Scholar in the Learning in Machines & Brains programme.