Original: M ODELLING THE INFLUENCE OF DATA STRUCTURE ON LEARNING IN NEURAL NETWORKS
Revised: M ODELLING THE INFLUENCE OF DATA STRUCTURE ON LEARNING IN NEURAL NETWORKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The lack of crisp mathematical models that capture the structure of real-world data sets is a major obstacle to the detailed theoretical understanding of deep neural networks.
Revised: The lack of crisp mathematical models that capture the structure of real-world data sets is a major obstacle to the detailed theoretical understanding of deep neural networks.

Original: Here, we ﬁrst demonstrate the effect of structured data sets by experimentally comparing the dynamics and the performance of two-layer networks trained on two different data sets: (i) an unstructured synthetic data set containing random i.i.d.
Revised: Here, we ﬁrst demonstrate the effect of structured data sets by experimentally comparing the dynamics and the performance of two-layer networks trained on two different data sets: (i) an unstructured synthetic data set containing random i.i.d.

Original: inputs, and (ii) a simple canonical data set such as MNIST images.
Revised: inputs, and (ii) a simple canonical data set containing MNIST images.

Original: Our analysis reveals two phenomena related to the dynamics of the networks and their ability to generalise that only appear when training on structured data sets.
Revised: Our analysis reveals two phenomena related to the dynamics of the networks and their ability to generalise that only appear when training on structured data sets.

Original: Second, we introduce a generative model for data sets, where high-dimensional inputs lie on a lower-dimensional manifold and have labels that depend only on their position within this manifold.
Revised: Second, we introduce a generative model for data sets, where high-dimensional inputs lie on a lower-dimensional manifold and have labels that depend only on their position within this manifold.

Original: We call it the hidden manifold model and we experimentally demonstrate that training networks on data sets drawn from this model reproduces both the phenomena seen during training on MNIST.
Revised: We call it the hidden manifold model and we experimentally demonstrate that training networks on data sets drawn from this model reproduces both the phenomena seen during training on MNIST

Original:  I NTRODUCTION 
Revised:  I NTRODUCTION AND RELATED WORK

Original: A major impediment for understanding the effectiveness of deep neural networks is our lack of mathematical models for the data sets on which neural networks are trained.
Revised: A major impediment for understanding the effectiveness of deep neural networks is our lack of mathematical models for the data sets on which neural networks are trained.

Original: This lack of tractable models prevents us from analysing the impact of data sets on the training of neural networks and their ability to generalise from examples, which remains an open problem both in statistical learning theory (Vapnik, 2013; Mohri et al., 2012), and in analysing the average-case behaviour of algorithms in synthetic data models (Seung et al., 1992; Engel & Van den Broeck, 2001; Zdeborov ´ a & Krzakala, 2016).
Revised: This lack of tractable models prevents us from analysing the impact of data sets on the training of neural networks and their ability to generalise from examples, which remains an open problem both in statistical learning theory (Vapnik, 2013; Mohri et al., 2012), and in analysing the average-case behaviour of algorithms in synthetic data models (Seung et al., 1992; Engel & Van den Broeck, 2001; Zdeborov ´ a & Krzakala, 2016).

Original: Indeed, most theoretical results on neural networks do not model the structure of the training data, while some works build on a setup where inputs are drawn component-wise i.i.d.
Revised: Indeed, most theoretical results on neural networks do not model the structure of the training data, while some works build on a setup where inputs are drawn component-wise i.i.d.

Original: from some probability distribution, and labels are either random or given by some random, but ﬁxed function of the inputs.
Revised: from some probability distribution, and labels are either random or given by some random, but ﬁxed function of the inputs.

Original: Despite providing valuable insights, these approaches are by construction blind to key structural properties of real-world data sets.
Revised: Despite providing valuable insights, these approaches are by construction blind to key structural properties of real-world data sets.

Original: Here, we focus on two types of data structure that can both already be illustrated by considering the simple canonical problem of classifying the handwritten digits in the MNIST database using a neural network N (LeCun & Cortes, 1998).
Revised: Here, we focus on two types of data structure that can both already be illustrated by considering the simple canonical problem of classifying the handwritten digits in the MNIST database using a neural network N (LeCun & Cortes, 1998).

Original: The input patterns are images with 28 × 28 pixels, so a priori we work in the high-dimensional R 784 .
Revised: The input patterns are images with 28 × 28 pixels, so a priori we work in the high-dimensional R 784 .

Original: However, the inputs that may be interpreted as handwritten digits, and hence constitute the “world” of our problem, span but a lower-dimensional manifold within R 784 which is not easily deﬁned.
Revised: However, the inputs that may be interpreted as handwritten digits, and hence constitute the “world” of our problem, span but a lower-dimensional manifold within R 784 which is not easily deﬁned.

Original: Its dimension can nevertheless be estimated to be around D
Revised: Its dimension can nevertheless be estimated to be around D

Original: ≈ 

Revised: ≈

Original: based on the neighbourhoods of inputs in the data set (Grassberger & Procaccia, 1983; Costa & Hero, 2004; Levina & Bickel, 2004;  Spigler et al., 2019).
Revised: based on the neighbourhoods of inputs in the data set (Grassberger & Procaccia, 1983; Costa & Hero, 2004; Levina & Bickel, 2004; Facco et al., 2017; Spigler et al., 2019).

Original: The intrinsic dimension being lower than the dimension of the input space is a property expected to be common to many real data sets used in machine leanring.
Revised: The intrinsic dimension being lower than the dimension of the input space is a property expected to be common to many real data sets used in machine leanring.

Original: We should not consider presenting N with an input that is outside of its world (or maybe we should train it to answer that the “input is outside of my world” in such cases).
Revised: We should not consider presenting N with an input that is outside of its world (or maybe we should train it to answer that the “input is outside of my world” in such cases).

Original: We will call inputs structured if they are concentrated on a lower-dimensional manifold and thus have a lower-dimensional latent representation.
Revised: We will call inputs structured if they are concentrated on a lower-dimensional manifold and thus have a lower-dimensional latent representation.

Original: The second type of the structure concerns the function of the inputs that is to be learnt, which we will call the learning task .
Revised: The second type of the structure concerns the function of the inputs that is to be learnt, which we will call the learning task.

Original: We will consider two models: the teacher task, where the label is obtained as a function of the high-dimensional input; and the latent task, where the label is a function of only the lower-dimensional latent representation of the input.
Revised: We will consider two models: the teacher task , where the label is obtained as a function of the high-dimensional input; and the latent task , where the label is a function of only the lower-dimensional latent representation of the input.

Original: [Table]
Revised: [Table]

Original:  manifold in
Revised: dimensional manifold in

Original: [Table]
Revised: [Table]

Original: for [Table] structured input, its coordinates in the 
Revised: for a structured input, its coordinates in the lower

Original: [Table]
Revised: [Table]

Original:  manifold task the function of the inputs to be learnt
Revised: dimensional manifold task the function of the inputs to be learnt

Original: [Table]
Revised: [Table]

Original: for structured inputs, labels are given as a function of the latent representation only
Revised: for structured inputs, labels are given as a function of the latent representation only

Original: [Table]
Revised: [Table]

Original: for all inputs, labels are obtained from a random, but ﬁxed function of the high-dimensional input without explicit dependence on the latent representation, if it exists
Revised: for all inputs, labels are obtained from a random, but ﬁxed function of the high-dimensional input without explicit dependence on the latent representation, if it exists

Original: [Table] task discriminating odd from even digits in the MNIST database 
Revised: MNIST task discriminating odd from even digits in the MNIST database [Table]

Original: [Table]
Revised: [Table]

Original: Generative model due to Gardner & Derrida (1989), where data sets consist of component-wise i.i.d.
Revised: Generative model due to Gardner & Derrida (1989), where data sets consist of component-wise i.i.d.

Original: inputs with labels given by a ﬁxed, but random neural network acting directly on the input hidden manifold model (HMF)
Revised: inputs with labels given by a ﬁxed, but random neural network acting directly on the input hidden manifold model (HMF)

Original: Generative model introduced in Sec. 4 for data sets consisting of structured inputs (Eq. 6) with latent labels (Eq. 7)
Revised: Generative model introduced in Sec. 4 for data sets consisting of structured inputs (Eq. 6) with latent labels (Eq. 7)

Original: We begin this paper by comparing neural networks trained on two different problems: the MNIST task , where one aims to discriminate odd from even digits in the in the MNIST data set; and the vanilla teacher-student setup , where inputs are drawn as vectors with i.i.d.
Revised: We begin this paper by comparing neural networks trained on two different problems: the MNIST task , where one aims to discriminate odd from even digits in the in the MNIST data set; and the vanilla teacher-student setup , where inputs are drawn as vectors with i.i.d.

Original: component from the
Revised: component from the

Original: Gaussian distribution and labels are given by a random, but ﬁxed, neural network acting on the high-dimensional inputs.
Revised: Gaussian distribution and labels are given by a random, but ﬁxed, neural network acting on the high-dimensional inputs.

Original: This model is an example of a teacher task on unstructured inputs.
Revised: This model is an example of a teacher task on unstructured inputs.

Original: It was introduced by Gardner & Derrida (1989) and has played a major role in theoretical studies of the generalisation ability of neural networks from an average-case perspective, particularly within the framework of statistical mechanics (Seung et al., 1992; Watkin et al., 1993; Engel & Van den Broeck, 2001; Zdeborov ´ a & Krzakala, 2016; Advani & Saxe, 2017; Aubin et al., 2018; Barbier et al., 2019;
Revised: It was introduced by Gardner & Derrida (1989) and has played a major role in theoretical studies of the generalisation ability of neural networks from an average-case perspective, particularly within the framework of statistical mechanics (Seung et al., 1992; Watkin et al., 1993; Engel & Van den Broeck, 2001; Zdeborov ´ a & Krzakala, 2016; Advani & Saxe, 2017; Aubin et al., 2018; Barbier et al., 2019;

Original: Goldt et al., 2019; Yoshida et al., 2019), and also in recent statistical learning theory works, e.g. (Ge et al., 2017; Li & Yuan, 2017; Mei & Montanari, 2019; Arora et al., 2019).
Revised: Goldt et al., 2019; Yoshida et al., 2019), and also in recent statistical learning theory works, e.g. (Ge et al., 2017; Li & Y., 2017; Mei & Montanari, 2019; Arora et al., 2019).

Original: We choose the MNIST data set because it is the simplest widely used example of a structured data set on which neural networks show signiﬁcantly different behaviour than when trained on synthetic data of the vanilla teacher-student setup.
Revised: We choose the MNIST data set because it is the simplest widely used example of a structured data set on which neural networks show signiﬁcantly different behaviour than when trained on synthetic data of the vanilla teacher-student setup.

Original: Our main contributions are then twofold:
Revised: Our reasoning then proceeds in two main steps: 1.

Original: We experimentally identify two key differences between networks trained in the vanilla teacherstudent setup and networks trained on the MNIST task (Sec. 3).
Revised: We experimentally identify two key differences between networks trained in the vanilla teacherstudent setup and networks trained on the MNIST task (Sec. 3).

Original: i)
Revised: i)

Original: Two identical networks trained on the same MNIST task, but starting from different initial conditions, will achieve the same test error on MNIST images, but they learn globally different functions.
Revised: Two identical networks trained on the same MNIST task, but starting from different initial conditions, will achieve the same test error on MNIST images, but they learn globally different functions.

Original: Their outputs coincide in those regions of input space where MNIST images tend to lie – the “world” of the problem, but differ signiﬁcantly when tested on Gaussian inputs.
Revised: Their outputs coincide in those regions of input space where MNIST images tend to lie – the “world” of the problem, but differ signiﬁcantly when tested on Gaussian inputs.

Original: In contrast, two networks trained on the teacher task learn the same functions globally to within a small error.
Revised: In contrast, two networks trained on the teacher task learn the same functions globally to within a small error.

Original: ii)
Revised: ii)

Original: In the vanilla teacher-student setup, the test error of a network is stationary during long periods of training before a sudden drop-off.
Revised: In the vanilla teacher-student setup, the test error of a network is stationary during long periods of training before a sudden drop-off.

Original: These plateaus are well-known features of this setup (Saad & Solla,  Engel & Van den Broeck ), but are not observed when training on the MNIST task nor on other datasets used commonly in machine learning.
Revised: These plateaus are well-known features of this setup (Saad & Solla, 1995; Engel & Van den Broeck, 2001), but are not observed when training on the MNIST task nor on other datasets used commonly in machine learning.

Original:  We introduce the hidden manifold model (HMF), a probabilistic model that generates data sets containing high-dimensional inputs which lie on a lower-dimensional manifold and whose labels depend only on their position within that manifold (Sec. 4).
Revised: Our main contribution: We introduce the hidden manifold model (HMF), a probabilistic model that generates data sets containing high-dimensional inputs which lie on a lower-dimensional manifold and whose labels depend only on their position within that manifold (Sec. 4).

Original: In this model, inputs are thus structured and labels depend on their lower-dimensional latent representation.
Revised: In this model, inputs are thus structured and labels depend on their lower-dimensional latent representation.

Original: We experimentally demonstrate that training networks on data sets drawn from this model reproduces both behaviours observed when training on MNIST.
Revised: We experimentally demonstrate that training networks on data sets drawn from this model reproduces both behaviours observed when training on MNIST.

Original: We also show that the structure of both, input space and the task to be learnt, play an important role for the dynamics and the performance of neural networks.
Revised: We also show that the structure of both, input space and the task to be learnt, play an important role for the dynamics and the performance of neural networks.

Original: Related work Several works have compared neural networks trained from different initial conditions on the same task by comparing the different features learnt in vision problems (Li et al., 2015; Raghu et al., 2017; Morcos et al., 2018), but these works did not compare the functions learned by the network.
Revised: Other related work Several works have compared neural networks trained from different initial conditions on the same task by comparing the different features learnt in vision problems (Li et al., 2015; Raghu et al., 2017; Morcos et al., 2018), but these works did not compare the functions learned by the network.

Original: On the theory side, several works have appreciated the need to model the inputs, and to go beyond the simple component-wise i.i.d.
Revised: On the theory side, several works have appreciated the need to model the inputs, and to go beyond the simple component-wise i.i.d.

Original: modelling (Bruna & Mallat, 2013; Patel et al., 2016; M´ezard, 2017; Gabri´e et al., 2018; Mossel, 2018).
Revised: modelling (Bruna & Mallat, 2013; Patel et al., 2016; M´ezard, 2017; Gabri´e et al., 2018; Mossel, 2018; Saxe et al., 2019).

Original: While we will focus on the ability of neural network to generalise from examples, two recent papers studied a network’s ability to store inputs with lower-dimensional structure and random labels: Chung et al.
Revised: While we will focus on the ability of neural network to generalise from examples, two recent papers studied a network’s ability to store inputs with lower-dimensional structure and random labels: Chung et al.

Original: (2018) studied the linear separability of general, ﬁnite-dimensional manifolds, while Rotondo et al.
Revised: (2018) studied the linear separability of general, ﬁnite-dimensional manifolds, while Rotondo et al.

Original: extended Cover’s argument (Cover, 1965) to count the number of learnable dichotomies when inputs are grouped in tuples of k inputs with the same label.
Revised: extended Cover’s argument (Cover, 1965) to count the number of learnable dichotomies when inputs are grouped in tuples of k inputs with the same label.

Original: Accessibility and reproducibility The full code of our experiments can be accessed via https: //drive.google.com/open?id=1L0UOtOoRTYSHZtTxMxKIQuZLEuVaoJl_ .
Revised: Accessibility and reproducibility The full code of our experiments can be accessed via https: //drive.google.com/open?id=1L0UOtOoRTYSHZtTxMxKIQuZLEuVaoJl_ .

Original: We give necessary parameter values to reproduce our ﬁgures beneath each plot.
Revised: We give necessary parameter values to reproduce our ﬁgures beneath each plot.

Original: For ease of reading, we adopt the notation from the textbook by Goodfellow et al.
Revised: For ease of reading, we adopt the notation from the textbook by Goodfellow et al.

Original:  S ETUP
Revised:  S ETUP

Original: In order to proceed on the question of what is a suitable model for structured data, we consider the setup of a feedforward neural network with one hidden layer with a few hidden units, as described below.
Revised: In order to proceed on the question of what is a suitable model for structured data, we consider the setup of a feedforward neural network with one hidden layer with a few hidden units, as described below.

Original: We chose this setting because it is the simplest one we found where we were able to identify key differences between training in the vanilla teacher-student setup and training on the MNIST task.
Revised: We chose this setting because it is the simplest one we found where we were able to identify key differences between training in the vanilla teacher-student setup and training on the MNIST task.

Original: So throughout this work, we focus on the dynamics and performance of fully-connected two-layer neural networks with K hidden units and ﬁrst- and second-layer weights W ∈ R K × N and v ∈ R K , resp.
Revised: So throughout this work, we focus on the dynamics and performance of fully-connected two-layer neural networks with K hidden units and ﬁrst- and second-layer weights W ∈ R K × N and v ∈ R K , resp.

Original: Given an input x ∈ R N , the output of a network with parameters θ =
Revised: Given an input x ∈ R N , the output of a network with parameters θ =

Original: ( W , v ) is given by
Revised: ( W , v ) is given by

Original: [Equation]
Revised: [Equation]

Original: where w k is the k th row of W , and g :
Revised: where w k is the k th row of W , and g :

Original: R → R is the non-linear activation function of the network, acting component-wise.
Revised: R → R is the non-linear activation function of the network.

Original: We will focus on sigmoidal networks with g ( x ) =
Revised: We will focus on sigmoidal networks with g ( x ) =

Original: erf( x/ √ 2) , or ReLU networks where g ( x ) = max(0 , x ) (see Appendix E).
Revised: erf( x/ √ 2) , or ReLU networks where g ( x ) = max(0 , x ) (see Appendix E).

Original: We will train the neural networks on data sets with P input-output pairs ( x i , y ∗ i ) , i = 1 , . . .
Revised: We will train the neural networks on data sets with P input-output pairs ( x i , y ∗ i ) , i = 1 , . . .

Original: , P , where we use the starred y ∗ i to denote the true label of an input x i .
Revised: , P , where we use the starred y ∗ i to denote the true label of an input x i .

Original: We train networks by minimising the quadratic training error E ( θ ) = 1 / 2 (cid:80) Pi =1 ∆ 2 i with ∆ i =
Revised: We train networks by minimising the quadratic training error E ( θ ) = 1 / 2 (cid:80) Pi =1 ∆ 2 i with ∆ i =

Original: [ φ ( x i , θ ) − y ∗
Revised: [ φ ( x i , θ ) − y ∗

Original: i ] using stochastic gradient descent (SGD) with constant learning rate η ,
Revised: i ] using stochastic gradient descent (SGD) with constant learning rate η ,

Original: [Equation]
Revised: [Equation]

Original: Initial weights for both layers of sigmoidal networks were always taken component-wise i.i.d.
Revised: Initial weights for both layers of sigmoidal networks were always taken component-wise i.i.d.

Original: from the normal distribution with mean 0 and variance 1.
Revised: from the normal distribution with mean 0 and variance 1.

Original: The initial weights of ReLU networks were also taken from the normal distribution, but with variance 10 − 6 to ensure convergence.
Revised: The initial weights of ReLU networks were also taken from the normal distribution, but with variance 10 − 6 to ensure convergence.

Original: The key quantity of interest is the test error or generalisation error of a network, for which we compare its predictions to the labels given in a test set that is composed of P ∗ input-output pairs ( x i , y ∗ i ) , i = 1 , . .
Revised: The key quantity of interest is the test error or generalisation error of a network, for which we compare its predictions to the labels given in a test set that is composed of P ∗ input-output pairs ( x i , y ∗ i ) , i = 1 , . .

Original: , P ∗ that are not used during training,
Revised: , P ∗ that are not used during training,

Original: [Equation]
Revised: [Equation]

Original: The test set might be composed of MNIST test images or generated by the same probabilistic model that generated the training data.
Revised: The test set might be composed of MNIST test images or generated by the same probabilistic model that generated the training data.

Original: For binary classiﬁcation tasks with y ∗ = ± 1 , this deﬁnition is easily amended to give the fractional generalisation error (cid:15) frac g ( θ ) ∝ (cid:80) P ∗ i Θ [ − φ ( x i , θ ) y ∗ i ] , where Θ( · ) is the Heaviside step function.
Revised: For binary classiﬁcation tasks with y ∗ = ± 1 , this deﬁnition is easily amended to give the fractional generalisation error (cid:15) frac g ( θ ) ∝ (cid:80) P ∗ i Θ [ − φ ( x i , θ ) y ∗ i ] , where Θ( · ) is the Heaviside step function.

Original: 2. L EARNING FROM REAL DATA OR FROM GENERATIVE MODELS ?
Revised: 2. L EARNING FROM REAL DATA OR FROM GENERATIVE MODELS ?

Original: We want to compare the behaviours of two-layer neural networks Eq.
Revised: We want to compare the behaviours of two-layer neural networks Eq.

Original: (1) trained either on real data sets or on unstructured tasks.
Revised: (1) trained either on real data sets or on unstructured tasks.

Original: As an example of a real data set, we will use the MNIST image database
Revised: As an example of a real data set, we will use the MNIST image database

Original: [Figure]
Revised: [Figure]

Original: 8 K 0.

Original: [Figure]

Original: [Figure]

Original: f r a c g       8 K 0.

Original: [-f r a c g frac g frac1,
-]
Revised: [-f r a c g frac g frac1,
-]

Original: (i.i.d. Gaussian) of handwritten digits (LeCun & Cortes, 1998) and focus on the task of discriminating odd from even digits.
Revised: of handwritten digits (LeCun & Cortes, 1998) and focus on the task of discriminating odd from even digits.

Original: Hence the inputs x i will be the MNIST images with labels y ∗
Revised: Hence the inputs x i will be the MNIST images with labels y ∗

Original: i
Revised: i

Original: = 1 , − 1 for odd and even digits, resp.
Revised: = 1 , − 1 for odd and even digits, resp.

Original: The joint probability distribution of input-output pairs ( x i , y ∗ i ) for this task is inaccessible, which prevents analytical control over the test error and other quantities of interest.
Revised: The joint probability distribution of input-output pairs ( x i , y ∗ i ) for this task is inaccessible, which prevents analytical control over the test error and other quantities of interest.

Original: To make theoretical progress, it is therefore promising to study the generalisation ability of neural networks for data arising from a probabilistic generative model.
Revised: To make theoretical progress, it is therefore promising to study the generalisation ability of neural networks for data arising from a probabilistic generative model.

Original: A classic model for data sets is the vanilla teacher-student setup (Gardner & Derrida, 1989), where unstructured i.i.d.
Revised: A classic model for data sets is the vanilla teacher-student setup (Gardner & Derrida, 1989), where unstructured i.i.d.

Original: inputs are fed through a random neural network called the teacher .
Revised: inputs are fed through a random neural network called the teacher .

Original: We will take the teacher to have two layers and M hidden nodes.
Revised: We will take the teacher to have two layers and M hidden nodes.

Original: We allow that M (cid:54) =
Revised: We allow that M (cid:54) =

Original: K and we will draw the components of the teacher’s weights θ ∗ =
Revised: K and we will draw the components of the teacher’s weights θ ∗ =

Original: ( v ∗ ∈ R M , W ∗ ∈ R M × N ) i.i.d.
Revised: ( v ∗ ∈ R M , W ∗ ∈ R M × N ) i.i.d.

Original: from the normal distribution with mean zero and unit variance.
Revised: from the normal distribution with mean zero and unit variance.

Original: Drawing the inputs i.i.d.
Revised: Drawing the inputs i.i.d.

Original: from the standard normal distribution N ( x ; 0 , I N ) , we will takey ∗ i = φ (
Revised: from the standard normal distribution N ( x ; 0 , I N ) , we will take

Original: [Equation]
Revised: [Equation]

Original: (4) for regression tasks, or y ∗ i = sgn
Revised: for regression tasks, or y ∗ i = sgn

Original: ( φ ( x i , θ ∗ )) for binary classiﬁcation tasks.
Revised: ( φ ( x i , θ ∗ )) for binary classiﬁcation tasks.

Original: This is hence an example of a teacher task.
Revised: This is hence an example of a teacher task.

Original: In this setting, the network with K hidden units that is trained using SGD Eq.
Revised: In this setting, the network with K hidden units that is trained using SGD Eq.

Original: (2) is traditionally called the student .
Revised: (2) is traditionally called the student .

Original: Notice that, if K ≥ M , there exist a student network that has zero generalisation error, the one with the same architecture and parameters as the teacher.
Revised: Notice that, if K ≥ M , there exists a student network that has zero generalisation error, the one with the same architecture and parameters as the teacher.

Original:  T WO CHARACTERISTIC BEHAVIOURS OF NEURAL NETWORKS TRAINED ON STRUCTURED DATA SETS
Revised:  T WO CHARACTERISTIC BEHAVIOURS OF NEURAL NETWORKS TRAINED ON STRUCTURED DATA SETS

Original: We now proceed to demonstrate experimentally two signiﬁcant differences in the dynamics and the performance of neural networks trained on realistic data sets and networks trained within the vanilla teacher-student setup.
Revised: We now proceed to demonstrate experimentally two signiﬁcant differences in the dynamics and the performance of neural networks trained on realistic data sets and networks trained within the vanilla teacher-student setup.

Original: 3. I NDEPENDENT NETWORKS ACHIEVE SIMILAR PERFORMANCE , BUT LEARN DIFFERENT FUNCTIONS WHEN TRAINED ON STRUCTURED TASKS
Revised: 3. I NDEPENDENT NETWORKS ACHIEVE SIMILAR PERFORMANCE , BUT LEARN DIFFERENT FUNCTIONS WHEN TRAINED ON STRUCTURED TASKS

Original: We trained two sigmoidal networks with K hidden units, starting from two independent draws of initial conditions to discriminate odd from even digits in the MNIST database.
Revised: We trained two sigmoidal networks with K hidden units, starting from two independent draws of initial conditions to discriminate odd from even digits in the MNIST database.

Original: We trained both networks using SGD with constant learning rate η , eq.
Revised: We trained both networks using SGD with constant learning rate η , eq.

Original: (2), until the generalisation error had converged to a stationary value.
Revised: (2), until the generalisation error had converged to a stationary value.

Original: We plot this asymptotic fractional test error (cid:15) frac g as blue circles on the left
Revised: We plot this asymptotic fractional test error (cid:15) frac g as blue circles on the left

Original: [Figure]
Revised: [Figure]

Original: experiment, but trained in Fig.
Revised: experiment, but trained in Fig.

Original: 1 (the averages are taken over both networks and over several realisations of the initial conditions).
Revised: 1 (the averages are taken over both networks and over several realisations of the initial conditions).

Original: We observed the same qualitative behaviour when we employed the early-stopping error to evaluate the networks, where we take the minimum of the generalisation error during training (see
Revised: We observed the same qualitative behaviour when we employed the early-stopping error to evaluate the networks, where we take the minimum of the generalisation error during training (see

Original: Appendix C).
Revised: Appendix C).

Original: First, we note that increasing the number of hidden units in the network decreases the test error on this task.
Revised: First, we note that increasing the number of hidden units in the network decreases the test error on this task.

Original: We also compared the networks to one another by counting the fraction of inputs which the two networks classify differently,
Revised: We also compared the networks to one another by counting the fraction of inputs which the two networks classify differently,

Original: [Equation]
Revised: [Equation]

Original: This is a measure of the degree to which both networks have learned the same function φ ( x , θ ) .
Revised: This is a measure of the degree to which both networks have learned the same function φ ( x , θ ) .

Original: Independent networks disagree on the classiﬁcation of MNIST test images at a rate that roughly corresponds to their test error for K ≥ 3 (orange crosses).
Revised: Independent networks disagree on the classiﬁcation of MNIST test images at a rate that roughly corresponds to their test error for K ≥ 3 (orange crosses).

Original: However, even though the additional parameters of bigger networks are helpful in the discrimination task (decreasing (cid:15) g ), both networks learn increasingly different functions when evaluated over the whole of R N using Gaussian inputs as the network size K increases (green diamonds).
Revised: However, even though the additional parameters of bigger networks are helpful in the discrimination task (decreasing (cid:15) g ), both networks learn increasingly different functions when evaluated over the whole of R N using Gaussian inputs as the network size K increases (green diamonds).

Original: The network learned the right function on the lower-dimensional manifold on which MNIST inputs concentrate, but not outside of it.
Revised: The network learned the right function on the lower-dimensional manifold on which MNIST inputs concentrate, but not outside of it.

Original: This behaviour is not reproduced if we substitute the MNIST data set with a data set of the same size drawn from the vanilla teacher-student setup from Sec.
Revised: This behaviour is not reproduced if we substitute the MNIST data set with a data set of the same size drawn from the vanilla teacher-student setup from Sec.

Original: 2.1 with M = 4 , leaving everything else the same (right of Fig. 1).
Revised: 2.1 with M = 4 , leaving everything else the same (right of Fig. 1).

Original: The ﬁnal test error decreases with K , and as soon as the expressive power of the network is at least equal to that of the teacher, i.e. K ≥ M , the asymptotic test error goes to zero, since the data set is large enough for the network to recover the teacher
Revised: The ﬁnal test error decreases with K , and as soon as the expressive power of the network is at least equal to that of the teacher, i.e. K ≥ M , the asymptotic test error goes to zero, since the data set is large enough for the network to recover the teacher

Original: ’s weights to within a very small error, leading to a small generalisation error.
Revised: ’s weights to within a very small error, leading to a small generalisation error.

Original: We also computed the (cid:15) frac1 , 2 evaluated using Gaussian i.i.d.
Revised: We also computed the (cid:15) frac1 , 2 evaluated using Gaussian i.i.d.

Original: inputs (green diamonds).
Revised: inputs (green diamonds).

Original: Networks with fewer parameters than the teacher ﬁnd different approximations to that function, yielding ﬁnite values of (cid:15) 1 , 2 .
Revised: Networks with fewer parameters than the teacher ﬁnd different approximations to that function, yielding ﬁnite values of (cid:15) 1 , 2 .

Original: If they have just enough parameters ( K = M ), they learn the same function.
Revised: If they have just enough parameters ( K = M ), they learn the same function.

Original: Remarkably, they also learn the same function when they have signiﬁcantly more parameters than the teacher.
Revised: Remarkably, they also learn the same function when they have signiﬁcantly more parameters than the teacher.

Original: The vanilla teacher-student setup is thus unable to reproduce the behaviour observed when training on MNIST.
Revised: The vanilla teacher-student setup is thus unable to reproduce the behaviour observed when training on MNIST.

Original: 3. T HE GENERALISATION ERROR EXHIBITS PLATEAUS DURING TRAINING ON I . I . D . INPUTS
Revised: 3. T HE GENERALISATION ERROR EXHIBITS PLATEAUS DURING TRAINING ON I . I . D . INPUTS

Original: We plot the generalisation dynamics, i.e. the test error as a function of training time, for neural networks of the form (1) in Fig.
Revised: We plot the generalisation dynamics, i.e. the test error as a function of training time, for neural networks of the form (1) in Fig.

Original: For a data set drawn from the vanilla teacher-student setup with
Revised: For a data set drawn from the vanilla teacher-student setup with

Original: M = 4 , (blue lines in the left-hand plot of Fig. 2), we observe that there is an extended period of training during which the test error (cid:15) g remains constant before a sudden drop.
Revised: M = 4 , (blue lines in the left-hand plot of Fig. 2), we observe that there is an extended period of training during which the test error (cid:15) g remains constant before a sudden drop.

Original: These “plateaus” are well-known in the literature for both SGD, where they appear as a function of time (Biehl &
Revised: These “plateaus” are well-known in the literature for both SGD, where they appear as a function of time (Biehl &

Original: Schwarze, 1995; Saad & Solla, 1995; Biehl et al., 1996), and in batch learning, where they appear as a function of the training set size (Schwarze, 1993; Engel & Van den Broeck, 2001).
Revised: Schwarze, 1995; Saad & Solla, 1995; Biehl et al., 1996), and in batch learning, where they appear as a function of the training set size (Schwarze, 1993; Engel & Van den Broeck, 2001).

Original: Their appearance is related to different stages of learning: After a brief exponential decay of the test error at the start of training, the network “believes” that data are linearly separable and all her hidden units have roughly the same overlap with all the teacher nodes.
Revised: Their appearance is related to different stages of learning: After a brief exponential decay of the test error at the start of training, the network “believes” that data are linearly separable and all her hidden units have roughly the same overlap with all the teacher nodes.

Original: Only after a longer time, the network picks up the additional structure of the teacher and “specialises”: each of its hidden units ideally becomes strongly correlated with one and only one hidden unit of the teacher before the generalisation error decreases exponentially to its ﬁnal value.
Revised: Only after a longer time, the network picks up the additional structure of the teacher and “specialises”: each of its hidden units ideally becomes strongly correlated with one and only one hidden unit of the teacher before the generalisation error decreases exponentially to its ﬁnal value.

Original: In contrast, the generalisation dynamics of the same network trained on the MNIST task (orange trajectories on the left of Fig. 2) shows no plateau.
Revised: In contrast, the generalisation dynamics of the same network trained on the MNIST task (orange trajectories on the left of Fig. 2) shows no plateau.

Original: In fact, plateaus are rarely seen during the training of neural networks (note that during training, we do not change any of the hyper-parameters, e.g. the learning rate η .)
Revised: In fact, plateaus are rarely seen during the training of neural networks (note that during training, we do not change any of the hyper-parameters, e.g. the learning rate η .)

Original: It has been an open question how to eliminate the plateaus from the dynamics of neural networks trained in the teacher-student setup.
Revised: It has been an open question how to eliminate the plateaus from the dynamics of neural networks trained in the teacher-student setup.

Original: The use of second-order gradient descent methods such  natural gradient descent (Yang & Amari, 1998) can shorten the plateau (Rattray et al., 1998), but we would like to focus on the more practically relevant case of ﬁrst-order SGD.
Revised: The use of second-order gradient descent methods such as natural gradient descent (Yang & Amari, 1998) can shorten the plateau (Rattray et al., 1998), but we would like to focus on the more practically relevant case of ﬁrst-order SGD.

Original: Yoshida et al.
Revised: Yoshida et al.

Original: (2019) recently showed that length and existence of the plateau depend on the dimensionality of the output of the network, but we would like a model where the plateau disappears independently of the output dimension.
Revised: (2019) recently showed that length and existence of the plateau depend on the dimensionality of the output of the network, but we would like a model where the plateau disappears independently of the output dimension.

Original:  T HE HIDDEN MANIFOLD MODEL
Revised:  T HE HIDDEN MANIFOLD MODEL

Original: We now introduce a new generative probabilistic model for structured data sets with the aim of reproducing the behaviour observed during training on MNIST, but with a synthetic data set.
Revised: We now introduce a new generative probabilistic model for structured data sets with the aim of reproducing the behaviour observed during training on MNIST, but with a synthetic data set.

Original: The main motivation for such a model is that a closed-form solution of the learning dynamics is expected to be accessible.
Revised: The main motivation for such a model is that a closed-form solution of the learning dynamics is expected to be accessible.

Original: To generate a data set containing P inputs in N dimensions, we ﬁrst choose D feature vectors in N dimensions and collect them in a feature matrix F ∈ R D × N .
Revised: To generate a data set containing P inputs in N dimensions, we ﬁrst choose D feature vectors in N dimensions and collect them in a feature matrix F ∈ R D × N .

Original: Next we draw P vectors
Revised: Next we draw P vectors

Original: c i with random i.i.d.
Revised: c i with random i.i.d.

Original: components and collect them in the matrix C ∈ R P × D .
Revised: components and collect them in the matrix C ∈ R P × D .

Original: The vector c i gives the coordinates of the i th input on the lower-dimensional manifold spanned by the feature vectors in F .
Revised: The vector c i gives the coordinates of the i th input on the lower-dimensional manifold spanned by the feature vectors in F .

Original: We will call c
Revised: We will call c

Original: i the latent representation of the input x i , which is given by the i th row of
Revised: i the latent representation of the input x i , which is given by the i th row of

Original: [Equation]
Revised: [Equation]

Original: where f is a non-linear function acting component-wise.
Revised: where f is a non-linear function acting component-wise.

Original: In this model, the “world” of the data on which the true label can depend is a D -dimensional manifold, which is obtained from the linear subspace of R N generated by the D lines of matrix F , through a folding process induced by the nonlinear function f .
Revised: In this model, the “world” of the data on which the true label can depend is a D -dimensional manifold, which is obtained from the linear subspace of R N generated by the D lines of matrix F , through a folding process induced by the nonlinear function f .

Original: As we discuss in Appendix A, the exact form of f does not seem to be important, as long as it is a nonlinear function.
Revised: As we discuss in Appendix A, the exact form of f does not seem to be important, as long as it is a nonlinear function.

Original: The latent labels are obtained by applying a two-layer neural network with weights ˜ θ  =
Revised: The latent labels are obtained by applying a two-layer neural network with weights ˜ θ ∗ =

Original: ( ˜ W  ∈ R M × D , ˜ v  ∈ R M ) within the unfolded hidden manifold according to
Revised: ( ˜ W ∗ ∈ R M × D , ˜ v ∗ ∈ R M ) within the unfolded hidden manifold according to

Original: [Equation]
Revised: [Equation]

Original: We draw the weights in both layers component-wise i.i.d.
Revised: We draw the weights in both layers component-wise i.i.d.

Original: from the normal distribution with unity variance, unless we note it otherwise.
Revised: from the normal distribution with unity variance, unless we note it otherwise.

Original: The key point here is the dependency of labels (cid:101)
Revised: The key point here is the dependency of labels (cid:101)

Original: y i on the coordinates of the lower-dimensional manifold C rather than on the high-dimensional data X .
Revised: y i on the coordinates of the lower-dimensional manifold C rather than on the high-dimensional data X .

Original: We believe that the exact form of this dependence is not crucial and we expect several other choices to yield similar results to the ones we will present in the next section.
Revised: We believe that the exact form of this dependence is not crucial and we expect several other choices to yield similar results to the ones we will present in the next section.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: f r a

Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original:  frac g frac1,2 (structured) frac1,2 (i.i.d. Gaussian)
Revised: f r a c g frac g frac1,2 (structured) frac1,2 (i.i.d. Gaussian)

Original: In the same experiment, structured inputs with teacher labels y ∗ i (4) ( M = 4 ) fail to reproduce the behaviour observed on MNIST (cf.
Revised: In the same experiment, structured inputs with teacher labels y ∗ i (4) ( M = 4 ) fail to reproduce the behaviour observed on MNIST (cf.

Original: Fig. 1).
Revised: Fig. 1).

Original: In both plots erf (cid:0) 
Revised: In both plots, f ( x ) = sgn( x ) g ( x ) = erf (cid:0) x/ √ 2 (cid:1) , D

Original: In the following, we choose the entries of both C and F to be i.i.d.
Revised: In the following, we choose the entries of both C and F to be i.i.d.

Original: draws from the normal distribution with mean zero and unit variance.
Revised: draws from the normal distribution with mean zero and unit variance.

Original: To ensure comparability of the data sets for different data-generating function f ( x ) , we always center the input matrix X by subtracting the mean value of the entire matrix from all components and we rescale inputs by dividing all entries by the covariance of  theentire matrix before training.
Revised: To ensure comparability of the data sets for different data-generating function f ( x ) , we always center the input matrix X by subtracting the mean value of the entire matrix from all components and we rescale inputs by dividing all entries by the covariance of all the entries in the matrix before training.

Original: 4. T HE IMPACT OF THE HIDDEN MANIFOLD MODEL ON NEURAL NETWORKS
Revised: 4. T HE IMPACT OF THE HIDDEN MANIFOLD MODEL ON NEURAL NETWORKS

Original: We repeated the experiments with two independent networks reported in Sec.
Revised: We repeated the experiments with two independent networks reported in Sec.

Original: 3.1 using data sets generated from the hidden manifold model with D = 10 latent dimensions (see Appendix D).
Revised: 3.1 using data sets generated from the hidden manifold model with D = 10 latent dimensions (see Appendix D).

Original: On the right of Fig.
Revised: On the right of Fig.

Original: 3, we plot the asymptotic performance of a network trained on structured inputs which lie on a manifold (6) with a teacher task: the labels are a function of the high-dimensional inputs and do not explicitly take the latent representation c i of an input into account, y ∗ i =
Revised: 3, we plot the asymptotic performance of a network trained on structured inputs which lie on a manifold (6) with a teacher task: the labels are a function of the high-dimensional inputs and do not explicitly take the latent representation c i of an input into account, y ∗ i =

Original: φ ( x i , θ ∗ ) .
Revised: φ ( x i , θ ∗ ) .

Original: The ﬁnal results are similar to those of networks trained on data from the vanilla teacher-student setup ( cf .
Revised: The ﬁnal results are similar to those of networks trained on data from the vanilla teacher-student setup ( cf .

Original: left of Fig. 1): given enough data, the network recovers the teacher function if the network has at least as many parameters as the teacher.
Revised: right of Fig. 1): given enough data, the network recovers the teacher function if the network has at least as many parameters as the teacher.

Original: Once the teacher weights are recovered by both networks, they achieve zero test error (blue circles) and they agree on the classiﬁcation of random Gaussian inputs because they do implement the same function.
Revised: Once the teacher weights are recovered by both networks, they achieve zero test error (blue circles) and they agree on the classiﬁcation of random Gaussian inputs because they do implement the same function.

Original: The left plot of Fig.
Revised: The left plot of Fig.

Original: 3 shows network performance when trained on the same inputs, but this time with a latent task where the labels are a function of the latent representation of the inputs: (cid:101)
Revised: 3 shows network performance when trained on the same inputs, but this time with a latent task where the labels are a function of the latent representation of the inputs: (cid:101)

Original: y
Revised: y

Original: i =
Revised: i =

Original: φ ( c i , (cid:101) θ ∗ ) .
Revised: φ ( c i , (cid:101) θ ∗ ) .

Original: The asymptotic performance of the networks then resembles that of networks trained on MNIST: after convergence, the two networks will disagree on structured inputs at a rate that is roughly their generalisation error, but as K increases, they also learn increasingly different functions, up to the point where they will agree on their classiﬁcation of a random Gaussian input in just half the cases.
Revised: The asymptotic performance of the networks then resembles that of networks trained on MNIST: after convergence, the two networks will disagree on structured inputs at a rate that is roughly their generalisation error, but as K increases, they also learn increasingly different functions, up to the point where they will agree on their classiﬁcation of a random Gaussian input in just half the cases.

Original: The hidden manifold model thus reproduces the behaviour of independent networks trained on MNIST.
Revised: The hidden manifold model thus reproduces the behaviour of independent networks trained on MNIST.

Original: A look at the right-hand plot Fig.
Revised: A look at the right-hand plot Fig.

Original: 2 reveals that in this model the plateaus are absent.
Revised: 2 reveals that in this model the plateaus are absent.

Original: Again, we repeat the experiment of Sec. 3.2, but we train networks on structured inputs X = sgn( CF ) with teacher ( y ∗ i ) and latent labels ( (cid:101)
Revised: Again, we repeat the experiment of Sec. 3.2, but we train networks on structured inputs X = sgn( CF ) with teacher ( y ∗ i ) and latent labels ( (cid:101)

Original: y ∗ i ), respectively.
Revised: y ∗ i ), respectively.

Original: It is clear from these plots that the plateaus only appear for the teacher task.
Revised: It is clear from these plots that the plateaus only appear for the teacher task.

Original: In Appendix B, we demonstrate that the lack of plateaus for latent tasks in Fig.
Revised: In Appendix B, we demonstrate that the lack of plateaus for latent tasks in Fig.

Original: 2 is not due to the fact that the network in the latent task asymptotes at a higher generalisation error than the teacher task.
Revised: 2 is not due to the fact that the network in the latent task asymptotes at a higher generalisation error than the teacher task.

Original: 4. L ATENT TASKS , STRUCTURED INPUTS ARE BOTH NECESSARY TO MODEL REAL DATA SETS
Revised: 4. L ATENT TASKS , STRUCTURED INPUTS ARE BOTH NECESSARY TO MODEL REAL DATA SETS

Original: Our quest to reproduce the behaviour of networks trained on MNIST has led us to consider three different setups so far: the vanilla teacher-student setup, i e. a teacher task on unstructured inputs; and teacher and latent tasks on structured inputs.
Revised: Our quest to reproduce the behaviour of networks trained on MNIST has led us to consider three different setups so far: the vanilla teacher-student setup, i.e. a teacher task on unstructured inputs; and teacher and latent tasks on structured inputs.

Original: While it is not strictly possible to test the case of a latent task with unstructured inputs, we can approximate this setup by training a network on the
Revised: While it is not strictly possible to test the case of a latent task with unstructured inputs, we can approximate this setup by training a network on the

Original: MNIST task and then using the resulting network as a teacher to generate labels y ∗
Revised: MNIST task and then using the resulting network as a teacher to generate labels y ∗

Original: i (4) for inputs drawn i.i.d.
Revised: i (4) for inputs drawn i.i.d.

Original: component-wise from the standard normal distribution.
Revised: component-wise from the standard normal distribution.

Original: To test this idea, we trained both layers sigmoidal networks with M = 4 hidden units using vanilla SGD on the MNIST task, where they reach a generalisation error of about 5% .
Revised: To test this idea, we trained both layers sigmoidal networks with M = 4 hidden units using vanilla SGD on the MNIST task, where they reach a generalisation error of about 5% .

Original: They have thus clearly learnt some of the structure of the MNIST task.
Revised: They have thus clearly learnt some of the structure of the MNIST task.

Original: However, as we show on the left of Fig. 4, independent students trained on a data set with i.i.d.
Revised: However, as we show on the left of Fig. 4, independent students trained on a data set with i.i.d.

Original: Gaussian inputs x i and true labels y ∗ i given by the pre-trained teacher network behave similarly to students trained in the vanilla teacher-student setup of Sec. 3.1.
Revised: Gaussian inputs x i and true labels y ∗ i given by the pre-trained teacher network behave similarly to students trained in the vanilla teacher-student setup of Sec. 3.1.

Original: Furthermore, the learning dynamics of a network trained in this setup display the plateaus that we observed in the vanilla teacher-student setup (inset of Fig. 4).
Revised: Furthermore, the learning dynamics of a network trained in this setup display the plateaus that we observed in the vanilla teacher-student setup (inset of Fig. 4).

Original: On the right of Fig. 4, we summarise the four different setups for synthetic data sets in supervised learning problems that we have analysed in this paper.
Revised: On the right of Fig. 4, we summarise the four different setups for synthetic data sets in supervised learning problems that we have analysed in this paper.

Original: Only the hidden manifold model, consisting of a latent task on structured inputs, reproduced the behaviour of neural networks trained on the MNIST task, leading us to conclude that a model for realistic data sets has to feature both, structured inputs and a latent task.
Revised: Only the hidden manifold model, consisting of a latent task on structured inputs, reproduced the behaviour of neural networks trained on the MNIST task, leading us to conclude that a model for realistic data sets has to feature both, structured inputs and a latent task.

Original:  C ONCLUDING PERSPECTIVES
Revised:  C ONCLUDING PERSPECTIVES

Original: We have introduced the hidden manifold model for structured data sets that is simple to write down, yet displays some of the phenomena that we observe when training neural networks on real-world inputs.
Revised: We have introduced the hidden manifold model for structured data sets that is simple to write down, yet displays some of the phenomena that we observe when training neural networks on real-world inputs.

Original: We saw that the model has two key ingredients, both of which are necessary: (1) highdimensional inputs which lie on a lower-dimensional manifold and (2) latent labels for these inputs that depend on the inputs’ position within the low dimensional manifold.
Revised: We saw that the model has two key ingredients, both of which are necessary: (1) highdimensional inputs which lie on a lower-dimensional manifold and (2) latent labels for these inputs that depend on the inputs’ position within the low dimensional manifold.

Original: We hope that this model is a step towards a more thorough understanding of how the structure we ﬁnd in real-world data sets impacts the training dynamics of neural networks and their ability to generalise.
Revised: We hope that this model is a step towards a more thorough understanding of how the structure we ﬁnd in real-world data sets impacts the training dynamics of neural networks and their ability to generalise.

Original: We see two main lines for future work.
Revised: We see two main lines for future work.

Original: On the one hand, the present work needs to be generalised to multi-layer networks to identify how depth helps to deal with structured data sets and to build a model capturing the key properties.
Revised: On the one hand, the present work needs to be generalised to multi-layer networks to identify how depth helps to deal with structured data sets and to build a model capturing the key properties.

Original: On the other hand, the key promise of the synthetic hidden manifold model is that the learning dynamics should be amenable to closed-form analysis in some limit.
Revised: On the other hand, the key promise of the synthetic hidden manifold model is that the learning dynamics should be amenable to closed-form analysis in some limit.

Original: Such analysis and its results would then provide further insights about the properties of learning beyond what is possible with numerical experiments.
Revised: Such analysis and its results would then provide further insights about the properties of learning beyond what is possible with numerical experiments.

Original: A T HE EXACT FORM OF THE DATA - GENERATING FUNCTION f ( · ) IS NOT IMPORTANT , AS LONG AS IT IS NON - LINEAR
Revised: A T HE EXACT FORM OF THE DATA - GENERATING FUNCTION f ( · ) IS NOT IMPORTANT , AS LONG AS IT IS NON - LINEAR

Original: Two questions arise when looking at the way we generate inputs in our data sets, X = f (cid:16) CF / √ D (cid:17) : is the non-linearity necessary f ( · ) necessary, and is the choice of non-linearity important?
Revised: Two questions arise when looking at the way we generate inputs in our data sets, X = f (cid:16) CF / √ D (cid:17) : is the non-linearity necessary f ( · ) necessary, and is the choice of non-linearity important?

Original: To answer the ﬁrst question, we plot the results of the experiment with independent networks described in Sec. 4.1.
Revised: To answer the ﬁrst question, we plot the results of the experiment with independent networks described in Sec. 4.1.

Original: The setup is exactly the same, except that we now take inputs to be
Revised: The setup is exactly the same, except that we now take inputs to be

Original: X = CF , (8) i.e. inputs are just a linear combination of the feature vectors, without applying a non-linearity.
Revised: X = CF , (8) i.e. inputs are just a linear combination of the feature vectors, without applying a non-linearity.

Original: In this case, two networks trained in the vanilla teacher-student setup will learn globally different functions, as can be seen from the fractional generalisation error between the networks (5) (green diamonds), which is 1 / 2 , i.e. not better than chance.
Revised: In this case, two networks trained in the vanilla teacher-student setup will learn globally different functions, as can be seen from the fractional generalisation error between the networks (5) (green diamonds), which is 1 / 2 , i.e. not better than chance.

Original: This is a direct consequence of using f ( x ) =
Revised: This is a direct consequence of using f ( x ) =

Original: x : to perfectly generalise with respect to the teacher, it is thus sufﬁcient to learn only the D components of the teacher weights w ∗ m in the direction F .
Revised: x : to perfectly generalise with respect to the teacher, it is thus sufﬁcient to learn only the D components of the teacher weights w ∗ m in the direction F .

Original: Thus the weights of the network in the weight space orthogonal to the directions F are unconstrained, and by starting from random initial conditions, will converge to different values for each network.
Revised: Thus the weights of the network in the weight space orthogonal to the directions F are unconstrained, and by starting from random initial conditions, will converge to different values for each network.

Original: We also checked that the qualitative behaviour of a neural networks trained on the hidden manifold model does not depend on the data-generating non-linearity f ( x ) .
Revised: We also checked that the qualitative behaviour of a neural networks trained on the hidden manifold model does not depend on the data-generating non-linearity f ( x ) .

Original: In Fig.
Revised: In Fig.

Original: 6, we therefore show the results of the same experiment described in Sec. 4.1, but this time using
Revised: 6, we therefore show the results of the same experiment described in Sec. 4.1, but this time using

Original: [Equation]
Revised: [Equation]

Original: (9) where the application of the non-linearity is again component-wise.
Revised: (9) where the application of the non-linearity is again component-wise.

Original: Indeed, the results mirror those when we used the sign function f ( x ) = sgn( x ) .
Revised: Indeed, the results mirror those when we used the sign function f ( x ) = sgn( x ) .

Original: [Figure]
Revised: [Figure]

Original: (i.i.d.
Revised: (i.i.d.

Original: Gaussian)       8 K 0.
Revised: Gaussian)       8 K 0.

Original: f r a c g frac g frac1,2 (structured) frac1,2 (i.i.d.
Revised: f r a c g frac g frac1,2 (structured) frac1,2 (i.i.d.

Original: Gaussian)
Revised: Gaussian)

Original: B T HE EXISTENCE OF PLATEAUS DOES NOT DEPEND ON THE ASYMPTOTIC GENERALISATION ERROR
Revised: B T HE EXISTENCE OF PLATEAUS DOES NOT DEPEND ON THE ASYMPTOTIC GENERALISATION ERROR

Original: We have demonstrated on the right of Fig.
Revised: We have demonstrated on the right of Fig.

Original: 2 that neural networks trained on data drawn from the hidden manifold model (HMF) introduced here do not show the plateau phenomenon, where the generalisation error stays stationary after an initial exponential decay, before dropping again.
Revised: 2 that neural networks trained on data drawn from the hidden manifold model (HMF) introduced here do not show the plateau phenomenon, where the generalisation error stays stationary after an initial exponential decay, before dropping again.

Original: Upon closer inspection, one might think that this is due to the fact that the student trained on data from the
Revised: Upon closer inspection, one might think that this is due to the fact that the student trained on data from the

Original: HMF asymptotes at a higher generalisation error than the student trained in the vanilla teacher-student setup.
Revised: HMF asymptotes at a higher generalisation error than the student trained in the vanilla teacher-student setup.

Original: This is not the case, as we demonstrate in Fig.
Revised: This is not the case, as we demonstrate in Fig.

Original: 7: we observe no plateau in a sigmoidal network
Revised: 7: we observe no plateau in a sigmoidal network

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: trained on data from the HMF even that network asymptotes at a generalisation error that is, within ﬂuctuations, the same as the generalisation error of a network of the same sized trained in the vanilla teacher-student setup and which shows a plateau.
Revised: trained on data from the HMF even that network asymptotes at a generalisation error that is, within ﬂuctuations, the same as the generalisation error of a network of the same sized trained in the vanilla teacher-student setup and which shows a plateau.

Original: C E ARLY - STOPPING YIELDS QUALITATIVELY SIMILAR RESULTS
Revised: C E ARLY - STOPPING YIELDS QUALITATIVELY SIMILAR RESULTS

Original: In Fig.
Revised: In Fig.

Original: we reproduce Fig.
Revised: we reproduce Fig.

Original: 3, where we compare the performance of independent neural networks trained on the MNIST task (Left) or trained on structured inputs with a latent task (Center) and a teacher task (Right) , respectively.
Revised: 3, where we compare the performance of independent neural networks trained on the MNIST task (Left) or trained on structured inputs with a latent task (Center) and a teacher task (Right) , respectively.

Original: This time, we the early-stopping generalisation error ˆ (cid:15) frac g rather than the asymptotic value at the end of training.
Revised: This time, we the early-stopping generalisation error ˆ (cid:15) frac g rather than the asymptotic value at the end of training.

Original: We deﬁne ˆ (cid:15) frac g as the minimum of (cid:15) frac g during the whole of training.
Revised: We deﬁne ˆ (cid:15) frac g as the minimum of (cid:15) frac g during the whole of training.

Original: Clearly, the qualitative result of Sec.
Revised: Clearly, the qualitative result of Sec.

Original: 4.1 is unchanged: although we use structured inputs (6) in both cases, independent students will learn different functions which agree on those inputs only when they are trained on a latent task (7) (Center) , but not when trained on a vanilla teacher task (4) (Right) .
Revised: 4.1 is unchanged: although we use structured inputs (6) in both cases, independent students will learn different functions which agree on those inputs only when they are trained on a latent task (7) (Center) , but not when trained on a vanilla teacher task (4) (Right) .

Original: Thus structured inputs and latent tasks are sufﬁcient to reproduce the behaviour observed when training on the MNIST task.
Revised: Thus structured inputs and latent tasks are sufﬁcient to reproduce the behaviour observed when training on the MNIST task.

Original: D D YNAMICS WITH A LARGE NUMBER OF FEATURES D ∼ N
Revised: D D YNAMICS WITH A LARGE NUMBER OF FEATURES D ∼ N

Original: It is of independent interest to investigate the behaviour of networks trained on data from the hidden manifold model when the number of feature vectors D is on the same order as the input dimension N .
Revised: It is of independent interest to investigate the behaviour of networks trained on data from the hidden manifold model when the number of feature vectors D is on the same order as the input dimension N .

Original: We call this the regime of extensive D .
Revised: We call this the regime of extensive D .

Original: It is a different regime from MNIST, where experimental studies consistently ﬁnd that inputs lie on a low-dimensional manifold of dimension D ∼ 14 , which is much smaller than the input dimension N = 784 (Costa & Hero, 2004; Levina & Bickel, 2004;
Revised: It is a different regime from MNIST, where experimental studies consistently ﬁnd that inputs lie on a low-dimensional manifold of dimension D ∼ 14 , which is much smaller than the input dimension N = 784 (Costa & Hero, 2004; Levina & Bickel, 2004;

Original: Spigler et al., 2019).
Revised: Spigler et al., 2019).

Original: We show the results of our numerical experiments with N = 500 , D = 250 in Fig.
Revised: We show the results of our numerical experiments with N = 500 , D = 250 in Fig.

Original: 9, where we reproduce Fig.
Revised: 9, where we reproduce Fig.

Original: 3 for the asymptotic (top row) and the early-stopping (bottow row) generalisation error.
Revised: 3 for the asymptotic (top row) and the early-stopping (bottow row) generalisation error.

Original: The behaviour of networks trained on a teacher task with structured inputs (right column) is unchanged w.r.t.
Revised: The behaviour of networks trained on a teacher task with structured inputs (right column) is unchanged w.r.t.

Original: to the case with D = 10 .
Revised: to the case with D = 10 .

Original: For the latent task, increasing the number of hidden units however increases the generalisation error, indicating severe over-ﬁtting, which is only partly mitigated by early stopping.
Revised: For the latent task, increasing the number of hidden units however increases the generalisation error, indicating severe over-ﬁtting, which is only partly mitigated by early stopping.

Original: The generalisation error on this task is generally much higher than in the low- D regime and clearly, increasing the width of the network is not the right way to learn a latent
Revised: The generalisation error on this task is generally much higher than in the low- D regime and clearly, increasing the width of the network is not the right way to learn a latent

Original: [Figure]
Revised: 8 K 0.

Original: [Figure]
Revised: f r a c g frac g frac1,2 (structured) frac1,2 (i.i.d.

Original: f r a c g   8 K 0.
Revised: Gaussian)   8 K 0.

Original: f r a c g   8 K 0.
Revised: f r a c g   8 K 0.

Original: [Figure]

Original: f r a c g frac g frac1,2 (structured) frac ,2 (i.i.d.
Revised: f r a c g frac g frac1,2 (structured) frac ,2 (i.i.d.

Original: Gaussian) task; instead, it would be intriguing to analyse the performance of deeper networks on this task where ﬁnding a good intermediate representation for inputs is key.
Revised: Gaussian) task; instead, it would be intriguing to analyse the performance of deeper networks on this task where ﬁnding a good intermediate representation for inputs is key.

Original: This is an intriguing avenue for future research.
Revised: This is an intriguing avenue for future research.

Original: E I NDEPENDENT STUDENTS WITH R E LU ACTIVATION FUNCTION
Revised: E I NDEPENDENT STUDENTS WITH R E LU ACTIVATION FUNCTION

Original: We also veriﬁed that the behaviour of independent networks we observed on MNIST with sigmoidal students persists when training networks with ReLU activation function and that the hidden manifold model is able to reproduce it for these networks.
Revised: We also veriﬁed that the behaviour of independent networks we observed on MNIST with sigmoidal students persists when training networks with ReLU activation function and that the hidden manifold model is able to reproduce it for these networks.

Original: We show the results of our numerical experiments in Fig.
Revised: We show the results of our numerical experiments in Fig.

Original: To that end, we trained both layers of a network φ ( x , θ ) with g ( x ) = max( x, 0) starting from small initial conditions, where we draw the weights component-wise i.i.d.
Revised: To that end, we trained both layers of a network φ ( x , θ ) with g ( x ) = max( x, 0) starting from small initial conditions, where we draw the weights component-wise i.i.d.

Original: from a normal distribution with variance 10 − 6 .
Revised: from a normal distribution with variance 10 − 6 .

Original: We see that the generalisation error of ReLU networks on the MNIST task ( Left of Fig. 10) decreases with increasing number of hidden units, while the generalisation error on MNIST inputs of the two independent students with respect to each other is comparable or less than the generalisation error of each individual network on the MNIST task.
Revised: We see that the generalisation error of ReLU networks on the MNIST task ( Left of Fig. 10) decreases with increasing number of hidden units, while the generalisation error on MNIST inputs of the two independent students with respect to each other is comparable or less than the generalisation error of each individual network on the MNIST task.

Original: On structured inputs with a teacher task ( Right of Fig. 10), where labels were generated by a teacher with M = 4 hidden units, the student recovers the teacher such that its generalisation error is less than 10 − 3 for K > 4 , and both independent students learn the same function, as evidenced by their generalisation errors with respect to each other.
Revised: On structured inputs with a teacher task ( Right of Fig. 10), where labels were generated by a teacher with M = 4 hidden units, the student recovers the teacher such that its generalisation error is less than 10 − 3 for K > 4 , and both independent students learn the same function, as evidenced by their generalisation errors with respect to each other.

Original: This is the same behaviour that we see in Fig.
Revised: This is the same behaviour that we see in Fig.

Original: 3 for sigmoidal networks.
Revised: 3 for sigmoidal networks.

Original: The ﬁnite value of the generalisation error for K = M = 4 is due to two out of ten runs taking a very long time to converge, longer than our simulation lasted for.
Revised: The ﬁnite value of the generalisation error for K = M = 4 is due to two out of ten runs taking a very long time to converge, longer than our simulation lasted for.

Original: Finally, we see that for a latent task on structured inputs, the generalisation error of the two networks with respect to each other increases beyond the generalisation error on structured inputs of each of them, as we observed on MNIST.
Revised: Finally, we see that for a latent task on structured inputs, the generalisation error of the two networks with respect to each other increases beyond the generalisation error on structured inputs of each of them, as we observed on MNIST.

Original: Thus we have recovered the phenomenology that we described for sigmoidal networks in ReLU networks, too.
Revised: Thus we have recovered the phenomenology that we described for sigmoidal networks in ReLU networks, too.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]