Original: Learning dynamics of deep linear networks with multiple pathways
Revised: Learning dynamics of deep linear networks with multiple pathways

Original: Anonymous Author(s)

Original: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Not only have deep networks become standard in machine learning, they are increasingly of interest in neuroscience as models of cortical computation that capturerelationships between structural and functional properties.
Revised: Not only have deep networks become standard in machine learning, they are increasingly of interest in neuroscience as models of cortical computation that capture relationships between structural and functional properties.

Original: In addition they are auseful target of theoretical research into the properties of network computation.
Revised: In addition they are a useful target of theoretical research into the properties of network computation.

Original: Deep networks typically have a serial or approximately serial organization acrosslayers, and this is often mirrored in models that purport to represent computationin mammalian brains.
Revised: Deep networks typically have a serial or approximately serial organization across layers, and this is often mirrored in models that purport to represent computation in mammalian brains.

Original: There are, however, multiple examples of parallel pathwaysin mammalian brains.
Revised: There are, however, multiple examples of parallel pathways in mammalian brains.

Original: In some cases, such as the mouse, the entire visual systemappears arranged in a largely parallel, rather than serial fashion.
Revised: In some cases, such as the mouse, the entire visual system appears arranged in a largely parallel, rather than serial fashion.

Original: While these pathways may be formed by differing cost functions that drive different computations,here we present a new mathematical analysis of learning dynamics in networksthat have parallel computational pathways driven by the same cost function.
Revised: While these pathways may be formed by differing cost functions that drive different computations, here we present a new mathematical analysis of learning dynamics in networks that have parallel computational pathways driven by the same cost function.

Original: Weuse the approximation of deep linear networks with large hidden layer sizes toshow that, as the depth of the parallel pathways increases, different features of thetraining set (defined by the singular values of the input-output correlation) willtypically concentrate in one of the pathways.
Revised: We use the approximation of deep linear networks with large hidden layer sizes to show that, as the depth of the parallel pathways increases, different features of the training set (defined by the singular values of the input-output correlation) will typically concentrate in one of the pathways.

Original: This result is derived analyticallyand demonstrated with numerical simulation.
Revised: This result is derived analytically and demonstrated with numerical simulation with both linear and non-linear networks.

Original: Thus, rather than sharing stimulusand task features across multiple pathways, parallel network architectures learn toproduce sharply diversified representations with specialized and specific pathways,a mechanism which may hold important consequences for codes in both biologicaland artificial systems.
Revised: Thus, rather than sharing stimulus and task features across multiple pathways, parallel network architectures learn to produce sharply diversified representations with specialized and specific pathways, a mechanism which may hold important consequences for codes in both biological and artificial systems.

Original:  Introduction
Revised:  Introduction

Original: Deep networks are increasingly used as models of cortical computation, particularly of the visualpathway.
Revised: Deep networks are increasingly used as models of cortical computation, particularly of the visual pathway.

Original: Indeed, the earliest convolutional neural network was a model of the mammalian visualsystem [6] and the use of CNNs to model the visual pathway has continued [12, 19, 20, 3, 2].
Revised: Indeed, the earliest convolutional neural network was a model of the mammalian visual system [6] and the use of CNNs to model the visual pathway has continued [14, 21, 22, 3, 2].

Original: Thesemodels are usually serial models of the primate ventral stream that describe the computation of thatsystem as being closely related to object recognition.
Revised: These models are usually serial models of the primate ventral stream that describe the computation of that system as being closely related to object recognition.

Original: However, processing of visual informationin the primate is associated with (at least) two parallel pathways, the “ventral” (or “what”) and the“dorsal” (or “where") pathways [4, 7, 10].
Revised: However, processing of visual information in the primate is associated with (at least) two parallel pathways, the “ventral” (or “what”) and the “dorsal” (or “where") pathways [4, 7, 10].

Original: In the mouse visual system, anatomical analysis alsosuggests that there are parallel pathways [8, 17, 18], although the specific functional role of thesepathways is less clear.
Revised: In the mouse visual system, anatomical analysis also suggests that there are parallel pathways [8, 19, 20], although the specific functional role of these pathways is less clear.

Original: Interestingly, recent research [1] has shown that self-supervised training inneural network architectures with parallel pathways can lead to the emergence of ventral-like anddorsal-like pathways.
Revised: Interestingly, recent research [1] has shown that self-supervised training in neural network architectures with parallel pathways can lead to the emergence of ventral-like and dorsal-like pathways.

Original: Another study shows that, with a similar self-supervised training objective,shallower architectures with parallel pathways lead to closer matches with functional data in the
Revised: Another study shows that, with a similar self-supervised training objective, shallower architectures with parallel pathways lead to closer matches with functional data in the

Original: Allen Brain Observatory than deep single pathway architectures [11].
Revised: Allen Brain Observatory than deep single pathway architectures [11].

Original: Furthermore, an anatomicallyconstrained deep neural network model for the mouse visual cortex has demonstrated that such parallel architectures produce more diverse representations of visual images, compared to singlepathway architectures [16].
Revised: Furthermore, an anatomically constrained deep neural network model for the mouse visual cortex has demonstrated that such parallel architectures produce more diverse representations of visual images, compared to single pathway architectures [18].

Original: Additionally, a coincident observation has been made for the original AlexNet [9] (a deep neuralnetwork designed for image processing) that also shows the emergence of functional specializationacross parallel pathways, with different pathways (defined in the model because it was implementedon multiple GPUs) learning different features of the input data.
Revised: Additionally, a coincident observation has been made for the original AlexNet [9] (a deep neural network designed for image processing) that also shows the emergence of functional specialization across parallel pathways, with different pathways (defined in the model because it was implemented on multiple GPUs) learning different features of the input data.

Original: Overall, in the road map fordeveloping future artificial intelligence systems, architectures with parallel pathways promise potentialsolutions for multi-tasking, multi-sensory, energy-efficient computing [13].
Revised: Overall, in the road map for developing future artificial intelligence systems, architectures with parallel pathways promise potential solutions for multi-tasking, multi-sensory, energy-efficient computing [15].

Original: But despite the richnessof information representation and the potential capabilities of the architectures with parallel pathways,there is a lack of research in understanding the fundamental learning behavior that occurs in sucharchitectures.
Revised: But despite the richness of information representation and the potential capabilities of the architectures with parallel pathways, there is a lack of research in understanding the fundamental learning behavior that occurs in such architectures.

Original: Here, we advance this frontier through a mathematical analysis of the behaviour ofdeep neural networks with parallel pathways, in the tractable setting of deep linear networks [14].
Revised: Here, we advance this frontier through a mathematical analysis of the behaviour of deep neural networks with parallel pathways, in the tractable setting of deep linear networks [16].

Original: Our starting point follows Saxe et al.
Revised: Our starting point follows Saxe et al.

Original: [15], which elegantly shows how, in such a linear network, thetraining data and learning dynamics for a given task can be fully described via a set of independentmodes resulting from singular value decomposition (SVD) of the input-output correlation matrix.
Revised: [17], which elegantly shows how, in such a linear network, the training data and learning dynamics for a given task can be fully described via a set of independent modes resulting from singular value decomposition (SVD) of the input-output correlation matrix.

Original: During gradient decent, a deep linear network will pick up the task related information from the databy acquiring knowledge from each mode [15].
Revised: During gradient decent, a deep linear network will pick up the task related information from the data by acquiring knowledge from each mode [17].

Original: Here, we use the same overall framework to study thedynamics of learning in linear networks with parallel pathways.
Revised: Here, we use the same overall framework to study the dynamics of learning in linear networks with parallel pathways.

Original: Our analysis and simulations yield the following main results.
Revised: Our analysis and simulations yield the following main results.

Original: For linear networks with multipleparallel pathways of hidden neurons that feed into the same output representation and cost function:
Revised: For linear networks with multiple parallel pathways of hidden neurons that feed into the same output representation and cost function:

Original: For wide networks with random weight initialization, the learning dynamics reduces to a coupled set of ordinary differential equations of dimension equal to the number of pathwaysfor each non-zero singular vector in the input-output correlation matrix of the training set.
Revised: For wide networks with random weight initialization, the learning dynamics reduces to a coupled set of ordinary differential equations of dimension equal to the number of pathways for each non-zero singular vector in the input-output correlation matrix of the training set.

Original: These dynamics are competitive in the sense that each pathway will compete to “explain” the singular values of the training set.
Revised: These dynamics are competitive in the sense that each pathway will compete to “explain” the singular values of the training set.

Original: For deeper networks, this competition becomes more severe, resulting in more trajectories in phase space flowing to extreme solutions, wherein features (defined by the singular values of the training set) are confined largely to a single pathway.4.
Revised: For deeper networks, this competition becomes more severe, resulting in more trajectories in phase space flowing to extreme solutions, wherein features (defined by the singular values of the training set) are confined largely to a single pathway.

Original: For the strict infinite-width limit, the system flows to a state wherein features are sharedevenly across the pathways.
Revised: For the strict infinite-width limit, the system flows to a state wherein features are shared evenly across the pathways.

Original: At finite width, fluctuations in the initial state put the systemonto a trajectory leading to a biased solution, where the bias is more severe at largerdepths.
Revised: At finite width, fluctuations in the initial state put the system onto a trajectory leading to a biased solution, where the bias is more severe at larger depths.

Original: Thus in large-width, large-depth networks, the system evinces a finite-size inducedspontaneous symmetry breaking.
Revised: Thus in large-width, large-depth networks, the system evinces a finite-size induced spontaneous symmetry breaking.

Revised: We demonstrate this result in simulations and show empiricially that the main result holds in networks with non-linearities.

Original: Thus, this depth-dependent mechanism reveals a dynamical link between network architecture andrepresentation learning.
Revised: Thus, this depth-dependent mechanism reveals a dynamical link between network architecture and representation learning.

Original:  Mathematical framework
Revised:  Mathematical framework

Original: Network description: We study the learning dynamics under gradient descent of a system withparallel pathways with one or more hidden layers in each pathway.We consider “linear" networks [Figure] [Equation]
Revised: Network description: We study the learning dynamics under gradient descent of a system with parallel pathways with one or more hidden layers in each pathway.

Original: [Figure]

Original: [Equation]

Original: [Figure]

Original: [Equation]

Original: [Figure]

Original: [Equation]

Original: [Figure]

Original: [Equation]

Original: [Figure]

Original: [Equation]

Original: [Figure]

Original: [Equation]

Original: [Figure]

Original: [Equation]

Original: [Figure]

Original: [Equation]

Original: [Table]

Original: =">AAAB+HicbVDLSsNAFL2pr1ofjbp0M1gEN5ZEiros6sJlBfuANobJdNIOnUzCzESoIV/ixoUibv0Ud/6N08dCWw9cOJxzL/feEyScKe0431ZhZXVtfaO4Wdra3tkt23v7LRWnktAmiXksOwFWlDNBm5ppTjuJpDgKOG0Ho+uJ336kUrFY3OtxQr0IDwQLGcHaSL5dznpBiNr5Q3Zz6uY+9u2KU3WmQMvEnZMKzNHw7a9ePyZpRIUmHCvVdZ1EexmWmhFO81IvVTTBZIQHtGuowBFVXjY9PEfHRumjMJamhEZT9fdEhiOlxlFgOiOsh2rRm4j/ed1Uh5dexkSSairIbFGYcqRjNEkB9ZmkRPOxIZhIZm5FZIglJtpkVTIhuIsvL5PWWdU9r9buapX61TyOIhzCEZyACxdQh1toQBMIpPAMr/BmPVkv1rv1MWstWPOZA/gD6/MH3GuSlQ

Original: [Figure] </latexit [Figure] in which the transformation of each layer to the next is given by a matrix multiplication, with nonon-linearity (i.e., linear transfer functions at each “node”).
Revised: We consider “linear" networks in which the transformation of each layer to the next is given by a matrix multiplication, with no non-linearity (i.e., linear transfer functions at each “node”).

Original: Given an input vector x (of dimension l x )and output vector y (of dimension l y ), the output of the system with M parallel pathways is given by
Revised: Given an input vector x (of dimension l x ) and output vector y (of dimension l y ), the output of the system with M parallel pathways is given by

Original: [Equation]
Revised: [Equation]

Original: where we have defined
Revised: where we have defined

Original: [Equation]
Revised: [Equation]

Original:  and where pathway a has D a − 1 hidden layers and D a weight matrices.
Revised: da (2) and where pathway a has D a − 1 hidden layers and D a weight matrices.

Original: Each weight matrix W dahas shape N da × N d − 1 a (where N 

Revised: Each weight matrix W da has shape N da × N d − 1 a (where N

Original: a = l x ).
Revised: a = l x ).

Original: In the product formula, we are assuming that matrices witha higher value of d are to the left of matrices with smaller values of d .
Revised: In the product formula, we are assuming that matrices with a higher value of d are to the left of matrices with smaller values of d .

Original: A schematic of this system isgiven in Figure 1.
Revised: A schematic of this system is given in Figure 1.

Original: As described below, we consider training this network using input-output vector pairs x and y thathave correlation matrix Σ yx = USV T , where we have used the singular value decomposition where
Revised: As described below, we consider training this network using input-output vector pairs x and y that have correlation matrix Σ yx = USV T , where we have used the singular value decomposition where

Original: S is diagonal and U , V are orthogonal matrices.
Revised: S is diagonal and U , V are orthogonal matrices.

Original: The gradient descent dynamics will lead to solutions
Revised: The gradient descent dynamics will lead to solutions

Original: U T ΩV
Revised: U T ΩV

Original: = S , which implies that in the steady state the pathway matrices K a ≡ U T Ω a V must sumto the singular values.
Revised: = S , which implies that in the steady state the pathway matrices K a ≡ U T Ω a V must sum to the singular values.

Original: A schematic of this decomposition is shown in Figure 2(top row), adaptedfrom [15].
Revised: A schematic of this decomposition is shown in Figure 2(top row), adapted from [17].

Original: Main question and illustration of main result:
Revised: Main question and illustration of main result:

Original: Given this description of how the network asa whole carries information about each input-output mode, we ask how this mode information isdistributed across parallel pathways.
Revised: Given this description of how the network as a whole carries information about each input-output mode, we ask how this mode information is distributed across parallel pathways.

Original: We illustrate two possibilities in Figure 2: (1) each mode’sinformation could be shared across multiple pathways (middle row) or (2) each mode’s informationcould be exclusively carried in individual pathways (bottom row).
Revised: We illustrate two possibilities in Figure 2: (1) each mode’s information could be shared across multiple pathways (middle row) or (2) each mode’s information could be exclusively carried in individual pathways (bottom row).

Original: We show that case (2) occurs in thelimit of large network size and large depth due to finite-size induced spontaneous symmetry breaking.
Revised: We show that case (2) occurs in the limit of large network size and large depth due to finite-size induced spontaneous symmetry breaking.

Original: 2.1 Learning dynamics under gradient descent
Revised: 2.1 Learning dynamics under gradient descent

Original: Using the same setting as in Saxe et al.
Revised: Using the same setting as in Saxe et al.

Original: [14], we train the network with a set of P examples { x i , y i } , i = 1 , 2 , . . .
Revised: [16], we train the network with a set of P examples { x i , y i } , i = 1 , 2 , . . .

Original: , P with gradient descent on the squared loss
Revised: , P with gradient descent on the squared loss

Original: [Equation]
Revised: [Equation]

Original: In the limit of small learning rate, the dynamics of gradient descent are approximated by thecontinuous time ordinary differential equations given below (for each pathway a ):
Revised: In the limit of small learning rate, the dynamics of gradient descent are approximated by the continuous time ordinary differential equations given below (for each pathway a ):

Original: [Equation]
Revised: [Equation]

Original: where (cid:81)
Revised: where (cid:81)

Original: d 2 i =
Revised: d 2 i =

Original: d 1 W ia
Revised: d 1 W ia

Original: =
Revised: =

Original: W l 2 a W l 2 − 1 a · · · W l 1 a when l 1 ≤ l 2 and (cid:81) l 2 i =
Revised: W l 2 a W l 2 − 1 a · · · W l 1 a when l 1 ≤ l 2 and (cid:81) l 2 i =

Original: l 1 W ia
Revised: l 1 W ia

Original: = I
Revised: = I

Original: when l 1 > l 2 .
Revised: when l 1 > l 2 .

Original: Σ yx =
Revised: Σ yx =

Original: E [ yx T ] and Σ x = E
Revised: E [ yx T ] and Σ x = E

Original: [ xx T ] .
Revised: [ xx T ] .

Original: We assume without loss of generality that Σ x = I .
Revised: We assume without loss of generality that Σ x = I .

Original: Define the initial conditions for the weight matrices as R da ≡ W da (0) .
Revised: Define the initial conditions for the weight matrices as R da ≡ W da (0) .

Original: We take the components R dato be independently and identically distributed according to a normal distribution with zero mean andvariance σ 2 N da , where N da is the number of units in the output of that layer.
Revised: We take the components R da to be independently and identically distributed according to a normal distribution with zero mean and variance σ 2 N da , where N da is the number of units in the output of that layer.

Original: We also denote the singularvalue decomposition of Σ yx = USV T , where S is diagonal and U , V are orthogonal matrices. 
Revised: We also denote the singular value decomposition of Σ yx = USV T , where S is diagonal and U , V are orthogonal matrices. [Equation] [Table]

Revised: +JDEdKjaPAdEZYD9WiNxH/87qpDi+9jIkk1VSQ2aIw5UjHaJIC6jNJieZjQzCRzNyKyBBLTLTJqmRCcBdfXiats6p7Xq3d1Sr1q3kcRTiEIzgBFy6gDrfQgCYQSOEZXuHNerJerHfrY9ZasOYzB/AH1ucP3e+Slg==</latexit>

Revised: [Figure]

Revised: <latexit sha1_base64="AGBYVGjY6a0QnJN2L/n5FUnY0NM=">AAAB+HicbVDLSsNAFL2pr1ofjbp0M1gEN5ZEiros6sJlBfuANobJdNIOnUzCzESoIV/ixoUibv0Ud/6N08dCWw9cOJxzL/feEyScKe0431ZhZXVtfaO4Wdra3tkt23v7LRWnktAmiXksOwFWlDNBm5ppTjuJpDgKOG0Ho+uJ336kUrFY3OtxQr0IDwQLGcHaSL5dznpBiNr5Q3Zz6uY+9u2KU3WmQMvEnZMKzNHw7a9ePyZpRIUmHCvVdZ1EexmWmhFO81IvVTTBZIQHtGuowBFVXjY9PEfHRumjMJamhEZT9fdEhiOlxlFgOiOsh2rRm4j/ed1Uh5dexkSSairIbFGYcqRjNEkB9ZmkRPOxIZhIZm5FZIglJtpkVTIhuIsvL5PWWdU9r9buapX61TyOIhzCEZyACxdQh1toQBMIpPAMr/BmPVkv1rv1MWstWPOZA/gD6/MH3GuSlQ==</latexit

Original:   Gradient descent solutions for large hidden layer size
Revised: [Figure] [Figure]  Gradient descent solutions for large hidden layer size

Original: Here we construct solutions to the equations (3).
Revised: Here we construct solutions to the equations (3).

Original: The general strategy is to choose coordinate systemsfor the neural representations in the vector space defined by the singular value decomposition.
Revised: The general strategy is to choose coordinate systems for the neural representations in the vector space defined by the singular value decomposition.

Original: Wethen show that the solutions to the gradient descent equations (3) are diagonal in these coordinates.[Figure]
Revised: We then show that the solutions to the gradient descent equations (3) are diagonal in these coordinates.

Original: 3.1 Choosing orthogonal coordinates
Revised: 3.1 Choosing orthogonal coordinates

Original: Each hidden layer d in each pathway a provides an N da dimensional representation of the input spacefrom which the vectors x are derived.
Revised: Each hidden layer d in each pathway a provides an N da dimensional representation of the input space from which the vectors x are derived.

Original: In order to construct solutions of the gradient descent equationsof motion (3), we will choose coordinates on these representations that map to the space of singularvalues of the input-output correlation matrix Σ yx .
Revised: In order to construct solutions of the gradient descent equations of motion (3), we will choose coordinates on these representations that map to the space of singular values of the input-output correlation matrix Σ yx .

Original: We construct these coordinates using the initialvalue matrices R da .
Revised: We construct these coordinates using the initial value matrices R da .

Original: Note that for large N da , we must have
Revised: Note that for large N da , we must have

Original: [Equation]
Revised: [Equation]

Original: (4) which is exact as N da → ∞ .
Revised: which is exact as N da → ∞ .

Original: We use the matrices U and V to project the initial and final weightmatrices onto the singular vectors: W 1 a → W 1 a V and W D a a → U T W D a a .
Revised: We use the matrices U and V to project the initial and final weight matrices onto the singular vectors: W 1 a → W 1 a V and W D a a → U T W D a a .

Original: In addition, we use theinitial state matrices R da to define a coordinate system in the singular value space for the intermediaterepresentations.
Revised: In addition, we use the initial state matrices R da to define a coordinate system in the singular value space for the intermediate representations.

Original: We define this map as follows:
Revised: We define this map as follows:

Original: [Equation]
Revised: [Equation]

Original: These coorinate transformations imply the following relations:
Revised: These coorinate transformations imply the following relations:

Original: [Equation]
Revised: [Equation]

Original: which together imply
Revised: which together imply

Original: [Equation]
Revised: [Equation]

Original: In the singular value coordinate space, the gradient descent equations of motion for each a are now
Revised: In the singular value coordinate space, the gradient descent equations of motion for each a are now

Original: [Equation]
Revised: [Equation]

Original: Note that in the steady state we have
Revised: Note that in the steady state we have

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: In the initial state we have in the large N da limit:
Revised: In the initial state we have in the large N da limit:

Original: [Equation]
Revised: [Equation]

Original:  where δ da → 0 in the large N da limit.
Revised: 14) where δ da → 0 in the large N da limit.

Original: So each ¯W da is diagonal (or zero) in the initial state.
Revised: So each ¯W da is diagonal (or zero) in the initial state.

Original: Byconstruction, these matrices are all of dimension l s × l s , where l s is the number of singular values ofΣ yx .
Revised: By construction, these matrices are all of dimension l s × l s , where l s is the number of singular values of Σ yx .

Original: 3.2 Diagonal solutions for the gradient descent equations
Revised: 3.2 Diagonal solutions for the gradient descent equations

Original: We propose the following ansatz for the solution:
Revised: We propose the following ansatz for the solution:

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original:  where Γ da ( t ) are all diagonal matrices, with diagonal entries given by the vector Γ da ( t ) , with components (cid:0) Γ da ( t ) (cid:1) α , where α is an index over the singular values.
Revised: 15) where Γ da ( t ) are all diagonal matrices, with diagonal entries given by the vector Γ da ( t ) , with components (cid:0) Γ da ( t ) (cid:1) α , where α is an index over the singular values.

Original: The terms ¯ δ da → 0 in the large N dalimit.
Revised: The terms ¯ δ da → 0 in the large N da limit.

Original: With this ansatz, in the N da → ∞ limit, we have for each pathway a and component α :[Equation]
Revised: With this ansatz, in the N da → ∞ limit, we have for each pathway a and component α :

Original: [Equation]
Revised: [Equation]

Original:  where S α and ¯Ω α are the diagonal components of S and ¯Ω , respectively.
Revised: ) where S α and ¯ Ω α are the diagonal components of S and ¯ Ω , respectively.

Original: For each α , each of the (cid:0) Γ da (cid:1) α for d < D a have the same initial conditions, so by symmetry we candefine q aα = (cid:0) Γ da (cid:1) α for all d < D a and p aα = (cid:0) Γ D a a (cid:1) α to reduce these equations to
Revised: For each α , each of the (cid:0) Γ da (cid:1) α for d < D a have the same initial conditions, so by symmetry we can define q aα = (cid:0) Γ da (cid:1) α for all d < D a and p aα = (cid:0) Γ D a a (cid:1) α to reduce these equations to

Original: [Equation]
Revised: [Equation]

Original: for each pathway a and component α .
Revised: for each pathway a and component α .

Original: The initial states of these variables are q aα = 1 and p aα = 0in the large N da limit.
Revised: The initial states of these variables are q aα = 1 and p aα =in the large N da limit.

Original: Moreover we have ¯Ω α = (cid:80) Ma
Revised: Moreover we have ¯Ω α = (cid:80) Ma

Original: =1 K aα =
Revised: =1 K aα =

Original: (cid:80) Ma =1 p aα q D a − 1 aα .
Revised: (cid:80) Ma =1 p aα q D a − 1 aα .

Original: Importantly, thesediagonal solutions are stable to off-diagonal perturbations, meaning that at finite size the smallcorrections ¯ δ da will decay to zero (see the section Simulations and Figure 4).
Revised: Importantly, these diagonal solutions are stable to off-diagonal perturbations, meaning that at finite size the small corrections ¯ δ da will decay to zero (see the section Simulations and Figure 4).

Original: Taking the ratio of each of the equations in (17) above yields the differential equation
Revised: Taking the ratio of each of the equations in (17) above yields the differential equation

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: and thus these values are related by
Revised: and thus these values are related by

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The variables p aα are thus sufficient to specify the solution and they obey the following equation:
Revised: The variables p aα are thus sufficient to specify the solution and they obey the following equation:

Original: [Equation]
Revised: [Equation]

Original: with ¯Ω α = (cid:80) Ma =1 p aα (cid:16)(cid:112) p 2 aα [Equation] [Equation]
Revised: with ¯Ω α = (cid:80) Ma =1 p aα (cid:16)(cid:112) p 2 aα + 1 (cid:17) D a − 1 .

Original: D a − 1 .

Original: Different pathways will then be related by the relationship:
Revised: Different pathways will then be related by the relationship:

Original: [Equation]
Revised: [Equation]

Original: which gives
Revised: which gives

Original: [Equation]
Revised: [Equation]

Original: where 2 F 1 is the hypergeometric function and we have defined the constant C .
Revised: where 2 F 1 is the hypergeometric function and we have defined the constant C .

Original: We have re-introducedthe finite size initial conditions via ¯ δ aα .
Revised: We have re-introduced the finite size initial conditions via ¯ δ aα .

Original: These are determined by the random matrix initializationsand will vanish in the large network size limit.
Revised: These are determined by the random matrix initializations and will vanish in the large network size limit.

Original: In the strict limit, the solutions therefore must beequal across all pathways.
Revised: In the strict limit, the solutions therefore must be equal across all pathways.

Original: Finite size fluctuations in the initial state will break this symmetry.
Revised: Finite size fluctuations in the initial state will break this symmetry.

Original: The phase planes for a two pathway network with common depths D = D 

Revised: The phase planes for a two pathway network with common depths D = D

Original: = D 2 is plotted forvarious values of D in Figure 3.
Revised: = D 2 is plotted for various values of D in Figure 3.

Original: Each row represents the same set of 100 initial conditions drawnfrom zero mean normal distributions of a given variance, with the larger variance in the top row.
Revised: Each row represents the same set of 100 initial conditions drawn from zero mean normal distributions of a given variance, with the larger variance in the top row.

Original: Notethat as the depth D increases, so does the curvature of the trajectories away from zero.
Revised: Note that as the depth D increases, so does the curvature of the trajectories away from zero.

Original: The sharpertrajectories push the steady state solution to one where one or the other pathway dominates ¯Ω α = S α ,so one can see the result already in the phase portrait that at larger depths the contribution to ¯Ω α willcome from a single pathway. 
Revised: The sharper trajectories push the steady state solution to one where one or the other pathway dominates ¯Ω α = S α , so one can see the result already in the phase portrait that at larger depths the contribution to ¯Ω α will come from a single pathway. [Figure]

Original: Asymptotic approximation While the result is already qualitatively clear from the phase portrait,we can obtain an analytic statement.
Revised: Asymptotic approximation While the result is already qualitatively clear from the phase portrait, we can obtain an analytic statement.

Original: For a more easily interpretable approach than analyzinghypergeometric functions (and a way of seeing the asymptotic behavior of the above equation),consider the case where S α is sufficiently greater than 1 so that we can approximate
Revised: For a more easily interpretable approach than analyzing hypergeometric functions (and a way of seeing the asymptotic behavior of the above equation), consider the case where S α is sufficiently greater than 1 so that we can approximate

Original: [Equation]
Revised: [Equation]

Original:  (One can check this with a post-hoc self consistency condition.
Revised: 23) (One can check this with a post-hoc self consistency condition.

Original: For the above relation to hold, S αwill need to be large.)
Revised: For the above relation to hold, S α will need to be large.)

Original: Then we have
Revised: Then we have

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: 24) which implies
Revised: which implies

Original: [Equation]
Revised: [Equation]

Original:  where C abα is a constant that depends upon initial conditions and which further implies
Revised: 25) where C abα is a constant that depends upon initial conditions and which further implies

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: When D a , D b are large we have
Revised: When D a , D b are large we have

Original: [Equation]
Revised: [Equation]

Original: 27) for some C ′ abα .
Revised: 27) for some C ′ abα .

Original: In steady state, we have S α = (cid:80) a K aα .
Revised: In steady state, we have S α = (cid:80) a K aα .

Original: Consider S ′
Revised: Consider S ′

Original: α = S
Revised: α = S

Original: α − (cid:80) c
Revised: α − (cid:80) c

Original: ̸
Revised: ̸

Original: = a,b K cα =
Revised: = a,b K cα =

Original: K aα + K bα .
Revised: K aα + K bα .

Original: Then in the large depth limit we can write for K bα
Revised: Then in the large depth limit we can write for K bα

Original: [Equation]
Revised: [Equation]

Original: 28) the solutions of which are given by the quadratic equation
Revised: 28) the solutions of which are given by the quadratic equation

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original:  which are
Revised: 29) which are

Original: [Equation]
Revised: [Equation]

Original: In the limit | C ′ abα | → ∞ and in the domain K bα ∈ [0 , S ′
Revised: In the limit | C ′ abα | → ∞ and in the domain K bα ∈ [0 , S ′

Original: α ] , these will be
Revised: α ] , these will be

Original: [Equation]
Revised: [Equation]

Original: (31) and thus one pathway or the other dominates the singular value α for each α , depending upon thesign of C abα .
Revised: and thus one pathway or the other dominates the singular value α for each α , depending upon the sign of C abα .

Original: As mentioned above, in the strict limit the constant C abα → 0 , leaving symmetricsolutions across the pathways.
Revised: As mentioned above, in the strict limit the constant C abα → 0 , leaving symmetric solutions across the pathways.

Original: In the finite size limit, fluctuations in the initial state will drive C abαto larger values, biasing the network to one pathway or the other.
Revised: In the finite size limit, fluctuations in the initial state will drive C abα to larger values, biasing the network to one pathway or the other.

Original: We note that in a real application,initial conditions are not necessarily drawn from the distributions that we assume, and we expecteffectively larger initial fluctuations than our initial state would imply, e.g. by choosing the initialvalues of each weight matrix from N (0 , σ ) , rather than scaling with network size (see the examplesin Simulations).
Revised: We note that in a real application, initial conditions are not necessarily drawn from the distributions that we assume, and we expect effectively larger initial fluctuations than our initial state would imply, e.g. by choosing the initial values of each weight matrix from N (0 , σ ) , rather than scaling with network size (see the examples in Simulations).

Original: Such larger fluctuations would only enhance the effect of the symmetry breaking.
Revised: Such larger fluctuations would only enhance the effect of the symmetry breaking.

Original:  Simulations
Revised:  Simulations

Original: We demonstrate our results with numerical simulations of networks with two pathways and multipledepths.
Revised: We demonstrate our results with numerical simulations of networks with two pathways and multiple depths.

Original: For these examples we use the same number of layers per pathway and N 1 = N 2 = 1000 .
Revised: For these examples we use the same number of layers per pathway and N 1 = N 2 = 1000 .

Original: Theinitial state of the weight matrices is drawn from a zero mean normal distribution with a fixed standarddeviation σ = 0 .
Revised: The initial state of the weight matrices is drawn from a zero mean normal distribution with a fixed standard deviation σ = 0 .

Original: The input vectors x are 8-dimensional and are the rows of the 8-dimensionalidentity matrix.
Revised: The input vectors x are 8-dimensional and are the rows of the 8-dimensional identity matrix.

Original: The output vectors y are 15-dimensional and are the rows of the matrix:
Revised: The output vectors y are 15-dimensional and are the rows of the matrix:

Original: [Equation]
Revised: [Equation]

Original:    1 1 0 1 0 0 0 1 0 0 0 0 0 01 1 0 1 0 0 0 0 1 0 0 0 0 01 1 0 0 1 0 0 0 0 1 0 0 0 01 1 0 0 1 0 0 0 0 0 1 0 0 01 0 1 0 0 1 0 0 0 0 0 1 0 01 0 1 0 0 1 0 0 0 0 0 0 1 01 0 1 0 0 0 1 0 0 0 0 0 0 11 0 1 0 0 0 1 0 0 0 0 0 

Revised:    1 1 0 1 0 0 0 1 0 0 0 0 0 01 1 0 1 0 0 0 0 1 0 0 0 0 01 1 0 0 1 0 0 0 0 1 0 0 0 01 1 0 0 1 0 0 0 0 0 1 0 0 01 0 1 0 0 1 0 0 0 0 0 1 0 01 0 1 0 0 1 0 0 0 0 0 0 1 01 0 1 0 0 0 1 0 0 0 0 0 0 11 0 1 0 0 0 1 0 0 0 0 0

Original: 0   (32)
Revised: 0   (32)

Original: Gradient descent is performed over 1000 epochs with learning rate l r = 0 .
Revised: Gradient descent is performed over 1000 epochs with learning rate l r = 0 .

Original: These simulations arenot compute intensive and are easily performed on a standard modern desktop or laptop.
Revised: These simulations are not compute intensive and are easily performed on a standard modern desktop or laptop.

Original: All code forsimulations and figures will be available on GitHub.
Revised: All code for simulations and figures is available on GitHub at https://github.com/AllenInstitute/

Original:  Figure (4) shows an example of the matrix K afor a specific pathway a , which is initialized as a uniform random matrix, becoming diagonal in thesingular value space.
Revised: Multipathway_NeurIPS2022 . Figure (4) shows an example of the matrix K a for a specific pathway a , which is initialized as a uniform random matrix, becoming diagonal in the singular value space.

Original: Figure 5 shows an example of learning dynamics for networks with differentdepths.
Revised: Figure 5 shows an example of learning dynamics for networks with different depths.

Original: For the network of depth 2 (for which “pathway” is an arbitrary concept, as there is onlyone hidden layer), the singular values are more or less divided evenly across the pathways.
Revised: For the network of depth 2 (for which “pathway” is an arbitrary concept, as there is only one hidden layer), the singular values are more or less divided evenly across the pathways.

Original: Thissymmetry is already broken for networks of depths 3 and 4.
Revised: This symmetry is already broken for networks of depths 3 and 4.

Original: Even by depth 7, the largest singularvalues are concentrated on one or the other pathway.
Revised: Even by depth 7, the largest singular values are concentrated on one or the other pathway.

Original: One can see that these results are essentiallyrandom by considering different seeds, which is shown in Figure 6 with two different initializationsof a depth 7 network.
Revised: Figure 6 shows two different initializations, demonstrating stochasticity.

Original: One can see that the largest singular values are typically in one pathway or theother, but not always the same pathway.
Revised: One can see that the largest singular values are typically in one pathway or the other, but not always the same pathway.

Original: Importantly, our analytic results apply strictly to the linearized network.
Revised: Importantly, our analytic results apply strictly to the linearized network.

Original: Since the standard situationwith deep networks is to use some form of nonlinear activation we test our results empirically usingeither of Tanh or ReLU.
Revised: Since the standard situation with deep networks is to use some form of nonlinear activation we test our results empirically using either of Tanh or ReLU.

Original: The general trend remains the same when using these nonlinearities.
Revised: The general trend remains the same when using these nonlinearities.

Original: Theresults are shown in Appendix A.
Revised: The results are shown in Figures 7 and 8 for ReLU and Tanh, respectively. Simulation parameters are as above with the exception that the corresponding nonlinearity is used as an activation function following each layer of each pathway and we train for longer (as shown in each Figure). [Figure] t

Revised: [Figure]

Revised: t

Original: [Figure]
Revised: [Figure]

Revised: t

Revised: [Figure]

Revised: t

Revised: [Figure]

Revised: t

Revised: [Figure]

Revised: t

Revised: [Figure]

Revised: [Figure]

Revised: d i

Revised: [Figure]

Revised: n s i o n α

Revised: [Figure]

Original:  Discussion
Revised:  Discussion

Original: Many biological neural systems have several computational pathways that are thought to mediatedifferent aspects of sensory processing, prominant examples being the “what" and “where”pathwaysof the macaque visual system [4, 7, 10], the “local” and “global” visual motion pathways [5], and theparallel anatomical pathways in the mouse [8].{+d i m e n s i o n α 
 1 2 3 4 5 6e p o c h K a , b α 0.00.51.01.52.02.53.+}
Revised: Many biological neural systems have several computational pathways that are thought to mediate different aspects of sensory processing, prominant examples being the “what" and “where” pathways {+d i m e n s i o n α 
 1 2 3 4 5 6e p o c h K a , b α 0.00.51.01.52.02.53.+}

Revised: D =7 K aα

Revised: K bα

Original: Here we have shown, using the linearized frameworkof [14], that depth alone can have a profound impact upon the learning dynamics in systems withmultiple channels by separating features in a learned data set by pathway.{+K b 0. d i m e n s i o n α 
 1 2 3 4 5 6e p o c h+}
Revised: K a {+K b 0. d i m e n s i o n α 
 1 2 3 4 5 6e p o c h+}

Revised: K a , b α 0.00.51.01.52.02.53.

Revised: D =7 K aα

Revised: K bα

Revised: K a

Revised: K b 0.

Revised: K a

Revised: K b

Revised: K a

Revised: K b

Revised: [Figure]

Revised: K

Revised: [Figure]

Revised: K a

Revised: K b

Revised: K a

Revised: K b

Revised: [Figure]

Revised: K a

Revised: K b

Revised: [Figure]

Revised: 0 1 2 3 4 5 6e p o c h

Revised: K a , b α 0.00.51.01.52.02.53.03.

Revised: D =7 K aα K bα

Revised: K a

Revised: K b

Revised: K a

Revised: K b

Revised: K a

Revised: K b

Revised: K a

Revised: K b  of the macaque visual system [4, 7, 10], the “local” and “global” visual motion pathways [5], and the parallel anatomical pathways in the mouse [8].

Revised: Here we have shown, using the linearized framework of [16], that depth alone can have a profound impact upon learning dynamics in systems with multiple channels by separating features in a learned data set by pathway.

Revised: It is useful to recall again the observation made for the AlexNet model [9] which featured two pathways as a result of building the architecture around two GPUs.

Revised: In that case, the authors noted that each pathway learned different large scale features of the data (one largely color, the other largely texture).

Revised: We have examined the supervised case using linear regression on the output features of the network.

Revised: In this framework a power iteration effect from multiple layers of computation (which can be seen in the weight matrix factors in equation 3) enhances the inherent “explaining away” effect in regression (as each pathway’s features contribute to accounting for the input-output relationship) so that with increasing depth there is a tendency for each singular vector to concentrate on one pathway.

Revised: Note that this does not imply that separate singular components will prefer different pathways in general.

Original: While we have used a very specific initialization scheme for mathematical purposes, we posit that(and have provided an example in which) our results are more generally applicable.
Revised: It is interesting to consider the parallels and differences between this model and the classic Oja rule that leads to PCA[13]. In that case, the learning dynamics magnifies the largest principal components of the input space and competition is introduced via weight normalization. This leads to the weights matching the largest principal components of the input data, i.e. the singular vectors of Σ x . (This happens in a deterministic fashion, although see [12] for a higher order generalization that does not lead to deterministic selection of the largest components.)

Revised: The mechanism we describe here operates in the supervised case and decomposes Σ yx .

Revised: While we have used a very specific initialization scheme for mathematical purposes, we posit that (and have provided an example in which) our results are more generally applicable.

Original: In some casesthis is easy to show as an extension of our work.
Revised: In some cases this is easy to show as an extension of our work.

Original: This work is an analysis of the analytic properties ofdeep networks in the infinite width limit.
Revised: This work is an analysis of the analytic properties of deep networks in the infinite width limit.

Original: As such, care must be taken in considering these results inthe context of systems in which corrections due to finite size are large.
Revised: As such, care must be taken in considering these results in the context of systems in which corrections due to finite size are large.

Original: The assumption of linearityis also important.
Revised: The assumption of linearity is also important.

Original: Deep networks in practice have non-linearities, which may be responsible fordeviations from the results described here. [Table] /KxcpNFkcOjuEEzsCDK6jAHVShBhSe4Ble4c0ZOS/Ou/Mxb11xspkj+APn8weLapH4</latexit> K a,b↵ <latexit sha1_base64="SLUzUIPxWVGCgF6jdV0IlXvxb8A=">AAAB9HicbVDLSgNBEOz1GeMr6tHLYBA8SNiVoB6DXgQvEcwDkiX0TibJkNnZdWY2EJZ8hxcPinj1Y7z5N06SPWhiQUNR1U13VxALro3rfjsrq2vrG5u5rfz2zu7efuHgsK6jRFFWo5GIVDNAzQSXrGa4EawZK4ZhIFgjGN5O/caIKc0j+WjGMfND7Eve4xSNlfz7TornQRtFPMBJp1B0S+4MZJl4GSlChmqn8NXuRjQJmTRUoNYtz42Nn6IynAo2ybcTzWKkQ+yzlqUSQ6b9dHb0hJxapUt6kbIlDZmpvydSDLUeh4HtDNEM9KI3Ff/zWonpXfspl3FimKTzRb1EEBORaQKkyxWjRowtQaq4vZXQASqkxuaUtyF4iy8vk/pFybsslR/KxcpNFkcOjuEEzsCDK6jAHVShBhSe4Ble4c0ZOS/Ou/Mxb11xspkj+APn8weLapH4</latexit [Figure]
Revised: Deep networks in practice have non-linearities.

Original: [Figure]

Original: [Equation]

Original: [Figure]

Original: [Equation]

Original: [Figure]

Original: [Equation]

Original: [Figure]

Original: [Equation]

Original: [Figure]

Original: [Table]

Original: vz8vVa

Original: [Figure] zOPJwBMdwCh5UoAp3UIM6MOjDM7zCmyOcF [Figure] fd+Zi35pxs5hD+wPn8AZTFjVk
Revised: One can argue that learning dynamics will push the system to operate in a regime that maximizes the dynamic range of the response.

Original: [Figure] </latexit [Figure] <
Revised: For a function like Tanh, this should be at the inflection point.

Original: [Figure] _base [Figure] "rbFGjpjxLrxjBxlU5j1N2CEoEF
Revised: If this is the case, the network should act in a roughly linear fashion.

Original: [Figure]

Original: [Figure] 0NBxqhjWWSxi1QqoRsEl1g03AluJQhoFApvB8GbqN59QaR7LRzNK0I9oX [Figure] Or2YpRFKwwTVuu25ifHHVBnOBE4KnVRjQtmQ9rFtqaQRan88O3VCTqzSI2GsbElDZurviTGNtB5Fge2MqBnoRW8q
Revised: This is just a high-level argument, however, and nonlinearities may be responsible for deviations from the results described here, although we have demonstrated in our examples that common linearities show the same empirical dynamics.

Original: [Figure] One obvious reason for  an architecture with multiple pathways is to create divergent representationsfor different cost functions from a common set of lower level features.
Revised: One obvious reason for having an architecture with multiple pathways is to create divergent representations for different cost functions from a common set of lower level features.

Original: In our case we consider analternative in which all pathways subserve the same cost function.
Revised: In our case we consider an alternative in which all pathways subserve the same cost function.

Original: If the result is to separate features,why would such an architecture be desirable?
Revised: If the result is to separate features, why would such an architecture be desirable?

Original: One answer may be that multiple pathways would serveas a form of regularization.
Revised: One answer may be that multiple pathways would serve as a form of regularization.

Original: Not only does this reduce the space of model complexity, but through themechanism we have demonstrated this can also lead to a separation of features, and perhaps moreeasily learnable representations.
Revised: Not only does this reduce the space of model complexity, but through the mechanism we have demonstrated this can also lead to a separation of features, and perhaps more easily learnable representations for downstream tasks.

Revised: This may be a useful component of pre-training procedures.

Revised: It may be that multiple pathways are a source of architectural efficiency, and particularly useful when the system has few components (suggesting a reason why the mouse architecture appears more parallel than mammals with much larger brains).

Revised: Quantifying these potential benefits would entail a proper model comparison across different architectures and examining performance as well as learning speed.

Revised: Looking towards future work, it is important to keep in mind that biological systems may be subject to different types of cost functions, and the dynamics of those may be quite different than what we see here.

Original: As this is an analytic approach to the theory of deep neural networks from a dynamical systemsperspective, we anticipate no negative societal impacts.
Revised: As this is an analytic approach to the theory of deep neural networks from a dynamical systems perspective, we anticipate no negative societal impacts.

Revised: Acknowledgments and Disclosure of Funding

Revised: We thank Gabriel Koch Ocker, Blake Richards, and Shahab Bakhtiari for helpful discussions.

Revised: We thank the Allen Institute founder, Paul G. Allen, for vision, encouragement, and support.

Original: Checklist
Revised: Checklist

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes](b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes](c) Did you discuss any potential negative societal impacts of your work?
Revised: [Yes] (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes](d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
Revised: [Yes] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes]2.
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes](b) Did you include complete proofs of all theoretical results?
Revised: [Yes] (b) Did you include complete proofs of all theoretical results?

Original: [Yes]3.
Revised: [Yes] 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] Not provided atinitial submission due to double-blind review, but a GitHub link will be provided in thefinal paper.(b)
Revised: [Yes] Code for simulations and figures is available at https://github.com/AllenInstitute/ Multipathway_NeurIPS2022 .

Original:  Did you specify all the training details (e .g. , data splits, hyperparameters, how theywere chosen)
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes ](c) Did you report error bars (e.g., with respect to the random seed 
Revised: [Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: after

Original: running experiments multiple times

Original: [N/A]

Original: d

Original: ) Did you

Original:  include the total amount of compute and the type of resources used (
Revised: [N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: e

Original: .g.

Original: type

Original: of GPUs, internal cluster, or cloud provider

Original: [Yes] 
Revised: [Yes] 4.

Original: If you are using existing assets (

Original: e

Original: g

Original: ., code, data, models) or curating/releasing new assets... a ) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [N/A] b ) Did you mention the license of the assets?
Revised: [N/A] (b) Did you mention the license of the assets?

Original: [N/A] c) Did you include any new assets either in the supplemental material or as a URL?
Revised: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A] (d) Did you discuss whether
Revised: [N/A] (d) Did you discuss whether

Original: and how consent was obtained from people whose data you’reusing/curating?
Revised: and how consent was obtained from people whose data you’re using/curating?

Original: [N/A](e) Did you discuss whether the data you are using/curating contains personally identifiableinformation or offensive content?
Revised: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [N/A]5.
Revised: [N/A] 5.

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A](b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A](c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]

Original: A Appendix: Simluation results with nonlinear activation functions.

Original: Figures 7 and 8 show simulations results similar to those in Figure 5 with the nonlinearities ReLU and

Original: Tanh, respectively.

Original: Simulation parameters are as in Section 4 with the exception that the correspondingnonlinearity is used as an activation function following each layer of each pathway.

Original: In addition, fordepths 2,3, and 4 we simulate 10000 epochs, whereas for depth 7 we simulation 20000 epochs.

Original: [Table]

Original: zOPJwBMdwCh5UoAp3UIM6MOjDM7zCmyOcF+fd+Zi35pxs5hD+wPn8AZTFjVk=</latexit> D =

Original: [Figure]

Original: [Table]

Original: =">AAAB6nicbVDLSgNBEOyNrxhfUY9eBoPgKeyKGC9CUA8eI5oHJEuYnfQmQ2Znl5lZIYR8ghcPinj1i7z5N06SPWhiQUNR1U13V5AIro3rfju5ldW19Y38ZmFre2d3r7h/0NBxqhjWWSxi1QqoRsEl1g03AluJQhoFApvB8GbqN59QaR7LRzNK0I9oX/KQM2qs9HB7VekWS27ZnYEsEy8jJchQ6xa/Or2YpRFKwwTVuu25ifHHVBnOBE4KnVRjQtmQ9rFtqaQRan88O3VCTqzSI2GsbElDZurviTGNtB5Fge2MqBnoRW8q/ue1UxNe+mMuk9SgZPNFYSqIicn0b9LjCpkRI0soU9zeStiAKsqMTadgQ/AWX14mjbOyd1E+vz8vVa+zOPJwBMdwCh5UoAp3UIM6MOjDM7zCmyOcF+fd+Zi35pxs5hD+wPn8AZTFjVk=</latexit> D =

Original: [Figure]