Original: Learning dynamics of deep linear networks with multiple pathways
Revised: Learning dynamics of deep linear networks with multiple pathways

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Not only have deep networks become standard in machine learning, they are increasingly of interest in neuroscience as models of cortical computation that capturerelationships between structural and functional properties.
Revised: Not only have deep networks become standard in machine learning, they are increasingly of interest in neuroscience as models of cortical computation that capturerelationships between structural and functional properties.

Original: In addition they are auseful target of theoretical research into the properties of network computation.
Revised: In addition they are auseful target of theoretical research into the properties of network computation.

Original: Deep networks typically have a serial or approximately serial organization acrosslayers, and this is often mirrored in models that purport to represent computationin mammalian brains.
Revised: Deep networks typically have a serial or approximately serial organization acrosslayers, and this is often mirrored in models that purport to represent computationin mammalian brains.

Original: There are, however, multiple examples of parallel pathwaysin mammalian brains.
Revised: There are, however, multiple examples of parallel pathwaysin mammalian brains.

Original: In some cases, such as the mouse, the entire visual systemappears arranged in a largely parallel, rather than serial fashion.
Revised: In some cases, such as the mouse, the entire visual systemappears arranged in a largely parallel, rather than serial fashion.

Original: While these pathways may be formed by differing cost functions that drive different computations,here we present a new mathematical analysis of learning dynamics in networksthat have parallel computational pathways driven by the same cost function.
Revised: While these pathways may be formed by differing cost functions that drive different computations,here we present a new mathematical analysis of learning dynamics in networksthat have parallel computational pathways driven by the same cost function.

Original: Weuse the approximation of deep linear networks with large hidden layer sizes toshow that, as the depth of the parallel pathways increases, different features of thetraining set (defined by the singular values of the input-output correlation) willtypically concentrate in one of the pathways.
Revised: Weuse the approximation of deep linear networks with large hidden layer sizes toshow that, as the depth of the parallel pathways increases, different features of thetraining set (defined by the singular values of the input-output correlation) willtypically concentrate in one of the pathways.

Original: This result is derived analyticallyand demonstrated with numerical simulation.
Revised: This result is derived analyticallyand demonstrated with numerical simulation.

Original: Thus, rather than sharing stimulusand task features across multiple pathways, parallel network architectures learn toproduce sharply diversified representations with specialized and specific pathways,a mechanism which may hold important consequences for codes in both biologicaland artificial systems.
Revised: Thus, rather than sharing stimulusand task features across multiple pathways, parallel network architectures learn toproduce sharply diversified representations with specialized and specific pathways,a mechanism which may hold important consequences for codes in both biologicaland artificial systems.

Original:  Introduction
Revised:  Introduction

Original: Deep networks are increasingly used as models of cortical computation, particularly of the visualpathway.
Revised: Deep networks are increasingly used as models of cortical computation, particularly of the visualpathway.

Original: Indeed, the earliest convolutional neural network was a model of the mammalian visualsystem [6] and the use of CNNs to model the visual pathway has continued [12, 19, 20, 3, 2].
Revised: Indeed, the earliest convolutional neural network was a model of the mammalian visualsystem [6] and the use of CNNs to model the visual pathway has continued [12, 19, 20, 3, 2].

Original: Thesemodels are usually serial models of the primate ventral stream that describe the computation of thatsystem as being closely related to object recognition.
Revised: Thesemodels are usually serial models of the primate ventral stream that describe the computation of thatsystem as being closely related to object recognition.

Original: However, processing of visual informationin the primate is associated with (at least) two parallel pathways, the “ventral” (or “what”) and the“dorsal” (or “where") pathways [4, 7, 10].
Revised: However, processing of visual informationin the primate is associated with (at least) two parallel pathways, the “ventral” (or “what”) and the“dorsal” (or “where") pathways [4, 7, 10].

Original: In the mouse visual system, anatomical analysis alsosuggests that there are parallel pathways [8, 17, 18], although the specific functional role of thesepathways is less clear.
Revised: In the mouse visual system, anatomical analysis alsosuggests that there are parallel pathways [8, 17, 18], although the specific functional role of thesepathways is less clear.

Original: Interestingly, recent research [1] has shown that self-supervised training inneural network architectures with parallel pathways can lead to the emergence of ventral-like anddorsal-like pathways.
Revised: Interestingly, recent research [1] has shown that self-supervised training inneural network architectures with parallel pathways can lead to the emergence of ventral-like anddorsal-like pathways.

Original: Another study shows that, with a similar self-supervised training objective,shallower architectures with parallel pathways lead to closer matches with functional data in the
Revised: Another study shows that, with a similar self-supervised training objective,shallower architectures with parallel pathways lead to closer matches with functional data in the

Original: Allen Brain Observatory than deep single pathway architectures [11].
Revised: Allen Brain Observatory than deep single pathway architectures [11].

Original: Furthermore, an anatomicallyconstrained deep neural network model for the mouse visual cortex has demonstrated that such parallel architectures produce more diverse representations of visual images, compared to singlepathway architectures [16].
Revised: Furthermore, an anatomicallyconstrained deep neural network model for the mouse visual cortex has demonstrated that such parallel architectures produce more diverse representations of visual images, compared to singlepathway architectures [16].

Original: Additionally, a coincident observation has been made for the original AlexNet [9] (a deep neuralnetwork designed for image processing) that also shows the emergence of functional specializationacross parallel pathways, with different pathways (defined in the model because it was implementedon multiple GPUs) learning different features of the input data.
Revised: Additionally, a coincident observation has been made for the original AlexNet [9] (a deep neuralnetwork designed for image processing) that also shows the emergence of functional specializationacross parallel pathways, with different pathways (defined in the model because it was implementedon multiple GPUs) learning different features of the input data.

Original: Overall, in the road map fordeveloping future artificial intelligence systems, architectures with parallel pathways promise potentialsolutions for multi-tasking, multi-sensory, energy-efficient computing [13].
Revised: Overall, in the road map fordeveloping future artificial intelligence systems, architectures with parallel pathways promise potentialsolutions for multi-tasking, multi-sensory, energy-efficient computing [13].

Original: But despite the richnessof information representation and the potential capabilities of the architectures with parallel pathways,there is a lack of research in understanding the fundamental learning behavior that occurs in sucharchitectures.
Revised: But despite the richnessof information representation and the potential capabilities of the architectures with parallel pathways,there is a lack of research in understanding the fundamental learning behavior that occurs in sucharchitectures.

Original: Here, we advance this frontier through a mathematical analysis of the behaviour ofdeep neural networks with parallel pathways, in the tractable setting of deep linear networks [14].
Revised: Here, we advance this frontier through a mathematical analysis of the behaviour ofdeep neural networks with parallel pathways, in the tractable setting of deep linear networks [14].

Original: Our starting point follows Saxe et al.
Revised: Our starting point follows Saxe et al.

Original: [15], which elegantly shows how, in such a linear network, thetraining data and learning dynamics for a given task can be fully described via a set of independentmodes resulting from singular value decomposition (SVD) of the input-output correlation matrix.
Revised: [15], which elegantly shows how, in such a linear network, thetraining data and learning dynamics for a given task can be fully described via a set of independentmodes resulting from singular value decomposition (SVD) of the input-output correlation matrix.

Original: During gradient decent, a deep linear network will pick up the task related information from the databy acquiring knowledge from each mode [15].
Revised: During gradient decent, a deep linear network will pick up the task related information from the databy acquiring knowledge from each mode [15].

Original: Here, we use the same overall framework to study thedynamics of learning in linear networks with parallel pathways.
Revised: Here, we use the same overall framework to study thedynamics of learning in linear networks with parallel pathways.

Original: Our analysis and simulations yield the following main results.
Revised: Our analysis and simulations yield the following main results.

Original: For linear networks with multipleparallel pathways of hidden neurons that feed into the same output representation and cost function:
Revised: For linear networks with multipleparallel pathways of hidden neurons that feed into the same output representation and cost function:

Original: For wide networks with random weight initialization, the learning dynamics reduces to a
Revised: For wide networks with random weight initialization, the learning dynamics reduces to a

Original: coupled set of ordinary differential equations of dimension equal to the number of pathwaysfor each non-zero singular vector in the input-output correlation matrix of the training set.
Revised: coupled set of ordinary differential equations of dimension equal to the number of pathwaysfor each non-zero singular vector in the input-output correlation matrix of the training set.

Original: These dynamics are competitive in the sense that each pathway will compete to “explain”
Revised: These dynamics are competitive in the sense that each pathway will compete to “explain”

Original: the singular values of the training set.
Revised: the singular values of the training set.

Original: For deeper networks, this competition becomes more severe, resulting in more trajectories
Revised: For deeper networks, this competition becomes more severe, resulting in more trajectories

Original: in phase space flowing to extreme solutions, wherein features (defined by the singular values
Revised: in phase space flowing to extreme solutions, wherein features (defined by the singular values

Original: of the training set) are confined
Revised: of the training set) are confined

Original: largely to a single pathway.4.
Revised: largely to a single pathway.4.

Original: For the strict infinite-width limit, the system flows to a state wherein features are sharedevenly across the pathways.
Revised: For the strict infinite-width limit, the system flows to a state wherein features are sharedevenly across the pathways.

Original: At finite width, fluctuations in the initial state put the systemonto a trajectory leading to a biased solution, where the bias is more severe at largerdepths.
Revised: At finite width, fluctuations in the initial state put the systemonto a trajectory leading to a biased solution, where the bias is more severe at largerdepths.

Original: Thus in large-width, large-depth networks, the system evinces a finite-size inducedspontaneous symmetry breaking.
Revised: Thus in large-width, large-depth networks, the system evinces a finite-size inducedspontaneous symmetry breaking.

Original: Thus, this depth-dependent mechanism reveals a dynamical link between network architecture andrepresentation learning.
Revised: Thus, this depth-dependent mechanism reveals a dynamical link between network architecture andrepresentation learning.

Original:  Mathematical framework
Revised:  Mathematical framework

Original: Network description: We study the learning dynamics under gradient descent of a system withparallel pathways with one or more hidden layers in each pathway.
Revised: Network description: We study the learning dynamics under gradient descent of a system withparallel pathways with one or more hidden layers in each pathway.

Original: We consider “linear" networks
Revised: We consider “linear" networks

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: =">AAAB+HicbVDLSsNAFL2pr1ofjbp0M1gEN5ZEiros6sJlBfuANobJdNIOnUzCzESoIV/ixoUibv0Ud/6N08dCWw9cOJxzL/feEyScKe0431ZhZXVtfaO4Wdra3tkt23v7LRWnktAmiXksOwFWlDNBm5ppTjuJpDgKOG0Ho+uJ336kUrFY3OtxQr0IDwQLGcHaSL5dznpBiNr5Q3Zz6uY+9u2KU3WmQMvEnZMKzNHw7a9ePyZpRIUmHCvVdZ1EexmWmhFO81IvVTTBZIQHtGuowBFVXjY9PEfHRumjMJamhEZT9fdEhiOlxlFgOiOsh2rRm4j/ed1Uh5dexkSSairIbFGYcqRjNEkB9ZmkRPOxIZhIZm5FZIglJtpkVTIhuIsvL5PWWdU9r9buapX61TyOIhzCEZyACxdQh1toQBMIpPAMr/BmPVkv1rv1MWstWPOZA/gD6/MH3GuSlQ
Revised: =">AAAB+HicbVDLSsNAFL2pr1ofjbp0M1gEN5ZEiros6sJlBfuANobJdNIOnUzCzESoIV/ixoUibv0Ud/6N08dCWw9cOJxzL/feEyScKe0431ZhZXVtfaO4Wdra3tkt23v7LRWnktAmiXksOwFWlDNBm5ppTjuJpDgKOG0Ho+uJ336kUrFY3OtxQr0IDwQLGcHaSL5dznpBiNr5Q3Zz6uY+9u2KU3WmQMvEnZMKzNHw7a9ePyZpRIUmHCvVdZ1EexmWmhFO81IvVTTBZIQHtGuowBFVXjY9PEfHRumjMJamhEZT9fdEhiOlxlFgOiOsh2rRm4j/ed1Uh5dexkSSairIbFGYcqRjNEkB9ZmkRPOxIZhIZm5FZIglJtpkVTIhuIsvL5PWWdU9r9buapX61TyOIhzCEZyACxdQh1toQBMIpPAMr/BmPVkv1rv1MWstWPOZA/gD6/MH3GuSlQ

Original: [Figure]
Revised: [Figure]

Original: </latexit
Revised: </latexit

Original: [Figure]
Revised: [Figure]

Original: in which the transformation of each layer to the next is given by a matrix multiplication, with nonon-linearity (i.e., linear transfer functions at each “node”).
Revised: in which the transformation of each layer to the next is given by a matrix multiplication, with nonon-linearity (i.e., linear transfer functions at each “node”).

Original: Given an input vector x (of dimension l x )and output vector y (of dimension l y ), the output of the system with M parallel pathways is given by
Revised: Given an input vector x (of dimension l x )and output vector y (of dimension l y ), the output of the system with M parallel pathways is given by

Original: [Equation]
Revised: [Equation]

Original: where we have defined
Revised: where we have defined

Original: [Equation]
Revised: [Equation]

Original: and where pathway a has D a − 1 hidden layers and D a weight matrices.
Revised: and where pathway a has D a − 1 hidden layers and D a weight matrices.

Original: Each weight matrix W dahas shape N da × N d − 1 a (where N 

Revised: Each weight matrix W dahas shape N da × N d − 1 a (where N

Original: a = l x ).
Revised: a = l x ).

Original: In the product formula, we are assuming that matrices witha higher value of d are to the left of matrices with smaller values of d .
Revised: In the product formula, we are assuming that matrices witha higher value of d are to the left of matrices with smaller values of d .

Original: A schematic of this system isgiven in Figure 1.
Revised: A schematic of this system isgiven in Figure 1.

Original: As described below, we consider training this network using input-output vector pairs x and y thathave correlation matrix Σ yx = USV T , where we have used the singular value decomposition where
Revised: As described below, we consider training this network using input-output vector pairs x and y thathave correlation matrix Σ yx = USV T , where we have used the singular value decomposition where

Original: S is diagonal and U , V are orthogonal matrices.
Revised: S is diagonal and U , V are orthogonal matrices.

Original: The gradient descent dynamics will lead to solutions
Revised: The gradient descent dynamics will lead to solutions

Original: U T ΩV
Revised: U T ΩV

Original: = S , which implies that in the steady state the pathway matrices K a ≡ U T Ω a V must sumto the singular values.
Revised: = S , which implies that in the steady state the pathway matrices K a ≡ U T Ω a V must sumto the singular values.

Original: A schematic of this decomposition is shown in Figure 2(top row), adaptedfrom [15].
Revised: A schematic of this decomposition is shown in Figure 2(top row), adaptedfrom [15].

Original: Main question and illustration of main result:
Revised: Main question and illustration of main result:

Original: Given this description of how the network asa whole carries information about each input-output mode, we ask how this mode information isdistributed across parallel pathways.
Revised: Given this description of how the network asa whole carries information about each input-output mode, we ask how this mode information isdistributed across parallel pathways.

Original: We illustrate two possibilities in Figure 2: (1) each mode’sinformation could be shared across multiple pathways (middle row) or (2) each mode’s informationcould be exclusively carried in individual pathways (bottom row).
Revised: We illustrate two possibilities in Figure 2: (1) each mode’sinformation could be shared across multiple pathways (middle row) or (2) each mode’s informationcould be exclusively carried in individual pathways (bottom row).

Original: We show that case (2) occurs in thelimit of large network size and large depth due to finite-size induced spontaneous symmetry breaking.
Revised: We show that case (2) occurs in thelimit of large network size and large depth due to finite-size induced spontaneous symmetry breaking.

Original: 2.1 Learning dynamics under gradient descent
Revised: 2.1 Learning dynamics under gradient descent

Original: Using the same setting as in Saxe et al.
Revised: Using the same setting as in Saxe et al.

Original: [14], we train the network with a set of P examples { x i , y i } , i = 1 , 2 , . . .
Revised: [14], we train the network with a set of P examples { x i , y i } , i = 1 , 2 , . . .

Original: , P with gradient descent on the squared loss
Revised: , P with gradient descent on the squared loss

Original: [Equation]
Revised: [Equation]

Original: In the limit of small learning rate, the dynamics of gradient descent are approximated by thecontinuous time ordinary differential equations given below (for each pathway a ):
Revised: In the limit of small learning rate, the dynamics of gradient descent are approximated by thecontinuous time ordinary differential equations given below (for each pathway a ):

Original: [Equation]
Revised: [Equation]

Original: where (cid:81)
Revised: where (cid:81)

Original: d 2 i =
Revised: d 2 i =

Original: d 1 W ia
Revised: d 1 W ia

Original: =
Revised: =

Original: W l 2 a W l 2 − 1 a · · · W l 1 a when l 1 ≤ l 2 and (cid:81) l 2 i =
Revised: W l 2 a W l 2 − 1 a · · · W l 1 a when l 1 ≤ l 2 and (cid:81) l 2 i =

Original: l 1 W ia
Revised: l 1 W ia

Original: = I
Revised: = I

Original: when l 1 > l 2 .
Revised: when l 1 > l 2 .

Original: Σ yx =
Revised: Σ yx =

Original: E [ yx T ] and Σ x = E
Revised: E [ yx T ] and Σ x = E

Original: [ xx T ] .
Revised: [ xx T ] .

Original: We assume without loss of generality that Σ x = I .
Revised: We assume without loss of generality that Σ x = I .

Original: Define the initial conditions for the weight matrices as R da ≡ W da (0) .
Revised: Define the initial conditions for the weight matrices as R da ≡ W da (0) .

Original: We take the components R dato be independently and identically distributed according to a normal distribution with zero mean andvariance σ 2 N da , where N da is the number of units in the output of that layer.
Revised: We take the components R dato be independently and identically distributed according to a normal distribution with zero mean andvariance σ 2 N da , where N da is the number of units in the output of that layer.

Original: We also denote the singularvalue decomposition of Σ yx = USV T , where S is diagonal and U , V are orthogonal matrices.
Revised: We also denote the singularvalue decomposition of Σ yx = USV T , where S is diagonal and U , V are orthogonal matrices.

Original:  Gradient descent solutions for large hidden layer size
Revised:  Gradient descent solutions for large hidden layer size

Original: Here we construct solutions to the equations (3).
Revised: Here we construct solutions to the equations (3).

Original: The general strategy is to choose coordinate systemsfor the neural representations in the vector space defined by the singular value decomposition.
Revised: The general strategy is to choose coordinate systemsfor the neural representations in the vector space defined by the singular value decomposition.

Original: Wethen show that the solutions to the gradient descent equations (3) are diagonal in these coordinates.
Revised: Wethen show that the solutions to the gradient descent equations (3) are diagonal in these coordinates.

Original: [Figure]
Revised: [Figure]

Original: 3.1 Choosing orthogonal coordinates
Revised: 3.1 Choosing orthogonal coordinates

Original: Each hidden layer d in each pathway a provides an N da dimensional representation of the input spacefrom which the vectors x are derived.
Revised: Each hidden layer d in each pathway a provides an N da dimensional representation of the input spacefrom which the vectors x are derived.

Original: In order to construct solutions of the gradient descent equationsof motion (3), we will choose coordinates on these representations that map to the space of singularvalues of the input-output correlation matrix Σ yx .
Revised: In order to construct solutions of the gradient descent equationsof motion (3), we will choose coordinates on these representations that map to the space of singularvalues of the input-output correlation matrix Σ yx .

Original: We construct these coordinates using the initialvalue matrices R da .
Revised: We construct these coordinates using the initialvalue matrices R da .

Original: Note that for large N da , we must have
Revised: Note that for large N da , we must have

Original: [Equation]
Revised: [Equation]

Original: (4) which is exact as N da → ∞ .
Revised: (4) which is exact as N da → ∞ .

Original: We use the matrices U and V to project the initial and final weightmatrices onto the singular vectors: W 1 a → W 1 a V and W D a a → U T W D a a .
Revised: We use the matrices U and V to project the initial and final weightmatrices onto the singular vectors: W 1 a → W 1 a V and W D a a → U T W D a a .

Original: In addition, we use theinitial state matrices R da to define a coordinate system in the singular value space for the intermediaterepresentations.
Revised: In addition, we use theinitial state matrices R da to define a coordinate system in the singular value space for the intermediaterepresentations.

Original: We define this map as follows:
Revised: We define this map as follows:

Original: [Equation]
Revised: [Equation]

Original: These coorinate transformations imply the following relations:
Revised: These coorinate transformations imply the following relations:

Original: [Equation]
Revised: [Equation]

Original: which together imply
Revised: which together imply

Original: [Equation]
Revised: [Equation]

Original: In the singular value coordinate space, the gradient descent equations of motion for each a are now
Revised: In the singular value coordinate space, the gradient descent equations of motion for each a are now

Original: [Equation]
Revised: [Equation]

Original: Note that in the steady state we have
Revised: Note that in the steady state we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: In the initial state we have in the large N da limit:
Revised: In the initial state we have in the large N da limit:

Original: [Equation]
Revised: [Equation]

Original: where δ da → 0 in the large N da limit.
Revised: where δ da → 0 in the large N da limit.

Original: So each ¯W da is diagonal (or zero) in the initial state.
Revised: So each ¯W da is diagonal (or zero) in the initial state.

Original: Byconstruction, these matrices are all of dimension l s × l s , where l s is the number of singular values ofΣ yx .
Revised: Byconstruction, these matrices are all of dimension l s × l s , where l s is the number of singular values ofΣ yx .

Original: 3.2 Diagonal solutions for the gradient descent equations
Revised: 3.2 Diagonal solutions for the gradient descent equations

Original: We propose the following ansatz for the solution:
Revised: We propose the following ansatz for the solution:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where Γ da ( t ) are all diagonal matrices, with diagonal entries given by the vector Γ da ( t ) , with components (cid:0) Γ da ( t ) (cid:1) α , where α is an index over the singular values.
Revised: where Γ da ( t ) are all diagonal matrices, with diagonal entries given by the vector Γ da ( t ) , with components (cid:0) Γ da ( t ) (cid:1) α , where α is an index over the singular values.

Original: The terms ¯ δ da → 0 in the large N dalimit.
Revised: The terms ¯ δ da → 0 in the large N dalimit.

Original: With this ansatz, in the N da → ∞ limit, we have for each pathway a and component α :
Revised: With this ansatz, in the N da → ∞ limit, we have for each pathway a and component α :

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where S α and ¯Ω α are the diagonal components of S and ¯Ω , respectively.
Revised: where S α and ¯Ω α are the diagonal components of S and ¯Ω , respectively.

Original: For each α , each of the (cid:0) Γ da (cid:1) α for d < D a have the same initial conditions, so by symmetry we candefine q aα = (cid:0) Γ da (cid:1) α for all d < D a and p aα = (cid:0) Γ D a a (cid:1) α to reduce these equations to
Revised: For each α , each of the (cid:0) Γ da (cid:1) α for d < D a have the same initial conditions, so by symmetry we candefine q aα = (cid:0) Γ da (cid:1) α for all d < D a and p aα = (cid:0) Γ D a a (cid:1) α to reduce these equations to

Original: [Equation]
Revised: [Equation]

Original: for each pathway a and component α .
Revised: for each pathway a and component α .

Original: The initial states of these variables are q aα = 1 and p aα = 0in the large N da limit.
Revised: The initial states of these variables are q aα = 1 and p aα = 0in the large N da limit.

Original: Moreover we have ¯Ω α = (cid:80) Ma
Revised: Moreover we have ¯Ω α = (cid:80) Ma

Original: =1 K aα =
Revised: =1 K aα =

Original: (cid:80) Ma =1 p aα q D a − 1 aα .
Revised: (cid:80) Ma =1 p aα q D a − 1 aα .

Original: Importantly, thesediagonal solutions are stable to off-diagonal perturbations, meaning that at finite size the smallcorrections ¯ δ da will decay to zero (see the section Simulations and Figure 4).
Revised: Importantly, thesediagonal solutions are stable to off-diagonal perturbations, meaning that at finite size the smallcorrections ¯ δ da will decay to zero (see the section Simulations and Figure 4).

Original: Taking the ratio of each of the equations in (17) above yields the differential equation
Revised: Taking the ratio of each of the equations in (17) above yields the differential equation

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: and thus these values are related by
Revised: and thus these values are related by

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: The variables p aα are thus sufficient to specify the solution and they obey the following equation:
Revised: The variables p aα are thus sufficient to specify the solution and they obey the following equation:

Original: [Equation]
Revised: [Equation]

Original: with ¯Ω α = (cid:80) Ma =1 p aα (cid:16)(cid:112)
Revised: with ¯Ω α = (cid:80) Ma =1 p aα (cid:16)(cid:112)

Original: p 2 aα
Revised: p 2 aα

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: D a − 1 .
Revised: D a − 1 .

Original: Different pathways will then be related by the relationship:
Revised: Different pathways will then be related by the relationship:

Original: [Equation]
Revised: [Equation]

Original: which gives
Revised: which gives

Original: [Equation]
Revised: [Equation]

Original: where 2 F 1 is the hypergeometric function and we have defined the constant C .
Revised: where 2 F 1 is the hypergeometric function and we have defined the constant C .

Original: We have re-introducedthe finite size initial conditions via ¯ δ aα .
Revised: We have re-introducedthe finite size initial conditions via ¯ δ aα .

Original: These are determined by the random matrix initializationsand will vanish in the large network size limit.
Revised: These are determined by the random matrix initializationsand will vanish in the large network size limit.

Original: In the strict limit, the solutions therefore must beequal across all pathways.
Revised: In the strict limit, the solutions therefore must beequal across all pathways.

Original: Finite size fluctuations in the initial state will break this symmetry.
Revised: Finite size fluctuations in the initial state will break this symmetry.

Original: The phase planes for a two pathway network with common depths D = D 

Revised: The phase planes for a two pathway network with common depths D = D

Original: = D 2 is plotted forvarious values of D in Figure 3.
Revised: = D 2 is plotted forvarious values of D in Figure 3.

Original: Each row represents the same set of 100 initial conditions drawnfrom zero mean normal distributions of a given variance, with the larger variance in the top row.
Revised: Each row represents the same set of 100 initial conditions drawnfrom zero mean normal distributions of a given variance, with the larger variance in the top row.

Original: Notethat as the depth D increases, so does the curvature of the trajectories away from zero.
Revised: Notethat as the depth D increases, so does the curvature of the trajectories away from zero.

Original: The sharpertrajectories push the steady state solution to one where one or the other pathway dominates ¯Ω α = S α ,so one can see the result already in the phase portrait that at larger depths the contribution to ¯Ω α willcome from a single pathway.
Revised: The sharpertrajectories push the steady state solution to one where one or the other pathway dominates ¯Ω α = S α ,so one can see the result already in the phase portrait that at larger depths the contribution to ¯Ω α willcome from a single pathway.

Original: Asymptotic approximation While the result is already qualitatively clear from the phase portrait,we can obtain an analytic statement.
Revised: Asymptotic approximation While the result is already qualitatively clear from the phase portrait,we can obtain an analytic statement.

Original: For a more easily interpretable approach than analyzinghypergeometric functions (and a way of seeing the asymptotic behavior of the above equation),consider the case where S α is sufficiently greater than 1 so that we can approximate
Revised: For a more easily interpretable approach than analyzinghypergeometric functions (and a way of seeing the asymptotic behavior of the above equation),consider the case where S α is sufficiently greater than 1 so that we can approximate

Original: [Equation]
Revised: [Equation]

Original: (One can check this with a post-hoc self consistency condition.
Revised: (One can check this with a post-hoc self consistency condition.

Original: For the above relation to hold, S αwill need to be large.)
Revised: For the above relation to hold, S αwill need to be large.)

Original: Then we have
Revised: Then we have

Original: [Equation]
Revised: [Equation]

Original: 24) which implies
Revised: 24) which implies

Original: [Equation]
Revised: [Equation]

Original: where C abα is a constant that depends upon initial conditions and which further implies
Revised: where C abα is a constant that depends upon initial conditions and which further implies

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: When D a , D b are large we have
Revised: When D a , D b are large we have

Original: [Equation]
Revised: [Equation]

Original: 27) for some C ′ abα .
Revised: 27) for some C ′ abα .

Original: In steady state, we have S α = (cid:80) a K aα .
Revised: In steady state, we have S α = (cid:80) a K aα .

Original: Consider S ′
Revised: Consider S ′

Original: α = S
Revised: α = S

Original: α − (cid:80) c
Revised: α − (cid:80) c

Original: ̸
Revised: ̸

Original: = a,b K cα =
Revised: = a,b K cα =

Original: K aα + K bα .
Revised: K aα + K bα .

Original: Then in the large depth limit we can write for K bα
Revised: Then in the large depth limit we can write for K bα

Original: [Equation]
Revised: [Equation]

Original: 28) the solutions of which are given by the quadratic equation
Revised: 28) the solutions of which are given by the quadratic equation

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: which are
Revised: which are

Original: [Equation]
Revised: [Equation]

Original: In the limit | C ′ abα | → ∞ and in the domain K bα ∈ [0 , S ′
Revised: In the limit | C ′ abα | → ∞ and in the domain K bα ∈ [0 , S ′

Original: α ] , these will be
Revised: α ] , these will be

Original: [Equation]
Revised: [Equation]

Original: (31) and thus one pathway or the other dominates the singular value α for each α , depending upon thesign of C abα .
Revised: (31) and thus one pathway or the other dominates the singular value α for each α , depending upon thesign of C abα .

Original: As mentioned above, in the strict limit the constant C abα → 0 , leaving symmetricsolutions across the pathways.
Revised: As mentioned above, in the strict limit the constant C abα → 0 , leaving symmetricsolutions across the pathways.

Original: In the finite size limit, fluctuations in the initial state will drive C abαto larger values, biasing the network to one pathway or the other.
Revised: In the finite size limit, fluctuations in the initial state will drive C abαto larger values, biasing the network to one pathway or the other.

Original: We note that in a real application,initial conditions are not necessarily drawn from the distributions that we assume, and we expecteffectively larger initial fluctuations than our initial state would imply, e.g. by choosing the initialvalues of each weight matrix from N (0 , σ ) , rather than scaling with network size (see the examplesin Simulations).
Revised: We note that in a real application,initial conditions are not necessarily drawn from the distributions that we assume, and we expecteffectively larger initial fluctuations than our initial state would imply, e.g. by choosing the initialvalues of each weight matrix from N (0 , σ ) , rather than scaling with network size (see the examplesin Simulations).

Original: Such larger fluctuations would only enhance the effect of the symmetry breaking.
Revised: Such larger fluctuations would only enhance the effect of the symmetry breaking.

Original:  Simulations
Revised:  Simulations

Original: We demonstrate our results with numerical simulations of networks with two pathways and multipledepths.
Revised: We demonstrate our results with numerical simulations of networks with two pathways and multipledepths.

Original: For these examples we use the same number of layers per pathway and N 1 = N 2 = 1000 .
Revised: For these examples we use the same number of layers per pathway and N 1 = N 2 = 1000 .

Original: Theinitial state of the weight matrices is drawn from a zero mean normal distribution with a fixed standarddeviation σ = 0 .
Revised: Theinitial state of the weight matrices is drawn from a zero mean normal distribution with a fixed standarddeviation σ = 0 .

Original: The input vectors x are 8-dimensional and are the rows of the 8-dimensionalidentity matrix.
Revised: The input vectors x are 8-dimensional and are the rows of the 8-dimensionalidentity matrix.

Original: The output vectors y are 15-dimensional and are the rows of the matrix:
Revised: The output vectors y are 15-dimensional and are the rows of the matrix:

Original: [Equation]
Revised: [Equation]

Original:    1 1 0 1 0 0 0 1 0 0 0 0 0 01 1 0 1 0 0 0 0 1 0 0 0 0 01 1 0 0 1 0 0 0 0 1 0 0 0 01 1 0 0 1 0 0 0 0 0 1 0 0 01 0 1 0 0 1 0 0 0 0 0 1 0 01 0 1 0 0 1 0 0 0 0 0 0 1 01 0 1 0 0 0 1 0 0 0 0 0 0 11 0 1 0 0 0 1 0 0 0 0 0 

Revised:    1 1 0 1 0 0 0 1 0 0 0 0 0 01 1 0 1 0 0 0 0 1 0 0 0 0 01 1 0 0 1 0 0 0 0 1 0 0 0 01 1 0 0 1 0 0 0 0 0 1 0 0 01 0 1 0 0 1 0 0 0 0 0 1 0 01 0 1 0 0 1 0 0 0 0 0 0 1 01 0 1 0 0 0 1 0 0 0 0 0 0 11 0 1 0 0 0 1 0 0 0 0 0

Original: 0   (32)
Revised: 0   (32)

Original: Gradient descent is performed over 1000 epochs with learning rate l r = 0 .
Revised: Gradient descent is performed over 1000 epochs with learning rate l r = 0 .

Original: These simulations arenot compute intensive and are easily performed on a standard modern desktop or laptop.
Revised: These simulations arenot compute intensive and are easily performed on a standard modern desktop or laptop.

Original: All code forsimulations and figures will be available on GitHub.
Revised: All code forsimulations and figures will be available on GitHub.

Original: Figure (4) shows an example of the matrix K afor a specific pathway a , which is initialized as a uniform random matrix, becoming diagonal in thesingular value space.
Revised: Figure (4) shows an example of the matrix K afor a specific pathway a , which is initialized as a uniform random matrix, becoming diagonal in thesingular value space.

Original: Figure 5 shows an example of learning dynamics for networks with differentdepths.
Revised: Figure 5 shows an example of learning dynamics for networks with differentdepths.

Original: For the network of depth 2 (for which “pathway” is an arbitrary concept, as there is onlyone hidden layer), the singular values are more or less divided evenly across the pathways.
Revised: For the network of depth 2 (for which “pathway” is an arbitrary concept, as there is onlyone hidden layer), the singular values are more or less divided evenly across the pathways.

Original: Thissymmetry is already broken for networks of depths 3 and 4.
Revised: Thissymmetry is already broken for networks of depths 3 and 4.

Original: Even by depth 7, the largest singularvalues are concentrated on one or the other pathway.
Revised: Even by depth 7, the largest singularvalues are concentrated on one or the other pathway.

Original: One can see that these results are essentiallyrandom by considering different seeds, which is shown in Figure 6 with two different initializationsof a depth 7 network.
Revised: One can see that these results are essentiallyrandom by considering different seeds, which is shown in Figure 6 with two different initializationsof a depth 7 network.

Original: One can see that the largest singular values are typically in one pathway or theother, but not always the same pathway.
Revised: One can see that the largest singular values are typically in one pathway or theother, but not always the same pathway.

Original: [Figure] [Figure]
Revised: Importantly, our analytic results apply strictly to the linearized network. Since the standard situationwith deep networks is to use some form of nonlinear activation we test our results empirically usingeither of Tanh or ReLU. The general trend remains the same when using these nonlinearities.

Original: [Table] +WjGMfND7Eve4xSNlfz7TornQRtFPMBJp1B0S+4MZJl4GSlChmqn8NXuRjQJmTRUoNYtz42Nn6IynAo2ybcTzWKkQ+yzlqUSQ6b9dHb0hJxapUt6kbIlDZmpvydSDLUeh4HtDNEM9KI3Ff/zWonpXfspl3FimKTzRb1EEBORaQKkyxWjRowtQaq4vZXQASqkxuaUtyF4iy8vk/pFybsslR/KxcpNFkcOjuEEzsCDK6jAHVShBhSe4Ble4c0ZOS/Ou/Mxb11xspkj+APn8weLapH4</latexit [Figure] [Table]
Revised: Theresults are shown in Appendix A.

Original: [Figure]
Revised: [Figure]

Original: K a and K b in the singular value space.

Original: Note that in the different initializations, the final assignment of the features to each pathway is random.

Original:  Discussion
Revised:  Discussion

Original: Many biological neural systems have several computational pathways that are thought to mediatedifferent aspects of sensory processing, prominant examples being the “what" and “where” pathwaysof the macaque visual system [4, 7, 10], the “local” and “global” visual motion pathways [5], and theparallel anatomical pathways in the mouse [8].
Revised: Many biological neural systems have several computational pathways that are thought to mediatedifferent aspects of sensory processing, prominant examples being the “what" and “where” pathwaysof the macaque visual system [4, 7, 10], the “local” and “global” visual motion pathways [5], and theparallel anatomical pathways in the mouse [8].

Original: Here we have shown, using the linearized frameworkof [14], that depth alone can have a profound impact upon the learning dynamics in systems withmultiple channels by separating features in a learned data set by pathway.
Revised: Here we have shown, using the linearized frameworkof [14], that depth alone can have a profound impact upon the learning dynamics in systems withmultiple channels by separating features in a learned data set by pathway.

Original: While we have used a very specific initialization scheme for mathematical purposes, we posit that(and have provided an example in which) our results are more generally applicable.
Revised: While we have used a very specific initialization scheme for mathematical purposes, we posit that(and have provided an example in which) our results are more generally applicable.

Original: In some casesthis is easy to show as an extension of our work.
Revised: In some casesthis is easy to show as an extension of our work.

Original: This work is an analysis of the analytic properties ofdeep networks in the infinite width limit.
Revised: This work is an analysis of the analytic properties ofdeep networks in the infinite width limit.

Original: As such, care must be taken in considering these results inthe context of systems in which corrections due to finite size are large.
Revised: As such, care must be taken in considering these results inthe context of systems in which corrections due to finite size are large.

Original: The assumption of linearityis also important.
Revised: The assumption of linearityis also important.

Original: Deep networks in practice have non-linearities, which may be responsible fordeviations from the results described here. 
Revised: Deep networks in practice have non-linearities, which may be responsible fordeviations from the results described here. [Table]

Original: One obvious reason for an architecture with multiple pathways is to create divergent representationsfor different cost functions from a common set of lower level features.
Revised: /KxcpNFkcOjuEEzsCDK6jAHVShBhSe4Ble4c0ZOS/Ou/Mxb11xspkj+APn8weLapH4</latexit> K a,b↵ <latexit sha1_base64="SLUzUIPxWVGCgF6jdV0IlXvxb8A=">AAAB9HicbVDLSgNBEOz1GeMr6tHLYBA8SNiVoB6DXgQvEcwDkiX0TibJkNnZdWY2EJZ8hxcPinj1Y7z5N06SPWhiQUNR1U13VxALro3rfjsrq2vrG5u5rfz2zu7efuHgsK6jRFFWo5GIVDNAzQSXrGa4EawZK4ZhIFgjGN5O/caIKc0j+WjGMfND7Eve4xSNlfz7TornQRtFPMBJp1B0S+4MZJl4GSlChmqn8NXuRjQJmTRUoNYtz42Nn6IynAo2ybcTzWKkQ+yzlqUSQ6b9dHb0hJxapUt6kbIlDZmpvydSDLUeh4HtDNEM9KI3Ff/zWonpXfspl3FimKTzRb1EEBORaQKkyxWjRowtQaq4vZXQASqkxuaUtyF4iy8vk/pFybsslR/KxcpNFkcOjuEEzsCDK6jAHVShBhSe4Ble4c0ZOS/Ou/Mxb11xspkj+APn8weLapH4</latexit [Figure] [Figure] [Equation]

Revised: [Figure]

Revised: [Equation]

Revised: [Figure]

Revised: [Equation]

Revised: [Figure]

Revised: [Equation]

Revised: [Figure]

Revised: [Table]

Revised: vz8vVa

Revised: [Figure]

Revised: zOPJwBMdwCh5UoAp3UIM6MOjDM7zCmyOcF

Revised: [Figure]

Revised: fd+Zi35pxs5hD+wPn8AZTFjVk

Revised: [Figure]

Revised: </latexit

Revised: [Figure]

Revised: <

Revised: [Figure]

Revised: _base

Revised: [Figure]

Revised: "rbFGjpjxLrxjBxlU5j1N2CEoEF

Revised: [Figure]

Revised: [Figure]

Revised: 0NBxqhjWWSxi1QqoRsEl1g03AluJQhoFApvB8GbqN59QaR7LRzNK0I9oX

Revised: [Figure]

Revised: Or2YpRFKwwTVuu25ifHHVBnOBE4KnVRjQtmQ9rFtqaQRan88O3VCTqzSI2GsbElDZurviTGNtB5Fge2MqBnoRW8q

Revised: [Figure]

Revised: One obvious reason for an architecture with multiple pathways is to create divergent representationsfor different cost functions from a common set of lower level features.

Original: In our case we consider analternative in which all pathways subserve the same cost function.
Revised: In our case we consider analternative in which all pathways subserve the same cost function.

Original: If the result is to separate features,why would such an architecture be desirable?
Revised: If the result is to separate features,why would such an architecture be desirable?

Original: One answer may be that multiple pathways would serveas a form of regularization.
Revised: One answer may be that multiple pathways would serveas a form of regularization.

Original: Not only does this reduce the space of model complexity, but through themechanism we have demonstrated this can also lead to a separation of features, and perhaps moreeasily learnable representations.
Revised: Not only does this reduce the space of model complexity, but through themechanism we have demonstrated this can also lead to a separation of features, and perhaps moreeasily learnable representations.

Original: As this is an analytic approach to the theory of deep neural networks from a dynamical systemsperspective, we anticipate no negative societal impacts.
Revised: As this is an analytic approach to the theory of deep neural networks from a dynamical systemsperspective, we anticipate no negative societal impacts.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes](b) Did you describe the limitations of your work?

Original: [Yes]
Revised: [Yes](c)

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: Did you discuss any potential negative societal impacts of your work?

Original: [Yes] (d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
Revised: [Yes](d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?

Original: [Yes]
Revised: [Yes]2.

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] (b) Did you include complete proofs of all theoretical results?
Revised: [Yes](b) Did you include complete proofs of all theoretical results?

Original: [Yes]
Revised: [Yes]3.

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] Not provided at initial submission due to double-blind review, but a GitHub link will be provided in the final paper.
Revised: [Yes] Not provided atinitial submission due to double-blind review, but a GitHub link will be provided in thefinal paper.(b)

Original: (b) Did you specify all the training details (e
Revised: Did you specify all the training details (e

Original: g., data splits, hyperparameters, how they were chosen)?
Revised: .g. , data splits, hyperparameters, how theywere chosen)

Original: [Yes]
Revised: [Yes

Original: (c) Did you report error bars (e
Revised: ](c) Did you report error bars (e.g.,

Original: g., with respect to the random seed after running experiments multiple times)?
Revised: with respect to the random seed after running experiments multiple times

Original: [N/A]
Revised: [N/A]

Revised: d

Revised: ) Did you

Original: (d) Did you include the total amount of compute and the type of resources used (e
Revised: include the total amount of compute and the type of resources used (

Original: g., typeof GPUs, internal cluster, or cloud provider)?
Revised: e .g. type of GPUs, internal cluster, or cloud provider

Original: [Yes]
Revised: [Yes]

Original: If you are using existing assets (e
Revised: If you are using existing assets ( e

Original: g., code, data, models) or curating/releasing new assets...
Revised: g ., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: a ) If your work uses existing assets, did you cite the creators?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you mention the license of the assets?
Revised: b ) Did you mention the license of the assets?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
Revised: [N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?

Original: [N/A]
Revised: [N/A](e)

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: Did you discuss whether the data you are using/curating contains personally identifiableinformation or offensive content?

Original: information

Original: or offensive

Original: content

Original: [N/A]
Revised: [N/A]5.

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?

Original: [N/A]
Revised: [N/A](b)

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A](c)

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
Revised: Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?

Original: [N/A]
Revised: [N/A]

Revised: A Appendix: Simluation results with nonlinear activation functions.

Revised: Figures 7 and 8 show simulations results similar to those in Figure 5 with the nonlinearities ReLU and

Revised: Tanh, respectively.

Revised: Simulation parameters are as in Section 4 with the exception that the correspondingnonlinearity is used as an activation function following each layer of each pathway.

Revised: In addition, fordepths 2,3, and 4 we simulate 10000 epochs, whereas for depth 7 we simulation 20000 epochs.

Revised: [Table]

Revised: zOPJwBMdwCh5UoAp3UIM6MOjDM7zCmyOcF+fd+Zi35pxs5hD+wPn8AZTFjVk=</latexit> D =

Revised: [Figure]

Revised: [Table]

Revised: =">AAAB6nicbVDLSgNBEOyNrxhfUY9eBoPgKeyKGC9CUA8eI5oHJEuYnfQmQ2Znl5lZIYR8ghcPinj1i7z5N06SPWhiQUNR1U13V5AIro3rfju5ldW19Y38ZmFre2d3r7h/0NBxqhjWWSxi1QqoRsEl1g03AluJQhoFApvB8GbqN59QaR7LRzNK0I9oX/KQM2qs9HB7VekWS27ZnYEsEy8jJchQ6xa/Or2YpRFKwwTVuu25ifHHVBnOBE4KnVRjQtmQ9rFtqaQRan88O3VCTqzSI2GsbElDZurviTGNtB5Fge2MqBnoRW8q/ue1UxNe+mMuk9SgZPNFYSqIicn0b9LjCpkRI0soU9zeStiAKsqMTadgQ/AWX14mjbOyd1E+vz8vVa+zOPJwBMdwCh5UoAp3UIM6MOjDM7zCmyOcF+fd+Zi35pxs5hD+wPn8AZTFjVk=</latexit> D =

Revised: [Figure]