Original: I NTERNEURONS ACCELERATE LEARNING DYNAMICS IN RECURRENT NEURAL NETWORKS FOR STATISTICAL ADAPTATION
Revised: I NTERNEURONS ACCELERATE LEARNING DYNAMICS IN RECURRENT NEURAL NETWORKS FOR STATISTICAL ADAPTATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Early sensory systems in the brain rapidly adapt to fluctuating input statistics, which requires recurrent communication between neurons.
Revised: Early sensory systems in the brain rapidly adapt to fluctuating input statistics, which requires recurrent communication between neurons.

Original: Mechanistically, such recurrent communication is often indirect and mediated by local interneurons.
Revised: Mechanistically, such recurrent communication is often indirect and mediated by local interneurons.

Original: In this work, we explore the computational benefits of mediating recurrent communication via interneurons compared with direct recurrent connections.
Revised: In this work, we explore the computational benefits of mediating recurrent communication via interneurons compared with direct recurrent connections.

Original: To this end, we consider two mathematically tractable recurrent linear neural networks that statistically whiten their inputs — one with direct recurrent connections and the other with interneurons that mediate recurrent communication.
Revised: To this end, we consider two mathematically tractable recurrent linear neural networks that statistically whiten their inputs — one with direct recurrent connections and the other with interneurons that mediate recurrent communication.

Original: By analyzing the corresponding continuous synaptic dynamics and numerically simulating the networks, we show that the network with interneurons is more robust to initialization than the network with direct recurrent connections in the sense that the convergence time for the synaptic dynamics in the network with interneurons (resp.
Revised: By analyzing the corresponding continuous synaptic dynamics and numerically simulating the networks, we show that the network with interneurons is more robust to initialization than the network with direct recurrent connections in the sense that the convergence time for the synaptic dynamics in the network with interneurons (resp.

Original: direct recurrent connections) scales logarithmically (resp. linearly) with the spectrum of their initialization.
Revised: direct recurrent connections) scales logarithmically (resp. linearly) with the spectrum of their initialization.

Original: Our results suggest that interneurons are computationally useful for rapid adaptation to changing input statistics.
Revised: Our results suggest that interneurons are computationally useful for rapid adaptation to changing input statistics.

Original: Interestingly, the network with interneurons is an overparameterized solution of the whitening objective for the network with direct recurrent connections, so our results can be viewed as a recurrent linear neural network analogue of the implicit acceleration phenomenon observed in overparameterized feedforward linear neural networks.
Revised: Interestingly, the network with interneurons is an overparameterized solution of the whitening objective for the network with direct recurrent connections, so our results can be viewed as a recurrent linear neural network analogue of the implicit acceleration phenomenon observed in overparameterized feedforward linear neural networks.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Efficient coding and redundancy reduction theories of neural coding hypothesize that early sensory systems decorrelate and normalize neural responses to sensory inputs (Barlow, 1961; Laughlin, 1989; Barlow & F ¨ oldi ´ ak, 1989; Simoncelli & Olshausen, 2001; Carandini & Heeger, 2012; Westrick et al., 2016), operations that are closely related to statistical whitening of inputs.
Revised: Efficient coding and redundancy reduction theories of neural coding hypothesize that early sensory systems decorrelate and normalize neural responses to sensory inputs (Barlow, 1961; Laughlin, 1989; Barlow & F ¨ oldi ´ ak, 1989; Simoncelli & Olshausen, 2001; Carandini & Heeger, 2012; Westrick et al., 2016), operations that are closely related to statistical whitening of inputs.

Original: Since the input statistics are often in flux due to dynamic environments, this calls for early sensory systems that can rapidly adapt (Wark et al., 2007; Whitmire & Stanley, 2016).
Revised: Since the input statistics are often in flux due to dynamic environments, this calls for early sensory systems that can rapidly adapt (Wark et al., 2007; Whitmire & Stanley, 2016).

Original: Decorrelating neural activities requires recurrent communication between neurons in a population, which is typically indirect and mediated by local interneurons (Christensen et al., 1993;
Revised: Decorrelating neural activities requires recurrent communication between neurons in a population, which is typically indirect and mediated by local interneurons (Christensen et al., 1993;

Original: Shepherd et al., 2004).
Revised: Shepherd et al., 2004).

Original: Why do neuronal circuits for statistical adaptation mediate recurrent communication using interneurons, which take up valuable space and metabolic resources, rather than using direct recurrent connections?
Revised: Why do neuronal circuits for statistical adaptation mediate recurrent communication using interneurons, which take up valuable space and metabolic resources, rather than using direct recurrent connections?

Original: A common explanation for why communication between neurons is mediated by interneurons is
Revised: A common explanation for why communication between neurons is mediated by interneurons is

Original: Dale’s principle, which states that each neuron only produces one type of neurotransmitter with exclusively inhibitory or excitatory effects on all of its targets (Strata & Harvey, 1999).
Revised: Dale’s principle, which states that each neuron only produces one type of neurotransmitter with exclusively inhibitory or excitatory effects on all of its targets (Strata & Harvey, 1999).

Original: Therefore, an excitatory neuron cannot directly inhibit another neuron; rather, the inhibition must be mediated by an inhibitory interneuron.
Revised: Therefore, an excitatory neuron cannot directly inhibit another neuron; rather, the inhibition must be mediated by an inhibitory interneuron.

Original: While Dale’s principle provides a physiological constraint that explains why recurrent interactions are mediated by interneurons, we seek a computational principle that can account for using interneurons rather than direct recurrent connections.
Revised: While Dale’s principle provides a physiological constraint that explains why recurrent interactions are mediated by interneurons, we seek a computational principle that can account for using interneurons rather than direct recurrent connections.

Original: This perspective is useful for a couple of reasons.
Revised: This perspective is useful for a couple of reasons.

Original: First, perhaps Dale’s principle is not a hard constraint and biology could have found an efficient mechanism for a neuron to produce more than one type of neurotransmitter.
Revised: First, perhaps Dale’s principle is not a hard constraint and biology could have found an efficient mechanism for a neuron to produce more than one type of neurotransmitter.

Original: In this case, a computational benefit of interneurons would provide a normative explanation for the existence of interneurons to mediate recurrent communication.
Revised: In this case, a computational benefit of interneurons would provide a normative explanation for the existence of interneurons to mediate recurrent communication.

Original: Second,  whitening is useful in statistical and machine learning methods (Hyv¨arinen & Oja, 2000; Krizhevsky, 2009; Ermolov et al., 2021), so our analysis is potentially relevant to the design of artificial neural networks and neuromorphic systems, which are not bound by Dale’s principle.
Revised: Second, decorrelation and whitening are useful in statistical and machine learning methods (Hyv¨arinen & Oja, 2000; Krizhevsky, 2009), especially recent self-supervised methods (Ermolov et al., 2021; Zbontar et al., 2021; Bardes et al., 2021; Hua et al., 2021), so our analysis is potentially relevant to the design of artificial neural networks and neuromorphic systems, which are not bound by Dale’s principle.

Original: In this work, to better understand the computational benefits of interneurons for statistical adaptation, we analyze the learning dynamics of two mathematically tractable recurrent neural networks that  whiten their inputs using Hebbian/anti-Hebbian learning rules — one with direct recurrent connections and the other with indirect recurrent interactions mediated by interneurons, Figure 1.
Revised: In this work, to better understand the computational benefits of interneurons for statistical adaptation, we analyze the learning dynamics of two mathematically tractable recurrent neural networks that statistically whiten their inputs using Hebbian/anti-Hebbian learning rules — one with direct recurrent connections and the other with indirect recurrent interactions mediated by interneurons, Figure 1.

Original: We show that the learning dynamics of the network with interneurons are more robust than the learning dynamics of the network with direct recurrent connections.
Revised: We show that the learning dynamics of the network with interneurons are more robust than the learning dynamics of the network with direct recurrent connections.

Original: In particular, we prove that the convergence time of the continuum limit of the network with direct lateral connections scales linearly with the spectrum of the initialization, whereas the convergence time of the continuum limit of the network with interneurons scales logarithmically with the spectrum of the initialization.
Revised: In particular, we prove that the convergence time of the continuum limit of the network with direct lateral connections scales linearly with the spectrum of the initialization, whereas the convergence time of the continuum limit of the network with interneurons scales logarithmically with the spectrum of the initialization.

Original: We also numerically test the networks and, consistent with the theoretical results, find that the network with interneurons is more robust to initialization.
Revised: We also numerically test the networks and, consistent with our theoretical results, find that the network with interneurons is more robust to initialization.

Original: Our results suggest that interneurons are computationally important for rapid adaptation to fluctuating input statistics.
Revised: Our results suggest that interneurons are computationally important for rapid adaptation to fluctuating input statistics.

Original: Our analysis is closely related to analyses of learning dynamics in feedforward linear networks trained using backpropagation (Saxe et al., 2014; Arora et al., 2018; Saxe et al., 2019; Gidel et al., 2019;
Revised: Our analysis is closely related to analyses of learning dynamics in feedforward linear networks trained using backpropagation (Saxe et al., 2014; Arora et al., 2018; Saxe et al., 2019; Gidel et al., 2019;

Original: Tarmoun et al., 2021).
Revised: Tarmoun et al., 2021).

Original: The optimization problems for deep linear networks are overparameterizations of linear problems and this overparameterization can accelerate convergence of gradient descent or gradient flow optimization — a phenomenon referred to as implicit acceleration .
Revised: The optimization problems for deep linear networks are overparameterizations of linear problems and this overparameterization can accelerate convergence of gradient descent or gradient flow optimization — a phenomenon referred to as implicit acceleration (Arora et al., 2018).

Original: Our results can be viewed as an analogous phenomenon forthe gradient flows corresponding to recurrent linear networks trained using Hebbian/anti-Hebbian learning rules.
Revised: Our results can be viewed as an analogous phenomenon for gradient flows corresponding to recurrent linear networks trained using Hebbian/anti-Hebbian learning rules.

Original: In our setting, the network with interneurons is naturally viewed as an overparamterized solution of the whitening objective for the network with direct recurrent connections.
Revised: In our setting, the network with interneurons is naturally viewed as an overparameterized solution of the whitening objective for the network with direct recurrent connections.

Original: In analogy with the feedforward setting, the interneurons can be viewed as a hidden layer that overparameterizes the optimization problem.
Revised: In analogy with the feedforward setting, the interneurons can be viewed as a hidden layer that overparameterizes the optimization problem.

Revised: In summary, our main contribution is a theoretical and numerical analysis of the synaptic dynamics of two linear recurrent neural networks for statistical whitening — one with direct lateral connections and one with indirect lateral connections mediated by interneurons.

Revised: Our analysis shows that the synaptic dynamics converge significantly faster in the network with interneurons than the network with direct lateral connections (logarithmic versus linear convergence times).

Revised: Our results have potential broader implications: (i) they suggest biological interneurons may facilitate rapid statistical adaptation; (ii) including interneurons in recurrent neural networks for solving other learning tasks may also accelerate learning; (iii) overparameterized whitening objectives may be useful for developing online self-supervised learning algorithms in machine learning.

Original:  S TATISTICAL WHITENING
Revised:  S TATISTICAL WHITENING

Original: Let x 1 , . . .
Revised: Let x 1 , . . .

Original: , x T be a sequence of n -dimensional centered inputs, for some n ≥ 1 , with positive definite (empirical) covariance matrix C xx :
Revised: , x T be a sequence of n -dimensional centered inputs, for some n ≥ 2 , with positive definite (empirical) covariance matrix C xx :

Original: = 1 T XX ⊤ , where X :=
Revised: = 1 T XX ⊤ , where X :=

Original: [ x 1 , . .
Revised: [ x 1 , . .

Original: , x T ] is the n × T data matrix of concatenated inputs.
Revised: , x T ] is the n × T data matrix of concatenated inputs.

Original: Whitening is a linear transformation of  the inputs to have identity covariance matrix. In particular, an n × n matrix F is a whitening matrix if the  outputs yt : = Fx t , for t = 1 , . . .
Revised: The goal of statistical whitening is to linearly transform the inputs so that the n -dimensional outputs y 1 , . . .

Original: ,  T, have identity covariance matrix; that is, C yy := 1 T YY ⊤ =
Revised: , y T have identity covariance; that is, C yy := 1 T YY ⊤ =

Original: I n , where Y :=
Revised: I n , where Y :=

Original: [ y 1 , . . .
Revised: [ y 1 , . . .

Original: , y T ]= FX is the  data matrix of concatenated outputs and I n denotes the n × n identity matrix.
Revised: , y T ] is the n × T data matrix of concatenated outputs.

Original: Whitening is not a unique transformation.
Revised: Statistical whitening is not a unique transformation.

Original: Let X = U x ( √ T S x ) V ⊤ x be the singular value decomposition of X , where U x is an n × northogonal matrix, √ T S x = diag ( √ Tλ 1 , . . . , √ Tλ n ) is the n × n diagonal matrix of singular values and V x is an T × n matrix with orthonormal column vectors. Then every whitening matrix F is of the form F = QS − 1 x U ⊤ x , where Q is an arbitrary n × n orthogonal matrix  (Kessy et al., 2018).
Revised: For example, if C yy = I n , then left multiplication of Y by any n × n orthogonal matrix results in another data matrix with identity covariance (Kessy et al., 2018).

Original: We focus on ZCA whitening (Bell & Sejnowski, 1997), also referred to as Mahalanobis whitening, which corresponds to the case Q =
Revised: We focus on Zero-phase Components Analysis (ZCA) whitening (Bell & Se- jnowski, 1997), also referred to as Mahalanobis whitening, given by Y

Original: U x , so that [Equation] ZCA whitening is a common choice of whitening transformationused in statistical analysis and machine learning (Krizhevsky, 2009; Huang et al., 2018; Ermolov et al., 2021) that minimizes the mean-square error between  X and  Y (Eldar & Oppenheim, 2003).
Revised: = C − 1 / 2 xx X , which is the whitening transformation that minimizes the mean-squared error between the inputs X and the outputs Y (Eldar & Oppenheim, 2003).

Original: In other words, Y ZCA is the unique solution to the objective 
Revised: In other words, it is the unique solution to the objective [Equation] Y

Original: [Equation]
Revised: [Equation]

Original: subject to
Revised: subject to

Original: [Equation]
Revised: [Equation]

Original: The goal of this work is a theoretical analysis of 2 recurrent neural networks that  perform ZCA whitening in the online or streaming setting using Hebbian/anti-Hebbian learning rules.
Revised: The goal of this work is to analyze the learning dynamics of 2 recurrent neural networks that learn to perform ZCA whitening in the online, or streaming, setting using Hebbian/anti-Hebbian learning rules.

Original: The derivations of the 2neural networks, which we include here for completeness, are closely related to derivations of PCA networks carried out in (Pehlevan & Chklovskii, 2015; Pehlevan et al., 2018).
Revised: The derivations of the 2 networks, which we include here for completeness, are closely related to derivations of PCA networks carried out in (Pehlevan & Chklovskii, 2015; Pehlevan et al., 2018).

Original: Our main theoretical and numerical results are presented in sections 6 and 7, respectively.
Revised: Our main theoretical and numerical results are presented in sections 6 and 7, respectively.

Original:  O BJECTIVES FOR DERIVING ZCA WHITENING NETWORKS
Revised:  O BJECTIVES FOR DERIVING ZCA WHITENING NETWORKS

Original: In this section, we rewrite the ZCA whitening objective 1 to obtain 2 objectives that will be the starting points for deriving our 2 recurrent whitening networks.
Revised: In this section, we rewrite the ZCA whitening objective 1 to obtain 2 objectives that will be the starting points for deriving our 2 recurrent whitening networks.

Original: 3. O BJECTIVE FOR THE NETWORK WITH DIRECT RECURRENT CONNECTIONS

Original: To derive a network with direct recurrent connections, we rewrite objective 1 to include the synaptic weight matrix of direct recurrent connections − M .

Original: Expanding the square in equation 1, substituting in with the constraint YY ⊤
Revised: We first expand the square in equation 1, substitute in with the constraint YY ⊤

Original: =
Revised: =

Original: T I n , dropping the terms that do not depend on Y and flipping the sign yields the following objective in terms of the neural activities data matrix Y :
Revised: T I n , drop the terms that do not depend on Y and finally flip the sign to obtain the following objective in terms of the neural activities data matrix Y :

Original: [Equation]
Revised: [Equation]

Original:  to
Revised: subject to

Original: [Equation]
Revised: [Equation]

Revised: 3. O BJECTIVE FOR THE NETWORK WITH DIRECT RECURRENT CONNECTIONS

Original: Next, we introduce the positive definite matrix M as a Lagrange multiplier to enforce  YY ⊤ ⪯ T I n :
Revised: To derive a network with direct recurrent connections, we introduce the positive definite matrix M as a Lagrange multiplier to enforce the upper bound YY ⊤ ⪯ T I n :

Original: [Equation]
Revised: [Equation]

Original: where S n ++ denotes the set of positive definite n × n matrices.
Revised: where S n ++ denotes the set of positive definite n × n matrices and

Original: The maximization over Y ensures that the upper bound YY ⊤ ⪯ T I n is in fact saturated.
Revised: [Equation]

Original: Since f ( M , Y ) is linear in M and strongly concave in Y ,  it satisfies the saddle point property (see Appendix A), so we can interchange the order of optimization.
Revised: Here, we have interchanged the order of optimization, which is justified because f ( M , Y ) is linear in M and strongly concave in Y , so it satisfies the saddle point property (see Appendix A).

Original: This results in the minimax objective
Revised: The maximization over Y ensures that the upper bound

Original: [Equation] which is a function of neural activities Y and synaptic weights M , and will be the starting point for our derivation of a  whitening network with direct recurrent connections.
Revised: YY ⊤ ⪯ T I n is in fact saturated, so the whitening constraint holds. The minimax objective equation 2 will be the starting point for our derivation of a statistical whitening network with direct recurrent connections in section 4.

Original: 3. O VERPARAMETERIZED OBJECTIVE FOR THE NETWORK WITH INTERNEURONS
Revised: 3. O VERPARAMETERIZED OBJECTIVE FOR THE NETWORK WITH INTERNEURONS

Original: In order to derive a network with k ≥ n interneurons, whose activities are encoded in the k × T data matrix Z =

Original: [ z 1 , . .

Original: , z T ] , we replace the matrix M in equation 2 with the overparameterizedmatrix product WW ⊤ , where W is an n × k matrix, which yields the minimax objective[Equation] The matrix − W (resp.
Revised: To derive a network with k ≥ n interneurons, we replace the matrix M in equation 2 with the overparameterized product WW ⊤ , where W is an n × k matrix, which yields the minimax objective

Original: W ⊤ ) will correspond to the synaptic weights between the interneurons (resp.

Original: principal neurons) and principal neurons (resp.

Original: interneurons).

Original: Finally, in order to arrive at an objective that also includes the interneuron activities Z we substitute in with the Legendre transform:

Original: [Equation]
Revised: [Equation]

Original: This results in the objective

Original: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: which is a function of neural activities Y , Z and synaptic weights W , and will be the starting point for the derivation of a  whitening network with interneurons.
Revised: The overparameterized minimax objective equation 3 will be the starting point for our derivation of a statistical whitening network with interneurons in section 5.

Original:  D ERIVATION OF A NETWORK WITH DIRECT RECURRENT CONNECTIONS
Revised:  D ERIVATION OF A NETWORK WITH DIRECT RECURRENT CONNECTIONS

Original: Starting from the  whitening objective  2, we derive an online algorithmthat maps onto a network with direct recurrent connections.4. O FFLINE ALGORITHM
Revised: Starting from the ZCA whitening objective in equation 2, we derive offline and online algorithms and we map the online algorithm onto a network with direct recurrent connections.

Original: To derive our algorithms, we assume  therecurrent neural dynamics operate ata much faster timescale than the synaptic updates.
Revised: We assume that the neural dynamics operate at faster timescale than the synaptic updates, so in both algorithms we first optimize over the neural activities and then take gradient-descent steps with respect to the synaptic weight matrix.

Original: In the offline setting, at each iteration, we first maximize f ( M , Y ) with respect to the neural activities Y by taking gradient ascent steps until convergence:
Revised: In the offline setting, at each iteration, we first maximize f ( M , Y ) with respect to the neural activities Y by taking gradient ascent steps until convergence:

Original: [Equation]
Revised: [Equation]

Original: where γ > 0 is a small constant.
Revised: where γ > 0 is a small constant.

Original: After the neural activities converge, we minimize f ( M , Y ) with respect to M by taking a gradient descent step with respect to M :
Revised: After the neural activities converge, we minimize f ( M , Y ) with respect to M by taking a gradient descent step with respect to M :

Original: [Equation]
Revised: [Equation]

Original: 4. O NLINE ALGORITHM

Original: In the online setting, at each time step t , we only have access to the input x t  .
Revised: In the online setting, at each time step t , we only have access to the input x t rather than the whole dataset X .

Original: In this case, we first optimize f ( M , Y ) with respect to the neural activities vector y t by running the fast neural dynamics until convergence:
Revised: In this case, we first optimize f ( M , Y ) with respect to the neural activities vector y t by taking gradient steps until convergence:

Original: [Equation]
Revised: [Equation]

Original: After the neural activities converge, the synaptic weight matrix M is updated by taking a stochastic gradient descent step:
Revised: After the neural activities converge, the synaptic weight matrix M is updated by taking a stochastic gradient descent step:

Original: [Equation]
Revised: [Equation]

Original: This results in Algorithm 1, which can be implemented in a network with direct recurrent connections − M , Figure 1 (left).
Revised: This results in Algorithm 1.

Original: Algorithm input centered inputs { x t } ; parameters γ , η initialize positive definite  M for t = 1 , . .
Revised: input centered inputs { x t } ; parameters γ , η initialize positive definite n × n matrix M for t = 1 , . .

Original: , T do repeat y t ← y t + γ ( x t − My t ) until convergence
Revised: , T do y t ←repeat y t ← y t + γ ( x t − My t ) until convergence

Original: M ← M + η ( y t y ⊤ t − I n ) end for
Revised: M ← M + η ( y t y ⊤ t − I n ) end for

Original: D ERIVATION OF A NETWORK WITH INTERNEURONS
Revised: Algorithm 1 can be implemented in a network with n principal neurons with direct recurrent connections − M , Figure 1 (left). At each time step t , the external input to the principal neurons is x t , the output of the principal neurons is y t and the recurrent input to the principal neurons is − My t .

Original: In this section, starting from the overeparameterized whitening objective 3, we derive an online algorithm that maps onto a network with interneurons.
Revised: Therefore, the total input to the principal neurons is encoded in the n -dimensional vector x t − My t . The neural outputs are updated according to the neural dynamics in Algorithm 1 until they converge at y t

Original: 5. O FFLINE ALGORITHM
Revised: =

Original: We first optimize g ( W , Y , Z ) in the offline setting by alternating optimizing over the neural activities data matrices ( Y , Z ) and taking a gradient descent step with respect to synaptic weight matrix 
Revised: M − 1 x t . After the neural activities converge, the synaptic weight matrix M is updated according to Algorithm 1.  D ERIVATION OF A NETWORK WITH INTERNEURONS Starting from the overparameterized ZCA whitening objective in equation 3, we derive offline and online algorithms and we map the online algorithm onto a network with k ≥ n interneurons.

Original: W .
Revised: As in the last section, we assume that the neural dynamics operate at a faster timescale than the synaptic updates.

Original: We first optimize g ( W , Y, Z ) with respect to the neural activitiesdata matrices ( Y, Z ) by taking gradient steps:
Revised: In the offline setting, at each iteration, we first maximize g ( W , Y ) with respect to the neural activities Y by taking gradient steps until convergence:

Original: [Equation]
Revised: [Equation]

Original: which eventually converge to Y

Original: =

Original: ( WW ⊤ ) − 1 X and Z = W ⊤ ( WW ⊤ )

Original: − 1 X .

Original: After the neural activities  ( Y , Z ) converge, we take a gradient descent step with respect to W :
Revised: After the neural activities converge, we minimize g ( W , Y ) with respect to W by taking a gradient descent step with respect to W :

Original: [Equation]
Revised: [Equation]

Original: In the online setting, at each time step t , we first optimize g ( W , Y, Z ) with respect to the neural activities vectors( y t, z t ) by running the gradient ascent-descent neural dynamics:
Revised: In the online setting, at each time step t , we first optimize g ( W , Y ) with respect to the neural activities vectors y t by running the gradient ascent-descent neural dynamics:

Original: [Equation]
Revised: [Equation]

Original: , which eventually converge to y t

Original: = ( WW ⊤ ) − 1 x t and z t = W ⊤ ( WW ⊤ ) − 1 x t .

Original: After convergence of the neural activities, the matrix W is updated by taking a stochastic gradient descent step:
Revised: After convergence of the neural activities, the matrix W is updated by taking a stochastic gradient descent step:

Original: [Equation]
Revised: [Equation]

Original: This results in Algorithm 2, which can be implemented in a recurrent neural network withrecurrent interactions mediated by interneurons, Figure 1 (right).
Revised: To implement the online algorithm in a recurrent neural network with interneurons, we let z t := W ⊤ y t denote the k -dimensional vector of (hidden) interneuron activities at time t . After substituting the interneuron activities into the neural dynamics and the synaptic update rules, we obtain

Original: Algorithm 2: A with input centered inputs { x t } ;  parameters γ , η initialize full rank  W for t = 1 , . ..
Revised: Algorithm 2. input centered inputs { x t } ; k ≥ n ; parameters γ , η initialize full rank n × k matrix W for t = 1 , . .

Original: , T dorepeat y t  ←  y t + γ ( x t − Wz t)
Revised: , T do y t ←repeat z t ← W ⊤ y t y t ← y t

Original: z t ← z t + γ ( W ⊤ y t − z t ) until convergence 
Revised: + γ ( x t − Wz t ) until convergence W ← W + η ( y t z ⊤ t − W ) end for

Revised: Algorithm 2 can be implemented in a network with n principal neurons and k interneurons, Figure(right).

Original:  W← W + η ( y t z ⊤ t − W ) end for
Revised: The principal neurons (resp. interneurons) are connected to the interneurons (resp. principal neurons) via the synaptic weight matrix W ⊤ (resp. − W ).

Original: A NALYSES OF CONTINUOUS SYNAPTIC DYNAMICS
Revised: At each time step t , the external input to the principal neurons is x t , the activity of the principal neurons is y t , the activity of the interneurons is z t = W ⊤ y t and the recurrent input to the principal neurons is − Wz t . Therefore, the total input to the principal neurons is encoded in the n -dimensional vector x t − Wz t . The neural activities are updated according to the neural dynamics in Algorithm 2 until they converge to y t = ( WW ⊤ ) − 1 x t and z t = W ⊤ y t .

Revised: After the neural activities converge, the synaptic weight matrix W is updated according to Algorithm 2.

Revised: Here, we require that the principal neuron-to-interneuron weight matrix W ⊤ be the negative transpose of the interneuron-to-principal neuron weight matrix − W , Figure 1 (right).

Revised: In general, enforcing such symmetry is not biologically plausible and commonly referred to as the weight transport problem.

Revised: In addition, we do not sign-constrain the weights, so the network can violate Dale’s principle.

Revised: In Appendix B, we modify our algorithm to be more biologically realistic.

Revised:  A NALYSES OF CONTINUOUS SYNAPTIC DYNAMICS

Original: In this section, we present our main theoretical results on  convergence of the  continuous synaptic dynamics for M and W  thatcorrespond to the  updates  in Algorithms 1 and 2, respectively.
Revised: We now present our main theoretical results on the convergence of the corresponding continuous synaptic dynamics for M and W . We first show that the synaptic updates are naturally viewed as (stochastic) gradient descent algorithms. We then analyze the corresponding continuous gradient flows. Detailed proofs of our results are provided in Appendix C

Original: 6. G RADIENT DESCENT ALGORITHMS
Revised: 6. G RADIENT DESCENT ALGORITHMS

Original: We first show that the offlineupdates in equations equation 4 and equation 5 are naturally viewed as gradient descent  algorithms for minimizing objectives U ( M ) and V ( W ) , respectively, where
Revised: We first show that the offline and online synaptic dynamics are naturally viewed as gradient descent and stochastic gradient descent algorithms for minimizing whitening objectives. Let U ( M ) be the convex function defined by

Original: [Equation]
Revised: [Equation]

Original: and V ( W )
Revised: Substituting the optimal neural activities Y

Original: =
Revised: =

Original: U ( WW ⊤ ) is the overparameterization of U ( M ) :
Revised: M − 1 X into the offline update in equation 4, we see that the offline algorithm is a gradient descent algorithm for minimizing U ( M ) :

Original: [Equation]
Revised: [Equation]

Original: In particular, substituting into the offline update equation 4 with the optimal Y = M − 1 X , we see that the offline update for  M is
Revised: By a similar argument, Algorithm 1 is naturally viewed as a stochastic gradient descent algorithm for minimizing U ( M ) . Next, let V ( W ) be the nonconvex function defined by

Original: [Equation]
Revised: [Equation]

Original: Substituting into the offline update equation 5 with the optimal  Y = ( WW ⊤ ) − 1 X and Z = W ⊤ Y , we seethat the offline update for  W is
Revised: Again, substituting the optimal neural activities Y = ( WW ⊤ ) − 1 X into the offline update in equation 5, we see the offline algorithm is a gradient descent algorithm for minimizing V ( W ) :

Original: [Equation]
Revised: [Equation]

Original: Similarly, Algorithms 1 and 2 are naturally viewed as stochastic gradient descent methods for minimizingU ( M ) and V ( W ) , respectively.
Revised: Similarly, Algorithm 2 is a stochastic gradient descent algorithm for minimizing V ( W ) .

Original: A common approach for studying stochastic gradient descent algorithms is to analyze the corresponding continuous gradient flows (Saxe et al., 2014; 2019; Tarmoun et al., 2021), which are more mathematically tractable and are useful approximations of the average behavior of the stochastic gradient descent dynamics when the step size is small.
Revised: A common approach for studying stochastic gradient descent algorithms is to analyze the corresponding continuous gradient flows (Saxe et al., 2014; 2019; Tarmoun et al., 2021), which are more mathematically tractable and are useful approximations of the average behavior of the stochastic gradient descent dynamics when the step size is small.

Original: In the remainder of this section, we analyze and compare the continuous gradient flows associated with Algorithms 1 and 2.
Revised: In the remainder of this section, we analyze and compare the continuous gradient flows associated with Algorithms 1 and 2.

Original: To further facilitate the analysis, wealso consider so called ‘spectral initializations’ that commute withC xx
Revised: To further facilitate the analysis, we consider so-called ‘spectral initializations’ that commute with

Original:  (Saxe et al., 2014; 2019; Gidel et al., 2019; Tarmoun et al., 2021).
Revised: C xx (Saxe et al., 2014; 2019; Gidel et al., 2019; Tarmoun et al., 2021).

Original: Specificially, we say that A 0 ∈ S n ++ is a spectral initialization if A 0 =
Revised: Specificially, we say that A 0 ∈ S n ++ is a spectral initialization if A 0 =

Original: U x diag ( σ 1(0) , . . .
Revised: U x diag ( σ 1 , . . .

Original: , σ n (0)) U ⊤ x , wherewe recall that U x is the n × n orthogonal matrix whose column vectors are the eigenvectors of C xx  .
Revised: , σ n ) U ⊤ x , where U x is the n × n orthogonal matrix of eigenvectors of C xx and σ 1 , . . . , σ n > 0 .

Original: To characterize the convergence rates of the gradient flows, we define the Lyapunov function
Revised: To characterize the convergence rates of the gradient flows, we define the Lyapunov function

Original: [Equation]
Revised: [Equation]

Original: 6. G RADIENT FLOW ANALYSIS OF A LGORITHM
Revised: 6. G RADIENT FLOW ANALYSIS OF A LGORITHM

Original: The gradient flow of U ( M ) is given by
Revised: The gradient flow of U ( M ) is given by

Original: [Equation] 
Revised: [Equation] [Equation]

Original: To analyze solutions of M ( t ) , we focus on spectral intializations.
Revised: To analyze solutions of M ( t ) , we focus on spectral intializations.

Original: Lemma 1.
Revised: Lemma 1.

Original: Suppose M 0 is a spectral initialization.
Revised: Suppose M 0 is a spectral initialization.

Original: Then the solution M ( t ) of the ODE 6 is of the form M ( t ) =
Revised: Then the solution M ( t ) of the ODE 6 is of the form M ( t ) =

Original: U x diag ( σ 1 ( t ) , . . .
Revised: U x diag ( σ 1 ( t ) , . . .

Original: , σ n ( t ))
Revised: , σ n ( t ))

Original: U ⊤ x where σ i ( t ) ,i = 1 , . . . ,  n  , are the solutions of the ODE[Equation]
Revised: U ⊤ x where σ 1 ( t ) , . . . , σ n ( t ) , are the solutions of the ODE

Original: n.

Original: [Equation]
Revised: [Equation]

Original: Consequently,
Revised: Consequently,

Original: [Equation]

Original: Proof.

Original: Let σ 1 ( t ) , . . .

Original: , σ n ( t ) be solutions of the

Original: ODE and define

Original: M ( t ) :=

Original: U x diag ( σ 1 ( t ) , . . .

Original: , σ n ( t ))

Original: U ⊤ x .

Original: Then

Original: [Equation]
Revised: [Equation]

Original: In particular, we see that M ( t ) must be the unique solution of the ODE 6, where uniqueness of solutions follows because the right-hand-side of equation 6 is locally Lipschitz continuous on its domain of definition.

Original: Equation 8 then follows from the ODE 7 and the chain rule.

Original: From Lemma 1, we see that for a spectral initialization with σ i (0) ≤ λ i , the dynamics of σ i ( t ) satisfy
Revised: From Lemma 1, we see that for a spectral initialization with σ i (0) ≤ λ i , the dynamics of σ i ( t ) satisfy

Original: [Equation]
Revised: [Equation]

Original: It follows that σ i ( t ) 2 converges to λ 2 i exponentially with convergence rate greater than 2  √ λ i .
Revised: It follows that σ i ( t ) 2 converges to λ 2 i exponentially with convergence rate greater than 2 / √ λ i .

Original: On the other hand, suppose σ i (0)
Revised: On the other hand, suppose σ i (0)

Original: ≫ λ i .
Revised: ≫ λ i .

Original: From equation 7, we see that while σ i ( t ) ≫ λ i , σ i ( t ) decays at approximately unit rate; that is, 
Revised: From equation 7, we see that while σ i ( t ) ≫ λ i , σ i ( t ) decays at approximately unit rate; that is, dσ i ( t ) dt

Original: [Equation]
Revised: [Equation]

Original:  Therefore, the time for σ i ( t ) to converge to λ i grows linearly with σ i (0) .
Revised: − 1 . Therefore, the time for σ i ( t ) to converge to λ i grows linearly with σ i (0) .

Original: We make these statements precise in the following proposition.
Revised: We make these statements precise in the following proposition.

Original: Proposition 1.
Revised: Proposition 1.

Original: Suppose M 0 is a spectral initialization and let M ( t ) denote the solution of the ODE 6 starting from M 0 .
Revised: Suppose M 0 is a spectral initialization and let M ( t ) denote the solution of the ODE 6 starting from M 0 .

Original: If σ i ≤ λ i for all i = 1 , . . . , n , then for ϵ < ℓ ( M 0 ) ,
Revised: If σ i ≤ λ i for all i = 1 , . . . , n , then for ϵ < ℓ ( M 0 ) ,

Original: [Equation]
Revised: [Equation]

Original: where λ max := max i λ i .
Revised: where λ max := max i λ i .

Original: On the other hand, if σ i > λ i for some i = 1 , . . . , n , then for ϵ < σ 2 i − λ 2 i ,

Original: [Equation]

Original: Proof.

Original: Suppose σ i (0) ≤ λ i for  i = 1 , . . . , n . By equation 8, σ  i ( t ) ≤ λ  i for all t ≥ 0 and so [Equation] [Equation]
Revised: On the other hand, if σ i > λ i for some i = 1 , . . . , n , then for ϵ < σ 2 i − λ 2 i ,

Original: n.

Original: Therefore,

Original: [Equation]
Revised: [Equation]

Original: It follows that equation 10 holds.

Original: Now suppose σ i (0) ≥ λ i for some i = 1 , . . . , n . By Lemma 1, [Equation]
Revised: [Figure]

Original: It follows that equation 11 holds.

Original: 6. G RADIENT FLOW ANALYSIS OF A LGORITHM
Revised: 6. G RADIENT FLOW ANALYSIS OF A LGORITHM

Original: The gradient flow of the overparameterized cost V ( W ) is given by
Revised: The gradient flow of the overparameterized cost V ( W ) is given by

Original: [Equation]
Revised: [Equation]

Original: Given a solution W ( t ) do the ODE 12, define A (t ) : = W ( t ) W ( t ) ⊤  .Then
Revised: Next, we show that ℓ ( W ( t ) W ( t ) ⊤ ) converges to zero exponentially for any initialization W 0 .

Original: [Equation]

Original: Note that the dynamics of A ( t ) do not depend on k ≥ n .

Original: In the following proposition, we show that ℓ ( A ( t )) converges to zero exponentially for any initialization W 0 .

Original: Proposition 2.
Revised: Proposition 2.

Original: Let W ( t ) denote the solution of the ODE 12 starting from any W 0 ∈ R n × k .
Revised: Let W ( t ) denote the solution of the ODE 12 starting from any W 0 ∈ R n × k .

Original: Let ϵ < ℓ ( W 0 W ⊤ 0 ) .Suppose W 0 W ⊤ 0 is a spectral initialization. Then [Equation]
Revised: Let ϵ < ℓ ( W 0 W ⊤ 0 ) .

Original: For general initializations W 0 W ⊤ 0 ,
Revised: For spectral initializations W 0 W ⊤ 0 ,

Original: [Equation]

Original: [Equation]

Original: Proof.

Original: By the chain rule and equation 13, for all t ≥ 0 ,

Original: [Equation]
Revised: [Equation]

Original: Suppose W 0 W ⊤ 0 is a spectral initialization.

Original: Then A ( t ) commutes with C xx and so

Original: [Equation]

Original: Thus, equation 14 holds.

Original: For general initializations,
Revised: For general initializations W 0 W ⊤ 0 ,

Original: [Equation]
Revised: [Equation]

Original: Therefore, inequality equation 15 holds.

Original: [Figure]

Original:  N UMERICAL EXPERIMENTS
Revised:  N UMERICAL EXPERIMENTS

Original: In this section, we numerically test the offline and online algorithms on synthetic datasets.
Revised: In this section, we numerically test the offline and online algorithms on synthetic datasets.

Original:  Whitening [Equation]
Revised: Let Whitening error ( t ) = ∥

Original: and compare the evolution of the eigenvalues for A  t 
Revised: A − 1 t C xx

Original: = M t , W t W ⊤ t  , where the subscript t indicates the value of matrices after the t th iterate.
Revised: A − 1 t − I n ∥ Frob . for A t ∈ { M t , W t W ⊤ t } , where the subscript t indicates the value of matrices after the t th iterate.

Original: To quantify the convergence time, we define
Revised: To quantify the convergence time, we define

Original: Convergence time :=
Revised: Convergence time :=

Original: min { t ≥ 1 : Whitening error ( t ) < 0 . 1 } .
Revised: min { t ≥ 1 : Whitening error ( t ) < 0 . 1 } .

Original: 7. O FFLINE ALGORITHMS
Revised: 7. O FFLINE ALGORITHMS

Original: Let n = 5 , k
Revised: Let n = 5 , k

Original: = 10 and T = 10 5 .
Revised: = 10 and T = 10 5 .

Original: We generate a data matrix X with i.i.d.
Revised: We generate a data matrix X with i.i.d.

Original: entries x t,i chosen uniformly from the interval (0 , 12) .
Revised: entries x t,i chosen uniformly from the interval (0 , 12) .

Original: The eigenvalues of C xx are λ 21 = 24 . 01 ,λ 22 = 16 . 42 ,λ 23 = 10 . 45 ,λ 24 = 6 . 
Revised: The eigenvalues of C xx are { 24 . 01 , 16 . 42 , 10 . 45 , 6 . 59 , 3 . 28 } .

Original: 59 , λ 25 = 3 .

Original: We initialize W [-
-]
Revised: We initialize W [-
-]0 =

Original: = Q √ α ΣP ⊤ , where α > 0 , Σ = diag (5 , 4 , 3 , 2 , 1) , Q is a random n × n orthogonal matrix and P is a random k × northogonal matrix with orthonormal column vectors.
Revised: Q √ α ΣP ⊤ , where α > 0 , Σ = diag (5 , 4 , 3 , 2 , 1) , Q is an n × n orthogonal matrix and P is a random k × n matrix with orthonormal column vectors.

Original: We set M 

Revised: We set M

Original: = W 0 W ⊤ 

Revised: = W 0 W ⊤

Original: = α QΣ 2 Q ⊤ .
Revised: = α QΣ 2 Q ⊤ .

Original: We considerboth spectral initializations (i.e., Q = U x ) and nonspectral initializations.
Revised: We consider spectral initializations (i.e., Q = U x ) and nonspectral initializations (i.e., Q is a random orthogonal matrix).

Original: We use step size η = 10 − 3 .
Revised: We use step size η = 10 − 3 .

Original: We first consider spectral (i.e., Q = U x ) and nonspectral initializations (i.e., Q is a random orthogonal matrix).

Original: The results of running the offline algorithms for α = 0 .
Revised: The results of running the offline algorithms for α = 0 .

Original: 1 , 1 ,5 , 10 are shown in Figure 2.
Revised: 1 , 1 , 10 are shown in Figure 2.

Original: Consistent with Propositions 1 and 2, the network with direct recurrent connections convergences slowly for ‘large’ α , whereas the network with interneurons converges exponentially for all α .
Revised: Consistent with Propositions 1 and 2, the network with direct recurrent connections convergences slowly for ‘large’ α , whereas the network with interneurons converges exponentially for all α .

Original: Furthermore, as predicted by equation 9, when the eigenvalues of M are ‘large’, i.e., σ i
Revised: Further, as predicted by equation 9, when the eigenvalues of M are ‘large’, i.e., σ i

Original: ≫ λ
Revised: ≫ λ

Original: i , they decay linearly.
Revised: i , they decay linearly.

Original: Interestingly,  contrast to the case of spectral initializations, the evolution of the eigenvalues σ i ( t ) is not always monotone for nonspectral initializations (see the bottom row of Figure 2b).
Revised: Interestingly, in contrast to the case of spectral initializations, the evolution of the eigenvalues σ i ( t ) is not always monotone for nonspectral initializations (see the bottom row of Figure 2b).

Original: In Figure 3, we plot the convergence times of the offline algorithms with spectral and nonspectral initializations, for α = 1 , 2 , . . .
Revised: In Figure 3, we plot the convergence times of the offline algorithms with spectral and nonspectral initializations, for α = 1 , 2 , . . .

Original: Consistent with our analysis of the gradient flows, the convergence times for network with direct lateral connections (resp.
Revised: Consistent with our analysis of the gradient flows, the convergence times for network with direct lateral connections (resp.

Original: interneurons) grows linearly (resp. logarithmically) with α .
Revised: interneurons) grows linearly (resp. logarithmically) with α .

Original: 7. O NLINE ALGORITHMS
Revised: 7. O NLINE ALGORITHMS

Original: To test our online algorithms, we generate a synthetic dataset with samples from 2 distributions.
Revised: To test our online algorithms, we generate a synthetic dataset with samples from 2 distributions.

Original: We first set n = 2 , k = 4 and fixed covariance matrices C A
Revised: We first set n = 2 , k = 4 and fixed covariance matrices C A

Original: = U
Revised: = U

Original: A diag (4 , 25) U ⊤ A and C B =
Revised: A diag (4 , 25) U ⊤ A and C B =

Original: U B diag (9 , 16) U ⊤ B , where U A , U B are random 2 × 2 rotation matrices.
Revised: U B diag (9 , 16) U ⊤ B , where U A , U B are random 2 × 2 rotation matrices.

Original: We set T = 10 5 and generated a 2 × 4 T dataset X =
Revised: We set T = 10 5 and generated a 2 × 4 T dataset X =

Original: [ x 1 , . .
Revised: [ x 1 , . .

Original: , x 4 T ] with i.i.d.
Revised: , x 4 T ] with i.i.d.

Original: samples x 1 , . .
Revised: samples x 1 , . .

Original: , x T , x 2 T +1 , . . .
Revised: , x T , x 2 T +1 , . . .

Original: , x 3 T (resp.
Revised: , x 3 T (resp.

Original: x T +1 , . . .
Revised: x T +1 , . . .

Original: , x 2 T , x 3 T +1 , . . .
Revised: , x 2 T , x 3 T +1 , . . .

Original: , x 4 T ) drawn according to a mean zero Gaussian distribution with covariance matrix C A (resp.
Revised: , x 4 T ) drawn according to a mean zero Gaussian distribution with covariance matrix C A (resp.

Original: C B ).
Revised: C B ).

Original: We evaluated our online algorithms with constant learning rates η = 10 − 4 and initializations M 0 =
Revised: We evaluated our online algorithms with constant learning rate η = 10 − 4 and initializations M 0 =

Original: W 0 W ⊤ 0 , where W 0 =
Revised: W 0 W ⊤ 0 , where W 0 =

Original: Q diag ( σ 1 , σ 2 ) P ⊤ , Q is a random 2 × 2 rotation matrix, P is a random 4 × 2 matrix with orthonormal column vectors and σ 1 , σ 2 are independent random variables chosen uniformly from the interval (1 , 1 . 5) .
Revised: Q diag ( σ 1 , σ 2 ) P ⊤ , Q is a random 2 × 2 rotation matrix, P is a random 4 × 2 matrix with orthonormal column vectors and σ 1 , σ 2 are independent random variables chosen uniformly from the interval (1 , 1 . 5) .

Original: The results are shown in
Revised: The results are shown in

Original: Figure 4.
Revised: Figure 4.

Original: Consistent with our theoretical analyses, we see that the network with interneurons adapts to changing distributions faster than the network with direct recurrent connections. 
Revised: Consistent with our theoretical analyses, we see that the network with interneurons adapts to changing distributions faster than the network with direct recurrent connections. 7. A PPLICATION TO PRINCIPAL SUBSPACE LEARNING

Revised: A challenge in unsupervised and self-supervised learning is preventing collapse (i.e., degenerate solutions).

Original: C ONCLUSION
Revised: A recent approach in self-supervised learning is to decorrelate or whiten the feature representation (Ermolov et al., 2021; Zbontar et al., 2021; Bardes et al., 2021; Hua et al., 2021). Here, using Oja’s online principal component algorithm (Oja, 1982) as a tractable example, we demonstrate that the speed of whitening can affect the accuracy of the learned representation. Consider a neuron whose input and output at time t are respectively s t ∈ R d and y t := v ⊤ s t , where v ∈ R d represents the synaptic weights connecting the inputs to the neuron.

Revised: Oja’s algorithm learns the top principal component of the inputs by updating the vector v as follows:

Revised: [Equation]

Revised: where ζ > 0 is the step size.

Revised: Next, consider a population of 2 ≤ n ≤ d neurons with outputs y t ∈ R n and feedforward synaptic weight vectors v 1 , . .

Revised: . , v n ∈ R d connecting the inputs s t to the n neurons.

Revised: The goal is to project the inputs onto their n -dimensional principal subspace.

Revised: Running n instances of Oja’s algorithm in parallel without lateral connections results in collapse — each synaptic weight vector v i converges to the top principal component.

Revised: One way to avoid collapse is to whiten the output y t using recurrent connections, Figure 5 (left).

Revised: Here we show that when the subspace projection and output whitening are learned simultaneously, it is critical that the whitening transformation is learned sufficiently fast.

Revised: We set d = 3 , n = 2 and generate i.i.d.

Revised: inputs s t ∼ N ( 0 , diag (5 , 2 , 1)) .

Revised: We initialize two random vectors v 1 , v 2 ∈ R 3 with independent N (0 , 1) entries.

Revised: At each time step t , we use the projection x t := ( v ⊤ 1 s t , v ⊤ 2 s t ) as the input to either Algorithm 1 or 2 and we let y t be the output; that is, y t

Revised: =

Revised: M − 1 x t or y t

Revised: = ( WW ⊤ ) − 1 x t .

Revised: For i = 1 , 2 , we update v i according to equation 15 with ζ = 10 − 3 and with v i (resp.

Revised: y t,i ) in place of v (resp.

Revised: y t ).

Revised: We update the recurrent weights M or W according to Algorithm 1 or 2.

Revised: To measure the performance, we define

Revised: Subspace error :

Revised: =

Revised: ∥ V ( V ⊤ V ) −

Revised: [Equation]

Revised: which is equal to zero when v 1 , v 2 span the 2-dimensional principal subspace of the inputs.

Revised: We plot the subspace error for Algorithms 1 and 2 using learning rates η = ζ and η = 10 ζ , Figure 5 (right).

Revised: The results suggest that in order to learn the correct subspace, the whitening transformation must be learned sufficiently fast relative to the feedforward vectors v i ; for additional results along these lines, see (Pehlevan et al., 2018, section 6).

Original:  We analyzed the gradient flow dynamics ofa 2 recurrent neural networks for ZCA whitening — one with direct recurrent connections and one with interneurons.
Revised: Therefore, since interneurons accelerate learning of the whitening transform, they are also useful for accurate or optimal representation learning.  D ISCUSSION We analyzed the gradient flow dynamics of 2 recurrent neural networks for ZCA whitening — one with direct recurrent connections and one with interneurons.

Original: For spectral initializations we can analytically estimate the convergence time for both gradient flows.
Revised: For spectral initializations we can analytically estimate the convergence time for both gradient flows.

Original: For nonspectral initializations, we show numerically that the convergence times are close to the spectral initializations with the same initial spectrum.
Revised: For nonspectral initializations, we show numerically that the convergence times are close to the spectral initializations with the same initial spectrum.

Original: Both our theoretical and numerical results show that the recurrent neural network with interneurons is more robust to initialization than the recurrent neural network with direct recurrent connections.
Revised: Both our theoretical and numerical results show that the recurrent neural network with interneurons is more robust to initialization.

Revised: In Appendix B, we modify the algorithm to be more biologically realistic and we map the modified algorithm onto the vertebrate olfactory bulb and show that the algorithm is consistent with several experimental observations, suggesting that biological interneurons may facilitate fast statistical adaptation.

Original: An interesting question is whether including interneurons in other classes of recurrent neural networks also accelerates the learning dynamics of those networks.
Revised: An interesting question is whether including interneurons in other classes of recurrent neural networks also accelerates the learning dynamics of those networks.

Original: We hypothesize that interneurons accelerate learning dynamics when the objective for the network with interneurons can be viewed as an overparameterization of the objective for the recurrent neural network with direct connections.
Revised: We hypothesize that interneurons accelerate learning dynamics when the objective for the network with interneurons can be viewed as an overparameterization of the objective for the recurrent neural network with direct connections.

Original: A S ADDLE POINT PROPERTY
Revised: A S ADDLE POINT PROPERTY

Original: Here we recall the following minimax property for a function that satisfies the saddle point property (Boyd & Vandenberghe, 2004, section 5.4).
Revised: Here we recall the following minimax property for a function that satisfies the saddle point property (Boyd & Vandenberghe, 2004, section 5.4).

Original: Theorem 1.
Revised: Theorem 1.

Original: Let V ⊆ R n 
Revised: Let V ⊆ R n , W ⊆

Original: W ⊆ R m and f : V × W → R .
Revised: R m and f : V × W → R .

Original: Suppose f satisfies the saddle point property; that is, there exists ( a ∗ , b ∗ )
Revised: Suppose f satisfies the saddle point property; that is, there exists ( a ∗ , b ∗ )

Original: ∈ V × W such that
Revised: ∈ V × W such that

Original: [Equation]
Revised: [Equation]

Original: for all ( a , b ) ∈ V ×
Revised: for all ( a , b ) ∈ V ×

Original: W.
Revised: W.

Original:  [Equation]
Revised: Then [Equation]

Revised: B R ELATION TO NEURONAL CIRCUITS

Revised: In this section, we modify Algorithm 2 to satisfy additional biological constraints and we map the modified algorithm onto the vertebrate olfactory bulb.

Revised: B. D ECOUPLING THE INTERNEURON SYNAPSES

Revised: The neural circuit implementation of Algorithm 2 requires that the principal neuron-to-interneuron synaptic weight matrix W ⊤ is the negative transpose of the interneuron-to-principal neuron synaptic weight matrix − W .

Revised: In general, enforcing this symmetry is not biologically plausible, and is commonly referred to as the weight transport problem.

Revised: Here, following (Golkar et al., 2020, appendix

Revised: D), we decouple the synapses and show that the (asymptotic) symmetry of the synaptic weight matrices follows from the symmetry of the local Hebbian/anti-Hebbian updates.

Revised: We begin by replacing W and W ⊤ in Algorithm 2 with W zy and W yz , respectively.

Revised: Then the neural activities are given by

Revised: [Equation]

Revised: which converge to ( W zy W yz )

Revised: − 1 x t and z t = W yz y t .

Revised: After the neural activities converge, the synaptic weights are updated according to the update rules

Revised: [Equation]

Revised: Let W zy,t and W yz,t denote the values of the synaptic weight matrices after t updates.

Revised: By iterating the above updates, we see that the difference between the weight matrices after t updates is given by

Revised: [Equation]

Revised: Therefore, provided 0 < η < 1 , the difference decays exponentially.

Revised: B. E NFORCING D ALE ’ S PRINCIPLE

Revised: In equation 16, the matrix W yz is preceded by a positive sign and the matrix W zy is preceded by a negative sign, which is consistent with the fact that the principal neuron-to-interneuron synapses are excitatory whereas the interneuron-to-principal neuron synapses are inhibitory.

Revised: However, this interpretation is superficial because the matrices are not constrained to be non-negative, so the network can violate Dale’s principle.

Revised: One approach to enforce Dale’s principle is to take projected gradient descent steps when updating the synaptic weight matrices, which results in the offline updates

Revised: [Equation]

Revised: and online updates

Revised: [Equation]

Revised: where [ · ] + denotes the element-wise rectification operation.

Revised: This results in Algorithm 3.

Revised: Algorithm 3: input centered inputs { x t } ; parameters γ , η initialize non-negative matrices W yz , W zy for t = 1 , . .

Revised: , T do repeat y t ← y t + γ ( x t − W zy z t )

Revised: z t ← z t + γ ( W yz y t − z t ) until convergence

Revised: W yz ← [ W yz + η

Revised: ( z t y ⊤ t − W yz )]

Revised: +

Revised: W zy ← [ W zy + η ( y t z ⊤ t − W zy )]

Revised: + end for

Revised: In general, this modification results in a network output that does not correspond to ZCA whitening of the inputs.

Revised: That being said, it is still worth comparing the performance of the network to the network with interneurons derived in section 5.

Revised: In Figure 6, we compare the performance of the offline versions of Algorithms 2 and 3.

Revised: We see that Algorithm 3 equilibrates at a rate that is comparable to

Revised: Algorithm 2 (and therefore much faster than Algorithm 1 for large α ).

Revised: In particular, this modified algorithm appears to also exhibit the accelerated dynamics due to the overparamterization of the objective.

Revised: As expected, the outputs of the projected gradient descent algorithm are not fully whitened — after the synaptic dynamics of Algorithm 3 equilibrate, the eigenvalues of the output covariance C

Revised: yy are { 1 . 39 , 1 . 01 , 1 . 00 , 0 . 98 , 0 . 61 } .

Revised: This can be compared with the eigenvalues of the input covariance matrix C xx : { 24 . 01 , 16 .

Revised: 42 , 10 . 45 , 6 . 59 , 3 . 28 } , which suggests that the network significantly normalizes the eigenvalues of the output covariance without exactly performing ZCA whitening.

Revised: In general, understanding the exact statistical transformation that results from the projected gradient descent algorithm is more mathematically challenging.

Revised: For instance, in the case of Algorithm 2, the weights W adapt so that WW ⊤ =

Revised: C 1 / 2 xx ; that is, the algorithm solves a symmetric matrix factorization problem whose solutions can be written in closed form in terms of the SVD of the covariance matrix C xx .

Revised: In the case of Algorithm 3, the algorithm is essentially solving for the optimal (approximate) non-negative symmetric matrix factorization of C 1 / 2 xx , for which there do not exist general closed form solutions.

Revised: That being said, the empirical results suggest that the more biologically realistic network still performs statistical adaptation.

Revised: B. M APPING A LGORITHM 3 ONTO THE OLFACTORY BULB

Revised: In vertebrates, an early stage of olfactory processing occurs in the olfactory bulb.

Revised: The olfactory bulb receives direct olfactory inputs, which it processes and transmits to higher order brain regions (Shepherd et al., 2004), axon structures glomeruli

Revised: Mitral are the main projection neurons of olfactory direct from interneurons called granule cells, which are axonless neurons that form reciprocal dendrodendritic synapses with the basal dendrites of mitral cells (Shepherd, 2009).

Revised: Experimental evidence indicates that the olfactory bulb transforms its inputs so that mitral cell responses to distinct odors are approximately orthogonal (Friedrich & Laurent, 2001; 2004;

Revised: Giridhar et al., 2011; Gschwend et al., 2015; Wanner & Friedrich, 2020), a transformation referred to as pattern separation , which is closely related to statistical whitening (Wick et al., 2010).

Revised: We explore the possibility that the mitral-granule cell microcircuit implements Algorithm 3, Figure 7.

Revised: In this case, the vectors x t and y t represent the inputs to and outputs from the mitral cells, respectively.

Revised: The vector z t represents the granule cell outputs.

Revised: We let W yz (resp.

Revised: − W zy ) denote the mitral cell-to-granule cell (resp.

Revised: granule cell-to-mitral cell) synaptic weights.

Revised: We compare our model with additional experimental observations.

Revised: First, Algorithm 3 learns by adapting the matrices W yz and W zy .

Revised: This is in line with experimental observations that granule cells in the olfactory bulb are highly plastic suggesting that the dendrodendritic synapses are a synaptic substrate for experience dependent modulation of the olfactory bulb output (Nissant et al., 2009; Sailor et al., 2016).

Revised: Second, a consequence of Algorithm 3 is that the mitral cellto-granule cell synapses and granule cell-to-mitral cell synapses are (asymptotically) symmetric; that is, W zy = W ⊤ yz .

Revised: There is experimental evidence in support of this symmetry — the dendrodendritic synapses are mainly present in reciprocal pairs (Woolf et al., 1991).

Revised: In other words, the connectivity matrix of mitral cell-to-granule cell synapses (the matrix obtained by setting the non-zero entries in W yz to 1) is approximately equal to the transpose of the connectivity matrix of granule cell-to-mitral cell synapses.

Revised: Finally, Algorithm 3 requires that k ≥ n ; that is, there are more granule cells than mitral cells.

Revised: This is consistent with measurements in the rodent olfactory bulb, which indicate that there are approximately 50–100 times more granule cells than mitral cells (Shepherd et al., 2004).

Revised: C P ROOFS OF OUR MAIN RESULTS C. P ROOF OF L EMMA

Revised: Proof.

Revised: Let σ 1 ( t ) , . . .

Revised: , σ n ( t ) be solutions of the ODE 7 and define M ( t ) :

Revised: = U x Σ ( t ) U ⊤ x , where Σ ( t ) := diag ( σ 1 ( t ) , . . .

Revised: , σ n ( t )) .

Revised: Then

Revised: [Equation]

Revised: In particular, we see that M ( t ) must be the unique solution of the ODE 6, where uniqueness of solutions follows because the right-hand-side of equation 6 is locally Lipschitz continuous on its domain of definition.

Revised: Equation 8 then follows from the ODE 7 and the chain rule.

Revised: C. P ROOF OF P ROPOSITION

Revised: Proof.

Revised: Suppose σ i (0) ≤ λ i for i = 1 , . . . , n .

Revised: By equation 8, σ i ( t ) ≤ λ i for all t ≥ 0 and so

Revised: [Equation]

Revised: Therefore,

Revised: [Equation]

Revised: It follows that equation 10 holds.

Revised: Now suppose σ i (0) ≥ λ

Revised: i for some i = 1 , . . . , n .

Revised: By equation 7,

Revised: [Equation]

Revised: It follows that equation 11 holds.

Revised: C. P ROOF OF P ROPOSITION

Revised: Proof.

Revised: Define A ( t ) :

Revised: = W ( t ) W ( t ) ⊤ .

Revised: By the product rule,

Revised: [Equation]

Revised: Then, by the chain rule and equation 17, for all t ≥ 0 ,

Revised: [Equation]

Revised: t ) 2 − C xx )

Revised: [Equation]

Revised: A

Revised: [Equation]

Revised: t )

Revised: [Equation]

Revised: A

Revised: [Equation]

Revised: t )

Revised: [Equation]

Revised: xx

Revised: A

Revised: [Equation]

Revised: t ) − 1 )

Revised: [Equation]

Revised: =

Revised: [Equation]

Revised: Tr

Revised: [Equation]

Revised: A

Revised: [Equation]

Revised: t )

Revised: [Equation]

Revised: xx )

Revised: [Equation]

Revised: t

Revised: [Equation]

Revised: t )

Revised: [Equation]

Revised: xx )

Revised: [Equation]

Revised: t )

Revised: [Equation]

Revised: [Equation]

Revised: t

Revised: [Equation]

Revised: t )

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: ( t )

Revised: [Equation]

Revised: xx )

Revised: [Equation]

Revised: [Equation]

Revised: ∥

Revised: [Equation]

Revised: Suppose W 0 W ⊤ 0 is a spectral initialization.

Revised: Then A ( t ) commutes with C xx and so

Revised: [Equation]

Revised: Thus, equation 13 holds.

Revised: For general initializations,

Revised: [Equation]

Revised: Therefore, inequality equation 14 holds.