Original: I NTERNEURONS ACCELERATE LEARNING DYNAMICS IN RECURRENT NEURAL NETWORKS FOR STATISTICAL ADAPTATION
Revised: I NTERNEURONS ACCELERATE LEARNING DYNAMICS IN RECURRENT NEURAL NETWORKS FOR STATISTICAL ADAPTATION

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Early sensory systems in the brain rapidly adapt to fluctuating input statistics, which requires recurrent communication between neurons.
Revised: Early sensory systems in the brain rapidly adapt to fluctuating input statistics, which requires recurrent communication between neurons.

Original: Mechanistically, such recurrent communication is often indirect and mediated by local interneurons.
Revised: Mechanistically, such recurrent communication is often indirect and mediated by local interneurons.

Original: In this work, we explore the computational benefits of mediating recurrent communication via interneurons compared with direct recurrent connections.
Revised: In this work, we explore the computational benefits of mediating recurrent communication via interneurons compared with direct recurrent connections.

Original: To this end, we consider two mathematically tractable recurrent linear neural networks that statistically whiten their inputs — one with direct recurrent connections and the other with interneurons that mediate recurrent communication.
Revised: To this end, we consider two mathematically tractable recurrent linear neural networks that statistically whiten their inputs — one with direct recurrent connections and the other with interneurons that mediate recurrent communication.

Original: By analyzing the corresponding continuous synaptic dynamics and numerically simulating the networks, we show that the network with interneurons is more robust to initialization than the network with direct recurrent connections in the sense that the convergence time for the synaptic dynamics in the network with interneurons (resp.
Revised: By analyzing the corresponding continuous synaptic dynamics and numerically simulating the networks, we show that the network with interneurons is more robust to initialization than the network with direct recurrent connections in the sense that the convergence time for the synaptic dynamics in the network with interneurons (resp.

Original: direct recurrent connections) scales logarithmically (resp. linearly) with the spectrum of their initialization.
Revised: direct recurrent connections) scales logarithmically (resp. linearly) with the spectrum of their initialization.

Original: Our results suggest that interneurons are computationally useful for rapid adaptation to changing input statistics.
Revised: Our results suggest that interneurons are computationally useful for rapid adaptation to changing input statistics.

Original: Interestingly, the network with interneurons is an overparameterized solution of the whitening objective for the network with direct recurrent connections, so our results can be viewed as a recurrent linear neural network analogue of the implicit acceleration phenomenon observed in overparameterized feedforward linear neural networks.
Revised: Interestingly, the network with interneurons is an overparameterized solution of the whitening objective for the network with direct recurrent connections, so our results can be viewed as a recurrent linear neural network analogue of the implicit acceleration phenomenon observed in overparameterized feedforward linear neural networks.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Efficient coding and redundancy reduction theories of neural coding hypothesize that early sensory systems decorrelate and normalize neural responses to sensory inputs (Barlow, 1961; Laughlin, 1989; Barlow & F ¨ oldi ´ ak, 1989; Simoncelli & Olshausen, 2001; Carandini & Heeger, 2012; Westrick et al., 2016), operationsthat are closely related to statistical whitening of inputs.
Revised: Efficient coding and redundancy reduction theories of neural coding hypothesize that early sensory systems decorrelate and normalize neural responses to sensory inputs (Barlow, 1961; Laughlin, 1989; Barlow & F¨oldi´ak, 1989; Simoncelli & Olshausen, 2001; Carandini & Heeger, 2012; Westrick et al., 2016; Chapochnikov et al., 2021), operations closely related to statistical whitening of inputs.

Original: Since the input statistics are often in flux due to dynamic environments, this calls for early sensory systems that can rapidly adapt (Wark et al., 2007; Whitmire & Stanley, 2016).
Revised: Since the input statistics are often in flux due to dynamic environments, this calls for early sensory systems that can rapidly adapt (Wark et al., 2007; Whitmire & Stanley, 2016).

Original: Decorrelating neural activities requires recurrent communication between neurons in a population, which is typically indirect and mediated by local interneurons (Christensen et al., 1993;
Revised: Decorrelating neural activities requires recurrent communication between neurons, which is typically indirect and mediated by local interneurons (Christensen et al., 1993;

Original: Shepherd et al., 2004).
Revised: Shepherd et al., 2004).

Original: Why do neuronal circuits for statistical adaptation mediate recurrent communication using interneurons, which take up valuable space and metabolic resources, rather than using direct recurrent connections?
Revised: Why do neuronal circuits for statistical adaptation mediate recurrent communication using interneurons, which take up valuable space and metabolic resources, rather than using direct recurrent connections?

Original: A common explanation for why communication between neurons is mediated by interneurons is
Revised: A common explanation for why communication between neurons is mediated by interneurons is

Original: Dale’s principle, which states that each neuron only produces one type of neurotransmitter with exclusively inhibitory or excitatory effects on all of its targets (Strata & Harvey, 1999).
Revised: Dale’s principle, which states that each neuron has exclusively inhibitory or excitatory effects on all of its targets (Strata & Harvey, 1999).

Original: Therefore, an excitatory neuron cannot directly inhibit another neuron; rather, the inhibition must be mediated by an inhibitory interneuron.

Original: While Dale’s principle provides a physiological constraint that explains why recurrent interactions are mediated by interneurons, we seek a computational principle that can account for using interneurons rather than direct recurrent connections.
Revised: While Dale’s principle provides a physiological constraint that explains why recurrent interactions are mediated by interneurons, we seek a computational principle that can account for using interneurons rather than direct recurrent connections.

Original: This perspective is useful for a couple of reasons.
Revised: This perspective is useful for a couple of reasons.

Original: First, perhaps Dale’s principle is not a hard constraint and biology could have found an efficient mechanism for a neuron to produce more than one type of neurotransmitter.
Revised: First, perhaps Dale’s principle is not a hard constraint; see (Saunders et al., 2015; Granger et al., 2020) for results along these lines.

Original: In this case, a computational benefit of interneurons would provide a normative explanation for the existence of interneurons to mediate recurrent communication.
Revised: In this case, a computational benefit of interneurons would provide a normative explanation for the existence of interneurons to mediate recurrent communication.

Original: Second, decorrelation and whitening are useful in statistical and machine learning methods (Hyv¨arinen & Oja, 2000; Krizhevsky, 2009), especially recent self-supervised methods (Ermolov et al., 2021; Zbontar et al., 2021;
Revised: Second, decorrelation and whitening are useful in statistical and machine learning methods (Hyv¨arinen & Oja, 2000; Krizhevsky, 2009), especially recent self-supervised methods (Ermolov et al., 2021; Zbontar et al., 2021;

Original: Bardes et al., 2021; Hua et al., 2021), so our analysis is potentially relevant to the design of artificial neural networks and neuromorphic systems, which are not bound by Dale’s principle.
Revised: Bardes et al., 2022; Hua et al., 2021), so our analysis is potentially relevant to the design of artificial neural networks, which are not bound by Dale’s principle.

Original: In this work, to better understand the computational benefits of interneurons for statistical adaptation, we analyze the learning dynamics of two mathematically tractable recurrent neural networks that statistically whiten their inputs using Hebbian/anti-Hebbian learning rules — one with direct recurrent connections and the other with indirect recurrent interactions mediated by interneurons,
Revised: In this work, to better understand the computational benefits of interneurons for statistical adaptation, we analyze the learning dynamics of two mathematically tractable recurrent neural networks that statistically whiten their inputs using Hebbian/anti-Hebbian learning rules — one with direct recurrent connections and the other with indirect recurrent interactions mediated by interneurons,

Original: Figure 1.
Revised: Figure 1.

Original: We show that the learning dynamics of the network with interneurons are more robust than the learning dynamics of the network with direct recurrent connections.
Revised: We show that the learning dynamics of the network with interneurons are more robust than the learning dynamics of the network with direct recurrent connections.

Original: In particular, we prove that the convergence time of the continuum limit of the network with direct lateral connections scales linearly with the spectrum of the initialization, whereas the convergence time of the continuum limit of the network with interneurons scales logarithmically with the spectrum of the initialization.
Revised: In particular, we prove that the convergence time of the continuum limit of the network with direct lateral connections scales linearly with the spectrum of the initialization, whereas the convergence time of the continuum limit of the network with interneurons scales logarithmically with the spectrum of the initialization.

Original: We also numerically test the networks and, consistent with our theoretical results, find that the network with interneurons is more robust to initialization.
Revised: We also numerically test the networks and, consistent with our theoretical results, find that the network with interneurons is more robust to initialization.

Original: Our results suggest that interneurons are computationally important for rapid adaptation to fluctuating input statistics.
Revised: Our results suggest that interneurons are computationally important for rapid adaptation to fluctuating input statistics.

Original: Our analysis is closely related to analyses of learning dynamics in feedforward linear networks trained using backpropagation (Saxe et al., 2014; Arora et al., 2018; Saxe et al., 2019; Gidel et al., 2019;
Revised: Our analysis is closely related to analyses of learning dynamics in feedforward linear networks trained using backpropagation (Saxe et al., 2014; Arora et al., 2018; Saxe et al., 2019; Gidel et al., 2019;

Original: Tarmoun et al., 2021).
Revised: Tarmoun et al., 2021).

Original: The optimization problems for deep linear networks are overparameterizations of linear problems and this overparameterization can accelerate convergence of gradient descent or gradient flow optimization — a phenomenon referred to as implicit acceleration (Arora et al., 2018).
Revised: The optimization problems for deep linear networks are overparameterizations of linear problems and this overparameterization can accelerate convergence of gradient descent or gradient flow optimization — a phenomenon referred to as implicit acceleration (Arora et al., 2018).

Original: Our results can be viewed as an analogous phenomenon for gradient flows corresponding to recurrent linear networks trained using Hebbian/anti-Hebbian learning rules.
Revised: Our results can be viewed as an analogous phenomenon for gradient flows corresponding to recurrent linear networks trained using Hebbian/anti-Hebbian learning rules.

Original: In our setting, the network with interneurons is naturally viewed as an overparameterized solution of the whitening objective for the network with direct recurrent connections.
Revised: In our setting, the network with interneurons is naturally viewed as an overparameterized solution of the whitening objective for the network with direct recurrent connections.

Original: In analogy with the feedforward setting, the interneurons can be viewed as a hidden layer that overparameterizes the optimization problem.
Revised: In analogy with the feedforward setting, the interneurons can be viewed as a hidden layer that overparameterizes the optimization problem.

Original: In summary, our main contribution is a theoretical and numerical analysis of the synaptic dynamics of two linear recurrent neural networks for statistical whitening — one with direct lateral connections and one with indirect lateral connections mediated by interneurons.
Revised: In summary, our main contribution is a theoretical and numerical analysis of the synaptic dynamics of two linear recurrent neural networks for statistical whitening — one with direct lateral connections and one with indirect lateral connections mediated by interneurons.

Original: Our analysis shows that the synaptic dynamics converge significantly faster in the network with interneurons than the network with direct lateral connections (logarithmic versus linear convergence times).
Revised: Our analysis shows that the synaptic dynamics converge significantly faster in the network with interneurons than the network with direct lateral connections (logarithmic versus linear convergence times).

Original: Our results have potential broader implications: (i) they suggest biological interneurons may facilitate rapid statistical adaptation; (ii) including interneurons in recurrent neural networks for solving other learning tasks may also accelerate learning; (iii) overparameterized whitening objectives may be useful for developing online self-supervised learning algorithms in machine learning.
Revised: Our results have potential broader implications: (i) they suggest biological interneurons may facilitate rapid statistical adaptation, see also (Duong et al., 2023); (ii) including interneurons in recurrent neural networks for solving other learning tasks may also accelerate learning; (iii) overparameterized whitening objectives may be useful for developing online self-supervised learning algorithms in machine learning.

Original:  S TATISTICAL WHITENING
Revised:  S TATISTICAL WHITENING

Original: Let  x 1 , . ..
Revised: Let n ≥ 2 and x 1 , . .

Original: , x T be a sequence of n -dimensional centered inputs, for some n ≥ 2 , with positive definite (empirical) covariance matrix C xx :
Revised: , x T be a sequence of n -dimensional centered inputs with positive definite (empirical) covariance matrix C xx :

Original: = 1 T XX ⊤ , where X :=
Revised: = 1 T XX ⊤ , where X :=

Original: [ x 1 , . .
Revised: [ x 1 , . .

Original: , x T ] is the n × T data matrix of concatenated inputs.
Revised: , x T ] is the n × T data matrix of concatenated inputs.

Original: The goal of statistical whitening is to linearly transform the inputs so that the n -dimensional outputs y 1 , . . .
Revised: The goal of statistical whitening is to linearly transform the inputs so that the n -dimensional outputs y 1 , . . .

Original: , y T have identity covariance; that is, C yy := 1 T YY ⊤ =
Revised: , y T have identity covariance; that is, C yy := 1 T YY ⊤ =

Original: I n , where Y :=
Revised: I n , where Y :=

Original: [ y 1 , . . .
Revised: [ y 1 , . . .

Original: , y T ] is the n × T data matrix of concatenated outputs.
Revised: , y T ] is the n × T data matrix of concatenated outputs.

Original: Statistical whitening is not a unique transformation.
Revised: Statistical whitening is not a unique transformation.

Original: For example, if C yy = I n , then left multiplication of Y by any n × n orthogonal matrix results in another data matrix with identity covariance (Kessy et al., 2018).
Revised: For example, if C yy = I n , then left multiplication of Y by any n × n orthogonal matrix results in another data matrix with identity covariance (Kessy et al., 2018).

Original: We focus on Zero-phase Components Analysis (ZCA) whitening (Bell & Se- jnowski, 1997), also referred to as Mahalanobis whitening, given by Y
Revised: We focus on Zero-phase Components Analysis (ZCA) whitening (Bell & Sejnowski, 1997), also referred to as Mahalanobis whitening, given by Y

Original: = C − 1 / 2 xx X , which is the whitening transformation that minimizes the mean-squared error between the inputs X and the outputs Y (Eldar & Oppenheim, 2003).
Revised: = C − 1 / 2 xx X , which is the whitening transformation that minimizes the mean-squared error between the inputs X and the outputs Y (Eldar & Oppenheim, 2003).

Original: In other words, it is the unique solution to the objective
Revised: In other words, it is the unique solution to the objective

Original: [Equation]
Revised: [Equation]

Original: Y

Original: [Equation]

Original: subject to
Revised: subject to

Original: [Equation]
Revised: [Equation]

Original: The goal of this work is to analyze the learning dynamics of 2 recurrent neural networks that learn to perform ZCA whitening in the online, or streaming, setting using Hebbian/anti-Hebbian learning rules.
Revised: The goal of this work is to analyze the learning dynamics of 2 recurrent neural networks that learn to perform ZCA whitening in the online, or streaming, setting using Hebbian/anti-Hebbian learning rules.

Original: The derivations of the 2 networks, which we include here for completeness, are closely related to derivations of PCA networks carried out in (Pehlevan & Chklovskii, 2015; Pehlevan et al., 2018).
Revised: The derivations of the 2 networks, which we include here for completeness, are closely related to derivations of PCA networks carried out in (Pehlevan & Chklovskii, 2015; Pehlevan et al., 2018).

Original: Our main theoretical and numerical results are presented in sections 6 and 7, respectively.
Revised: Our main theoretical and numerical results are presented in sections 6 and 7, respectively.

Original:  O BJECTIVES FOR DERIVING ZCA WHITENING NETWORKS
Revised:  O BJECTIVES FOR DERIVING ZCA WHITENING NETWORKS

Original: In this section, we rewrite the ZCA whitening objective 1 to obtain 2 objectives that will be the starting points for deriving our 2 recurrent whitening networks.
Revised: In this section, we rewrite the ZCA whitening objective 1 to obtain 2 objectives that will be the starting points for deriving our 2 recurrent whitening networks.

Original: We first expand the square in equation 1, substitute in with the constraint YY ⊤
Revised: We first expand the square in equation 1, substitute in with the constraint YY ⊤

Original: =
Revised: =

Original: T I n , drop the terms that do not depend on Y and finally flip the sign to obtain the following objective in terms of the neural activities data matrix Y :
Revised: T I n , drop the terms that do not depend on Y and finally flip the sign to obtain the following objective in terms of the neural activities data matrix Y :

Original: [Equation]
Revised: [Equation]

Original: subject to
Revised: to

Original: [Equation]
Revised: [Equation]

Original: 3. O BJECTIVE FOR THE NETWORK WITH DIRECT RECURRENT CONNECTIONS
Revised: 3. O BJECTIVE FOR THE NETWORK WITH DIRECT RECURRENT CONNECTIONS

Original: To derive a network with direct recurrent connections, we introduce the positive definite matrix M as a Lagrange multiplier to enforce the upper bound YY ⊤ ⪯ T I n :
Revised: To derive a network with direct recurrent connections, we introduce the positive definite matrix M as a Lagrange multiplier to enforce the upper bound YY ⊤ ⪯ T I n :

Original: [Equation]
Revised: [Equation]

Original: where S n ++ denotes the set of positive definite n × n matrices and
Revised: where S n ++ denotes the set of positive definite n × n matrices and

Original: [Equation]
Revised: [Equation]

Original: Here, we have interchanged the order of optimization, which is justified because f ( M , Y ) is linear in M and strongly concave in Y , so it satisfies the saddle point property (see Appendix A).
Revised: Here, we have interchanged the order of optimization, which is justified because f ( M , Y ) is linear in M and strongly concave in Y , so it satisfies the saddle point property (see Appendix A).

Original: The maximization over Y ensures that the upper bound
Revised: The maximization over Y ensures that the upper bound

Original: YY ⊤ ⪯ T I n is in fact saturated, so the whitening constraint holds.
Revised: YY ⊤ ⪯ T I n is in fact saturated, so the whitening constraint holds.

Original: The minimax objective equation 2 will be the starting point for our derivation of a statistical whitening network with direct recurrent connections in section 4.
Revised: The minimax objective equation 2 will be the starting point for our derivation of a statistical whitening network with direct recurrent connections in section 4.

Original: 3. O VERPARAMETERIZED OBJECTIVE FOR THE NETWORK WITH INTERNEURONS
Revised: 3. O VERPARAMETERIZED OBJECTIVE FOR THE NETWORK WITH INTERNEURONS

Original: To derive a network with k ≥ n interneurons, we replace the matrix M in equation 2 with the overparameterized product WW ⊤ , where W is an n × k matrix, which yields the minimax objective 
Revised: To derive a network with k ≥ n interneurons, we replace the matrix M in equation 2 with the overparameterized product WW ⊤ , where W is an n × k matrix, which yields the minimax objective [Equation]

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: The overparameterized minimax objective equation 3 will be the starting point for our derivation of a statistical whitening network with interneurons in section 5.
Revised: The overparameterized minimax objective equation 3 will be the starting point for our derivation of a statistical whitening network with interneurons in section 5.

Original:  D ERIVATION OF A NETWORK WITH DIRECT RECURRENT CONNECTIONS
Revised:  D ERIVATION OF A NETWORK WITH DIRECT RECURRENT CONNECTIONS

Original: Starting from the ZCA whitening objective in equation 2, we derive offline and online algorithms andwe map the online algorithm onto a network with direct recurrent connections.
Revised: Starting from the ZCA whitening objective in equation 2, we derive offline and online algorithms and map the online algorithm onto a network with direct recurrent connections.

Original: We assume that the neural dynamics operate at faster timescale than the synaptic updates, soin both algorithms we first optimize over the neural activities and then take gradient-descent steps with respect to the synaptic weight matrix.
Revised: We assume that the neural dynamics operate at faster timescale than the synaptic updates, so we first optimize over the neural activities and then take gradient-descent steps with respect to the synaptic weight matrix.

Original: In the offline setting, at each iteration, we first maximize f ( M , Y ) with respect to the neural activities Y by taking gradient ascent steps until convergence:
Revised: In the offline setting, at each iteration, we first maximize f ( M , Y ) with respect to the neural activities Y by taking gradient ascent steps until convergence:

Original: [Equation]
Revised: [Equation]

Original: where γ > 0 is a small constant.
Revised: where γ > 0 is a small constant.

Original: After the neural activities converge, we minimize f ( M , Y ) with respect to M by taking a gradient descent step with respect to M :
Revised: After the neural activities converge, we minimize f ( M , Y ) with respect to M by taking a gradient descent step with respect to M :

Original: [Equation]
Revised: [Equation]

Original: In the online setting, at each time step t , we only have access to the input x t rather than the whole dataset X .
Revised: In the online setting, at each time step t , we only have access to the input x t rather than the whole dataset X .

Original: In this case, we first optimize f ( M , Y ) with respect to the neural activities vector y t by taking gradient steps until convergence:
Revised: In this case, we first optimize f ( M , Y ) with respect to the neural activities vector y t by taking gradient steps until convergence:

Original: [Equation]
Revised: [Equation]

Original: After the neural activities converge, the synaptic weight matrix M is updated by taking a stochastic gradient descent step:
Revised: After the neural activities converge, the synaptic weight matrix M is updated by taking a stochastic gradient descent step:

Original: [Equation]
Revised: [Equation]

Original: This results in Algorithm 1.
Revised: This results in Algorithm 1.

Original: input centered inputs { x t } ; parameters γ , η initialize positive definite n × n matrix M for t = 1 , . .
Revised: input centered inputs { x t } ; parameters γ , η initialize positive definite n × n matrix M for t = 1 , . .

Original: , T do y t ←repeat y t ← y t + γ ( x t − My t ) until convergence
Revised: , T do y t ←repeat y t ← y t + γ ( x t − My t ) until convergence

Original: M ← M + η ( y t y ⊤ t − I n ) end for
Revised: M ← M + η ( y t y ⊤ t − I n ) end for

Original: Algorithm 1 can be implemented in a network with n principal neurons with direct recurrent connections − M , Figure 1 (left).
Revised: Algorithm 1 can be implemented in a network with n principal neurons with direct recurrent connections − M , Figure 1 (left).

Original: At each time step t , the external input to the principal neurons is x t , the output of the principal neurons is y t and the recurrent input to the principal neurons is − My t .
Revised: At each time step t , the external input to the principal neurons is x t , the output of the principal neurons is y t and the recurrent input to the principal neurons is − My t .

Original: Therefore, the total input to the principal neurons is encoded in the n -dimensional vector x t − My t .
Revised: Therefore, the total input to the principal neurons is encoded in the n -dimensional vector x t − My t .

Original: The neural outputs are updated according to the neural dynamics in Algorithm 1 until they converge at y t
Revised: The neural outputs are updated according to the neural dynamics in Algorithm 1 until they converge at y t

Original: =
Revised: =

Original: M − 1 x t .
Revised: M − 1 x t .

Original: After the neural activities converge, the synaptic weight matrix M is updated according to Algorithm 1.
Revised: After the neural activities converge, the synaptic weight matrix M is updated according to Algorithm 1.

Original:  D ERIVATION OF A NETWORK WITH INTERNEURONS
Revised:  D ERIVATION OF A NETWORK WITH INTERNEURONS

Original: Starting from the overparameterized ZCA whitening objective in equation 3, we derive offline and online algorithms and we map the online algorithm onto a network with k ≥ n interneurons.
Revised: Starting from the overparameterized ZCA whitening objective in equation 3, we derive offline and online algorithms and we map the online algorithm onto a network with k ≥ n interneurons.

Original: As in the last section, we assume that the neural dynamics operate at a faster timescale than the synaptic updates.
Revised: As in the last section, we assume that the neural dynamics operate at a faster timescale than the synaptic updates.

Original: In the offline setting, at each iteration, we first maximize g ( W , Y ) with respect to the neural activities Y by taking gradient steps until convergence:
Revised: In the offline setting, at each iteration, we first maximize g ( W , Y ) with respect to the neural activities Y by taking gradient steps until convergence:

Original: [Equation]
Revised: [Equation]

Original: After the neural activities converge, we minimize g ( W , Y ) with respect to W by taking a gradient descent step with respect to W :
Revised: After the neural activities converge, we minimize g ( W , Y ) with respect to W by taking a gradient descent step with respect to W :

Original: [Equation]
Revised: [Equation]

Original: In the online setting, at each time step t , we first optimize g ( W , Y ) with respect to the neural activities vectors y t by running the gradient ascent-descent neural dynamics:
Revised: In the online setting, at each time step t , we first optimize g ( W , Y ) with respect to the neural activities vectors y t by running the gradient ascent-descent neural dynamics:

Original: [Equation]
Revised: [Equation]

Original: After convergence of the neural activities, the matrix W is updated by taking a stochastic gradient descent step:
Revised: After convergence of the neural activities, the matrix W is updated by taking a stochastic gradient descent step:

Original: [Equation]
Revised: [Equation]

Original: To implement the online algorithm in a recurrent neural network with interneurons, we let z t :=
Revised: To implement the online algorithm in a recurrent neural network with interneurons, we let z t :=

Original: W ⊤ y t denote the k -dimensional vector of(hidden) interneuron activities at time t .
Revised: W ⊤ y t denote the k -dimensional vector of interneuron activities at time t .

Original: After substitutingthe interneuron activities into the neural dynamics and the synaptic update rules, we obtain Algorithm 2.
Revised: After substituting into the neural dynamics and the synaptic update rules, we obtain Algorithm 2.

Original:  input centered inputs { x t } ; k ≥ n ; parameters γ , η initialize full rank n × k matrix W for t = 1 , . .
Revised: Algorithm 2: A whitening network with interneurons input centered inputs { x t } ; k ≥ n ; parameters γ , η initialize full rank n × k matrix W for t = 1 , . .

Original: , T do y t ←repeat
Revised: , T do y t ←repeat

Original: z
Revised: z

Original: t ← W ⊤ y t
Revised: t ← W ⊤ y t

Original: y t ← y t
Revised: y t ← y t

Original: + γ ( x t − Wz t ) until convergence
Revised: + γ ( x t − Wz t ) until convergence

Original: W ← W + η ( y t z ⊤ t − W ) end for
Revised: W ← W + η ( y t z ⊤ t − W ) end for

Original: Algorithm 2 can be implemented in a network with n principal neurons and k interneurons, Figure(right).
Revised: Algorithm 2 can be implemented in a network with n principal neurons and k interneurons, Figure(right).

Original: The principal neurons (resp.
Revised: The principal neurons (resp.

Original: interneurons) are connected to the interneurons (resp.
Revised: interneurons) are connected to the interneurons (resp.

Original: principal neurons) via the synaptic weight matrix W ⊤ (resp.
Revised: principal neurons) via the synaptic weight matrix W ⊤ (resp.

Original: − W ).
Revised: − W ).

Original: At each time step t , the external input to the principal neurons is x t , the activity of the principal neurons is y t , the activity of the interneurons is z t = W ⊤ y t and the recurrent input to the principal neurons is − Wz t .
Revised: At each time step t , the external input to the principal neurons is x t , the activity of the principal neurons is y t , the activity of the interneurons is z t = W ⊤ y t and the recurrent input to the principal neurons is − Wz t .

Original: Therefore, the total input to the principal neurons is encoded in the n -dimensional vector x t − Wz t .

Original: The neural activities are updated according to the neural dynamics in Algorithm 2 until they converge to y t
Revised: The neural activities are updated according to the neural dynamics in Algorithm 2 until they converge to y t

Original: = ( WW ⊤ ) − 1 x t and z t = W ⊤ y t .
Revised: = ( WW ⊤ ) − 1 x t and z t = W ⊤ y t .

Original: After the neural activities converge, the synaptic weight matrix W is updated according to Algorithm 2.
Revised: After the neural activities converge, the synaptic weight matrix W is updated according to Algorithm 2.

Original: Here,we require that the principal neuron-to-interneuron weight matrix W ⊤ be the negative transpose of the interneuron-to-principal neuron weight matrix − W , Figure 1 (right).
Revised: Here, the principal neuron-to-interneuron weight matrix W ⊤ is the negative transpose of the interneuron-to-principal neuron weight matrix − W , Figure 1 (right).

Original: In general, enforcing such symmetry is not biologically plausible and commonly referred to as the weight transport problem.
Revised: In general, enforcing such symmetry is not biologically plausible and commonly referred to as the weight transport problem.

Original: In addition, we do not sign-constrain the weights, so the network can violate Dale’s principle.
Revised: In addition, we do not sign-constrain the weights, so the network can violate Dale’s principle.

Original: In Appendix B, we modify our algorithm to be more biologically realistic.
Revised: In Appendix B, we modify the algorithm to be more biologically realistic and we map the modified algorithm onto the vertebrate olfactory bulb and show that the algorithm is consistent with several experimental observations.

Original:  A NALYSES OF CONTINUOUS SYNAPTIC DYNAMICS
Revised:  A NALYSES OF CONTINUOUS SYNAPTIC DYNAMICS

Original: We now present our main theoretical results on the convergence of the corresponding continuous synaptic dynamics for M and W .
Revised: We now present our main theoretical results on the convergence of the corresponding continuous synaptic dynamics for M and W .

Original: We first show that the synaptic updates are naturally viewed as (stochastic) gradient descent algorithms.
Revised: We first show that the synaptic updates are naturally viewed as (stochastic) gradient descent algorithms.

Original: We then analyze the corresponding continuous gradient flows.
Revised: We then analyze the corresponding continuous gradient flows.

Original: Detailed proofs of our results are provided in Appendix C
Revised: Detailed proofs of our results are provided in Appendix C

Original: 6. G RADIENT DESCENT ALGORITHMS
Revised: 6. G RADIENT DESCENT ALGORITHMS

Original: We first show that the offline and online synaptic dynamics are naturally viewed as gradient descent and stochastic gradient descent algorithms for minimizing whitening objectives.
Revised: We first show that the offline and online synaptic dynamics are naturally viewed as gradient descent and stochastic gradient descent algorithms for minimizing whitening objectives.

Original: Let U ( M ) be the convex function defined by
Revised: Let U ( M ) be the convex function defined by

Original: [Equation]
Revised: [Equation]

Original: Substituting the optimal neural activities Y
Revised: Substituting the optimal neural activities Y

Original: =
Revised: =

Original: M − 1 X into the offline update in equation 4, we see that the offline algorithm is a gradient descent algorithm for minimizing U ( M ) :
Revised: M − 1 X into the offline update in equation 4, we see that the offline algorithm is a gradient descent algorithm for minimizing U ( M ) :

Original: [Equation]
Revised: [Equation]

Original: By a similar argument, Algorithm 1 isnaturally viewed as a stochastic gradient descent algorithm for minimizing U ( M ) .
Revised: Similarly, Algorithm 1 is a stochastic gradient descent algorithm for minimizing U ( M ) .

Original: Next, let V ( W ) be the nonconvex function defined by
Revised: Next, let V ( W ) be the nonconvex function defined by

Original: [Equation]
Revised: [Equation]

Original: Again, substituting the optimal neural activities Y = ( WW ⊤ ) − 1 X into the offline update in equation 5, we see the offline algorithm is a gradient descent algorithm for minimizing V ( W ) :
Revised: Again, substituting the optimal neural activities Y = ( WW ⊤ ) − 1 X into the offline update in equation 5, we see the offline algorithm is a gradient descent algorithm for minimizing V ( W ) :

Original: [Equation]
Revised: [Equation]

Original: Similarly, Algorithm 2 is a stochastic gradient descent algorithm for minimizing V ( W ) .
Revised: Similarly, Algorithm 2 is a stochastic gradient descent algorithm for minimizing V ( W ) .

Original: A common approach for studying stochastic gradient descent algorithms is to analyze the corresponding continuous gradient flows (Saxe et al., 2014; 2019; Tarmoun et al., 2021), which are more mathematically tractable and are useful approximations of the average behavior of the stochastic gradient descent dynamics when the step size is small.
Revised: A common approach for studying stochastic gradient descent algorithms is to analyze the corresponding continuous gradient flows (Saxe et al., 2014; 2019; Tarmoun et al., 2021), which are more mathematically tractable and are useful approximations of the average behavior of the stochastic gradient descent dynamics when the step size is small.

Original: In the remainder of this section, we analyze and compare the continuous gradient flows associated with Algorithms 1 and 2.
Revised: In the remainder of this section, we analyze and compare the continuous gradient flows associated with Algorithms 1 and 2.

Original: To further facilitate the analysis, we consider so-called ‘spectral initializations’ that commute with
Revised: To further facilitate the analysis, we consider so-called ‘spectral initializations’ that commute with

Original: C xx (Saxe et al., 2014; 2019; Gidel et al., 2019; Tarmoun et al., 2021).
Revised: C xx (Saxe et al., 2014; 2019; Gidel et al., 2019; Tarmoun et al., 2021).

Original: Specificially, we say that
Revised: Specificially, we say that

Original: A 0 ∈ S n ++ is a spectral initialization if A 0 =
Revised: [Equation] + is a spectral initialization if A 0 =

Original: U x diag ( σ 1 , . . .
Revised: U x diag ( σ 1 , . . .

Original: , σ n ) U ⊤ x , where U x is the n × n orthogonal matrix of eigenvectors of C xx and σ 1 , . . .
Revised: , σ n ) U ⊤ x , where U x is the n × n orthogonal matrix of eigenvectors of C xx and σ 1 , . . .

Original: , σ n > 0 .
Revised: , σ n > 0 .

Original: To characterize the convergence rates of the gradient flows, we define the Lyapunov function
Revised: To characterize the convergence rates of the gradient flows, we define the Lyapunov function

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: 6. G RADIENT FLOW ANALYSIS OF A LGORITHM
Revised: G RADIENT FLOW ANALYSIS OF A LGORITHM

Original: The gradient flow of U ( M ) is given by
Revised: The gradient flow of U ( M ) is given by

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: To analyze solutions of M ( t ) , we focus on spectral intializations.
Revised: To analyze solutions of M ( t ) , we focus on spectral intializations.

Original: Lemma 1.
Revised: Lemma 1.

Original: Suppose M 0 is a spectral initialization.
Revised: Suppose M 0 is a spectral initialization.

Original: Then the solution M ( t ) of the ODE 6 is of the form M ( t ) =
Revised: Then the solution M ( t ) of the ODE 6 is of the form M ( t ) =

Original: U x diag ( σ 1 ( t ) , . . .
Revised: U x diag ( σ 1 ( t ) , . . .

Original: , σ n ( t ))
Revised: , σ n ( t ))

Original: U ⊤ x where σ 1 ( t ) , . . .
Revised: U ⊤ x where σ 1 ( t ) , . . .

Original: , σ n ( t ) , are the solutions of the ODE 
Revised: , σ n ( t ) , are the solutions of the ODE dσ

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Consequently,
Revised: Consequently,

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: From Lemma 1, we see that for a spectral initialization with σ i (0) ≤ λ i , the dynamics of σ i ( t ) satisfy
Revised: From Lemma 1, we see that for a spectral initialization with σ i (0) ≤ λ i , the dynamics of σ i ( t ) satisfy

Original: [Equation]
Revised: [Equation]

Original: It follows that σ i ( t ) 2 converges to λ 2 i exponentially with convergence rate greater than 2 / √ λ i .
Revised: It follows that σ i ( t ) 2 converges to λ 2 i exponentially with convergence rate greater than 2 / √ λ i .

Original: On the other hand, suppose σ i (0)
Revised: On the other hand, suppose σ i (0)

Original: ≫ λ i .
Revised: ≫ λ i .

Original: From equation 7, we see that while σ i ( t ) ≫ λ i , σ i ( t ) decays at approximately unit rate; that is,dσ i ( t ) dt
Revised: From equation 7, we see that while σ i ( t ) ≫ λ i , σ i ( t ) decays at approximately unit rate; that is,

Original: [Equation]
Revised: [Equation]

Original: − 1 .

Original: Therefore, the time for σ i ( t ) to converge to λ i grows linearly with σ i (0) .
Revised: Therefore, the time for σ i ( t ) to converge to λ i grows linearly with σ i (0) .

Original: We make these statements precise in the following proposition.
Revised: We make these statements precise in the following proposition.

Original: Proposition 1.
Revised: Proposition 1.

Original: Suppose M 0 is a spectral initialization and let M ( t ) denote the solution of the ODE 6 starting from M 0 .
Revised: Suppose M 0 is a spectral initialization and let M ( t ) denote the solution of the ODE 6 starting from M 0 .

Original: If σ i ≤ λ i for all i = 1 , . . . , n , then for ϵ < ℓ ( M 0 ) ,
Revised: If σ i ≤ λ i for all i = 1 , . . . , n , then for ϵ < ℓ ( M 0 ) ,

Original: [Equation]
Revised: [Equation]

Original: where λ max := max i λ i .
Revised: where λ max := max i λ i .

Original: On the other hand, if σ i > λ i for some i = 1 , . . . , n , then for ϵ < σ 2 i − λ 2 i ,
Revised: On the other hand, if σ i > λ i for some i = 1 , . . . , n , then for ϵ < σ 2 i − λ 2 i ,

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Equation]

Original: 6. G RADIENT FLOW ANALYSIS OF A LGORITHM
Revised: 6. G RADIENT FLOW ANALYSIS OF A LGORITHM

Original: The gradient flow of the overparameterized cost V ( W ) is given by
Revised: The gradient flow of the overparameterized cost V ( W ) is given by

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Next, we show that ℓ ( W ( t ) W ( t ) ⊤ ) converges to zero exponentially for any initialization W 0 .
Revised: Next, we show that ℓ ( W ( t ) W ( t ) ⊤ ) converges to zero exponentially for any initialization W 0 .

Original: Proposition 2.
Revised: Proposition 2.

Original: Let W ( t ) denote the solution of the ODE 12 starting from any W 0 ∈ R n × k .
Revised: Let W ( t ) denote the solution of the ODE 12 starting from any W 0 ∈ R n × k .

Original: Let ϵ < ℓ ( W 0 W ⊤ 0 ) .
Revised: Let ϵ < ℓ ( W 0 W ⊤ 0 ) .

Original: For spectral initializations W 0 W ⊤ 0 ,
Revised: For spectral initializations W 0 W ⊤ 0 ,

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: For general initializations W 0 W ⊤ 0 ,
Revised: For general initializations W 0 W ⊤ 0 ,

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original:  N UMERICAL EXPERIMENTS
Revised:  N UMERICAL EXPERIMENTS

Original: In this section, we numerically test the offline and online algorithms on synthetic datasets.
Revised: In this section, we numerically test the offline and online algorithms on synthetic datasets.

Original: Let
Revised: Let

Original: Whitening error ( t ) = ∥
Revised: Whitening error [Equation]

Original: A − 1 t C xx A − 1 t − I n ∥ Frob . for A t ∈ { M t , W t W ⊤ t } , where the subscript t indicates the value of matrices after the t th iterate.
Revised: where A t is the value of matrix after the t th iterate.

Original: To quantify the convergence time,we define
Revised: To quantify the convergence time, define

Original: Convergence time :=
Revised: Convergence time :=

Original: min { t ≥ 1 : Whitening error ( t ) < 0 . 1 } .
Revised: min { t ≥ 1 : Whitening error ( t ) < 0 . 1 } .

Revised: In plots with multiple runs, lines and shaded regions respectively denote the means and 95% confidence intervals over 10 runs.

Original: 7. O FFLINE ALGORITHMS
Revised: 7. O FFLINE ALGORITHMS

Original: Let n = 5 , k
Revised: Let n = 5 , k

Original: = 10 and T = 10 5 .
Revised: = 10 and T = 10 5 .

Original: We generate a data matrix X with i.i.d.
Revised: We generate a data matrix X with i.i.d.

Original: entries x t,i chosen uniformly from the interval (0 , 12) .
Revised: entries x t,i chosen uniformly from the interval (0 , 12) .

Original: The eigenvalues of C xx are { 24 . 01 , 16 . 42 , 10 . 45 , 6 . 59 , 3 . 28 } .
Revised: The eigenvalues of C xx are { 24 . 01 , 16 . 42 , 10 . 45 , 6 . 59 , 3 . 28 } .

Original: We initialize W 0 =
Revised: We initialize W 0 =

Original: Q √ α ΣP ⊤ , where α > 0 , Σ = diag (5 , 4 , 3 , 2 , 1) , Q is an n × n orthogonal matrix and P is a random k × n matrix with orthonormal column vectors.
Revised: Q √ α ΣP ⊤ , where α > 0 , Σ = diag (5 , 4 , 3 , 2 , 1) , Q is an n × n orthogonal matrix and P is a random k × n matrix with orthonormal column vectors.

Original: We set M 

Revised: We set M

Original: = W 0 W ⊤ 

Revised: = W 0 W ⊤

Original: = α QΣ 2 Q ⊤ .
Revised: = α QΣ 2 Q ⊤ .

Original: We consider spectral initializations (i.e., Q = U x ) and nonspectral initializations (i.e.,
Revised: We consider spectral initializations (i.e., Q = U x ) and nonspectral initializations (i.e.,

Original: Q is a random orthogonal matrix).
Revised: Q is a random orthogonal matrix).

Original: We use step size η = 10 − 3 .
Revised: We use step size η = 10 − 3 .

Original: The results of running the offline algorithms for α = 0 .
Revised: The results of running the offline algorithms for α = 0 .

Original: 1 , 1 , 10 are shown in Figure 2.
Revised: 1 , 1 , 10 are shown in Figure 2.

Original: Consistent with Propositions 1 and 2, the network with direct recurrent connections convergences slowly for ‘large’ α , whereas the network with interneurons converges exponentially for all α .
Revised: Consistent with Propositions 1 and 2, the network with direct recurrent connections convergences slowly for ‘large’ α , whereas the network with interneurons converges exponentially for all α .

Original: Further, as predicted by equation 9, when the eigenvalues of M are ‘large’, i.e., σ i
Revised: Further, as predicted by equation 9, when the eigenvalues of M are ‘large’, i.e., σ i

Original: ≫ λ
Revised: ≫ λ

Original: i , they decay linearly.
Revised: i , they decay linearly.

Original: Interestingly, in contrast to the case of spectral initializations, the evolution of the eigenvalues σ i ( t ) is not always monotone for nonspectral initializations (see the bottom row of Figure 2b).

Original: In Figure 3, we plot the convergence times of the offline algorithms with spectral and nonspectral initializations, for α = 1 , 2 , . . .
Revised: In Figure 3, we plot the convergence times of the offline algorithms with spectral and nonspectral initializations, for α = 1 , 2 , . . .

Original: Consistent with our analysis of the gradient flows, the convergence times for network with direct lateral connections (resp.
Revised: Consistent with our analysis of the gradient flows, the convergence times for network with direct lateral connections (resp.

Original: interneurons) grows linearly (resp. logarithmically) with α .
Revised: interneurons) grows linearly (resp. logarithmically) with α .

Revised: [Figure]

Original: 7. O NLINE ALGORITHMS
Revised: O NLINE ALGORITHMS

Original: To test our online algorithms, we generate a synthetic dataset with samples from 2 distributions.
Revised: We test our online algorithms on a synthetic dataset with samples from 2 distributions.

Original: We first set n = 2 , k = 4 and fixed covariance matrices C A
Revised: Let n = 2 , k = 4 and T = 10 5 . Fix covariance matrices C A

Original: = U
Revised: = U

Original: A diag (4 , 25) U ⊤ A and C B =
Revised: A diag (4 , 25) U ⊤ A and C B =

Original: U B diag (9 , 16) U ⊤ B , where U A , U B are random 2 × 2 rotation matrices.
Revised: U B diag (9 , 16) U ⊤ B , where U A , U B are random 2 × 2 rotation matrices.

Original: We set T = 10 5 and generated a 2 × 4 T dataset X =
Revised: We generate a 2 × 4 T dataset X =

Original: [ x 1 , . .
Revised: [ x 1 , . .

Original: , x 4 T ] with i.i.d. samples x 1 , . . 
Revised: , x 4 T ] with independent samples x 1 , . . .

Original: , x T , x 2 T +1 , . . .
Revised: , x T , x 2 T +1 , . . .

Original: , x 3 T (resp. x T +1 , . . .
Revised: , x 3 T ∼ N ( 0 , C A ) and x T +1 , . . .

Original: , x 2 T , x 3 T +1 , . . .
Revised: , x 2 T , x 3 T +1 , . . .

Original: , x 4 T ) drawn according to a mean zero Gaussian distribution with covariance matrix C A (resp. C B ).
Revised: , x 4 T ∼ N ( 0 , C B ) .

Original: We evaluated our online algorithms with constant learning rate η = 10 − 4 and initializations M 0 =
Revised: We evaluated our online algorithms with step size η = 10 − 4 and initializations M 0 =

Original: W 0 W ⊤ 0 , where W 0 =
Revised: W 0 W ⊤ 0 , where W 0 =

Original: Q diag ( σ 1 , σ 2 ) P ⊤ , Q is a random 2 × 2 rotation matrix, P is a random 4 × 2 matrix with orthonormal column vectors and σ 1 , σ 2 are independent random variables chosen uniformly from the interval (1 , 1 . 5) .
Revised: Q diag ( σ 1 , σ 2 ) P ⊤ , Q is a random 2 × 2 rotation matrix, P is a random 4 × 2 matrix with orthonormal column vectors and σ 1 , σ 2 are independent random variables chosen uniformly from the interval (1 , 1 . 5) .

Original: The results are shown in
Revised: The results are shown in

Original: Figure 4.
Revised: Figure 4.

Original: Consistent with our theoretical analyses, we see that the network with interneurons adapts to changing distributions faster than the network with direct recurrent connections.
Revised: Consistent with our theoretical analyses, we see that the network with interneurons adapts to changing distributions faster than the network with direct recurrent connections.

Original: 7. A PPLICATION TO PRINCIPAL SUBSPACE LEARNING
Revised: 7. A PPLICATION TO PRINCIPAL SUBSPACE LEARNING

Original: A challenge in unsupervised and self-supervised learning is preventing collapse (i.e., degenerate solutions).
Revised: A challenge in unsupervised and self-supervised learning is preventing collapse (i.e., degenerate solutions).

Original: A recent approach in self-supervised learning is to decorrelate or whiten the feature representation (Ermolov et al., 2021; Zbontar et al., 2021;
Revised: A recent approach in self-supervised learning is to decorrelate or whiten the feature representation (Ermolov et al., 2021; Zbontar et al., 2021;

Original: Bardes et al., 2021; Hua et al., 2021).
Revised: Bardes et al., 2022; Hua et al., 2021).

Original: Here, using Oja’s online principal component algorithm (Oja, 1982) as a tractable example, we demonstrate that the speed of whitening can affect the accuracy of the learned representation.
Revised: Here, using Oja’s online principal component algorithm (Oja, 1982) as a tractable example, we demonstrate that the speed of whitening can affect the accuracy of the learned representation.

Original: Consider a neuron whose input and output at time t are respectively s t ∈ R d and y t := v ⊤ s t , where v ∈ R d represents the synaptic weights connecting the inputs to the neuron.
Revised: Consider a neuron whose input and output at time t are respectively s t ∈ R d and y t := v ⊤ s t , where v ∈ R d represents the synaptic weights connecting the inputs to the neuron.

Original: Oja’s algorithm learns the top principal component of the inputs by updating the vector v as follows:
Revised: Oja’s algorithm learns the top principal component of the inputs by updating the vector v as follows:

Original: [Equation]
Revised: [Equation]

Original: where ζ > 0 is the step size.
Revised: where ζ > 0 is the step size.

Original: Next, consider a population of 2 ≤ n ≤ d neurons with outputs y t ∈ R n and feedforward synaptic weight vectors v 1 , . .
Revised: Next, consider a population of 2 ≤ n ≤ d neurons with outputs y t ∈ R n and feedforward synaptic weight vectors v 1 , . .

Original: . , v n ∈ R d connecting the inputs s t to the n neurons.
Revised: . , v n ∈ R d connecting the inputs s t to the n neurons.

Original: The goal is to project the inputs onto their n -dimensional principal subspace.
Revised: The goal is to project the inputs onto their n -dimensional principal subspace.

Original: Running n instances of Oja’s algorithm in parallel without lateral connections results in collapse — each synaptic weight vector v i converges to the top principal component.
Revised: Running n instances of Oja’s algorithm in parallel without lateral connections results in collapse — each synaptic weight vector v i converges to the top principal component.

Original: One way to avoid collapse is to whiten the output y t using recurrent connections, Figure 5 (left).
Revised: One way to avoid collapse is to whiten the output y t using recurrent connections, Figure 5 (left).

Original: Here we show that when the subspace projection and output whitening are learned simultaneously, it is critical that the whitening transformation is learned sufficiently fast.
Revised: Here we show that when the subspace projection and output whitening are learned simultaneously, it is critical that the whitening transformation is learned sufficiently fast.

Original: We set d = 3 , n = 2 and generate i.i.d.
Revised: We set d = 3 , n = 2 and generate i.i.d.

Original: inputs s t ∼ N ( 0 , diag (5 , 2 , 1)) .
Revised: inputs s t ∼ N ( 0 , diag (5 , 2 , 1)) .

Original: We initialize two random vectors v 1 , v 2 ∈ R 3 with independent N (0 , 1) entries.
Revised: We initialize two random vectors v 1 , v 2 ∈ R 3 with independent N (0 , 1) entries.

Original: At each time step t , we use the projection x t := ( v ⊤ 1 s t , v ⊤ 2 s t ) as the input to either Algorithm 1 or 2 and we let y t be the output; that is, y t
Revised: At each time step t , we use the projection x t := ( v ⊤ 1 s t , v ⊤ 2 s t ) as the input to either Algorithm 1 or 2 and we let y t be the output; that is, y t

Original: =
Revised: =

Original: M − 1 x t or y t
Revised: M − 1 x t or y t

Original: = ( WW ⊤ ) − 1 x t .
Revised: = ( WW ⊤ ) − 1 x t .

Original: For i = 1 , 2 , we update v i according to equation 15 with ζ = 10 − 3 and with v i (resp.
Revised: For i = 1 , 2 , we update v i according to equation 15 with ζ = 10 − 3 and with v i (resp.

Original: y t,i ) in place of v (resp.
Revised: y t,i ) in place of v (resp.

Original: y t ).
Revised: y t ).

Original: We update the recurrent weights M or W according to Algorithm 1 or 2.
Revised: We update the recurrent weights M or W according to Algorithm 1 or 2.

Original: To measure the performance, we define
Revised: To measure the performance, we define

Original: Subspace error :
Revised: Subspace error :

Original: =
Revised: =

Original: ∥ V ( V ⊤ V ) − 
Revised: ∥ V ( V ⊤ V ) − 1 V ⊤ − diag (1 , 1 , 0) ∥ 2 Frob ,

Original: [Equation]
Revised: V :=

Original:  which is equal to zero when v 1 , v 2 span the 2-dimensional principal subspace of the inputs.
Revised: [ v 1 , v 2 ] ∈ R 3 × 2 , which is equal to zero when v 1 , v 2 span the 2-dimensional principal subspace of the inputs.

Original: We plot the subspace error for Algorithms 1 and 2 using learning rates η = ζ and η = 10 ζ , Figure 5 (right).
Revised: We plot the subspace error for Algorithms 1 and 2 using learning rates η = ζ and η = 10 ζ , Figure 5 (right).

Original: The results suggest that in order to learn the correct subspace, the whitening transformation must be learned sufficiently fast relative to the feedforward vectors v i ; for additional results along these lines, see (Pehlevan et al., 2018, section 6).
Revised: The results suggest that in order to learn the correct subspace, the whitening transformation must be learned sufficiently fast relative to the feedforward vectors v i ; for additional results along these lines, see (Pehlevan et al., 2018, section 6).

Original: Therefore, since interneurons accelerate learning of the whitening transform, they arealso useful for accurate or optimal representation learning.
Revised: Therefore, since interneurons accelerate learning of the whitening transform, they are useful for accurate or optimal representation learning.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: We analyzed the gradient flow dynamics of 2 recurrent neural networks for ZCA whitening — one with direct recurrent connections and one with interneurons.
Revised: We analyzed the gradient flow dynamics of 2 recurrent neural networks for ZCA whitening — one with direct recurrent connections and one with interneurons.

Original: For spectral initializations we can analytically estimate the convergence time for both gradient flows.
Revised: For spectral initializations we can analytically estimate the convergence time for both gradient flows.

Original: For nonspectral initializations, we show numerically that the convergence times are close to the spectral initializations with the same initial spectrum.
Revised: For nonspectral initializations, we show numerically that the convergence times are close to the spectral initializations with the same initial spectrum.

Original: Both our theoretical and numerical results show that the recurrent neural network with interneurons is more robust to initialization.
Revised: Our results show that the recurrent neural network with interneurons is more robust to initialization.

Original: In Appendix B, we modify the algorithm to be more biologically realistic and we map the modified algorithm onto the vertebrate olfactory bulb and show that the algorithm is consistent with several experimental observations, suggesting that biological interneurons may facilitate fast statistical adaptation.

Original: An interesting question is whether including interneurons in other classes of recurrent neural networks also accelerates the learning dynamics of those networks.
Revised: An interesting question is whether including interneurons in other classes of recurrent neural networks also accelerates the learning dynamics of those networks.

Original: We hypothesize that interneurons accelerate learning dynamics when the objective for the network with interneurons can be viewed as an overparameterization of the objective for the recurrent neural network with direct connections.
Revised: We hypothesize that interneurons accelerate learning dynamics when the objective for the network with interneurons can be viewed as an overparameterization of the objective for the recurrent neural network with direct connections.

Revised: A CKNOWLEDGMENTS

Revised: We thank Yanis Bahroun, Nikolai Chapochnikov, Lyndon Duong, Johannes Friedrich, Siavash

Revised: Golkar, Jason Moore and Tiberiu Tes¸ileanu for helpful feedback on an earlier draft of this work.

Revised: C. Pehlevan acknowledges support from the Intel Corporation.

Original: A S ADDLE POINT PROPERTY
Revised: A S ADDLE POINT PROPERTY

Original: Here we recall the following minimax property for a function that satisfies the saddle point property (Boyd & Vandenberghe, 2004, section 5.4).
Revised: Here we recall the following minimax property for a function that satisfies the saddle point property (Boyd & Vandenberghe, 2004, section 5.4).

Original: Theorem 1.
Revised: Theorem 1.

Original: Let V ⊆ R n , W ⊆
Revised: Let V ⊆ R n , W ⊆

Original: R m and f : V × W → R .
Revised: R m and f : V × W → R .

Original: Suppose f satisfies the saddle point property; that is, there exists ( a ∗ , b ∗ )
Revised: Suppose f satisfies the saddle point property; that is, there exists ( a ∗ , b ∗ )

Original: ∈ V × W such that
Revised: ∈ V × W such that

Original: [Equation]
Revised: [Equation]

Original: for all ( a , b) ∈ V ×
Revised: for all [Equation] b

Original: W.
Revised: [Equation] V × W [Equation]

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: B R ELATION TO NEURONAL CIRCUITS
Revised: B R ELATION TO NEURONAL CIRCUITS

Original: In this section, we modify Algorithm 2 to satisfy additional biological constraints and we map the modified algorithm onto the vertebrate olfactory bulb.
Revised: In this section, we modify Algorithm 2 to satisfy additional biological constraints and we map the modified algorithm onto the vertebrate olfactory bulb.

Original: B. D ECOUPLING THE INTERNEURON SYNAPSES
Revised: B. D ECOUPLING THE INTERNEURON SYNAPSES

Original: The neural circuit implementation of Algorithm 2 requires that the principal neuron-to-interneuron synaptic weight matrix W ⊤ is the negative transpose of the interneuron-to-principal neuron synaptic weight matrix − W .
Revised: The neural circuit implementation of Algorithm 2 requires that the principal neuron-to-interneuron synaptic weight matrix W ⊤ is the negative transpose of the interneuron-to-principal neuron synaptic weight matrix − W .

Original: In general, enforcing this symmetry is not biologically plausible, and is commonly referred to as the weight transport problem.
Revised: In general, enforcing this symmetry is not biologically plausible, and is commonly referred to as the weight transport problem.

Original: Here, following (Golkar et al., 2020, appendix
Revised: Here, following (Golkar et al., 2020, appendix

Original: D), we decouple the synapses and show that the (asymptotic) symmetry of the synaptic weight matrices follows from the symmetry of the local Hebbian/anti-Hebbian updates.
Revised: D), we decouple the synapses and show that the (asymptotic) symmetry of the synaptic weight matrices follows from the symmetry of the local Hebbian/anti-Hebbian updates.

Original: We begin by replacing W and W ⊤ in Algorithm 2 with W zy and W yz , respectively.
Revised: We begin by replacing W and W ⊤ in Algorithm 2 with W zy and W yz , respectively.

Original: Then the neural activities are given by
Revised: Then the neural activities are given by

Original: [Equation]
Revised: [Equation]

Original: which converge to ( W zy W yz )
Revised: which converge to ( W zy W yz )

Original: − 1 x t and z t = W yz y t .
Revised: − 1 x t and z t = W yz y t .

Original: After the neural activities converge, the synaptic weights are updated according to the update rules
Revised: After the neural activities converge, the synaptic weights are updated according to the update rules

Original: [Equation]
Revised: [Equation]

Original: Let W zy,t and W yz,t denote the values of the synaptic weight matrices after t updates.
Revised: Let W zy,t and W yz,t denote the values of the synaptic weight matrices after t updates.

Original: By iterating the above updates, we see that the difference between the weight matrices after t updates is given by
Revised: By iterating the above updates, we see that the difference between the weight matrices after t updates is given by

Original: [Equation]
Revised: [Equation]

Original: Therefore, provided 0 < η < 1 , the difference decays exponentially.
Revised: Therefore, provided 0 < η < 1 , the difference decays exponentially.

Original: B.E NFORCING D ALE ’ S PRINCIPLE
Revised: B. S IGN - CONSTRAINING THE INTERNEURON WEIGHTS

Original: In equation 16, the matrix W yz is preceded by a positive sign and the matrix W zy is preceded by a negative sign, which is consistent with the fact that the principal neuron-to-interneuron synapses are excitatory whereas the interneuron-to-principal neuron synapses are inhibitory.
Revised: In equation 16, the matrix W yz is preceded by a positive sign and the matrix W zy is preceded by a negative sign, which is consistent with the fact that the principal neuron-to-interneuron synapses are excitatory whereas the interneuron-to-principal neuron synapses are inhibitory.

Original: However, this interpretation is superficial because the matrices are not constrained to be non-negative, so the network can violate Dale’s principle.
Revised: However, this interpretation is superficial because the matrices are not constrained to be non-negative, so the network can violate Dale’s principle.

Original: One approach to enforce Dale’s principle is to take projected gradient descent steps when updating the synaptic weight matrices, which results in the offline updates
Revised: One approach to enforce Dale’s principle is to take projected gradient descent steps when updating the synaptic weight matrices, which results in the offline updates

Original: [Equation]
Revised: [Equation]

Original:  and online updates
Revised: (cid:19)(cid:21) + , and online updates

Original: [Equation]
Revised: [Equation]

Original: where [ · ] + denotes the element-wise rectification operation.
Revised: where [ · ] + denotes the element-wise rectification operation.

Original: This results in Algorithm 3.
Revised: This results in Algorithm 3.

Original: Algorithm3: input centered inputs { x t } ; parameters γ , η initialize non-negative matrices W yz , W zy for t = 1 , . .
Revised: Algorithm input centered inputs { x t } ; parameters γ , η initialize non-negative matrices W yz , W zy for t = 1 , . .

Original: , T do repeat y t ← y t + γ ( x t − W zy z t )
Revised: , T do repeat y t ← [Equation] z [Equation]

Original: z t ← z t + γ ( W yz y t − z t ) until convergence
Revised: ← z t + [Equation] y t − z t ) until convergence

Original: W yz ←[ W yz + η
Revised: W [Equation] ←

Original: ( z t y ⊤ t − W yz )] +
Revised: [Equation]

Original: W zy ← [ W zy + η ( y t z ⊤ t − W zy )]
Revised: W [Equation] ← [Equation]

Original: + end for
Revised: end for

Original: In general, this modification results in a network output that does not correspond to ZCA whitening of the inputs.
Revised: In general, this modification results in a network output that does not correspond to ZCA whitening of the inputs.

Original: That being said, it is still worth comparing the performance of the network to the network with interneurons derived in section 5.
Revised: That being said, it is still worth comparing the performance of the network to the network with interneurons derived in section 5.

Original: In Figure 6, we compare the performance of the offline versions of Algorithms 2 and 3.
Revised: In Figure 6, we compare the performance of the offline versions of Algorithms 2 and 3.

Original: We see that Algorithm 3 equilibrates at a rate that is comparable to
Revised: We see that Algorithm 3 equilibrates at a rate that is comparable to

Original: Algorithm 2 (and therefore much faster than Algorithm 1 for large α ).
Revised: Algorithm 2 (and therefore much faster than Algorithm 1 for large α ).

Original: In particular, this modified algorithm appears to also exhibit the accelerated dynamics due to the overparamterization of the objective.
Revised: In particular, this modified algorithm appears to also exhibit the accelerated dynamics due to the overparamterization of the objective.

Original: As expected, the outputs of the projected gradient descent algorithm are not fully whitened — after the synaptic dynamics of Algorithm 3 equilibrate, the eigenvalues of the output covariance C
Revised: As expected, the outputs of the projected gradient descent algorithm are not fully whitened — after the synaptic dynamics of Algorithm 3 equilibrate, the eigenvalues of the output covariance C

Original: yy are { 1 . 39 , 1 . 01 , 1 . 00 , 0 . 98 , 0 . 61 } .
Revised: yy are { 1 . 39 , 1 . 01 , 1 . 00 , 0 . 98 , 0 . 61 } .

Original: This can be compared with the eigenvalues of the input covariance matrix C xx : { 24 . 01 , 16 .
Revised: This can be compared with the eigenvalues of the input covariance matrix C xx : { 24 . 01 , 16 .

Original: 42 , 10 . 45 , 6 . 59 , 3 . 28 } , which suggests that the network significantly normalizes the eigenvalues of the output covariance without exactly performing ZCA whitening.
Revised: 42 , 10 . 45 , 6 . 59 , 3 . 28 } , which suggests that the network significantly normalizes the eigenvalues of the output covariance without exactly performing ZCA whitening.

Original: In general, understanding the exact statistical transformation that results from the projected gradient descent algorithm is more mathematically challenging.
Revised: In general, understanding the exact statistical transformation that results from the projected gradient descent algorithm is more mathematically challenging.

Original: For instance, in the case of Algorithm 2, the weights W adapt so that WW ⊤ =
Revised: For instance, in the case of Algorithm 2, the weights W adapt so that WW ⊤ =

Original: C 1 / 2 xx ; that is, the algorithm solves a symmetric matrix factorization problem whose solutions can be written in closed form in terms of the SVD of the covariance matrix C xx .
Revised: C 1 / 2 xx ; that is, the algorithm solves a symmetric matrix factorization problem whose solutions can be written in closed form in terms of the SVD of the covariance matrix C xx .

Original: In the case of Algorithm 3, the algorithm is essentially solving for the optimal (approximate) non-negative symmetric matrix factorization of C 1 / 2 xx , for which there do not exist general closed form solutions.
Revised: In the case of Algorithm 3, the algorithm is essentially solving for the optimal (approximate) non-negative symmetric matrix factorization of C 1 / 2 xx , for which there do not exist general closed form solutions.

Original: That being said, the empirical results suggest that the more biologically realistic network still performs  statistical adaptation.
Revised: That being said, the empirical results suggest that the more biologically realistic network still performs rapid statistical adaptation.

Original: B. M APPING A LGORITHM 3 ONTO THE OLFACTORY BULB
Revised: B. M APPING A LGORITHM 3 ONTO THE OLFACTORY BULB

Original: In vertebrates, an early stage of olfactory processing occurs in the olfactory bulb.
Revised: In vertebrates, an early stage of olfactory processing occurs in the olfactory bulb.

Original: The olfactory bulb receives direct olfactory inputs, which it processes and transmits to higher order brain regions (Shepherd et al., 2004), axon structures glomeruli
Revised: The olfactory bulb receives direct olfactory inputs, which it processes and transmits to higher order brain regions (Shepherd et al., 2004), Figure 7.

Original:  Mitral  are the main projection neurons of  olfactory  direct  from interneurons called granule cells, which are axonless neurons that form reciprocal dendrodendritic synapses with the basal dendrites of mitral cells (Shepherd, 2009).
Revised: Olfactory receptor neurons project to the olfactory bulb, where their axon terminals cluster by receptor type into spherical structures called glomeruli. Mitral cells, which are the main projection neurons of the olfactory bulb, receive direct inputs from the olfactory receptor neuron axons and output to the rest of the brain. Each mitral cell extends its apical dendrite into a single glomerulus, where it forms synapses with the axons of the olfactory receptor neurons expressing a common receptor type. The mitral cell activities are modulated by lateral inhibition from interneurons called granule cells, which are axonless neurons that form reciprocal dendrodendritic synapses with the basal dendrites of mitral cells (Shepherd, 2009).

Original: Experimental evidence indicates that the olfactory bulb transforms its inputs so that mitral cell responses to distinct odors are approximately orthogonal (Friedrich & Laurent, 2001; 2004;
Revised: Experimental evidence indicates that the olfactory bulb transforms its inputs so that mitral cell responses to distinct odors are approximately orthogonal (Friedrich & Laurent, 2001; 2004;

Original: Giridhar et al., 2011; Gschwend et al., 2015; Wanner & Friedrich, 2020), a transformation referred to as pattern separation , which is closely related to statistical whitening (Wick et al., 2010).
Revised: Giridhar et al., 2011; Gschwend et al., 2015; Wanner & Friedrich, 2020), a transformation referred to as pattern separation , which is closely related to statistical whitening (Wick et al., 2010).

Original: We explore the possibility that the mitral-granule cell microcircuit implements Algorithm 3, Figure 7.
Revised: We explore the possibility that the mitral-granule cell microcircuit implements Algorithm 3, Figure 7.

Original: In this case, the vectors x t and y t represent the inputs to and outputs from the mitral cells, respectively.
Revised: In this case, the vectors x t and y t represent the inputs to and outputs from the mitral cells, respectively.

Original: The vector z t represents the granule cell outputs.
Revised: The vector z t represents the granule cell outputs.

Original: We let W yz (resp.
Revised: We let W yz (resp.

Original: − W zy ) denote the mitral cell-to-granule cell (resp.
Revised: − W zy ) denote the mitral cell-to-granule cell (resp.

Original: granule cell-to-mitral cell) synaptic weights.
Revised: granule cell-to-mitral cell) synaptic weights.

Original: We compare our model with additional experimental observations.
Revised: We compare our model with additional experimental observations.

Original: First, Algorithm 3 learns by adapting the matrices W yz and W zy .
Revised: First, Algorithm 3 learns by adapting the matrices W yz and W zy .

Original: This is in line with experimental observations that granule cells in the olfactory bulb are highly plastic suggesting that the dendrodendritic synapses are a synaptic substrate for experience dependent modulation of the olfactory bulb output (Nissant et al., 2009; Sailor et al., 2016).
Revised: This is in line with experimental observations that granule cells in the olfactory bulb are highly plastic suggesting that the dendrodendritic synapses are a synaptic substrate for experience dependent modulation of the olfactory bulb output (Nissant et al., 2009; Sailor et al., 2016).

Original: Second, a consequence of Algorithm 3 is that the mitral cellto-granule cell synapses and granule cell-to-mitral cell synapses are (asymptotically) symmetric; that is, W zy = W ⊤ yz .
Revised: Second, a consequence of Algorithm 3 is that the mitral cell-togranule cell synapses and granule cell-to-mitral cell synapses are (asymptotically) symmetric; that is, W zy = W ⊤ yz .

Original: There is experimental evidence in support of this symmetry — the dendrodendritic synapses are mainly present in reciprocal pairs (Woolf et al., 1991).
Revised: There is experimental evidence in support of this symmetry — the dendrodendritic synapses are mainly present in reciprocal pairs (Woolf et al., 1991).

Original: In other words, the connectivity matrix of mitral cell-to-granule cell synapses (the matrix obtained by setting the non-zero entries in W yz to 1) is approximately equal to the transpose of the connectivity matrix of granule cell-to-mitral cell synapses.
Revised: In other words, the connectivity matrix of mitral cell-to-granule cell synapses (the matrix obtained by setting the non-zero entries in W yz to 1) is approximately equal to the transpose of the connectivity matrix of granule cell-tomitral cell synapses.

Original: Finally, Algorithm 3 requires that k ≥ n ; that is, there are more granule cells than mitral cells.
Revised: Finally, Algorithm 3 requires that k ≥ n ; that is, there are more granule cells than mitral cells.

Original: This is consistent with measurements in therodent olfactory bulb, which indicate that there are approximately 50–100 times more granule cells than mitral cells (Shepherd et al., 2004).
Revised: This is consistent with measurements in the olfactory bulb indicating that there are approximately 50–100 times more granule cells than mitral cells (Shepherd et al., 2004).

Original: C P ROOFS OF OUR MAIN RESULTS C. P ROOF OF L EMMA
Revised: C P ROOFS OF OUR MAIN RESULTS C. P ROOF OF L EMMA

Original: Proof.
Revised: Proof.

Original: Let σ 1 ( t ) , . . .
Revised: Let σ 1 ( t ) , . . .

Original: , σ n ( t ) be solutions of the ODE 7 and define M ( t ) :
Revised: , σ n ( t ) be solutions of the ODE 7 and define M ( t ) :

Original: = U x Σ ( t ) U ⊤ x , where Σ ( t ) := diag ( σ 1 ( t ) , . . .
Revised: = U x Σ ( t ) U ⊤ x , where Σ ( t ) := diag ( σ 1 ( t ) , . . .

Original: , σ n ( t )) .
Revised: , σ n ( t )) .

Original: Then 
Revised: Then d M ( t ) dt = U x diag (cid:18)

Original: [Equation]
Revised: [Equation]

Original: In particular, we see that M ( t ) must be the unique solution of the ODE 6, where uniqueness of solutions follows because the right-hand-side of equation 6 is locally Lipschitz continuous on its domain of definition.
Revised: In particular, we see that M ( t ) must be the unique solution of the ODE 6, where uniqueness of solutions follows because the right-hand-side of equation 6 is locally Lipschitz continuous on its domain of definition.

Original: Equation 8 then follows from the ODE 7 and the chain rule.
Revised: Equation 8 then follows from the ODE 7 and the chain rule.

Original: C. P ROOF OF P ROPOSITION
Revised: C. P ROOF OF P ROPOSITION

Original: Proof.
Revised: Proof.

Original: Suppose σ i (0) ≤ λ i for i = 1 , . . . , n .
Revised: Suppose σ i (0) ≤ λ i for i = 1 , . . . , n .

Original: By equation 8, σ i ( t ) ≤ λ i for all t ≥ 0 and so
Revised: By equation 8, σ i ( t ) ≤ λ i for all t ≥ 0 and so

Original: [Equation]
Revised: [Equation]

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: It follows that equation 10 holds.
Revised: It follows that equation 10 holds.

Original: Now suppose σ i (0) ≥ λ
Revised: Now suppose σ i (0) ≥ λ

Original: i for some i = 1 , . . . , n .
Revised: i for some i = 1 , . . . , n .

Original: By equation 7,
Revised: By equation 7,

Original: [Equation]
Revised: [Equation]

Original: It follows that equation 11 holds.
Revised: It follows that equation 11 holds.

Original: C. P ROOF OF P ROPOSITION
Revised: C. P ROOF OF P ROPOSITION

Original: Proof.
Revised: Proof.

Original: Define A ( t ) :
Revised: Define A ( t ) :

Original: = W ( t ) W ( t ) ⊤ .
Revised: = W ( t ) W ( t ) ⊤ .

Original: By the product rule,
Revised: By the product rule,

Original: [Equation]
Revised: [Equation]

Original: Then, by the chain rule and equation 17, for all t ≥ 0 ,
Revised: Then, by the chain rule and equation 17, for all t ≥ 0 ,

Original: [Equation]

Original: t ) 2 − C xx )

Original: [Equation]

Original: A

Original: [Equation]

Original: t )

Original: [Equation]

Original: A

Original: [Equation]

Original: t )

Original: [Equation]

Original: xx

Original: A

Original: [Equation]

Original: t ) − 1 )

Original: [Equation]

Original: =

Original: [Equation]

Original: Tr

Original: [Equation]
Revised: [Equation]

Original: A

Original: [Equation]

Original: t )

Original: [Equation]

Original: xx )

Original: [Equation]

Original: t

Original: [Equation]

Original: t )

Original: [Equation]

Original: xx )

Original: [Equation]

Original: t )

Original: [Equation]

Original: [Equation]

Original: t

Original: [Equation]

Original: t )

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: ( t )

Original: [Equation]

Original: xx )

Original: [Equation]

Original: [Equation]

Original: ∥

Original: [Equation]

Original: Suppose W 0 W ⊤ 0 is a spectral initialization.
Revised: Suppose W 0 W ⊤ 0 is a spectral initialization.

Original: Then A ( t ) commutes with C xx and so
Revised: Then A ( t ) commutes with C xx and so

Original: [Equation]
Revised: [Equation]

Original: Thus, equation 13 holds.
Revised: Thus, equation 13 holds.

Original: For general initializations,
Revised: For general initializations,

Original: [Equation]
Revised: [Equation]

Original: Therefore, inequality equation 14 holds.
Revised: Therefore, inequality equation 14 holds.